Crowd positioning method and device, electronic device and storage medium

By extracting feature in crowd video clips and fusion of optical flow maps and combining with neural networks for crowd positioning, the problem of low accuracy in crowd positioning in the prior art is solved, and the accuracy of crowd counting and group behavior analysis is improved.

CN113297983BActive Publication Date: 2024-11-26SHANGHAI SENSETIME INTELLIGENT TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202110586959.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-27
Publication Date
2024-11-26
Estimated Expiration
2041-05-27

AI Technical Summary

Technical Problem

The prior art has low accuracy in crowd positioning, and cannot fully utilize the timing information in crowd video clips, resulting in insufficient accuracy of crowd counting and group behavior analysis.

Method used

By extracting at least two frames of images in the crowd video clip, the optical flow map between the target population and the non-target population is determined, the feature map is fused to obtain the spatiotemporal information of the target population, and the neural network is used to locate the population to improve the accuracy of positioning sex.

Benefits of technology

It improves the accuracy of crowd positioning, can more effectively utilize the timing information in crowd video clips, and enhances the accuracy of crowd counting and group behavior analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113297983B_ABST
    Figure CN113297983B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a crowd positioning method and device, an electronic device and a storage medium, the method comprising: extracting features from at least two frames of crowd images obtained from a crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include a frame of target crowd image and at least one frame of non-target crowd image; determining a first optical flow map between the target crowd image and the at least one frame of non-target crowd image; according to the first optical flow map, fusing the at least two first feature maps to obtain a second feature map corresponding to the target crowd image; positioning the crowd according to the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the position of the human body included in the target crowd image. The embodiments of the present disclosure can effectively improve the accuracy of crowd positioning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a crowd positioning method and device, an electronic device, and a storage medium. Background Art

[0002] With the growth of population and the acceleration of urbanization, the number of large crowd gatherings is increasing and the scale is getting larger. Crowd analysis is of great significance to public safety and urban planning. Common crowd analysis tasks include crowd counting, group behavior analysis, crowd positioning, etc. Among them, crowd positioning is the basis of other crowd analysis tasks. Crowd positioning refers to estimating the position of the human body included in the image or video through computer vision algorithms, determining the coordinates of the human body included in the image or video, and providing data basis for subsequent crowd analysis tasks such as crowd counting and group behavior analysis. The accuracy of crowd positioning directly affects the accuracy of crowd counting and the results of crowd behavior analysis. Therefore, a crowd positioning method with high accuracy is urgently needed. Summary of the invention

[0003] The present disclosure proposes a crowd positioning method and device, an electronic device, and a storage medium technical solution.

[0004] According to one aspect of the present disclosure, a crowd positioning method is provided, comprising: performing feature extraction on at least two frames of crowd images obtained from a crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include a frame of target crowd image and at least one frame of non-target crowd image; determining a first optical flow map between the target crowd image and the at least one frame of non-target crowd image; based on the first optical flow map, fusing the at least two first feature maps to obtain a second feature map corresponding to the target crowd image; performing crowd positioning based on the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the position of a human body included in the target crowd image.

[0005] In a possible implementation, the fusing of the at least two first feature maps according to the first optical flow map to obtain a second feature map corresponding to the target population image includes: spatially transforming the first feature map corresponding to the at least one frame of non-target population image according to the first optical flow map to obtain at least one third feature map corresponding to the at least one frame of non-target population image; and fusing the target first feature map and the at least one third feature map to obtain the second feature map, wherein the target first feature map is the first feature map corresponding to the target population image.

[0006] In a possible implementation, the fusing the target first feature map and the at least one third feature map to obtain the second feature map includes: fusing the target first feature map and the at least one third feature map along the channel dimension to obtain a fourth feature map, wherein the number of channels of the fourth feature map is the sum of the number of channels of the target first feature map and the at least one third feature map; performing dimensionality reduction processing on the fourth feature map along the channel dimension to obtain the second feature map, wherein the second feature map and the target first feature map have the same size and number of channels.

[0007] In a possible implementation, the method of performing crowd positioning according to the second feature map to obtain a target positioning map corresponding to the target crowd image includes: performing crowd positioning according to the second feature map to obtain a first positioning probability map, wherein the first positioning probability map is used to indicate the probability that each pixel in the target crowd image is a human body; and performing image processing on the first positioning probability map according to a probability threshold to obtain the target positioning map.

[0008] In a possible implementation, the first positioning probability map is subjected to image processing according to a probability threshold to obtain the target positioning map, including: performing an average pooling operation on the first positioning probability map to obtain a mean pooling map; performing a maximum pooling operation on the mean pooling map to obtain a maximum pooling map; obtaining a second positioning probability map according to the mean pooling map and the maximum pooling map; and performing threshold segmentation on the second positioning probability map according to the probability threshold to obtain the target positioning map.

[0009] In a possible implementation, the crowd positioning method is implemented by a crowd positioning neural network, and the training samples of the crowd positioning neural network include a crowd sample video clip and a real positioning map corresponding to a target crowd sample image in the crowd sample video clip; the training method of the crowd positioning neural network is as follows: through the crowd positioning neural network, a predicted positioning probability map corresponding to the target crowd sample image is determined, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the target crowd sample image is a human body; based on the predicted positioning probability map and the real positioning map, the positioning loss is determined; based on the positioning loss, the crowd positioning neural network is optimized.

[0010] In a possible implementation, determining the predicted positioning probability map corresponding to the target crowd sample image through the crowd positioning neural network includes: performing feature extraction on at least two frames of crowd sample images obtained from the crowd sample video clip to obtain at least two fifth feature maps, wherein the at least two frames of crowd images include the target crowd sample image and at least one frame of non-target crowd sample image; determining a second optical flow map between the target crowd sample image and the at least one frame of non-target crowd sample image; based on the second optical flow map, fusing the at least two fifth feature maps to obtain a sixth feature map corresponding to the target crowd sample image; and positioning the crowd based on the sixth feature map to obtain the predicted positioning probability map.

[0011] In a possible implementation, determining the positioning loss based on the predicted positioning probability map and the true positioning map includes: determining the positioning loss using a cross entropy loss function according to the predicted positioning probability map, the true positioning map, and the positive sample weight, wherein the positive sample weight is the weight corresponding to the pixel point in the true positioning map used to indicate the position of the human body.

[0012] According to one aspect of the present disclosure, a crowd positioning device is provided, comprising: a feature extraction module, used to perform feature extraction on at least two frames of crowd images obtained from a crowd video clip, to obtain at least two first feature maps, wherein the at least two frames of crowd images include a frame of target crowd image and at least one frame of non-target crowd image; an optical flow determination module, used to determine a first optical flow map between the target crowd image and the at least one frame of non-target crowd image; a feature fusion module, used to fuse the at least two first feature maps according to the first optical flow map, to obtain a second feature map corresponding to the target crowd image; and a crowd positioning module, used to perform crowd positioning according to the second feature map, to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the position of a human body included in the target crowd image.

[0013] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0014] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0015] In an embodiment of the present disclosure, feature extraction is performed on at least two frames of crowd images obtained from a crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include a frame of target crowd image and at least one frame of non-target crowd image; a first optical flow map between the target crowd image and the at least one frame of non-target crowd image is determined; based on the first optical flow map, the at least two first feature maps are fused to obtain a second feature map corresponding to the target crowd image; crowd positioning is performed based on the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the positions of human bodies included in the target crowd image. In the crowd positioning process, the first optical flow map between the target crowd image and at least one frame of non-target crowd image obtained from the crowd video clip is used to fuse at least two first feature maps corresponding to the target crowd image and at least one frame of non-target crowd image. The fused second feature map can reflect both the inter-frame temporal information of the target crowd image and other non-target crowd images and the intra-frame spatial information of the target crowd image, so that after the second feature map is used to locate the crowd, a target positioning map with a higher accuracy corresponding to the target crowd image can be obtained, thereby effectively improving the accuracy of crowd positioning.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0018] Figure 1 A flow chart showing a crowd positioning method according to an embodiment of the present disclosure is shown;

[0019] Figure 2 A schematic diagram showing a crowd positioning neural network according to an embodiment of the present disclosure;

[0020] Figure 3 A block diagram of a crowd positioning device according to an embodiment of the present disclosure is shown;

[0021] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0022] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0024] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0025] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0026] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0027] Figure 1 A flow chart of a crowd positioning method according to an embodiment of the present disclosure is shown. The crowd positioning method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The crowd positioning method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the crowd positioning method can be executed by a server. Figure 1 As shown, the crowd positioning method may include:

[0028] In step S11, feature extraction is performed on at least two frames of crowd images obtained from the crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include a frame of target crowd image and at least one frame of non-target crowd image.

[0029] The crowd video clip here is a video clip including multiple frames of crowd images, which can be obtained after an image acquisition device captures a dense crowd in a certain spatial range (for example, squares, shopping malls, subway stations, tourist attractions and other places with large traffic flow), or it can be obtained by other means, and the present disclosure does not make specific limitations on this.

[0030] In one example, after an image acquisition device acquires a video of a dense crowd and obtains an original crowd video, when the frame rate of the original crowd video is high, the difference between the crowd images of adjacent frames in the original crowd video is small. In order to better utilize the spatiotemporal relationship between crowd images of different frames, the original crowd video with a high frame rate can be downsampled to obtain a crowd video segment with a frame rate less than a threshold, so that the difference between the crowd images of adjacent frames in the crowd video segment is large, so that the spatiotemporal relationship between the crowd images of different frames in the crowd video segment can be better utilized, and better crowd positioning can be achieved. For example, the frame rate of the crowd video segment is 5 frames per second.

[0031] In one example, after the original crowd video is frame-rate downsampled to obtain a crowd video clip, since each frame of the crowd image of the crowd video clip may include other background parts in addition to the dense crowd, in order to better locate the crowd, each frame of the crowd image can be cropped to retain the dense crowd part included in each frame of the crowd image. However, the cropped frames of the crowd image need to have the same scale to ensure that subsequent image processing operations such as feature extraction and feature fusion can be performed on each frame of the crowd image.

[0032] The crowd video clip includes multiple frames of crowd images, and at least two frames of crowd images are obtained from the crowd video clip for feature extraction, wherein the number of frames of crowd images for feature extraction obtained from the crowd video clip can be determined according to actual conditions, for example, 2 frames, 3 frames, 5 frames, 7 frames, etc., and the present disclosure does not make specific limitations on this.

[0033] Feature extraction is performed on at least two frames of crowd images, specifically, the feature extraction module in the convolutional neural network is used to extract features from at least two frames of crowd images to obtain at least two first feature maps. The feature extraction process will be described in detail later in conjunction with possible implementations of the present disclosure, and will not be described here.

[0034] In step S12, a first optical flow map between the target crowd image and at least one frame of non-target crowd image is determined.

[0035] The target crowd image here is a certain frame of at least two frames of crowd images obtained from the crowd video clip. For example, it can be the first frame of the at least two frames of crowd images; it can be the last frame of the at least two frames of crowd images; in the case where the crowd images obtained from the crowd video clip are odd-numbered frames greater than or equal to 3 frames, the target crowd image can be the middle frame of the odd-numbered frames greater than or equal to 3 frames; it can also be any one of the at least two frames of crowd images, and the present disclosure does not make specific limitations on this.

[0036] The non-target crowd image here may be other frames of crowd image other than the target crowd image among at least two frames of crowd image acquired from the crowd video clip.

[0037] In the related art, when crowd positioning is performed using crowd video clips, at least two first feature maps obtained after feature extraction of at least two frames of crowd images obtained from the crowd video clips are only simply fused in the channel dimension, which fails to fully utilize the temporal information between different frames of crowd images in the crowd video clips.

[0038] Since the optical flow can reflect the change of pixels in the image sequence in the time domain, in the embodiment of the present disclosure, the first optical flow map between the target crowd image and other non-target crowd images is determined based on at least two frames of crowd images obtained from the crowd video clip, so that the spatiotemporal information between the target crowd image and other non-target crowd images can be obtained. The process of determining the first optical flow map will be described in detail later in conjunction with possible implementation methods of the present disclosure, and will not be repeated here.

[0039] In step S13, at least two first feature maps are fused according to the first optical flow map to obtain a second feature map corresponding to the target population image.

[0040] Since the first optical flow map can reflect the spatiotemporal information between the target population image and other non-target population images, at least two first feature maps are fused according to the first optical flow map, so that in the feature fusion process, both the intra-frame spatial information of the target population image and the inter-frame temporal information between different frame population images can be used to obtain a second feature map with more robust semantic features corresponding to the target population image. The fusion of at least two first feature maps according to the first optical flow map will be described in detail later in conjunction with possible implementation methods of the present disclosure, which will not be repeated here.

[0041] In step S14, crowd positioning is performed according to the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the positions of human bodies included in the target crowd image.

[0042] Since the second feature map obtained after feature fusion can reflect the spatiotemporal relationship between the target crowd image and other non-target crowd images, after using the second feature map to locate the crowd, a target location map with a high accuracy corresponding to the target crowd image can be obtained. The crowd location process will be described in detail later in conjunction with possible implementation methods of the present disclosure, which will not be repeated here.

[0043] In an embodiment of the present disclosure, feature extraction is performed on at least two frames of crowd images obtained from a crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include a frame of target crowd image and at least one frame of non-target crowd image; a first optical flow map between the target crowd image and the at least one frame of non-target crowd image is determined; based on the first optical flow map, the at least two first feature maps are fused to obtain a second feature map corresponding to the target crowd image; crowd positioning is performed based on the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the positions of human bodies included in the target crowd image.

[0044] Compared with the method of performing crowd positioning based on a single crowd image in the related art, using crowd video clips to perform crowd positioning of target crowd images can mine the temporal information in the crowd video clips, so as to improve the crowd positioning accuracy.

[0045] In addition, compared with the method of simply fusing at least two first feature maps at the channel level in the related art, in the crowd positioning process, the optical flow map between the target crowd image and other non-target crowd images can be used to capture the correlation between the pixels in the crowd video clip in terms of temporal information and spatial information, so that the fusion of at least two first feature maps can mine more sufficient and robust feature information, so that the fused second feature map can reflect the spatiotemporal relationship between the target crowd image and other non-target crowd images, and then after using the second feature map to locate the crowd, a target positioning map with a higher accuracy corresponding to the target crowd image can be obtained, thereby effectively improving the accuracy of crowd positioning.

[0046] In a possible implementation, a feature extraction module in a convolutional neural network may be used to extract features from at least two frames of crowd images obtained from a crowd video clip to obtain at least two first feature maps.

[0047] In one example, the feature extraction module in the convolutional neural network can be composed of convolutional layers (e.g., the first 13 convolutional layers of the VGG-16 network) in a deep convolutional neural network trained on a public image dataset (e.g., ImageNet) in the field of computer vision.

[0048] For example, three crowd images I1, I2 and I3 are obtained from a crowd video clip, where I1, I2, I3∈R H×W×3 , H and W are the height and width of the crowd image respectively, and the number of channels of the crowd image is 3 (for example, RGB three channels). The three frames of crowd images I1, I2 and I3 are input into the feature extraction module of the convolutional neural network for feature extraction, and three first feature maps can be obtained: the first feature map F1 corresponding to the crowd image I1, the first feature map F2 corresponding to the crowd image I2, and the first feature map F3 corresponding to the crowd image I3. Among them, the size of the first feature map is reduced to 1 / 8 of the corresponding frame crowd image, and the number of channels is 512, that is, F1, F2,

[0049] In addition to being composed of the first 13 convolutional layers of the VGG-16 network, the structure of the feature extraction module can also be set to other network structures according to actual conditions, and the present disclosure does not make specific restrictions on this. For feature extraction modules with different network structures, the size and number of channels of the first feature map obtained after feature extraction of crowd video clips can be different, and the present disclosure does not make specific restrictions on this.

[0050] In a possible implementation manner, an optical flow calculation module in a convolutional neural network may be used to determine a first optical flow map between the target population image and the non-target population image.

[0051] In one example, the optical flow calculation module may be a pre-trained PWC-Net module for calculating optical flow. The optical flow calculation module may also be a pre-trained module with other optical flow calculation functions, which is not specifically limited in the present disclosure. The training process of the optical flow calculation module may be trained according to the network training method of the relevant technology, which is not specifically limited in the present disclosure.

[0052] Still taking the above example of obtaining three frames of crowd images I1, I2 and I3 from the crowd video clip, assuming that the middle frame crowd image I2 is the target crowd image, then the crowd images I1 and I3 are non-target crowd images. The three frames of crowd images I1, I2 and I3 are input into the optical flow calculation module, and the optical flow calculation module is used to determine the first optical flow map M between the target crowd image I2 and the non-target crowd image I1. f (forward optical flow map), and the first optical flow map M between the target crowd image I2 and the non-target crowd image I3 b (Backward optical flow map). Among them, the first optical flow map M f and the first optical flow map M b The size of the original crowd image is reduced to 1 / 8 of that of the original crowd image, namely, f , In addition to using the above-mentioned optical flow calculation module, the first optical flow map may also be determined by using other optical flow calculation methods, which is not specifically limited in the present disclosure.

[0053] In a possible implementation, at least two first feature maps are fused according to the first optical flow map to obtain a second feature map corresponding to the target population image, including: according to the first optical flow map, spatially transforming the first feature map corresponding to at least one frame of non-target population image to obtain at least one third feature map corresponding to at least one frame of non-target population image; fusing the target first feature map and the at least one third feature map to obtain a second feature map, wherein the target first feature map is the first feature map corresponding to the target population image.

[0054] According to the first optical flow map between the target population image and the non-target population image, the first feature map corresponding to the non-target population image is spatially transformed to achieve feature alignment between the first feature map corresponding to the non-target population image and the target first feature map corresponding to the target population image, and obtain the aligned third feature map, so that the target first feature map and the third feature map can be more effectively fused to obtain the second feature map corresponding to the target population image.

[0055] In one example, an optical flow feature fusion module in a convolutional neural network can be used to fuse at least two first feature maps according to a first optical flow map to obtain a second feature map corresponding to the target population image.

[0056] Still using the above first feature graphs F1, F2, F3, the first optical flow graph M f , Μ b For example, the first feature map F1, F2, F3, and the first optical flow map M f , Μ b Input optical flow feature fusion module, since the first optical flow map M f It is the forward optical flow map between the target population image I2 and the non-target population image I1. Therefore, the optical flow feature fusion module is based on the first optical flow map M. f , using the following formula (1), the first feature map F1 corresponding to the non-target population image I1 is spatially transformed to achieve feature alignment between the first feature map F1 corresponding to the non-target population image I1 and the target first feature map F2 corresponding to the target population image I2, and the third feature map F1' corresponding to the non-target population image I1 is obtained:

[0057]

[0058] Among them, WARP(·) is the image affine transformation function, which is used to achieve spatial transformation.

[0059] Similarly, due to the first optical flow map Mb It is the backward optical flow map between the target group image I2 and the non-target group image I3. Therefore, the optical flow feature fusion module is based on the first optical flow map M. b , using the following formula (2), the first feature map F3 corresponding to the non-target population image I3 is spatially transformed to achieve feature alignment between the first feature map F3 corresponding to the non-target population image I3 and the target first feature map F2 corresponding to the target population image I2, and the third feature map F3' corresponding to the non-target population image I3 is obtained:

[0060]

[0061] According to the first optical flow map between the target population image and the non-target population image, the first feature map corresponding to the non-target population image is spatially transformed to obtain a third feature map corresponding to the non-target population image. In addition to sampling the above-mentioned WARP(·) image affine transformation function, other spatial transformation methods can also be sampled, which is not specifically limited in the present disclosure.

[0062] In a possible implementation, a target first feature map and at least one third feature map are fused to obtain a second feature map, including: fusing the target first feature map and at least one third feature map along a channel dimension to obtain a fourth feature map, wherein the number of channels of the fourth feature map is the sum of the number of channels of the target first feature map and the at least one third feature map; performing dimensionality reduction processing on the fourth feature map along the channel dimension to obtain a second feature map, wherein the second feature map and the target first feature map have the same size and number of channels.

[0063] Since the third feature map is determined after spatial transformation of the first optical flow map between the target population image and the non-target population image, it can reflect the spatiotemporal relationship between the target population image and the non-target population image. Therefore, compared with the method of simply fusing multiple first feature maps at the channel level in the related art, the target first feature map and the third feature map corresponding to the target population image are fused along the channel dimension, so as to obtain a fourth feature map with more complete feature information and greater robustness.

[0064] Since the number of channels of the fourth feature map obtained after feature fusion along the channel dimension is the sum of the number of channels of at least two first feature maps, in order to make the fused feature map have the same size and number of channels as the feature map before fusion, the fourth feature map is subjected to dimensionality reduction processing along the channel dimension, thereby obtaining the second feature map after feature enhancement corresponding to the target population image.

[0065] Among them, the dimensionality reduction processing along the channel dimension can be achieved by using convolution activation processing, and can also be achieved by other channel dimensionality reduction processing methods, which is not specifically limited in the present disclosure.

[0066] Still taking the above-mentioned first feature map F2 and the third feature map F1' and F3' as examples, the optical flow feature fusion module can use the following formula (3) to fuse the first feature map F2 and the third feature map F1' and F3' of the target along the channel dimension to obtain the fourth feature map F c :

[0067]

[0068] Among them, cat(·) is used to indicate the feature fusion operation along the channel dimension. Using the convolution-activation layer in the optical flow feature module, the fourth feature map F c Perform convolution activation processing to obtain a second feature map with the same size and number of channels as the target first feature map F2 The second feature map F can reflect the spatiotemporal relationship between the target population image I2 and other non-target population images I1 and I3.

[0069] In another embodiment, when only two frames of crowd images I1 and I2 are obtained from the crowd video clip, and the crowd image I2 is the target crowd image, and the crowd image I1 is the non-target crowd image, the method for obtaining the second feature map corresponding to the target crowd image I2 is as follows: determining the first feature maps F1 and F2 corresponding to the crowd images I1 and I2; determining the first optical flow map M between the target crowd image I2 and the non-target crowd image I1 through the optical flow calculation module f (Forward optical flow map); the first feature map F1 and F2, and the first optical flow map M f Input the optical flow feature fusion module, use the above formula (1) to obtain the third feature map F1'; fuse the first feature map F2 and the third feature map F1' to obtain the second feature map F corresponding to the target population image I2. The method of fusing the first feature map F2 and the third feature map F1' to obtain the second feature map F is similar to the method of the above formula (3), the only difference is that the number of input feature maps is different, which will not be repeated here.

[0070] In another embodiment, when only two frames of crowd images I2 and I3 are obtained from the crowd video clip, and the crowd image I2 is the target crowd image, and the crowd image I3 is the non-target crowd image, the method for obtaining the second feature map corresponding to the target crowd image I2 is as follows: determining the first feature maps F2 and F3 corresponding to the crowd images I2 and I3; determining the first optical flow map M between the target crowd image I2 and the non-target crowd image I3 through the optical flow calculation module b (Backward optical flow map); The first feature map F2 and F3, and the first optical flow map M bInput the optical flow feature fusion module, use the above formula (2) to obtain the third feature map F3'; fuse the first feature map F2 and the third feature map F3' to obtain the second feature map F corresponding to the target population image I2. The method of fusing the first feature map F2 and the third feature map F3' to obtain the second feature map F is similar to the method of the above formula (3), the only difference is that the number of input feature maps is different, which will not be repeated here.

[0071] In a possible implementation, crowd positioning is performed according to the second feature map to obtain a target positioning map corresponding to the target crowd image, including: performing crowd positioning according to the second feature map to obtain a first positioning probability map, wherein the first positioning probability map is used to indicate the probability that each pixel in the target crowd image is a human body; performing image processing on the first positioning probability map according to a probability threshold to obtain a target positioning map.

[0072] Since the second feature map can reflect the spatiotemporal relationship between the target crowd image and other non-target crowd images, the second feature map is used to locate the crowd, and a first positioning probability map with a higher accuracy corresponding to the target crowd image can be obtained. However, since the first positioning probability map is only used to indicate the probability that each pixel in the target crowd image is a human body, the first positioning probability map is processed by a pre-set probability threshold, so that a target positioning map for indicating the position of the human body included in the target crowd image can be effectively obtained.

[0073] The process of locating a crowd based on the second feature map is described in detail below in conjunction with possible implementations of the present disclosure.

[0074] In a possible implementation, crowd positioning is performed according to the second feature map to obtain a first positioning probability map, including: performing convolution processing on the second feature map to obtain a seventh feature map; performing transposed convolution processing on the seventh feature map to obtain an eighth feature map, wherein the eighth feature map and the target crowd image have the same size; and performing convolution processing on the eighth feature map to obtain a first positioning probability map.

[0075] By performing convolution processing on the second feature map to achieve further feature extraction, a seventh feature map is obtained. In order to determine the probability that each pixel in the target population image is a human body, the seventh feature map is transposed and convolved so that an eighth feature map having the same size as the target population image can be obtained. The eighth feature map is then convolved to effectively obtain a first positioning probability map indicating the probability that each pixel in the target population image is a human body.

[0076] The first positioning probability map has the same size as the target crowd image. Taking the above target crowd image I2 as an example, when the target crowd image is I2∈R H×W×3In the case of, the first positioning probability map can be written as in, Used to indicate the probability that I2(x) is a human body, x is the target population image I2 and the first positioning probability map The coordinates of the pixels with the same relative position in .

[0077] In a possible implementation, a positioning prediction module in a convolutional neural network can be used to locate the crowd on the second feature map to obtain a first positioning probability map.

[0078] In one example, the positioning prediction module in the convolutional neural network may include a convolution layer for performing convolution processing and a transposed convolution layer for performing transposed convolution processing. The specific structure of the positioning prediction module can be set according to actual conditions (for example, the number of convolution layers, and the arrangement of each layer, etc.), and the present disclosure does not specifically limit this.

[0079] In one example, still taking the above-mentioned second feature map F as an example, the second feature map F is input into the positioning prediction module in the convolutional neural network. The process of the positioning prediction module for crowd positioning the second feature map F can be specifically described as follows: using three convolutional layers (the convolution kernel size is 3, the hole rate is 2, and the number of channels is 512), the second feature map Z is convolved to achieve further feature extraction and obtain the seventh feature map; then using three transposed convolutional layers (the convolution kernel size is 4, the step size is 2, and the number of channels is 256, 128, and 64, respectively), and a convolutional layer connected after each transposed convolutional layer (the convolution kernel size is 3, the hole rate is 2, and the number of channels is 256, 128, and 64, respectively), the seventh feature map is transposed and convolved to achieve the transformation of the feature into the size of the target crowd image, and the eighth feature map with the same size as the target crowd image is obtained. Finally, a 1×1 convolutional layer is used to convolve the eighth feature map to transform the number of channels of the feature to 1, and finally output the first positioning probability map.

[0080] Since the first positioning probability map is only used to indicate the probability that each pixel in the target population image is a human body, the first positioning probability map is processed by a pre-set probability threshold, so that a target positioning map for indicating the position of the human body included in the target population image can be effectively obtained. The specific value of the probability threshold can be determined according to actual conditions, and the present disclosure does not specifically limit this.

[0081] In a possible implementation, the first positioning probability map is subjected to image processing according to a probability threshold to obtain a target positioning map, including: performing an average pooling operation on the first positioning probability map to obtain a mean pooling map; performing a maximum pooling operation on the mean pooling map to obtain a maximum pooling map; obtaining a second positioning probability map according to the mean pooling map and the maximum pooling map; and performing threshold segmentation on the second positioning probability map according to the probability threshold to obtain a target positioning map.

[0082] An average pooling operation is performed on the first positioning probability map to obtain a mean pooling map, and a maximum pooling operation is performed on the mean pooling map to obtain a maximum pooling map, thereby effectively suppressing image noise.

[0083] In one example, first, an average pooling operation is performed on the first positioning probability map using a first pooling kernel with a size of 3 and a step size of 1 to obtain a mean pooling map; then, a maximum pooling operation is performed on the mean pooling map using a second pooling kernel with a size of 3 and a step size of 1 to obtain a maximum pooling map. The specific values ​​of the size and step size of the first pooling kernel and the second pooling kernel can be set according to actual conditions, and the present disclosure does not specifically limit this.

[0084] Peak screening is performed on the mean pooling map and the maximum pooling map, so as to obtain a second positioning probability map with higher accuracy.

[0085] In one example, the pixel values ​​of the pixels in the mean pooling map and the maximum pooling map are used to indicate the probability that the corresponding pixel in the target population image is a human body. The mean pooling map and the maximum pooling map are compared pixel by pixel. For the pixels at the same relative position in the mean pooling map and the maximum pooling map, if the pixel values ​​are the same probability values, the pixel values ​​of the pixels at the same relative position in the second positioning probability map are determined as the probability value; for the pixels at the same relative position in the mean pooling map and the maximum pooling map, if the pixel values ​​are different probability values, the pixel values ​​of the pixels at the same relative position in the second positioning probability map are determined as 0.

[0086] For example, when the pixel value of the pixel point (i, j) (the pixel point at the i-th row and the j-th column) in the mean pooling map is 0.7 and the pixel value of the pixel point (i, j) in the maximum pooling map is 0.7, the pixel value of the pixel point (i, j) in the second positioning probability map is determined to be 0.7.

[0087] For another example, when the pixel value of the pixel point (i, j) in the mean pooling map is 0.7 and the pixel value of the pixel point (i, j) in the maximum pooling map is 0.5, the pixel value of the pixel point (i, j) in the second positioning probability map is determined to be 0.

[0088] Finally, the second positioning probability map is threshold segmented by using a preset probability threshold, so that the final required target positioning map can be obtained, thereby effectively determining the position of the human body included in the target crowd image.

[0089] In one example, the probability value corresponding to each pixel in the second positioning probability map is compared with the probability threshold. When the pixel value of a certain pixel is greater than or equal to the probability threshold, the pixel value of the pixel with the same relative position in the target positioning map is determined to be 1; when the pixel value of a certain pixel is less than the probability threshold, the pixel value of the pixel with the same relative position in the target positioning map is determined to be 0.

[0090] The target positioning map and the target population image have the same size, and the position of the pixel point with a pixel value of 1 in the target positioning map is used to indicate the position of the human body included in the target population image. For example, when the pixel value of the pixel point (i, j) in the target positioning map is 1, the position of the pixel point (i, j) in the target population image corresponds to the human body; when the pixel value of the pixel point (i, j) in the target positioning map is 0, the position of the pixel point (i, j) in the target population image corresponds to the part other than the head of the human body.

[0091] According to the target positioning map, the positions of the human bodies included in the target crowd image can be determined to achieve crowd positioning and provide data basis for other crowd analysis tasks (e.g., crowd counting, group behavior analysis, etc.). For example, by counting the number of pixels with a pixel value of 1 in the target positioning map, the number of human bodies included in the target crowd image can be obtained to achieve crowd counting of the target crowd image. For another example, by counting the distribution of pixel points with a pixel value of 1 in the target positioning map, the behavior trajectory of the human bodies included in the target crowd image can be obtained to achieve crowd behavior analysis of the target crowd image.

[0092] In a possible implementation, the crowd positioning method is implemented by a crowd positioning neural network, and the training samples of the crowd positioning neural network include crowd sample video clips and real positioning maps corresponding to target crowd sample images in the crowd sample video clips; the training method of the crowd positioning neural network is as follows: through the crowd positioning neural network, a predicted positioning probability map corresponding to the target crowd sample image is determined, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the target crowd sample image is a human body; based on the predicted positioning probability map and the real positioning map, the positioning loss is determined; based on the positioning loss, the crowd positioning neural network is optimized.

[0093] In order to quickly realize crowd positioning, a crowd positioning neural network can be pre-trained based on the above crowd positioning method, and then in practical applications, the crowd positioning neural network can be used to quickly and effectively realize the recognition and positioning of crowd images.

[0094] A training sample for network training of a crowd positioning neural network is pre-constructed, wherein the training sample includes a crowd sample video clip and a real positioning map corresponding to a target crowd sample image in the crowd sample video clip.

[0095] The crowd sample video clip here is a video clip including multiple frames of crowd sample images, which can be obtained after the image acquisition device performs video acquisition on dense crowds in a certain spatial range (for example, squares, shopping malls, subway stations, tourist attractions and other places with large traffic flow), or it can be obtained by other means, and the present disclosure does not make specific limitations on this.

[0096] In a possible implementation, the crowd positioning method further includes: acquiring an original crowd sample video; downsampling the original crowd sample video to obtain a crowd sample video segment, wherein the frame rate of the crowd sample video segment is less than a threshold.

[0097] After the image acquisition device acquires the video of the dense crowd and obtains the original crowd sample video, when the frame rate of the original crowd sample video is high, the difference between the crowd sample images of adjacent frames in the original crowd sample video is small. In order to better utilize the spatiotemporal relationship between the crowd sample images of different frames, the original crowd sample video with a high frame rate can be frame-rate downsampled to obtain a crowd sample video segment with a frame rate less than a threshold, so that the difference between the crowd sample images of adjacent frames in the crowd sample video segment is large, so that the spatiotemporal relationship between the crowd sample images of different frames in the crowd sample video segment can be better utilized to train the crowd positioning neural network. For example, the frame rate of the crowd sample video segment is 5 frames per second.

[0098] After the original crowd sample video is frame-rate downsampled to obtain the crowd sample video clip, since each frame of the crowd sample image of the crowd sample video clip may include other background parts in addition to the dense crowd, in order to better train the crowd positioning neural network, each frame of the crowd sample image can be cropped to retain the dense crowd part included in each frame of the crowd sample image. However, the cropped crowd sample images need to have the same scale to ensure that subsequent image processing operations such as feature extraction and feature fusion can be performed on each frame of the crowd sample image.

[0099] The crowd sample video clip includes multiple frames of crowd sample images, and at least two frames of crowd sample images are obtained from the crowd sample video clip for feature extraction, wherein the specific number of the at least two frames of crowd sample images for feature extraction can be determined according to actual conditions, for example, 2 frames, 3 frames, 5 frames, 7 frames, etc., and the present disclosure does not make any specific limitation on this.

[0100] Among at least two frames of crowd sample images obtained from the crowd sample video clip, one frame of crowd sample image is randomly selected to be determined as the target crowd sample image. For example, the target crowd sample image may be the first frame of the at least two frames of crowd sample images; or the last frame of the at least two frames of crowd sample images; in the case where the crowd sample images obtained from the crowd sample video clip are odd-numbered frames greater than or equal to 3 frames, the target crowd sample image may be the middle frame of the odd-numbered frames greater than or equal to 3 frames, or may be any one of the at least two frames of crowd sample images, and the present disclosure does not specifically limit this.

[0101] The real positioning map and the target population sample image have the same size. The pixel values ​​of the pixels in the real positioning map are 0 or 1. The positions of the pixels with a pixel value of 1 are used to indicate the positions of the human body included in the target population sample image; the positions of the pixels with a pixel value of 0 are used to indicate other positions other than the human body included in the target population sample image. The relationship between the real positioning map and the target population sample image is similar to the relationship between the target positioning map and the target population image, and will not be repeated here.

[0102] In a possible implementation, the crowd positioning method further includes: determining a labeling result corresponding to a sample image of a target crowd, wherein the labeling result includes coordinates of a human body in the sample image of the target crowd; and determining a true positioning map according to the labeling result.

[0103] By annotating the human body coordinates of the sample images of the target population, the real positioning map corresponding to the sample images of the target population can be effectively determined according to the annotation results, and then the training samples for network training of the population positioning neural network can be effectively constructed according to the sample images of the target population and the real positioning map.

[0104] In one example, the target population sample image is I2'∈R H×W×3 , H and W are the height and width of the target population sample image I2' respectively, and the number of channels of the target population sample image is 3. Label the human body included in the target population sample image I2' to obtain the labeling result corresponding to the target population sample image I2' Among them, a i is the coordinate of the i-th human body in the target population sample image I2', and m is the number of human bodies included in the target population sample image I2'.

[0105] In one example, the labeling result corresponding to the target population sample image I2' can be Using the following formula (4), determine the real positioning map Y∈R corresponding to the target population sample image I2′ H×W :

[0106]

[0107] in, y is the coordinate of the pixel points with the same relative position in the target population sample image I2' and the true positioning map Y, K = [0, 1, 0; 1, 1, 1; 0, 1, 0] is the convolution kernel, ψ(·) is the convolution result map, δ(·) is the multivariate delta function, and its specific form can be shown in the following formula (5):

[0108]

[0109] According to the annotation results corresponding to the sample images of the target population, in addition to using the above formula (4) to determine the true positioning map, other methods can also be used to determine it, and the present disclosure does not make specific limitations on this.

[0110] After determining the above-mentioned training samples, the crowd localization neural network is trained using the above-mentioned training samples. First, the crowd localization neural network is used to determine the predicted localization probability map corresponding to the target crowd sample image included in the training samples, which is used to indicate the probability that each pixel point in the target crowd sample image is a human body.

[0111] In a possible implementation, a predicted positioning probability map corresponding to a target crowd sample image is determined through a crowd positioning neural network, including: extracting features from at least two frames of crowd sample images obtained from a crowd sample video clip to obtain at least two fifth feature maps, wherein the at least two frames of crowd images include a target crowd sample image and at least one frame of non-target crowd sample image; determining a second optical flow map between the target crowd sample image and the at least one frame of non-target crowd sample image; based on the second optical flow map, fusing the at least two fifth feature maps to obtain a sixth feature map corresponding to the target crowd sample image; and positioning the crowd based on the sixth feature map to obtain a predicted positioning probability map.

[0112] The crowd localization neural network includes a feature extraction module. After at least two frames of crowd sample images obtained from the crowd sample video clip are subjected to feature extraction by the feature extraction module, at least two fifth feature maps can be obtained. For example, three frames of crowd sample images are obtained from the crowd sample video clip: I1', I'2 and I'3, wherein the crowd sample image I'2 is the target crowd sample image. Using the feature extraction module in the crowd localization neural network, feature extraction is performed on the crowd sample images I'1, I'2 and I'3 to obtain three fifth feature maps: the fifth feature map F1' corresponding to the crowd sample image I'1, the fifth feature map F2' corresponding to the crowd image I'2 and the fifth feature map F3' corresponding to the crowd image I'3.

[0113] The network structure of the feature extraction module in the crowd positioning neural network is similar to the network structure of the feature extraction module in the above-mentioned convolutional neural network. The feature extraction process of the feature extraction module in the crowd positioning neural network for at least two frames of crowd sample images is similar to the feature extraction process of the feature extraction module in the above-mentioned convolutional neural network for at least two frames of crowd images, which will not be repeated here.

[0114] The crowd positioning neural network includes an optical flow calculation module, which is used to determine the second optical flow graph M' between the target crowd sample image I'2 and the non-target crowd sample image I'1 f (forward optical flow map), and the second optical flow map M′ between the target population sample image I′2 and the non-target population sample image I′3 b (Backward optical flow graph).

[0115] The network structure of the optical flow calculation module in the crowd positioning neural network is similar to the network structure of the optical flow calculation module in the above-mentioned convolutional neural network. The process of determining the second optical flow by the optical flow calculation module in the crowd positioning neural network is similar to the process of determining the first optical flow by the optical flow calculation module in the above-mentioned convolutional neural network, and will not be repeated here.

[0116] The crowd positioning neural network includes an optical flow feature fusion module, according to the second optical flow map M' f and the second optical flow map M′ b , the three fifth feature maps F1', F2' and F3' are fused to obtain the sixth feature map F' corresponding to the target population sample image I'2.

[0117] The network structure of the optical flow feature fusion module in the crowd positioning neural network is similar to the network structure of the optical flow feature fusion module in the above-mentioned convolutional neural network. The process of the optical flow feature fusion module in the crowd positioning neural network fusing multiple fifth feature maps according to the second optical flow map is similar to the process of the optical flow feature fusion module in the above-mentioned convolutional neural network fusing multiple first feature maps according to the first optical flow map, which will not be repeated here.

[0118] The crowd positioning neural network also includes a positioning prediction module. The positioning prediction module is used to perform crowd positioning on the sixth feature map to obtain a predicted positioning probability map indicating the probability that each pixel in the target crowd sample image is a human body. Still taking the sixth feature map F' as an example, the positioning prediction module in the crowd positioning neural network is used to perform crowd positioning on the sixth feature map F' to obtain a predicted positioning probability map

[0119] The network structure of the positioning prediction module in the crowd positioning neural network is similar to the network structure of the positioning prediction module in the above-mentioned convolutional neural network. The crowd positioning process of the sixth feature map by the positioning prediction module in the crowd positioning neural network is similar to the crowd positioning process of the second feature map by the positioning prediction module in the above-mentioned convolutional neural network, which will not be repeated here.

[0120] Since the predicted positioning probability map is used to indicate the probability that each pixel in the target population sample image is a human body, and the true positioning map is used to indicate the position of the human body included in the target population sample image, the positioning loss of the crowd neural network can be determined based on the predicted positioning probability map and the true positioning map, and then based on the positioning loss, the network parameters of the crowd positioning neural network can be adjusted to optimize the crowd positioning neural network.

[0121] After the crowd localization neural network obtains the predicted localization probability map corresponding to the target crowd sample image, the localization loss of the crowd localization neural network can be determined according to the difference between the predicted localization probability map and the true localization map.

[0122] In a possible implementation, the positioning loss is determined based on the predicted positioning probability map and the true positioning map, including: determining the positioning loss using a cross entropy loss function according to the predicted positioning probability map, the true positioning map and the positive sample weight, wherein the positive sample weight is the weight corresponding to the pixel point in the true positioning map used to indicate the position of the human body.

[0123] In one example, based on the predicted positioning probability map and the true positioning map Y, the positioning loss L can be determined by the cross entropy loss function shown in the following formula (6):

[0124]

[0125] Among them, H and W are the predicted positioning probability maps and the height and width of the true positioning map Y, y is the predicted positioning probability map The coordinates of the pixel points with the same relative position as the real positioning image Y, Y(y) takes the value of 0 or 1. When Y(y) takes the value of 1, it can indicate that the pixel point y in the target population sample image is the human body position. When Y(y) takes the value of 0, it can indicate that the pixel point y in the target population sample image is other than the human body. The value is [0,1], It is used to indicate the probability that the pixel y in the sample image of the target population is a human body, and λ is the positive sample weight.

[0126] In the sample images of the target population, the pixels where the human body is located can be considered as positive samples, and the other pixels are considered as negative samples. Since the proportion of the background in the sample images of the target population may be much larger than the proportion of the human body, that is, the number of negative samples is much larger than the number of positive samples, by setting the weight of the positive samples, the positive and negative samples can be balanced during the training process.

[0127] In addition to the cross entropy loss function shown in the above formula (6), the positioning loss can also be determined by other loss functions, which is not specifically limited in the present disclosure.

[0128] After determining the positioning loss, the network parameters corresponding to the crowd positioning neural network can be adjusted according to the positioning loss to achieve the optimization of the crowd positioning network, and iterative training can be performed using the above-mentioned network training method until the iterative training meets the preset training conditions, and finally the trained crowd positioning neural network is obtained.

[0129] In a possible implementation, the network parameters are adjusted using a gradient descent method according to the positioning loss.

[0130] For example, the network parameters for the i-th iteration training are θ i , using the network parameters θ i The positioning loss determined after network training is L, then the network parameter θ during the i+1th iteration training is i+1 It can be determined by the following formula (7):

[0131]

[0132] Wherein, ▽ represents the gradient operator symbol, and γ is the network learning rate. The specific value of the network learning rate γ can be determined according to actual conditions, for example, γ=0.0001, which is not limited in the present disclosure.

[0133] In a possible implementation, the preset training condition may be network convergence. For example, the above network training method is used to perform iterative training until the network parameters no longer change, and the network is considered to have reached convergence, and the trained crowd positioning neural network is determined to be obtained.

[0134] In a possible implementation, the preset training condition may be an iteration threshold. For example, the above network training method is used to perform iterative training until the number of iterations reaches the iteration threshold, and the trained crowd positioning neural network is determined to be obtained.

[0135] In a possible implementation, the preset training condition may be a positioning threshold. For example, the above network training method is used to perform iterative training until the positioning accuracy rate corresponding to the network is greater than the positioning threshold, and a trained crowd positioning neural network is determined to be obtained.

[0136] In addition to the above-mentioned network convergence, iteration threshold, or positioning threshold, the preset training conditions can also be set to other training conditions according to actual conditions, and the present disclosure does not make specific limitations on this.

[0137] Figure 2 FIG. 2 is a schematic diagram of a crowd positioning neural network according to an embodiment of the present disclosure. Figure 2 As shown, the crowd positioning neural network 20 includes a feature extraction module 21, an optical flow calculation module 22, an optical flow feature fusion module 23 and a positioning prediction module 24.

[0138] like Figure 2 As shown, at least two frames of crowd images obtained from a crowd video clip that need to be positioned are input into a crowd positioning neural network 20, and a feature extraction module 21 extracts features from the at least two frames of crowd images to obtain at least two first feature maps; an optical flow calculation module 22 determines a first optical flow (for example, it may include a forward optical flow and / or a backward optical flow) between a target crowd image and a non-target crowd image in at least two frames of crowd images; an optical flow feature fusion module 23 fuses at least two first feature maps according to the first optical flow to obtain a second feature map corresponding to the target crowd image; and a positioning prediction module 24 performs crowd positioning on the second feature map to obtain a target positioning map for indicating the position of a human body included in the target crowd image. The specific processing procedures of the feature extraction module 21, the optical flow calculation module 22, the optical flow feature fusion module 23, and the positioning prediction module 24 are similar to the above-mentioned related descriptions and will not be repeated here.

[0139] In the disclosed embodiment, a crowd positioning neural network is used. During the crowd positioning process, at least two first feature maps corresponding to at least two frames of crowd images are fused using a first optical flow map between a target crowd image and a non-target crowd image in at least two frames of crowd images obtained from a crowd video clip. The fused second feature map can reflect both the inter-frame temporal information of the target crowd image and other non-target crowd images and the intra-frame spatial information of the target crowd image. After the crowd is positioned using the second feature map, a target positioning map with a higher accuracy corresponding to the target crowd image can be obtained, thereby effectively improving the accuracy of crowd positioning.

[0140] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0141] In addition, the present disclosure also provides a crowd positioning device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any crowd positioning method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0142] Figure 3 FIG. 2 is a block diagram of a crowd positioning device according to an embodiment of the present disclosure. Figure 3 As shown, the device 30 includes:

[0143] A feature extraction module 31 is used to extract features from at least two frames of crowd images obtained from the crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include one frame of target crowd image and at least one frame of non-target crowd image;

[0144] An optical flow determination module 32, configured to determine a first optical flow map between a target crowd image and at least one frame of a non-target crowd image;

[0145] A feature fusion module 33 is used to fuse at least two first feature maps according to the first optical flow map to obtain a second feature map corresponding to the target population image;

[0146] The crowd positioning module 34 is used to perform crowd positioning according to the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the positions of the human bodies included in the target crowd image.

[0147] In a possible implementation, the feature fusion module 33 includes:

[0148] A spatial transformation submodule, configured to perform spatial transformation on a first feature map corresponding to at least one frame of non-target crowd image according to the first optical flow map, to obtain at least one third feature map corresponding to at least one frame of non-target crowd image;

[0149] The feature fusion submodule is used to fuse the target first feature map and at least one third feature map to obtain a second feature map, wherein the target first feature map is the first feature map corresponding to the target population image.

[0150] In a possible implementation, the feature fusion submodule is specifically used to:

[0151] The target first feature map and the at least one third feature map are fused along the channel dimension to obtain a fourth feature map, wherein the number of channels of the fourth feature map is the sum of the number of channels of the target first feature map and the at least one third feature map;

[0152] The fourth feature map is subjected to dimensionality reduction processing along the channel dimension to obtain a second feature map, wherein the second feature map and the target first feature map have the same size and number of channels.

[0153] In a possible implementation, the crowd positioning module 34 includes:

[0154] A first determination submodule is used to locate the crowd according to the second feature map to obtain a first positioning probability map, wherein the first positioning probability map is used to indicate the probability that each pixel in the target crowd image is a human body;

[0155] The second determination submodule is used to perform image processing on the first positioning probability map according to the probability threshold to obtain a target positioning map.

[0156] In a possible implementation, the second determining submodule is specifically configured to:

[0157] Perform an average pooling operation on the first positioning probability map to obtain a mean pooling map;

[0158] Perform a maximum pooling operation on the mean pooling map to obtain a maximum pooling map;

[0159] According to the mean pooling map and the maximum pooling map, a second positioning probability map is obtained;

[0160] According to the probability threshold, the second positioning probability map is threshold segmented to obtain a target positioning map.

[0161] In a possible implementation, the device 30 implements the crowd positioning method through a crowd positioning neural network, and the training samples of the crowd positioning neural network include a crowd sample video clip and a real positioning map corresponding to a target crowd sample image in the crowd sample video clip;

[0162] The device 30 also includes a network training module, including:

[0163] A third determination submodule is used to determine a predicted positioning probability map corresponding to the target population sample image through a crowd positioning neural network, wherein the predicted positioning probability map is used to indicate the probability that each pixel in the target population sample image is a human body;

[0164] A fourth determination submodule, used to determine the positioning loss based on the predicted positioning probability map and the true positioning map;

[0165] The optimization submodule is used to optimize the crowd localization neural network based on the localization loss.

[0166] In a possible implementation, the third determining submodule is specifically configured to:

[0167] Performing feature extraction on at least two frames of crowd sample images obtained from the crowd sample video clip to obtain at least two fifth feature maps, wherein the at least two frames of crowd images include a target crowd sample image and at least one frame of non-target crowd sample image;

[0168] Determine a second optical flow map between the target population sample image and at least one frame of non-target population sample image;

[0169] According to the second optical flow map, at least two fifth feature maps are fused to obtain a sixth feature map corresponding to the target population sample image;

[0170] Crowd positioning is performed according to the sixth characteristic map to obtain a predicted positioning probability map.

[0171] In a possible implementation, the fourth determining submodule is specifically configured to:

[0172] The positioning loss is determined using the cross entropy loss function based on the predicted positioning probability map, the true positioning map and the positive sample weights, where the positive sample weights are the weights corresponding to the pixel points in the true positioning map that indicate the location of the human body.

[0173] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0174] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0175] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0176] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0177] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0178] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 4 As shown, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0179] Reference Figure 4, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0180] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0181] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0182] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0183] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0184] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0185] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0186] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0187] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0188] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0189] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0190] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 5 As shown, the electronic device 1900 may be provided as a server. Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0191] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003).TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0192] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0193] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0194] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0195] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0196] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0197] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0198] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0199] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0200] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0201] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0202] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A crowd positioning method, It is characterized in that include: Performing feature extraction on at least two frames of crowd images obtained from the crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include one frame of target crowd image and at least one frame of non-target crowd image; Determine a first optical flow map between the target crowd image and the at least one frame of non-target crowd image; According to the first optical flow map, the at least two first feature maps are fused to obtain a second feature map corresponding to the target population image, wherein the second feature map is used to reflect the spatiotemporal relationship between the target population image and the non-target population image; Performing crowd positioning according to the second feature map to obtain a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate positions of human bodies included in the target crowd image; The step of fusing the at least two first feature maps according to the first optical flow map to obtain a second feature map corresponding to the target population image includes: According to the first optical flow map, spatially transform the first feature map corresponding to the at least one frame of the non-target crowd image to obtain at least one third feature map corresponding to the at least one frame of the non-target crowd image; The target first feature map and the at least one third feature map are fused to obtain the second feature map, wherein the target first feature map is the first feature map corresponding to the target population image.

2. The method according to claim 1, It is characterized in that The step of fusing the target first feature map and the at least one third feature map to obtain the second feature map includes: Fusing the target first feature map and the at least one third feature map along a channel dimension to obtain a fourth feature map, wherein the number of channels of the fourth feature map is the sum of the number of channels of the target first feature map and the at least one third feature map; Perform dimensionality reduction processing on the fourth feature map along the channel dimension to obtain the second feature map, wherein the second feature map and the target first feature map have the same size and number of channels.

3. The method according to claim 1 or 2, It is characterized in that The performing crowd positioning according to the second feature map to obtain a target positioning map corresponding to the target crowd image includes: Performing crowd positioning according to the second feature map to obtain a first positioning probability map, wherein the first positioning probability map is used to indicate the probability that each pixel in the target crowd image is a human body; According to the probability threshold, image processing is performed on the first positioning probability map to obtain the target positioning map.

4. The method according to claim 3, It is characterized in that The step of performing image processing on the first positioning probability map according to the probability threshold to obtain the target positioning map includes: Performing an average pooling operation on the first positioning probability map to obtain a mean pooling map; Performing a maximum pooling operation on the mean pooling map to obtain a maximum pooling map; Obtaining a second positioning probability map according to the mean pooling map and the maximum pooling map; According to the probability threshold, the second positioning probability map is threshold segmented to obtain the target positioning map.

5. The method according to claim 1 or 2, It is characterized in that The crowd positioning method is implemented by a crowd positioning neural network, wherein the training samples of the crowd positioning neural network include a crowd sample video clip and a real positioning map corresponding to a target crowd sample image in the crowd sample video clip; The training method of the crowd positioning neural network is as follows: Determine, by means of the crowd positioning neural network, a predicted positioning probability map corresponding to the target crowd sample image, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the target crowd sample image is a human body; Determining a positioning loss based on the predicted positioning probability map and the actual positioning map; Based on the positioning loss, the crowd positioning neural network is optimized.

6. The method according to claim 5, It is characterized in that Determining the predicted positioning probability map corresponding to the target crowd sample image through the crowd positioning neural network includes: Performing feature extraction on at least two frames of crowd sample images obtained from the crowd sample video clip to obtain at least two fifth feature maps, wherein the at least two frames of crowd images include the target crowd sample image and at least one frame of non-target crowd sample image; Determine a second optical flow map between the target population sample image and the at least one frame of non-target population sample image; According to the second optical flow map, the at least two fifth feature maps are fused to obtain a sixth feature map corresponding to the target population sample image; Crowd positioning is performed according to the sixth characteristic map to obtain the predicted positioning probability map.

7. The method according to claim 5, It is characterized in that The determining of the positioning loss based on the predicted positioning probability map and the true positioning map includes: The positioning loss is determined using a cross entropy loss function according to the predicted positioning probability map, the true positioning map and the positive sample weight, wherein the positive sample weight is a weight corresponding to a pixel point in the true positioning map used to indicate the position of a human body.

8. A crowd positioning device, It is characterized in that include: A feature extraction module, configured to extract features from at least two frames of crowd images obtained from a crowd video clip to obtain at least two first feature maps, wherein the at least two frames of crowd images include one frame of a target crowd image and at least one frame of a non-target crowd image; An optical flow determination module, used to determine a first optical flow graph between the target crowd image and the at least one frame of non-target crowd image; a feature fusion module, configured to fuse the at least two first feature maps according to the first optical flow map to obtain a second feature map corresponding to the target population image, wherein the second feature map is used to reflect the spatiotemporal relationship between the target population image and the non-target population image; A crowd positioning module, used for positioning a crowd according to the second feature map, and obtaining a target positioning map corresponding to the target crowd image, wherein the target positioning map is used to indicate the positions of human bodies included in the target crowd image; Wherein, the feature fusion module includes: a spatial transformation submodule, configured to perform spatial transformation on the first feature map corresponding to the at least one frame of the non-target crowd image according to the first optical flow map, so as to obtain at least one third feature map corresponding to the at least one frame of the non-target crowd image; The feature fusion submodule is used to fuse the target first feature map and the at least one third feature map to obtain the second feature map, wherein the target first feature map is the first feature map corresponding to the target population image.

9. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Video human body key point detection method and device, computer device and storage medium

    CN109977912A

  • Image processing method and device, electronic device and storage medium

    CN109978891A

  • Method for detecting crowd abnormal behaviors by using fused convolutional neural network

    CN112329685A

  • Crowd abnormal behavior detection method and system based on appearance motion fusion network

    CN112668431A

  • Crowd distribution prediction method and system based on double-flow convolutional neural network

    CN112767451A