Social distancing detection method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a social distance detection method and device, electronic equipment and storage medium, the method comprises: acquiring an image to be detected; predicting the positioning point information and the head frame information of each target person in the image to be detected through a pre-trained neural network model, and extracting the corresponding head depth information according to the head frame information; calculating the height information of the corresponding target person according to the positioning point information; according to the height information and the head depth information, the head of the target person is three-dimensionally reconstructed to obtain the three-dimensional head information of the target person; based on the three-dimensional head information of the target person, the social distance between multiple target persons is calculated. By calculating the height information of the target person, more accurate target head information is extracted for three-dimensional reconstruction, so that the position of the three-dimensional target head in the three-dimensional space is more accurate, thereby improving the accuracy of the social distance between the target persons.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a social distance detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In some scenarios, it is necessary to control the distance between people, such as ticket windows, bank counters, public places during flu season, and special locations with restricted gatherings. Current methods for detecting distance between people mainly rely on visual estimation by staff from the scene or surveillance video, or on image processing techniques. However, visual estimation by staff is not only time-consuming and labor-intensive, with high labor costs, but is also susceptible to subjective factors, resulting in low accuracy. Existing image processing methods primarily measure the approximate actual distance by detecting the distance between people in two-dimensional images, which also has low accuracy. Therefore, current methods for detecting distance between people suffer from low accuracy. Summary of the Invention

[0003] This invention provides a social distance detection method that can improve the accuracy of social distance detection between people.

[0004] In a first aspect, embodiments of the present invention provide a social distance detection method, the method comprising:

[0005] Acquire an image to be detected, the image to be detected including image depth information and a head to be detected, the head to be detected including the heads of multiple target persons;

[0006] The pre-trained neural network model predicts the location point information and head bounding box information of each target person in the image to be detected, and extracts the corresponding head depth information based on the head bounding box information.

[0007] Based on the location point information, calculate the height information of the corresponding target person;

[0008] Based on the height information and the head depth information, a three-dimensional reconstruction of the target person's head is performed to obtain the three-dimensional head information of the target person;

[0009] Based on the three-dimensional head information of the target individuals, the social distance between multiple target individuals is calculated.

[0010] Optionally, the pre-trained neural network model includes a common network and a first branch network, a second branch network, and a third branch network connected to the common network. The common network is used to extract common features of the first branch network, the second branch network, and the third branch network. The first branch network and the second branch network are used to extract the location point information of the human head, and the third branch network is used to extract the head bounding box information.

[0011] Optionally, the head location information includes the head center point, the head projection center point, and key points of the feet. The step of predicting the location information of each target person in the image to be detected using a pre-trained neural network model includes:

[0012] Common features of the image to be detected are extracted through the public network;

[0013] The shared features are input into the first branch network, and the Gaussian heat map of the human head is predicted through the first branch network. The center point of the human head is obtained based on the Gaussian heat map of the human head.

[0014] The shared features are input into the second branch network, and the center point of the human head projection and the key points of the human feet are predicted by the second branch network. The second branch network performs offset prediction based on the center point of the human head.

[0015] Optionally, predicting the head bounding box information of each target person in the image to be detected using a pre-trained neural network model includes:

[0016] The shared features are input into the third branch network, and the head bounding box information is predicted by the third branch network. The third branch network performs offset prediction based on the center point of the head.

[0017] Optionally, calculating the height information of the corresponding target person based on the positioning point information includes:

[0018] By combining the center point of the head, the center point of the head projection, and the key points of the feet, the height information of the corresponding target person is calculated.

[0019] Optionally, common features are extracted through the public network, including:

[0020] The image to be detected is sequentially subjected to convolution downsampling at a first multiple, a second multiple, and a third multiple to obtain a first shared feature map, a second shared feature map, and a third shared feature map. The value of the first multiple is less than the value of the second multiple, the value of the second multiple is less than the value of the third multiple, the tensor size of the first shared feature map is greater than the tensor size of the second shared feature map, and the tensor size of the second shared feature map is greater than the tensor size of the third shared feature map.

[0021] Optionally, the step of inputting the shared features into the first branch network and predicting the Gaussian heatmap of the human head through the first branch network includes:

[0022] The third shared feature map is upsampled and then convolved to obtain a first upsampled feature map with the same tensor size as the second shared feature map.

[0023] The second shared feature map is convolved and fused with the first upsampled feature map to obtain the first fused feature map.

[0024] The first fused feature is upsampled and then convolutional to obtain a second upsampled feature map with the same tensor size as the first shared feature map.

[0025] The first shared feature map is convolved and fused with the second upsampled feature to obtain the second fused feature map.

[0026] Based on the second fused feature map, a Gaussian heatmap is obtained.

[0027] Optionally, the step of performing three-dimensional reconstruction of the target person's head based on the height information and the head depth information to obtain the target person's three-dimensional head information includes:

[0028] Based on the height information, determine the reconstructed position of the target person's head in three-dimensional space;

[0029] At the reconstruction location in the three-dimensional space, the head of the target person is reconstructed in three dimensions based on the head depth information to obtain the three-dimensional head information of the target person.

[0030] Optionally, the method further includes:

[0031] The ground in the three-dimensional space is constructed based on the center point of the human head projection.

[0032] Secondly, embodiments of the present invention also provide a social distancing detection device, the device comprising:

[0033] The acquisition module is used to acquire the image to be detected, the image to be detected including image depth information and the head to be detected, the head to be detected including the heads of multiple target persons;

[0034] The prediction module is used to predict the location point information and head bounding box information of each target person in the image to be detected through a pre-trained neural network model, and extract the corresponding head depth information based on the head bounding box information.

[0035] The first calculation module is used to calculate the height information of the corresponding target person based on the positioning point information;

[0036] The reconstruction module is used to perform three-dimensional reconstruction of the target person's head based on the height information and the head depth information to obtain the three-dimensional head information of the target person.

[0037] The second calculation module is used to calculate the social distance between multiple target individuals based on the three-dimensional head information of the target individuals.

[0038] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the social distance detection method provided in embodiments of the present invention.

[0039] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the social distancing detection method provided in the embodiments of the present invention.

[0040] In this embodiment of the invention, an image to be detected is acquired, which includes image depth information and heads to be detected, comprising multiple target heads. A pre-trained neural network model is used to predict the location points and head bounding boxes of each target person in the image, and the corresponding head depth information is extracted based on the head bounding box information. The height information of the corresponding target person is calculated based on the location point information. Based on the height information and the head depth information, the head of the target person is reconstructed in three dimensions to obtain the three-dimensional head information of the target person. Based on the three-dimensional head information of the target person, the social distance between multiple target persons is calculated. By calculating the height information of the target persons, more accurate target head information is extracted for three-dimensional reconstruction, making the position of the three-dimensional target head in three-dimensional space more accurate, thereby improving the accuracy of the social distance between target persons. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a social distance detection method provided in an embodiment of the present invention;

[0043] Figure 2a This is a schematic diagram of the processing flow of a neural network model provided in an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of a method for predicting location point information provided in an embodiment of the present invention;

[0045] Figure 2b This is a schematic diagram of the processing flow of a first branch network provided in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart of a method for predicting head frame information provided in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of a social distancing detection device provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of a prediction module provided in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of a second processing submodule provided in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the structure of a reconstruction module provided in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of another social distancing detection device provided in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1 , Figure 1 This is a flowchart of a social distance detection method provided in an embodiment of the present invention, such as... Figure 1 As shown, this method is used for real-time social distance detection and includes the following steps:

[0055] 101. Obtain the image to be detected.

[0056] In this embodiment of the invention, the image to be detected includes image depth information and a head to be detected, which includes the heads of multiple target persons.

[0057] A depth camera (also known as a 3D camera) can be used to acquire images of the target area, resulting in the image to be detected, which includes the image depth information.

[0058] In one possible embodiment, the number of people in the image captured by the depth camera can be detected first. If only one person is detected, the image is not processed, meaning that the social distance of the person in the image does not need to be detected. If two or more people are detected, the image is used as the image to be detected.

[0059] The image to be detected can be a real-time captured image or an image uploaded by the user. Furthermore, the image to be detected can be a series of consecutive frames (video stream images) or independent frames (photographs).

[0060] 102. Predict the location information and head bounding box information of each target person in the image to be detected by a pre-trained neural network model, and extract the corresponding head depth information based on the head bounding box information.

[0061] In this embodiment of the invention, the location information of the target person can be head location information and foot location information. Furthermore, the location information of the target person also includes head projection location information.

[0062] The head bounding box information of the target person mentioned above can be a head detection box (x, y, h, w), where x and y are the coordinates of the center point of the head detection box, h is the height of the head detection box, and w is the width of the head detection box. The head detection box can be understood as a small image within the target image, with each small image containing the head image of a target person.

[0063] In one possible embodiment, the pre-trained neural network model can be multiple, including a head detection network model for detecting heads in the image to be detected, and a localization point extraction network model for extracting localization point information from the image to be detected.

[0064] In this embodiment of the invention, to improve the acquisition speed of head bounding box information and location point information of the target person, an optional neural network model is provided. The input is the image to be detected, and the output is the location point information and head bounding box information of the target person in the image to be detected. That is, the aforementioned location point information and head bounding box information are obtained by processing the same neural network model. The input of the aforementioned pre-trained neural network model is the image to be detected. The aforementioned neural network model includes at least a common network, a first branch network, and a third branch network. The common network is connected to both the first branch network and the third branch network. The first branch network is used to output the location point information, and the third branch network is used to output the head bounding box information.

[0065] Furthermore, the location information of the target person can be head location information and foot location information. To improve the acquisition speed of the head and foot location information of the target person, an optional neural network model is provided. The input is the image to be detected, and the output is the head location information, foot location information, and head bounding box information of the target person in the image to be detected. That is, the location information and head bounding box information are obtained by processing the same neural network model. The input of the pre-trained neural network model is the image to be detected. The neural network model includes a common network, and a first branch network, a second branch network, and a third branch network connected to the common network. The common network is used to extract the common features of the first branch network, the second branch network, and the third branch network. The first and second branch networks are used to extract the head location information. Specifically, the first branch network is used to extract the head location information, the second branch network is used to extract the foot location information, and the third branch network is used to extract the head bounding box information.

[0066] Furthermore, the location information of the target person can be head location information, head projection location information, and foot location information. To improve the acquisition speed of the target person's head location information, head projection location information, and foot location information, an optional neural network model is provided. The input is the image to be detected, and the output is the corresponding head location information, head projection location information, foot location information, and head bounding box information of the target person in the image to be detected. That is, the above-mentioned location information and head bounding box information are obtained by processing the same neural network model. The input of the pre-trained neural network model is the image to be detected. The aforementioned neural network model includes a common network, and a first branch network, a second branch network, and a third branch network connected to the common network. The common network is used to extract common features from the first, second, and third branch networks. The first and second branch networks are used to extract the head's location point information. Specifically, the first branch network is used to extract the head's location point information, the second branch network is used to extract the head's projection location point information and the foot's location point information, and the third branch network is used to extract the head bounding box information. Figure 2a As shown.

[0067] Specifically, this embodiment of the invention uses the positioning point information of the target person as an example, namely, the positioning point information of the head, the positioning point information of the head projection, and the positioning point information of the feet. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart of a method for predicting location point information provided in an embodiment of the present invention, such as... Figure 2 As shown, it includes the following steps:

[0068] 201. Extract common features of the image to be detected through public networks.

[0069] In this embodiment of the invention, the aforementioned common feature can be understood as a hidden feature including head features, head positioning point features, head projection positioning point features, and foot positioning point features, without any background features.

[0070] The aforementioned public network can be a convolutional neural network, which extracts features from the image to be detected through convolutional kernels, and obtains implicit features, including head features, head positioning point features, head projection positioning point features, and foot positioning point features, as shared features.

[0071] Furthermore, the public network in this embodiment of the invention can be a convolutional neural network with MobileNet v3 (a lightweight network) as its backbone. The public network sequentially performs convolutional downsampling on the image to be detected by a first multiple, a second multiple, and a third multiple, obtaining a first shared feature map, a second shared feature map, and a third shared feature map. The first multiple is smaller than the second multiple, the second multiple is smaller than the third multiple, the tensor size of the first shared feature map is larger than the tensor size of the second shared feature map, and the tensor size of the second shared feature map is larger than the tensor size of the third shared feature map. The above convolutional downsampling can be achieved by first performing a convolution operation on the image to obtain a feature map, and then performing downsampling to obtain a smaller feature map. Of course, the above convolution can also be achieved by changing the stride of the convolution kernel to reduce the size of the obtained feature map while performing the convolution operation on the image. For example, if the stride of the convolution kernel is 2, the obtained feature map will be half the size of the input image.

[0072] Furthermore, the first multiplier can be 4 times, the second multiplier can be 8 times, and the third multiplier can be 16 times. For example, if the input image tensor is 1024*1024, after 4x convolution downsampling, the tensor of the feature map (the first shared feature map) is 256*256; after 8x convolution downsampling, the tensor of the feature map (the second shared feature map) is 128*128; and after 16x convolution downsampling, the tensor of the feature map (the third shared feature map) is 64*64.

[0073] 202. Input the shared features into the first branch network, predict the Gaussian heatmap of the head through the first branch network, and obtain the center point of the head based on the Gaussian heatmap of the head.

[0074] In this embodiment of the invention, the first branch network predicts the Gaussian heatmap for predicting human heads. Since the Gaussian heatmap output feature map is large and has strong spatial generalization ability, the Gaussian heatmap output method is more accurate than direct regression of coordinate points.

[0075] Specifically, such as Figure 2bAs shown, the aforementioned shared features include a first shared feature map, a second shared feature map, and a third shared feature map. The third shared feature map can be upsampled and then convolved to obtain a first upsampled feature map with the same tensor size as the second shared feature map. The second shared feature map can be convolved and then fused with the first upsampled feature map to obtain a first fused feature map. The first fused feature map can be upsampled and then convolved to obtain a second upsampled feature map with the same tensor size as the first shared feature map. The first shared feature map can be convolved and then fused with the second upsampled feature map to obtain a second fused feature map. Based on the second fused feature map, a Gaussian heatmap is obtained. Taking the following sampling multiplier as an example (first multiplier: 4x, second multiplier: 8x, third multiplier: 16x), and the input image tensor as 1024*1024, then the tensor of the first shared feature map is 256*256, the tensor of the second shared feature map is 128*128, and the tensor of the third shared feature map is 64*64. The tensor of the first upsampled feature map is 128*128, and the tensor of the second upsampled feature map is 256*256. Feature fusion can be achieved by adding the feature maps together. The tensor of the resulting feature map does not change; only the values ​​within the tensor change. Therefore, the final output Gaussian heatmap tensor is 256*256.

[0076] After obtaining the Gaussian heatmap, image processing can be used to find the head positioning coordinates (x, y) for each target person as head positioning point information. In one possible embodiment, the aforementioned person positioning point can be the center point of the head. After obtaining the Gaussian heatmap, image processing can be used to find the center coordinates (x, y) of the head for each target person. center ,y center This serves as head positioning information.

[0077] It should be noted that the first branch network is a convolutional neural network used to predict the Gaussian heatmap of a human head, so the shared features extracted by the public network also include the implicit features of the Gaussian heatmap for predicting the human head.

[0078] 203. Input the shared features into the second branch network, and use the second branch network to predict the center point of the human head projection and the key points of the human feet.

[0079] In this embodiment of the invention, the center point of the head projection is used as the head projection positioning point, and the key points of the feet are used as the foot positioning points. The second branch network performs offset prediction based on the head center point. The second branch network can also extract the latent features of the predicted Gaussian heatmap of the head in the common network and predict the coordinates of the head center (x, y). center ,y centerThis serves as the basis for offset prediction. The second branch network outputs the center point of the head projection and the coordinates (x, y) of the foot key points relative to the head center. center y center The offset of the center point of the head projection relative to the center coordinates of the head center (x, y) is (Δx3, Δy3, Δx4, Δy4, Δx5, Δy5). Where (Δx3, Δy3) represents the offset of the center point of the head projection relative to the center coordinates of the head center (x, y). center y center The offset of (Δx4, Δy4) is the coordinate of the left foot key point relative to the center of the head (x). center y center The offset of (Δx5, Δy5) is the coordinate of the right foot key point relative to the center of the head (x). center ,y center The offset of ) can be used to obtain the coordinates of the center point of the head projection as (x) center +Δx3,y center +Δy3), the coordinates of the left foot key point are (x center +Δx4,y center +Δy4), the coordinates of the right foot key point are (x center +Δx5,y center +Δy5).

[0080] Understandably, the second branch network has a similar structure to the first branch network for predicting the coordinates of the head center (x). center ,y center However, the second branch network has an additional convolutional structure compared to the first branch network, which is used for offset prediction and regression of the corresponding offset values ​​(Δx3, Δy3, Δx4, Δy4, Δx5, Δy5).

[0081] In this embodiment of the invention, since the head projection positioning point and the foot positioning point are predicted by the offset of the head center coordinate, the obtained results are the relative coordinates of the head projection positioning point and the foot positioning point, which ensures the consistency of the head center point, the head projection positioning point and the foot positioning point, and improves the accuracy of prediction.

[0082] Specifically, the aforementioned neural network model includes a third branch network, which is used to extract the head bounding box information. Please refer to [link / reference]. Figure 3 , Figure 3 This is a flowchart of a method for predicting head frame information provided by an embodiment of the present invention, such as... Figure 3 As shown, it includes the following steps:

[0083] 301. Extract common features of the image to be detected through public networks.

[0084] In this embodiment of the invention, the shared features extracted from the public network also include the implicit features of the predicted human head Gaussian heatmap.

[0085] 302. Input the shared features into the third branch network, and use the third branch network to predict the head frame information.

[0086] The third branch network mentioned above performs offset prediction based on the head center point. It is understood that the third branch network has a similar structure to the first branch network for predicting the head center coordinates (x, y). center ,y center However, the third branch network has an additional convolutional structure compared to the first branch network, used for offset prediction and regression of the offset values ​​(Δx1, Δy1, Δx2, Δy2) of the person's head bounding box diagonal points. Here, (Δx1, Δy1) can be the predicted offset of the person relative to the top-left corner of the bounding box, and (Δx2, Δy2) can be the predicted offset of the person relative to the top-right corner of the bounding box. Based on the predicted offsets, the position coordinates of the top-left corner point can be obtained as follows:

[0087]

[0088] The coordinates of the top right corner are:

[0089]

[0090] In this embodiment of the invention, since the head frame is predicted by the offset of the head center coordinates, the relative coordinates of the corner points of the head frame are obtained, which ensures the consistency between the head frame and the head center point and improves the accuracy of the head frame prediction.

[0091] It should be noted that although both the second and third branches predict the offset of coordinate points, they do not need to be merged. In this case, the Wing loss function used for training the second branch will yield more accurate results; the Giou loss function used for the third branch will yield more accurate results.

[0092] Using the head bounding box information, the corresponding head depth information can be extracted from the position corresponding to the head bounding box in the image to be detected.

[0093] 103. Calculate the height information of the corresponding target person based on the location point information.

[0094] In this embodiment of the invention, the above-mentioned positioning point information may be head positioning point information and foot positioning point information, that is, the positioning point may be head positioning point and foot positioning point.

[0095] In one possible embodiment, the second branch network described above can output (Δx4, Δy4) as the coordinates of the left foot keypoint relative to the center of the head (x...). center ,y centerThe offset of (Δx5, Δy5) is the coordinate of the right foot key point relative to the center of the head (x). center ,y center The offset of Δy4 and Δy5 can be calculated as the height information of the target person. This height information can be the image height of the target person in the image to be detected.

[0096] In one possible embodiment, the aforementioned positioning point information can be head positioning point information, head projection positioning point information, and foot positioning point information; that is, the positioning points can be head positioning points, head projection positioning points, and foot positioning points. The height information of the target person can be calculated using a triangle calculation method, with the predicted head positioning point, head projection positioning point, and foot positioning point serving as the three corner points of a triangle. This avoids the need to calculate the height by comparing the child's head positioning with the adult's foot positioning when an adult is holding a child, thus improving the accuracy of the height information.

[0097] Furthermore, the aforementioned head positioning point, head projection positioning point, and foot positioning point are respectively the head center point, head projection center point, left foot key point, and right foot key point. The left foot key point and right foot key point can be fitted into a foot fitting point. The height information of the target person can be calculated by using the three points of the head center point, head projection center point, and foot fitting point as the three corner points of a triangle.

[0098] 104. Based on the height and head depth information, perform three-dimensional reconstruction of the target person's head to obtain the target person's three-dimensional head information.

[0099] In this embodiment of the invention, the head of a target person can be reconstructed in three dimensions in a preset three-dimensional space. The three-dimensional space can be constructed based on the coordinate system of the depth camera and the world coordinate system.

[0100] In one possible embodiment, since the head projection is located on the ground, the construction of the ground in three-dimensional space can be aided by using the head projection positioning point. Specifically, the head projection positioning point can be the center point of the head projection, which can be used to assist in the construction of the ground in three-dimensional space.

[0101] Specifically, based on the height information mentioned above, the reconstructed position of the target person's head in three-dimensional space can be determined; at the reconstructed position in the three-dimensional space, the target person's head can be reconstructed in three dimensions based on the head depth information mentioned above, to obtain the three-dimensional head information of the target person.

[0102] More specifically, the height of the target person's head in three-dimensional space can be determined using height information. This height is the height from the reconstructed head position to the ground in three-dimensional space. This allows for accurate reconstruction of the target person's three-dimensional head, obtaining the target person's three-dimensional head information in three-dimensional space. After determining the head reconstruction height, the target head can be reconstructed in three dimensions using head depth information, resulting in a three-dimensional target head.

[0103] 105. Calculate the social distance between multiple target individuals based on their 3D head information.

[0104] In this embodiment of the invention, the social distance between target individuals can be calculated based on the reconstructed 3D target heads. This can be achieved by projecting multiple reconstructed 3D target heads onto a preset plane (the ground of the 3D space) in 3D space to obtain multiple target head projections; calculating the distance between different target head projections to obtain the projection distance between target individuals; and converting the projection distance between target individuals into the social distance between target individuals according to a preset ratio. The aforementioned 3D space can be constructed based on calibrated camera coordinates, and the origin of the 3D space can be the optical center point of the camera, or the center point of two optical centers.

[0105] Specifically, the geometric center of the target person's head projection can be calculated as the head projection point. The distances between each head projection point on the 3D projection surface are then calculated using Euclidean distance. The aforementioned preset ratio is a 3D spatial measurement compared to the actual spatial measurement, which can be determined based on depth information; the greater the depth value, the larger the matching ratio. The distances of the head projection points are then converted into the distances of the target person's head in reality using the matched preset ratio, thus obtaining the social distance of the target person in the real-world scene.

[0106] In this embodiment of the invention, an image to be detected is acquired, the image including image depth information and heads to be detected, the heads including multiple target heads; a pre-trained neural network model is used to predict the location point information and head bounding box information of each target person in the image to be detected, and the corresponding head depth information is extracted based on the head bounding box information; the height information of the corresponding target person is calculated based on the location point information; the head of the target person is reconstructed in three dimensions based on the height information and the head depth information to obtain the three-dimensional head information of the target person; based on the three-dimensional head information of the target person, the social distance between multiple target persons is calculated. By calculating the height information of the target persons, more accurate target head information is extracted for three-dimensional reconstruction, making the position of the three-dimensional target head in three-dimensional space more accurate, thereby improving the accuracy of the social distance between target persons.

[0107] It should be noted that the social distance detection method provided in this embodiment of the invention can be applied to devices such as mobile phones, monitors, computers, and servers that are capable of social distance detection.

[0108] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a social distancing detection device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes:

[0109] The acquisition module 401 is used to acquire an image to be detected, the image to be detected including image depth information and a head to be detected, the head to be detected including the heads of multiple target persons;

[0110] The prediction module 402 is used to predict the localization point information and head bounding box information of each human head in the image to be detected through a pre-trained neural network model, and extract the corresponding head depth information based on the head bounding box information.

[0111] The first calculation module 403 is used to calculate the height information of the corresponding target person based on the positioning point information.

[0112] The reconstruction module 404 is used to perform three-dimensional reconstruction of the head of the target person based on the height information and the head depth information to obtain the three-dimensional head information of the target person.

[0113] The second calculation module 405 is used to calculate the social distance between multiple target persons based on the three-dimensional head information of the target persons.

[0114] Optionally, the pre-trained neural network model includes a common network and a first branch network, a second branch network, and a third branch network connected to the common network. The common network is used to extract common features of the first branch network, the second branch network, and the third branch network. The first branch network and the second branch network are used to extract the location point information of the human head, and the third branch network is used to extract the head bounding box information.

[0115] Optionally, the positioning information of the human head includes the center point of the human head, the center point of the human head projection, and key points of the human feet, such as... Figure 5 As shown, the prediction module 402 includes:

[0116] The first processing submodule 4021 is used to extract common features of the image to be detected through the public network;

[0117] The second processing submodule 4022 is used to input the common features into the first branch network, predict the Gaussian heat map of the human head through the first branch network, and obtain the center point of the human head based on the Gaussian heat map of the human head.

[0118] The third processing submodule 4023 is used to input the shared features into the second branch network, and predict the center point of the human head projection and the key points of the human feet through the second branch network. The second branch network performs offset prediction based on the center point of the human head.

[0119] Optionally, the prediction module 402 is further configured to input the common features into the third branch network and predict the head bounding box information through the third branch network, wherein the third branch network performs offset prediction based on the head center point.

[0120] Optionally, the first calculation module 403 is further used to calculate the height information of the corresponding target person by combining the center point of the head, the center point of the head projection, and the key points of the feet.

[0121] Optionally, the first processing submodule 4021 is further configured to sequentially perform convolution downsampling of the image to be detected by a first multiple, a second multiple, and a third multiple, to obtain a first common feature map, a second common feature map, and a third common feature map, wherein the value of the first multiple is less than the value of the second multiple, the value of the second multiple is less than the value of the third multiple, the tensor size of the first common feature map is greater than the tensor size of the second common feature map, and the tensor size of the second common feature map is greater than the tensor size of the third common feature map.

[0122] Optional, such as Figure 6 As shown, the second processing submodule 4022 includes:

[0123] The first computing unit 40221 is used to perform convolution calculation on the third common feature map after upsampling to obtain a first upsampled feature map with the same tensor size as the second common feature map.

[0124] The first fusion unit 40222 is used to perform convolution calculation on the second common feature map and fuse it with the first upsampled feature map to obtain the first fused feature map.

[0125] The second computing unit 40223 is used to upsample the first fused feature and then perform convolution calculation to obtain a second upsampled feature map with the same tensor size as the first shared feature map.

[0126] The second fusion unit 40224 is used to perform convolution calculation on the first common feature map and fuse it with the second upsampled feature to obtain a second fused feature map.

[0127] Processing unit 40225 is used to obtain a Gaussian heatmap based on the second fused feature map.

[0128] Optional, such as Figure 7 As shown, the reconstruction module 404 includes:

[0129] The determination submodule 4041 is used to determine the reconstructed position of the target person's head in three-dimensional space based on the height information;

[0130] The reconstruction submodule 4042 is used to perform three-dimensional reconstruction of the head of the target person at the reconstruction position in the three-dimensional space based on the head depth information, so as to obtain the three-dimensional head information of the target person.

[0131] Optional, such as Figure 8 As shown, the device further includes:

[0132] The construction module 406 is used to construct the ground in the three-dimensional space based on the center point of the human head projection.

[0133] It should be noted that the social distancing detection device provided in this embodiment of the invention can be applied to devices such as mobile phones, monitors, computers, and servers that can perform social distancing detection.

[0134] The social distancing detection device provided in this embodiment of the invention can implement all the processes of the social distancing detection method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0135] See Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 9 As shown, it includes: a memory 902, a processor 901, and a computer program stored in the memory 902 and executable on the processor 901, wherein:

[0136] The processor 901 is used to call the computer program stored in the memory 902 and execute the following steps:

[0137] Acquire an image to be detected, the image to be detected including image depth information and a head to be detected, the head to be detected including the heads of multiple target persons;

[0138] The pre-trained neural network model predicts the location point information and head bounding box information of each target person in the image to be detected, and extracts the corresponding head depth information based on the head bounding box information.

[0139] Based on the location point information, calculate the height information of the corresponding target person;

[0140] Based on the height information and the head depth information, a three-dimensional reconstruction of the target person's head is performed to obtain the three-dimensional head information of the target person;

[0141] Based on the three-dimensional head information of the target individuals, the social distance between multiple target individuals is calculated.

[0142] Optionally, the pre-trained neural network model includes a common network and a first branch network, a second branch network, and a third branch network connected to the common network. The common network is used to extract common features of the first branch network, the second branch network, and the third branch network. The first branch network and the second branch network are used to extract the location point information of the human head, and the third branch network is used to extract the head bounding box information.

[0143] Optionally, the head location information includes the head center point, the head projection center point, and the foot key points. The processor 901's execution of predicting the location information of each target person in the image to be detected using a pre-trained neural network model includes:

[0144] Common features of the image to be detected are extracted through the public network;

[0145] The shared features are input into the first branch network, and the Gaussian heat map of the human head is predicted through the first branch network. The center point of the human head is obtained based on the Gaussian heat map of the human head.

[0146] The shared features are input into the second branch network, and the center point of the human head projection and the key points of the human feet are predicted by the second branch network. The second branch network performs offset prediction based on the center point of the human head.

[0147] Optionally, the process executed by processor 901 to predict the head bounding box information of each target person in the image to be detected using a pre-trained neural network model includes:

[0148] The shared features are input into the third branch network, and the head bounding box information is predicted by the third branch network. The third branch network performs offset prediction based on the center point of the head.

[0149] Optionally, the step of processor 901 calculating the height information of the corresponding target person based on the positioning point information includes:

[0150] By combining the center point of the head, the center point of the head projection, and the key points of the feet, the height information of the corresponding target person is calculated.

[0151] Optionally, the processor 901 performs the extraction of common features through the public network, including:

[0152] The image to be detected is sequentially subjected to convolution downsampling at a first multiple, a second multiple, and a third multiple to obtain a first shared feature map, a second shared feature map, and a third shared feature map. The value of the first multiple is less than the value of the second multiple, the value of the second multiple is less than the value of the third multiple, the tensor size of the first shared feature map is greater than the tensor size of the second shared feature map, and the tensor size of the second shared feature map is greater than the tensor size of the third shared feature map.

[0153] Optionally, the step of processor 901 inputting the shared features into the first branch network and predicting a Gaussian heatmap of the human head through the first branch network includes:

[0154] The third shared feature map is upsampled and then convolved to obtain a first upsampled feature map with the same tensor size as the second shared feature map.

[0155] The second shared feature map is convolved and fused with the first upsampled feature map to obtain the first fused feature map.

[0156] The first fused feature is upsampled and then convolutional to obtain a second upsampled feature map with the same tensor size as the first shared feature map.

[0157] The first shared feature map is convolved and fused with the second upsampled feature to obtain the second fused feature map.

[0158] Based on the second fused feature map, a Gaussian heatmap is obtained.

[0159] Optionally, the step of processor 901 performing three-dimensional reconstruction of the target person's head based on the height information and the head depth information to obtain the target person's three-dimensional head information includes:

[0160] Based on the height information, determine the reconstructed position of the target person's head in three-dimensional space;

[0161] At the reconstruction location in the three-dimensional space, the head of the target person is reconstructed in three dimensions based on the head depth information to obtain the three-dimensional head information of the target person.

[0162] Optionally, processor 901 also performs the following:

[0163] The ground in the three-dimensional space is constructed based on the center point of the human head projection.

[0164] It should be noted that the aforementioned electronic devices can be mobile phones, monitors, computers, servers, and other devices that can be used for social distancing detection.

[0165] The electronic device provided in this embodiment of the invention can implement all the processes of the social distance detection method in the above method embodiments and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0166] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the social distancing detection method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0167] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0168] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A social distance detection method, characterized in that, Includes the following steps: Acquire an image to be detected, the image to be detected including image depth information and a head to be detected, the head to be detected including the heads of multiple target persons; The pre-trained neural network model predicts the location point information and head bounding box information of each target person in the image to be detected, and extracts the corresponding head depth information based on the head bounding box information. Based on the location point information, calculate the height information of the corresponding target person; Based on the height information and the head depth information, a three-dimensional reconstruction of the target person's head is performed to obtain the three-dimensional head information of the target person; Based on the three-dimensional head information of the target individuals, the social distance between multiple target individuals is calculated; Based on the three-dimensional head information of the target individuals, the social distance between multiple target individuals is calculated, including: Multiple 3D target heads are projected onto a preset plane in 3D space to obtain multiple target head projections; the distance between different target head projections is calculated to obtain the projection distance between target individuals; the projection distance between target individuals is converted into social distance between target individuals according to a preset ratio; The pre-trained neural network model includes a common network, and a first branch network, a second branch network, and a third branch network connected to the common network; The location information of the human head includes the center point of the head, the center point of the head projection, and key points of the feet. The prediction of the location information of each target person in the image to be detected using a pre-trained neural network model includes: Common features of the image to be detected are extracted through the public network; The shared features are input into the first branch network, and the Gaussian heat map of the human head is predicted through the first branch network. The center point of the human head is obtained based on the Gaussian heat map of the human head. The shared features are input into the second branch network, and the center point of the human head projection and the key points of the human feet are predicted by the second branch network. The second branch network performs offset prediction based on the center point of the human head. Common features extracted through the public network include: The image to be detected is sequentially subjected to convolution downsampling at a first multiple, a second multiple, and a third multiple to obtain a first common feature map, a second common feature map, and a third common feature map. The value of the first multiple is less than the value of the second multiple, the value of the second multiple is less than the value of the third multiple, the tensor size of the first common feature map is greater than the tensor size of the second common feature map, and the tensor size of the second common feature map is greater than the tensor size of the third common feature map. The step of inputting the shared features into the first branch network and predicting the Gaussian heatmap of the human head through the first branch network includes: The third shared feature map is upsampled and then convolved to obtain a first upsampled feature map with the same tensor size as the second shared feature map. The second shared feature map is convolved and fused with the first upsampled feature map to obtain the first fused feature map. The first fused feature is upsampled and then convolutional to obtain a second upsampled feature map with the same tensor size as the first shared feature map. The first shared feature map is convolved and fused with the second upsampled feature to obtain the second fused feature map. Based on the second fused feature map, a Gaussian heatmap is obtained; The above feature fusion involves adding the feature maps together, and the resulting feature map tensor remains unchanged.

2. The method as described in claim 1, characterized in that, The common network is used to extract the common features of the first branch network, the second branch network and the third branch network. The first branch network and the second branch network are used to extract the positioning point information of the human head, and the third branch network is used to extract the human head frame information.

3. The method as described in claim 1, characterized in that, The step of predicting the head bounding box information of each target person in the image to be detected using a pre-trained neural network model includes: The shared features are input into the third branch network, and the head bounding box information is predicted by the third branch network. The third branch network performs offset prediction based on the center point of the head.

4. The method as described in claim 1, characterized in that, The step of calculating the height information of the corresponding target person based on the positioning point information includes: By combining the center point of the head, the center point of the head projection, and the key points of the feet, the height information of the corresponding target person is calculated.

5. The method as described in claim 1, characterized in that, The step of performing three-dimensional reconstruction of the target person's head based on the height information and the head depth information to obtain the target person's three-dimensional head information includes: Based on the height information, determine the reconstructed position of the target person's head in three-dimensional space; At the reconstruction location in the three-dimensional space, the head of the target person is reconstructed in three dimensions based on the head depth information to obtain the three-dimensional head information of the target person.

6. The method as described in claim 5, characterized in that, The method further includes: The ground in the three-dimensional space is constructed based on the center point of the human head projection.

7. A social distancing detection device, characterized in that, The device includes: The acquisition module is used to acquire the image to be detected, the image to be detected including image depth information and the head to be detected, the head to be detected including the heads of multiple target persons; The prediction module is used to predict the localization point information and head bounding box information of each human head in the image to be detected through a pre-trained neural network model, and extract the corresponding head depth information based on the head bounding box information. The first calculation module is used to calculate the height information of the corresponding target person based on the positioning point information; The reconstruction module is used to perform three-dimensional reconstruction of the target person's head based on the height information and the head depth information to obtain the three-dimensional head information of the target person. The second calculation module is used to calculate the social distance between multiple target individuals based on the three-dimensional head information of the target individuals; In calculating the social distance between multiple target individuals based on their three-dimensional head information, the second calculation module is specifically used for: Multiple 3D target heads are projected onto a preset plane in 3D space to obtain multiple target head projections; the distance between different target head projections is calculated to obtain the projection distance between target individuals; the projection distance between target individuals is converted into social distance between target individuals according to a preset ratio; The pre-trained neural network model includes a common network, and a first branch network, a second branch network, and a third branch network connected to the common network; The location information of the human head includes the center point of the head, the center point of the head projection, and key points of the feet. The prediction of the location information of each target person in the image to be detected using a pre-trained neural network model includes: Common features of the image to be detected are extracted through the public network; The shared features are input into the first branch network, and the Gaussian heat map of the human head is predicted through the first branch network. The center point of the human head is obtained based on the Gaussian heat map of the human head. The shared features are input into the second branch network, and the center point of the human head projection and the key points of the human feet are predicted by the second branch network. The second branch network performs offset prediction based on the center point of the human head. Common features extracted through the public network include: The image to be detected is sequentially subjected to convolution downsampling at a first multiple, a second multiple, and a third multiple to obtain a first common feature map, a second common feature map, and a third common feature map. The value of the first multiple is less than the value of the second multiple, the value of the second multiple is less than the value of the third multiple, the tensor size of the first common feature map is greater than the tensor size of the second common feature map, and the tensor size of the second common feature map is greater than the tensor size of the third common feature map. The step of inputting the shared features into the first branch network and predicting the Gaussian heatmap of the human head through the first branch network includes: The third shared feature map is upsampled and then convolved to obtain a first upsampled feature map with the same tensor size as the second shared feature map. The second shared feature map is convolved and fused with the first upsampled feature map to obtain the first fused feature map. The first fused feature is upsampled and then convolutional to obtain a second upsampled feature map with the same tensor size as the first shared feature map. The first shared feature map is convolved and fused with the second upsampled feature to obtain the second fused feature map. Based on the second fused feature map, a Gaussian heatmap is obtained; The above feature fusion involves adding the feature maps together, and the resulting feature map tensor remains unchanged.

8. An electronic device, characterized in that, include: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the social distancing detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the social distancing detection method as described in any one of claims 1 to 6.

Images