Human figure detection method in video conference scene based on deep learning

Abstract

The present invention discloses a method for detecting human figures in a video conference scene based on deep learning, which is used to extract panoramic images in a video conference scene and detect the positions of various persons in the image, which is helpful to achieve local picture focusing, auxiliary voice enhancement, etc. The method uses a camera to collect and correct the original panoramic image from the conference scene; the corrected rectangular panoramic image is spliced ​​and mapped into a square image with equal length and width, and normalization preprocessing and data enhancement are performed; a deep learning model based on a residual network-feature pyramid network is constructed; a bounding box regression network is established, and the positions of the borders around the human target as well as the confidence and center weighting are calculated at the same time; the model is trained using an adaptive focus loss, and the conference scene image is input for training. The present invention guides the model to adapt to human targets in special scenes through boundary regression, center weighting and adaptive focus loss, thereby improving the accuracy and recall rate of detection in densely populated conference scenes, and has good application prospects.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and specifically designs a method for detecting human figures in video conferencing scenes based on deep learning. Background Art

[0002] Human detection in video conferencing is often used to achieve speaker focus, auxiliary voice directional enhancement and other functions. However, in complex conference room scenarios, due to factors such as the large number of participants, dense seating, uneven lighting, and random movement of people, the performance of conventional target detection algorithms in this application scenario is greatly reduced. At the same time, the cameras used in conference rooms are usually ultra-wide-angle or panoramic cameras with a wide frame. The proportion of a single human target in the picture is small, and the available features are limited. In addition, people in conference rooms are often seated, and their bodies are partially blocked by conference facilities such as tables, chairs, and computers, so the effective information that can be provided is even scarcer. Therefore, the task of human detection in video conferencing scenarios has become a common problem.

[0003] Traditional human detection algorithms often use manually designed operators to extract some features for analysis. The ViolaJones detector uses a sliding window to view all possible positions and proportions in the image to check whether the target exists in the window. This method combines three important technologies: "integral image", "feature selection" and "detection cascade", which greatly improves the detection speed. The Histogram of Directed Gradients (HOG) feature descriptor has also been used to solve the problem of pedestrian detection. It can be used to balance feature invariance (including translation, scale, illumination, etc.) and nonlinearity (distinguishing different object categories). The DPM algorithm is improved and extended on the basis of the HOG algorithm. It consists of a main filter and multiple auxiliary filters. The detection accuracy is improved through hard negative mining, bounding box regression and context priming technology. As the optimal traditional detection algorithm, the DPM method has a fast operation speed and can adapt to object deformation, but it cannot adapt to large rotations, so it has poor stability.

[0004] In recent years, the rapid development of deep learning algorithms has also been widely used in the detection field. Object detection based on deep learning methods overcomes the shortcomings of traditional algorithms that rely on manually designed features. There are currently two types of object detection: single-stage and two-stage. The two-stage method means that the detection algorithm needs to be completed in two steps. First, the candidate area needs to be obtained, and then the classification is performed, such as the R-CNN series; in contrast, the single-stage detection does not require a separate search for candidate areas, and typical examples include the SSD and YOLO series. For the above two methods, the two-stage method based on the candidate area has an advantage in detection accuracy and positioning accuracy, and the end-to-end single-stage algorithm has an advantage in speed. However, these algorithms often solve general multi-category object detection, and can only achieve good results when the object features are rich, the target is large and sparsely distributed, and the lighting is consistent. Due to the complexity of reality, the actual conference room scene is likely to have a large deviation from the general training set. Although self-built data sets can be used to make up for some deficiencies, the general deep learning detection model is always insufficient in dense detection, small target detection, and capture of occluded human bodies.

[0005] Therefore, how to solve problems such as dense human targets, uneven lighting, small targets in a wide frame, and occlusion by irregular objects in the conference room is the key to improving the human detection effect in video conferencing scenarios. It has important research significance for improving the human detection effect, improving the quality of video conference calls, and improving the experience of participants. Summary of the invention

[0006] The purpose of the present invention is to solve the problems of dense human targets, uneven lighting, wide frame, small targets, and irregular object occlusion in the human figure detection task in the video conference scene, so as to improve the performance of the detection algorithm, improve the detection accuracy and recall rate, as well as the IOU precision of the detection and other indicators. The human figure detection model for video conference scenes based on deep learning of the present invention uses a bounding box regression network, and simultaneously calculates the position of the borders around the human target as well as the confidence and center weighting, so as to better use the densely arranged human targets and avoid missed detection and repeated detection; at the same time, an adaptive focus loss is introduced to train the model to solve the problem of sample imbalance, mainly including the problem of whether the positive and negative samples and the difficult and easy samples are balanced, and finally better adapts to the human figure detection scene, greatly improving the detection performance, the method is ingenious and novel, and has good application prospects.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] The method for detecting human figures in video conference scenes based on deep learning includes the following steps:

[0009] Step (A), using a camera to obtain a rectangular panoramic image from the conference scene and correcting it;

[0010] Step (B), mapping the rectified rectangular panoramic images into square images with equal length and width, and performing normalization preprocessing and data enhancement operations, followed by normalization processing, wherein the normalization preprocessing and data enhancement operations specifically include random flipping, region cropping, and region masking and reorganization;

[0011] Step (C), constructing a deep learning model based on residual network-feature pyramid network as a baseline model, inputting the image processed in step (B), and outputting the rectangular position frames of all human bodies in the image;

[0012] Step (D), introducing a bounding box regression network based on the baseline model in step (C), and calculating the position of the bounding box around the human target as well as the confidence and center weighting;

[0013] Step (E), introducing an adaptive focus loss training model, inputting the conference scene image obtained in step (B) according to the trained human figure detection model, and detecting the positions of the corresponding participants.

[0014] The aforementioned method for detecting human figures in a video conference scene based on deep learning comprises the following steps:

[0015] Step (A), using a camera to obtain a rectangular panoramic image from the conference scene and correcting it, includes the following steps:

[0016] Step (A1), placing a camera device with a 180-degree fisheye lens in the center of the conference room to capture a distorted panoramic image of the conference room;

[0017] Step (A2), using the OpenCV chessboard calibration method to perform correction analysis on the collected panoramic image of the conference room, using a fisheye camera to shoot a chessboard picture of a fixed size, binarizing the image, and then performing corrosion and expansion operations on the image, traversing the outline of each square and obtaining the corner points of all small squares, calibrating the lens parameters according to the distortion of these corner points, obtaining the camera lens parameters, inputting the image to be corrected, and performing coordinate transformation through the coordinate correspondence before and after the lens distortion to obtain a normal non-distorted rectangular panoramic photo.

[0018] Step (B), mapping the rectified rectangular panoramic images into square images with equal length and width, and performing normalization preprocessing and data enhancement operations, followed by normalization processing, wherein the normalization preprocessing and data enhancement operations are specifically random flipping, region cropping and region masking and reorganization, including the following steps,

[0019] Step (B1), taking the rectangular panoramic image with a side length of 3000×1000 containing the 360° annular scene information in the conference room output by the camera after correction in step (A), vertically cropping two original rectangular images with a side length of 2000×1000, and then splicing them up and down into a square image with a side length of 2000×2000, for adapting the input shape ratio of the deep learning detector;

[0020] Step (B2), mapping the stitched square image to the original rectangular image position one by one, the upper half of the square image is directly mapped to the x∈[0,2000) position of the original rectangular image, and the lower half is an image with a side length of 2000×1000 formed by stitching multiple segments of x∈[0,500) and x∈[1500,3000) of the original image, so as to avoid the panoramic image being split;

[0021] Step (B3), after the rectified image is mapped to the original image position, non-maximum suppression is performed to avoid repeated detection of the image after splicing in step (B1);

[0022] Step (B4), data enhancement is performed on the spliced ​​square image, the original image is randomly flipped up and down and left and right, and then the partial image area containing the human target is randomly cropped, and the image area not containing the human target is smeared or covered in a mosaic manner;

[0023] Step (B5), normalize the square image after data enhancement, convert each pixel value into a decimal in the interval (0, 1), and then compress the input image to a size of 512×512 in side length as the input image of the model.

[0024] Step (C), constructing a deep learning model based on the residual network-feature pyramid network as a baseline model, inputting the image processed in step (B), and outputting the rectangular position borders of all human bodies in the image, including the following steps,

[0025] Step (C1), input the image processed in step (B), then build a baseline model, and connect the residual convolution network and the feature pyramid network in sequence;

[0026] Step (C2), using the residual convolutional network used to learn the spatial semantic features of the original image as the backbone network, using the feature pyramid network to achieve multi-scale feature fusion of the image, and modeling the features from different scales;

[0027] Step (C3) uses the fully connected layer as a detection head through a shallow convolutional network to obtain the position of the target human figure, uses the k-means clustering algorithm to obtain the anchor box of the human figure in the adaptation data set, and then outputs the rectangular position frame of all human bodies in the image.

[0028] Step (D), based on the baseline model in step (C), introduces a bounding box regression network to calculate the position, confidence and center weight of the bounding box around the human target, including the following steps:

[0029] Step (D1), introducing a bounding box regression network, inputting the feature map output by the first-layer feature pyramid network and the first-pass multi-layer convolution to regress the boundary of the target candidate box area by changing the number of channels of the upper-layer input, and outputting a tensor of the first-pass convolution shape of H×W×5, where H and W are the length and width values ​​of the output of the previous layer, and 5 is the channel number, which is used as the distance (l, t, r, b) and confidence between the upper, lower, left, and right boundaries of the target human body and the center point of the current detection area;

[0030] Step (D2), the feature map and the original image position are matched one by one, the reduction ratio of the original image and the feature map is set to d, and the coordinates (x, y) in the feature map correspond to the center point of the original image. The real human body in the current area is regressed through the bounding box regression network to get the distance between the upper, lower, left, and right boundaries of the object and the center point of the current detection area as (l, t, r, b). These four values ​​correspond to the coordinate points of the four corners of the bounding box around the human target.

[0031]

[0032]

[0033]

[0034]

[0035] Among them, (x1, y1) (x2, y2) are the coordinates of the upper left corner and lower right corner of the border around the human target, d is the reduction ratio of the original image to the feature map, and H and W are the length and width of the network input feature map;

[0036] Step (D3) uses the same input as step (D1), inputs the feature map output by the second-layer feature pyramid network, and regresses the boundary of the target candidate box area by changing the number of channels of the upper-layer input. Then, the second-layer convolution outputs a tensor with a shape of H×W×1, which represents the distance coefficient between the center point of the current area and the center point of the real box, as the center weight, and its representation value is ω, which is used to ensure that each detection area detects only the real human body closest to itself as much as possible.

[0037] Step (E), introducing an adaptive focus loss training model, inputting the conference scene image obtained in step (B) according to the trained human detection model, and detecting the position of the corresponding participant, including the following steps,

[0038] Step (E1), adding a loss function training model, and adding two hyperparameters, namely weight coefficients α and γ, based on the cross entropy of the loss function, and then introducing adaptive focal loss to perform binary classification on the loss function to determine whether there is an object in the current detection area;

[0039] Step (E2), input the conference scene image obtained in step (B), and use the center weight value ω output in step (D3) as a parameter of the loss function to participate in the calculation of the loss function. The farther the distance between the center of the detection area and the center of the real human body is, the smaller the loss function is; when there is no human body in the detection area, ω = 0;

[0040] The loss function formula is as follows:

[0041]

[0042] Among them, the weight α is used to balance the imbalance of positive and negative samples, the weight γ is used to distinguish difficult and easy samples, p is the predicted value of confidence, ω is the central weight value output by step (D2), and σ is any small number greater than 0 to prevent division by zero.

[0043] Step (E3), in the initial state, the γ value is 0. When γ increases, the adjustment factor also increases, that is, the loss generated by the simple sample is gradually suppressed. As the γ value increases, the loss generated by the simple sample is greatly reduced.

[0044] The beneficial effects of the present invention are as follows: the human figure detection method for video conferencing scenes based on deep learning first abandons the anchor box clustering in the original detection algorithm based on deep learning, and instead uses a regression method, using a distance operator to represent the position of the human body, and on the basis of regressing the distance, also regresses and calculates the confidence and center weighted parameters at the same time; secondly, an adaptive focus loss is designed to replace the previously commonly used cross-entropy loss function selection training, thereby improving the speed of model convergence and improving the detection accuracy in complex scenes; therefore, the method of the present invention can improve the robustness and performance of human body detection, the method is ingenious and novel, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a method for detecting human figures in a video conference scene based on deep learning in the present invention;

[0046] Figure 2 It is a schematic diagram of cutting and splicing rectangular images output by the panoramic camera in the present invention;

[0047] Figure 3 It is a complete structural diagram of the model proposed in the present invention;

[0048] Figure 4This is a network structure diagram of the detection head designed by the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings.

[0050] like Figure 1 and Figure 3 As shown, the method for detecting human figures in a video conference scene based on deep learning of the present invention comprises the following steps: step (A), using a camera to obtain a rectangular panoramic image from the conference scene and correcting it, comprising the following steps,

[0051] Step (A1), placing a camera device with a 180-degree fisheye lens in the center of the conference room to capture a distorted panoramic image of the conference room;

[0052] Step (A2), using the OpenCV chessboard calibration method to perform correction analysis on the collected panoramic image of the conference room, using a fisheye camera to shoot a chessboard picture of a fixed size, binarizing the image, and then performing corrosion and expansion operations on the image, traversing the outline of each square and obtaining the corner points of all small squares, calibrating the lens parameters according to the distortion of these corner points, obtaining the camera lens parameters, inputting the image to be corrected, and performing coordinate transformation through the coordinate correspondence before and after the lens distortion to obtain a normal non-distorted rectangular panoramic photo.

[0053] Step B: Figure 2 As shown, the rectified rectangular panoramic images are spliced ​​and mapped into square images with equal length and width, and normalized preprocessing and data enhancement operations are performed, followed by normalization processing, wherein the normalization preprocessing and data enhancement operations are specifically random flipping, region cropping and region masking and reorganization, including the following steps:

[0054] Step (B1), taking the rectangular panoramic image with a side length of 3000×1000 containing 360° annular scene information in the conference room output by the image acquisition device after correction in step (A), vertically cropping two original rectangular images with a side length of 2000×1000, and then splicing them up and down into a square image with a side length of 2000×2000, for adapting the input shape ratio of the deep learning detector;

[0055] Step (B2), mapping the stitched square image to the original rectangular image position one by one, the upper half of the square image is directly mapped to the x∈[0,2000) position of the original rectangular image, and the lower half is an image with a side length of 2000×1000 formed by stitching multiple segments of x∈[0,500) and x∈[1500,3000) of the original image, so as to avoid the panoramic image being split;

[0056] Step (B3), after the rectified image is mapped to the original image position, non-maximum suppression is performed to avoid repeated detection of the image after splicing in step (B1);

[0057] Step (B4), data enhancement is performed on the spliced ​​square image, the original image is randomly flipped up and down and left and right, and then the partial image area containing the human target is randomly cropped, and the image area not containing the human target is smeared or covered in a mosaic manner;

[0058] Step (B5), normalize the square image after data enhancement, convert each pixel value into a decimal in the interval (0, 1), and then compress the input image to a size of 512×512 in side length as the input image of the model.

[0059] Step (C), constructing a deep learning model based on the residual network-feature pyramid network as a baseline model, inputting the image processed in step (B), and outputting the rectangular position borders of all human bodies in the image, including the following steps,

[0060] Step (C1), input the image processed in step (B), then build a baseline model, and connect the residual convolution network and the feature pyramid network in sequence;

[0061] Step (C2), using the residual convolutional network used to learn the spatial semantic features of the original image as the backbone network, using the feature pyramid network to achieve multi-scale feature fusion of the image, and modeling the features from different scales;

[0062] Step (C3) uses the fully connected layer as a detection head through a shallow convolutional network to obtain the position of the target human figure, uses the k-means clustering algorithm to obtain the anchor box of the human figure in the adaptation data set, and then outputs the rectangular position frame of all human bodies in the image.

[0063] Step (D) introduces a bounding box regression network based on the baseline model in step (C) to calculate the position, confidence and center weight of the bounding box around the human target, such as Figure 4 As shown, the following steps are included:

[0064] Step (D1), introducing a bounding box regression network, inputting the feature map output by the first-layer feature pyramid network and the first-pass multi-layer convolution to regress the boundary of the target candidate box area by changing the number of channels of the upper-layer input, and outputting a tensor of the first-pass convolution shape of H×W×5, where H and W are the length and width values ​​of the output of the previous layer, and 5 is the channel number, which is used as the distance (l, t, r, b) and confidence between the upper, lower, left, and right boundaries of the target human body and the center point of the current detection area;

[0065] Step (D2), the feature map and the original image position are matched one by one, the reduction ratio of the original image and the feature map is set to d, and the coordinates (x, y) in the feature map correspond to the center point of the original image. The real human body in the current area is regressed through the bounding box regression network to get the distance between the upper, lower, left, and right boundaries of the object and the center point of the current detection area as (l, t, r, b). These four values ​​correspond to the coordinate points of the four corners of the bounding box around the human target.

[0066]

[0067]

[0068]

[0069]

[0070] Among them, (x1, y1) (x2, y2) are the coordinates of the upper left corner and lower right corner of the border around the human target, d is the reduction ratio of the original image to the feature map, and H and W are the length and width of the network input feature map;

[0071] Step (D3) uses the same input as step (D1), inputs the feature map output by the second-layer feature pyramid network, and regresses the boundary of the target candidate box area by changing the number of channels of the upper-layer input. Then, the second-layer convolution outputs a tensor with a shape of H×W×1, which represents the distance coefficient between the center point of the current area and the center point of the real box as the center weight. The value represented by it is ω, which is used to ensure that each detection area detects only the real human body closest to itself as much as possible.

[0072] Step (E), introducing an adaptive focus loss training model, inputting the conference scene image obtained in step (B) according to the trained human figure detection model, and detecting the position of the corresponding participant, including the following steps;

[0073] Step (E1), adding a loss function training model, and adding two hyperparameters, namely weight coefficients α and γ, based on the cross entropy of the loss function, and then introducing adaptive focal loss to perform binary classification on the loss function to determine whether there is an object in the current detection area;

[0074] Step (E2), input the conference scene image obtained in step (B), and use the center weight value ω output in step (D3) as a parameter of the loss function to participate in the calculation of the loss function. The farther the distance between the center of the detection area and the center of the real human body is, the smaller the loss function is; when there is no human body in the detection area, ω = 0;

[0075] The loss function formula is as follows:

[0076]

[0077] Among them, the weight α is used to balance the imbalance of positive and negative samples, the weight γ is used to distinguish difficult and easy samples, p is the predicted value of confidence, ω is the central weight value output by step (D2), and σ is any small number greater than 0 to prevent division by zero.

[0078] Step (E3), in the initial state, the γ value is 0. When γ increases, the adjustment factor also increases, that is, the loss generated by the simple sample is gradually suppressed. As the γ value increases, the loss generated by the simple sample is greatly reduced.

[0079] In order to fully compare the performance of the human figure detection method for video conferencing scenes based on deep learning proposed in the present invention, the experiment is set up on a self-built conference room scene human figure detection dataset, which contains 9183 pictures, and each conference room picture contains about 9 people on average; the experiment uses the average precision mean (AP) of the model on the test set on the dataset as the evaluation index of the model performance, the average precision mean (mAP@0.5) under the condition of IOU threshold of 0.5, and similarly, mAP@.75 under the condition of threshold of 0.75; after determining the IOU threshold, it can be judged according to this threshold whether the intersection of the predicted box and the real box of each category exceeds the threshold, so as to calculate the accuracy and recall rate under different confidence levels (conf), and then the results under each threshold (IOU=.50:.05:0.95, that is, the AP value is obtained every 0.05 step in the range of IOU from 0.5 to 0.95) are averaged to obtain the technical indicator mAP adopted in the experimental steps of the present invention. Experimental results show that the mAP of the baseline model proposed in this invention is 48.7; after introducing the boundary regression network, center weight network and adaptive focal loss, the accuracy is improved to 73.6.

[0080] To sum up, the human figure detection method for video conferencing scenes based on deep learning proposed in the present invention first abandons the anchor box clustering in the original detection algorithm based on deep learning, and instead uses a regression method, using a distance operator to represent the position of the human body, and on the basis of regressing the distance, it also simultaneously regresses and calculates the confidence and center weighted parameters; secondly, an adaptive focus loss is designed to replace the previously commonly used cross-entropy loss function selection training, thereby improving the speed of model convergence and improving the detection accuracy in complex scenes; therefore, the method of the present invention can improve the robustness and performance of human body detection, the method is ingenious and novel, and has good application prospects.

[0081] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for detecting human figures in video conferencing scenes based on deep learning, characterized in that: The steps include: Step (A), using a camera to obtain a rectangular panoramic image from the conference scene and correcting it; Step (B), mapping the rectified rectangular panoramic images into square images with equal length and width, and performing normalization preprocessing and data enhancement operations, followed by normalization processing, wherein the normalization preprocessing and data enhancement operations specifically include random flipping, region cropping, and region masking and reorganization; Step (C), constructing a deep learning model based on the residual network-feature pyramid network as a baseline model, inputting the image processed in step (B), and outputting the rectangular position frames of all human bodies in the image, including the following steps, Step (C1), input the image processed in step (B), then build a baseline model, and connect the residual convolution network and the feature pyramid network in sequence; Step (C2), using the residual convolutional network used to learn the spatial semantic features of the original image as the backbone network, using the feature pyramid network to achieve multi-scale feature fusion of the image, and modeling the features from different scales; Step (C3), using the fully connected layer as a detection head through a shallow convolutional network to obtain the position of the target human figure, using the k-means clustering algorithm to obtain the anchor box of the human figure in the adaptation data set, and then outputting the rectangular position frame of all human bodies in the image; Step (D), based on the baseline model in step (C), a bounding box regression network is introduced to calculate the position of the bounding box around the human target as well as the confidence and center weighting, including the following steps: Step (D1), introducing a bounding box regression network, inputting the feature map output by the first-layer feature pyramid network and the first-pass multi-layer convolution to regress the boundary of the target candidate box area by changing the number of channels of the upper-layer input, and outputting a tensor of the first-pass convolution shape of H×W×5, where H and W are the length and width values ​​of the output of the previous layer, and 5 is the channel number, which is used as the distance (l, t, r, b) and confidence between the upper, lower, left, and right boundaries of the target human body and the center point of the current detection area; Step (D2), the feature map and the original image position are matched one by one, the reduction ratio of the original image and the feature map is set to d, and the coordinates (x, y) in the feature map correspond to the center point of the original image. The real human body in the current area is regressed through the bounding box regression network to get the distance between the upper, lower, left, and right boundaries of the human body and the center point of the current detection area as (l, t, r, b). These four values ​​correspond to the coordinate points of the four corners of the bounding box around the human target. Among them, (x1, y1) (x2, y2) are the coordinates of the upper left corner and lower right corner of the border around the human target, d is the reduction ratio of the original image to the feature map, and H and W are the length and width of the network input feature map; Step (D3) is the same as step (D1), and the feature map output by the second-layer feature pyramid network is input. The second multi-layer convolution regresses the boundary of the target candidate box area by changing the number of channels of the upper layer input, and then outputs the second multi-layer convolution tensor with a shape of H×W×1, which represents the distance coefficient between the center point of the current area and the center point of the real box as the center weight, and its representation value is ω, which is used to ensure that each detection area detects only the real human body closest to itself as much as possible; Step (E), introducing an adaptive focus loss training model, inputting the conference scene image obtained in step (B) according to the trained human figure detection model, and detecting the positions of the corresponding participants.

2. According to the method of deep learning based human figure detection in video conference scene of claim 1, it is characterized in that: Step (A), using a camera to obtain a rectangular panoramic image from the conference scene and correcting it, includes the following steps: Step (A1), placing a camera device with a 180-degree fisheye lens in the center of the conference room to capture a distorted panoramic image of the conference room; Step (A2), using the OpenCV chessboard calibration method to perform correction analysis on the collected panoramic image of the conference room, using a fisheye camera to shoot a chessboard picture of a fixed size, binarizing the image, and then performing corrosion and expansion operations on the image, traversing the outline of each square and obtaining the corner points of all small squares, calibrating the lens parameters according to the distortion of these corner points, obtaining the camera lens parameters, inputting the image to be corrected, and performing coordinate transformation through the coordinate correspondence before and after the lens distortion to obtain a normal non-distorted rectangular panoramic photo.

3. According to the method of deep learning-based human figure detection in video conferencing scenes as described in claim 1, it is characterized in that: Step (B), mapping the rectified rectangular panoramic images into square images with equal length and width, and performing normalization preprocessing and data enhancement, followed by normalization processing, wherein the normalization preprocessing and data enhancement operations are specifically random flipping, region cropping and region masking and reorganization, including the following steps, Step (B1), taking the rectangular panoramic image with a side length of 3000×1000 containing the 360° annular scene information in the conference room output by the camera after correction in step (A), vertically cropping two original rectangular images with a side length of 2000×1000, and then splicing them up and down into a square image with a side length of 2000×2000, for adapting the input shape ratio of the deep learning detector; Step (B2), mapping the stitched square image to the original rectangular image position one by one, the upper half of the square image is directly mapped to the x∈[0,2000) position of the original rectangular image, and the lower half is an image with a side length of 2000×1000 formed by stitching multiple segments of x∈[0,500) and x∈[1500,3000) of the original image, so as to avoid the panoramic image being split; Step (B3), after the rectified image is mapped to the original image position, non-maximum suppression is performed to avoid repeated detection of the image after splicing in step (B1); Step (B4), data enhancement is performed on the spliced ​​square image, the original image is randomly flipped up and down and left and right, and then the partial image area containing the human target is randomly cropped, and the image area not containing the human target is smeared or covered in a mosaic manner; Step (B5), normalize the square image after data enhancement, convert each pixel value into a decimal in the interval (0, 1), and then compress the input image to a size of 512×512 in side length as the input image of the model.

4. According to the method of human figure detection in video conference scene based on deep learning in claim 1, it is characterized in that: Step (E), introducing an adaptive focus loss training model, inputting the conference scene image obtained in step (B) according to the trained human detection model, and detecting the position of the corresponding participant, including the following steps, Step (E1), adding a loss function training model, and adding two hyperparameters, namely weight coefficients α and γ, based on the cross entropy of the loss function, and then introducing adaptive focal loss to perform binary classification on the loss function to determine whether there is an object in the current detection area; Step (E2), input the conference scene image obtained in step (B), and use the center weight value ω output in step (D3) as a parameter of the loss function to participate in the calculation of the loss function. The farther the distance between the center of the detection area and the center of the real human body is, the smaller the loss function is; when there is no human body in the detection area, ω = 0; The loss function formula is as follows: Among them, the weight α is used to balance the imbalance of positive and negative samples, the weight γ is used to distinguish difficult and easy samples, p is the predicted value of confidence, ω is the central weight value output by step (D2), and σ is any small number greater than 0 to prevent division by zero. Step (E3), in the initial state, the γ value is 0. When γ increases, the adjustment factor also increases, that is, the loss generated by the simple sample is gradually suppressed. As the γ value increases, the loss generated by the simple sample is greatly reduced.

Images