A Smart Broadcast Switching Method Based on Multi-Person Pose Estimation and Motion Amplitude Detection

Abstract

This invention provides an intelligent broadcast switching method based on multi-person pose estimation and motion amplitude detection. First, the video signal is framed into images. These images are then input into a pre-built and trained multi-person pose estimation model to detect keypoint coordinates and related information. The motion amplitude detection module takes into account the differences in image depth and human scale. Based on the keypoint coordinates and related information, a motion amplitude detection algorithm is used to normalize and calculate pose feature values. Based on these values, the motion amplitude is judged. A threshold method is used to determine whether a motion is large or small. If a pose feature value exceeds a system-set threshold, the motion is considered large; otherwise, it is considered small. Finally, the intelligent broadcast system switches shots based on the detection results. This invention focuses on improving the accuracy of human keypoint localization and motion amplitude detection, achieving automated or assisted program production.

Description

Technical Field

[0001] This invention relates to an intelligent broadcast switching method based on multi-person posture estimation and motion amplitude detection, belonging to the field of artificial intelligence technology. Background Technology

[0002] Performing arts programs typically employ multi-camera shooting from different angles on stage, with the director selecting and switching between shots for broadcast. The director's choice of shots plays a crucial role in the overall effect and expressiveness of the program. However, traditional directing methods require experienced directors and teams, consuming significant manpower and resources. The production cycle is lengthy and cumbersome, involving multi-camera shooting, director switching, and post-production editing. This is especially true in live performances, where directors need to quickly assess multiple camera feeds to capture the performers' most memorable moments, making the selection and switching of shots paramount. Intelligent directing based on artificial intelligence aims to solve the intelligent directing problems of various forms of performing arts programs, providing automated and intelligent decision-making methods. It uses real-time recognition of the characters' states within the scene to facilitate director switching. This technology will significantly reduce production cycles and manpower and resource costs, while improving program production efficiency and quality. Unlike motion recognition and motion detection, which aim to detect the type of motion, this method's motion amplitude detection aims to detect the extension and variation range of human movements in specific scenarios, representing a new application direction in the field of artificial intelligence. The magnitude of facial movements can influence an actor's performance style and help them express more emotion and dynamism on stage. By detecting facial movements and analyzing video frame by frame, real-time status recognition of hosts and guests can be achieved. The results are then fed back to an intelligent directing system, enabling automated or assisted program production. This has significant research value and practical application potential.

[0003] With the development of deep learning, there are currently two main approaches to motion amplitude detection methods based on deep learning: First, image classification-based motion amplitude detection methods use image classification technology to identify and detect actors' movements, achieving simple motion amplitude detection. However, this method lacks analysis of pose features, and its real-time performance and ability to analyze multiple actors' movements remain challenging. Second, multi-person pose estimation-based motion amplitude detection methods primarily detect the joints of the human body, such as shoulders, knees, and ankles. Since the amplitude changes of performing arts movements are mainly reflected in the joints below the head, analyzing the relative positions and movements between these joints achieves motion amplitude detection. Therefore, this method has the advantages of better reducing background interference, focusing on the actor themselves, and offering fast operation and accurate detection.

[0004] Chinese patent CN201910530100.8 discloses a method for measuring the amplitude of human movements. The method includes the following steps: parsing a first video and a second video into a frame sequence; extracting the last frame of the first video and the first frame of the second video; using a joint detection algorithm to find the corresponding joints in the last frame and the first frame; calculating the average displacement of each pair of corresponding joints; and normalizing the average displacement Dis to obtain the amplitude values ​​of the two frames. The beneficial effect is that the method of this invention can finally calculate the amplitude of the movement of the next frame relative to the previous frame. This amplitude is also the measurement standard for our frame interpolation position. The two frames with larger amplitudes are cropped, and the two frames with smaller amplitudes are interpolated. The final video spliced ​​together has better smoothness.

[0005] As described in the invention patent, current motion amplitude detection mainly relies on the relative displacement of joints in adjacent video frames. However, in performing arts scenarios, due to the wide variety of movements, existing motion amplitude detection algorithms are prone to misjudgment, resulting in low reliability. Furthermore, current multi-person pose estimation algorithms used in artificial intelligence applications employ two-stage models: top-down and bottom-up. The top-down model suffers from high memory requirements, poor real-time performance, and high computational costs, while the bottom-up model is significantly affected by complex backgrounds, making it prone to misjudgment and matching errors. Therefore, existing multi-person pose estimation technologies require further improvement. Currently, there is a lack of motion amplitude detection algorithms based on multi-person pose estimation that achieve high speed and accuracy to meet the needs of intelligent broadcasting systems in performing arts scenarios.

[0006] In summary, the technical solution for motion amplitude detection has the following drawbacks:

[0007] 1. Consumes a large amount of computing resources:

[0008] The model requires manual operation, such as cropping, nonmaximum suppression, and grouping. It relies on object detection models to assist in motion amplitude detection, which consumes a lot of computing resources.

[0009] 2. Multi-person pose estimation models cannot achieve a balance between speed and accuracy:

[0010] Current methods for motion analysis based on multi-person pose estimation (including motion recognition, motion classification, motion amplitude detection, etc.) almost all use two-stage multi-person pose estimation models, which cannot achieve fully end-to-end multi-person pose estimation and achieve a balance between speed and accuracy.

[0011] 3. Low accuracy in motion amplitude detection:

[0012] Motion amplitude detection is mainly achieved through image classification or relative displacement of joints in adjacent video frames. However, it lacks human posture feature analysis, which leads to problems such as misjudgment, poor reliability, and low accuracy in motion amplitude detection. Summary of the Invention

[0013] To address the aforementioned technical problems, this invention provides an intelligent director switching method based on multi-person pose estimation and motion amplitude detection. It focuses on resolving issues such as the inability of multi-person pose estimation models to achieve a balance between speed and accuracy, leading to excessive computational resource consumption and low motion amplitude detection accuracy, and achieves excellent results. This invention applies an end-to-end multi-person pose estimation (PETR) model based on Transformers to motion amplitude detection, effectively considering the differences in image depth and human scale. Based on keypoint coordinates, it calculates pose feature values ​​through normalization processing using a motion amplitude detection algorithm, and then uses these features to determine motion amplitude, thereby achieving intelligent director switching. This enables automated or assisted program production, saving manpower and resources and improving work efficiency.

[0014] The technical solution adopted in this invention is an intelligent broadcast switching method based on multi-person posture estimation and motion amplitude detection, which includes the following steps:

[0015] Step 1: Extract images by frame extraction from the performance video captured by the camera;

[0016] Step 2: Build and train a multi-person pose estimation model;

[0017] Step 3: Input the image from Step 1 into the multi-person pose estimation model in Step 2, and calculate the pose features using the motion amplitude detection algorithm;

[0018] Step 4: Determine the amplitude of movement based on the posture features obtained in Step 3;

[0019] Step 5: Switch the camera angle based on the judgment of the movement amplitude.

[0020] Step 1 specifically includes:

[0021] Multiple cameras are used to acquire video of the posture and movement amplitude of multiple people, and frames are extracted from this video to obtain a multi-person posture and movement amplitude image.

[0022] Step 1.1: Select video frames of the multi-person posture and movement amplitude video at intervals of n frames, and discard other video frames directly due to information redundancy; where n is a positive integer greater than 1, for example, n is 10.

[0023] Step 1.2: Name each video frame according to the camera position information to obtain multiple images. Each image represents the current program screen of one camera position. For example, 1_01.jpg represents the first frame image of camera number 1.

[0024] Step 2 specifically includes:

[0025] The multi-person pose estimation model, also known as the PETR model, mainly includes a backbone network module, a position encoding module, a visual feature encoder module, a pose decoder module, and a joint decoder module. Based on the image obtained in step 1, the PETR model can output the coordinates of the human joints in the image and draw the human pose in the image according to the joint association information.

[0026] Step 2.1: The backbone network module is used to input the image from Step 1 and outputs a multi-scale feature map. The backbone network module is a ResNet-50, which is a 50-layer residual network used to extract feature maps from the image. The backbone network module is used to extract high-resolution multi-scale feature maps.

[0027] Step 2.2: The visual feature encoder generates a position code for each pixel based on the multi-scale feature map obtained in step 2.1 and the position encoding module, and generates a multi-scale feature token and pose query.

[0028] Step 2.3: The pose decoder is based on the multi-scale feature token F obtained in step 2.2 and N randomly initialized pose queries Q. pose ∈R N×D And obtain the poses of N bodies. in Let represent the coordinates of the K key points of the i-th person, and D represent the dimension of the query key. The pose decoder estimates the pose coordinates layer by layer using all decoder layers, with each layer refining the pose based on the predictions of the previous layer.

[0029] Step 2.4: The joint decoder uses the K joint information predicted by the pose decoder in Step 2.3 for each group of human poses as randomly initialized joint queries, and refines the joint position information and joint structure information. The joint queries and key feature information are continuously updated through self-attention and deformable cross-attention. Finally, the joint queries are aggregated by bias to further refine the joint position information and joint structure information.

[0030] Step 2.5, during training, a ensemble Hungarian loss-based approach is used to force a unique prediction for each true pose, reducing false positives and negative detections at key points. The classification loss function is denoted as Li. clsIt is used in the classification head. To eliminate the relative error of the prediction results, the PETR model uses OKS loss, which is calculated as the regression loss between the joint coordinates of the model's predicted human pose and the true joint coordinates. L1 loss is denoted as L reg The OKS loss is denoted as L. oks These are used as loss functions for the attitude regression head and the joint regression head, respectively, to enable the attitude decoder and the joint decoder to converge better.

[0031] Using heatmap-based regression to assist training adds directional guidance to the model's training. The closer to the target joint, the higher the activation value, allowing the model to quickly approach the target joint according to the guided direction, achieving rapid convergence. Therefore, the PETR model uses a deformable transformer encoder to generate heatmap predictions, and then calculates a variant Focal Loss, denoted as L, between the predicted and actual heatmaps. hm The heatmap branch is only used to assist training and will be removed during the inference phase. In summary, all loss functions of this PETR model can be expressed as follows:

[0032] L = L cls +λ1L reg +λ2L oks +λ3L hm

[0033] Where λ1, λ2, and λ3 represent the corresponding loss weights.

[0034] Step 2.6: Construct the COCO joint dataset as the training set and train and test the PETR model. Specifically, the COCO joint dataset contains a total of 200,000 images, of which 64,355 are joint images representing the joints of 250,000 people. Each person has 17 labeled joints, such as head, shoulder, arm, hand, knee, and ankle joints. The PETR model was trained for a total of 100 iterations. The learning rate was 2.5e-5. To improve the convergence of the PETR model, the learning rate was reduced to 2.5e-6 after 80 epochs. After training, the PETR model detects the coordinates of human joints in each input image and generates a human pose.

[0035] Step 3 specifically includes:

[0036] The input for this step is the coordinates of the human joints from step 2, and the output is the pose feature value of the human body in the image. The algorithm takes into account the differences in image depth and human scale, and consists of four detection conditions.

[0037] Step 3.1, the pose estimation joint connection format adopts the COCO human joint connection format, such as... Figure 5 As shown, (x i ,y i Let x and y be the x and y coordinates of the i-th joint of the human body. Calculate the coordinates of the midpoint s of the left and right shoulder joints (joint number: 5, 6), the midpoint h of the left and right hip joints (joint number: 11, 12), and the midpoint k of the left and right knee joints (joint number: 13, 14).

[0038] Step 3.2, Detection Condition 1: Calculate the ratio of the hand opening length to the body center trunk length. The hand opening length is determined by the joints of the left and right wrists (joint numbers: 9, 10), and the body center trunk length is determined by s and h. Based on the joint coordinates, the hand opening length d_hand and the body center trunk length d_body are calculated. When the ratio of the hand opening length to the body center trunk length is greater than or equal to the first threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on experimental results, the first threshold is set to 1.8, and the discrimination formula is as follows:

[0039]

[0040] Step 3.3, Detection Condition Two: Calculate the body torso tilt angle. The vector x1 composed of s and h, and the vector x2 composed of h and k, represent the body torso tilt angle. When the body torso tilt angle_1 is less than or equal to the second threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on experimental results, the second threshold is set to 150°. The discrimination formula is as follows:

[0041]

[0042] Step 3.4, Detection Condition Three: Calculate the opening and closing angles of the arms and torso. The opening and closing angle of the left arm is composed of the vector x3 of the left elbow and left shoulder (joint number: 7, 5) and the vector x4 of the left shoulder and left hip (joint number: 5, 11). The opening and closing angle of the right arm is composed of the vector x5 of the right elbow and right shoulder (joint number: 8, 6) and the vector x6 of the right shoulder and right hip (joint number: 6, 12). When the opening and closing angles of the left arm (angle_2) and the right arm (angle_3) are greater than or equal to the third threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on the experimental results, the third threshold is set to 90°. The discrimination formula is as follows:

[0043]

[0044] Step 3.5, Detection Condition 4: Calculate the opening angle between the legs and torso. The opening angle of the left leg is composed of the vector x7 of the left knee and left hip (joint number: 13, 11) and the vector x8 of the left hip and right hip (joint number: 11, 12). The opening angle of the right leg is composed of the vector x9 of the right knee and right hip (joint number: 14, 12) and the vector x8 of the right hip and left hip (joint number: 11, 12). 10 Composition. When the opening angle of the left leg (angle_4) and the opening angle of the right leg (angle_5) are greater than or equal to the fourth threshold, it is judged as a large-amplitude movement; otherwise, it is judged as a small-amplitude movement. Based on experimental results, the fourth threshold is set to 125°, and the discrimination formula is as follows:

[0045]

[0046] Step 4 specifically includes:

[0047] This step takes the human posture feature values ​​from the image output in step 3 as input and outputs the motion amplitude detection results. By calculating the amplitude of the posture feature changes, a threshold method is used to determine whether it is a large-amplitude motion.

[0048] Step 4.1: Compare the output of Step 3 with the set threshold. If any of the posture feature values ​​among the four detection conditions exceeds the set threshold, the performance action is determined to be a large-amplitude action; otherwise, it is a small-amplitude action.

[0049] Step 5 specifically includes:

[0050] The input for this step is the motion amplitude detection result output from step 4. The intelligent broadcasting system controls the camera to switch the lens to the camera view corresponding to the image with large motion amplitude.

[0051] Step 5.1, as described in step 1.2, involves naming each image according to the camera position information. Therefore, each image represents the current program view of a single camera position. For example, 1_01.jpg represents the first frame image of camera number 1. Thus, the intelligent broadcast control system can switch the camera view to the camera corresponding to the image with significant motion based on the image name and motion amplitude detection results.

[0052] This invention applies a fully end-to-end multi-person posture estimation model to motion amplitude detection, achieving functional and performance improvements, and focusing on enhancing the accuracy of human joint localization and motion amplitude detection. Based on joint coordinates, posture features are accurately and quickly calculated, and motion amplitude is determined accordingly. The intelligent broadcasting system achieves automated or assisted program production, saving significant manpower and resources and improving work efficiency.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0054] (1) High performance: Compared with the motion amplitude detection method based on image classification and the motion amplitude detection method based on pose estimation, this invention can better reduce the interference of the background environment and focus on the person itself. This model omits manual operation and does not need to rely on the target detection model, thus reducing a lot of computing resources. It has the advantages of high speed and low memory consumption.

[0055] (2) Complete end-to-end, with a trade-off between speed and accuracy: This invention applies the PETR model to the direction of motion amplitude detection. Compared with two-stage models such as top-down and bottom-up multi-person pose estimation, this model summarizes human pose estimation into a hierarchical set prediction problem. Human instances and fine-grained human joint coordinates are processed in a unified manner, realizing complete end-to-end multi-person pose estimation. Speed ​​and accuracy have reached a high level, with a trade-off between the two.

[0056] (3) High accuracy: Traditional motion amplitude detection mainly relies on the relative displacement of joints in adjacent video frames, and the number of motion categories detected is limited. This invention takes into account the different depth of field and scale of characters, and proposes a motion amplitude detection algorithm. By normalizing the motion amplitude detection through posture characteristic analysis, it achieves better detection results and higher accuracy in performance scenarios. Attached Figure Description

[0057] Figure 1 A flowchart illustrating a specific embodiment of the present invention;

[0058] Figure 2 This is a diagram showing the overall structure of the PETR model of this invention;

[0059] Figure 3 This is a structural diagram of the attitude decoder of the present invention;

[0060] Figure 4 This is a structural diagram of the joint decoder of the present invention;

[0061] Figure 5 This is a diagram showing the connection points of the COCO human body joints in this invention. Detailed Implementation

[0062] The method will be described in detail below with reference to the accompanying drawings and embodiments.

[0063] The flowchart of the implementation method is as follows Figure 1 As shown, it includes the following steps:

[0064] Step S10: Video frame extraction;

[0065] Step S20: Build and train a multi-person pose estimation model;

[0066] Step S30: Calculate pose features using a motion amplitude detection algorithm;

[0067] Step S40: Determine the range of motion;

[0068] Step S50: Based on the judgment result, the intelligent broadcasting system switches the camera angle.

[0069] Intelligent broadcasting based on artificial intelligence technology aims to solve problems in the production of various forms of performing arts programs, including real-time status recognition of hosts and guests in different scenarios and automatic broadcast switching. The core of this technology lies in using multi-person posture estimation to identify dynamic changes in human joints, thereby achieving automatic broadcast switching for various programs. The application of this technology will significantly reduce production cycles and manpower and material costs, and improve program production efficiency and quality. Motion recognition and motion detection aim to detect the type of motion. Unlike these, motion amplitude detection aims to detect the range of motion changes in the human body, belonging to a new application direction in the field of artificial intelligence. The magnitude of motion amplitude can affect an actor's performance style and help them express more emotion and dynamism on stage. Through motion amplitude detection, frame-by-frame analysis of video is performed to achieve real-time status recognition of hosts and guests, feeding the results back to the intelligent broadcasting system to achieve automatic or assisted program production.

[0070] The video frame extraction step S10 of the implementation method specifically includes:

[0071] This step takes video from multiple cameras as input and outputs images obtained by extracting video frames. The images obtained by extracting video frames are used as input to the PETR model in step S20.

[0072] Step S100: The image naming method is as follows: each image is named according to the camera position information, so each image represents the current program screen of a camera position; for example, 1_01.jpg represents the first frame image of camera No. 1.

[0073] The video frame extraction method in steps S110 and S100 is as follows: video frames are selected at intervals of n frames, and other video frames are directly discarded due to information redundancy; where n is a positive integer greater than 1, for example, n is 10.

[0074] Step S20, which involves building and training a multi-person pose estimation model, specifically includes:

[0075] The input for this step is the image obtained in step S10, and the output is the coordinates of the human joints. The human pose is then drawn onto the image according to the joint association information. The PETR model mainly includes a backbone network module, a position encoder module, a visual feature encoder module, a pose decoder module, and a joint decoder module; the overall structure diagram of the PETR model is shown below. Figure 2 As shown.

[0076] Step S200: The input to this step is the image obtained in step S10, and the output is a multi-scale feature map. ResNet-50 is used as the backbone network of the PETR model. ResNet-50 is a 50-layer residual network that can be used to extract feature maps from images. The model of this invention is used to extract multi-scale feature maps for the last three stages. Input image Where H is the image height and W is the width, the last three stages of multi-scale feature maps are extracted using the ResNet-50 network; ResNet-50 is a 50-layer residual network that can be used to extract feature maps of an image.

[0077] Step S210: Visual Feature Encoder of the PETR Model. The input to this step is the multi-scale feature map obtained in step S200 and the positional encoding generated for each pixel by the positional encoding module. The output is multi-scale feature information and a pose lookup key. Since the multi-head attention module has a quadratic complexity of the input scale, the visual feature encoder uses a deformable multi-head attention module to implement feature encoding. The multi-scale feature information F∈R... L×256 Where L is the total number of tokens. Finally, F is input together with the attitude query key into the attitude decoder for attitude prediction.

[0078] Step S220: Pose Decoder for the PETR Model. The input to this step is the multi-scale feature F obtained in step S210 and N randomly initialized pose queries Q. pose ∈R N×D The pose decoder outputs the poses of N bodies. in Let represent the coordinates of the K key points of the i-th person, and D represent the dimension of the query key.

[0079] The structure diagram of the attitude decoder is as follows: Figure 3 As shown, the pose query is first input into SelfAttention for interaction between pose queries, i.e., Pose-To-Pose SelfAttention. Then, each pose query extracts K feature map pixels from the multi-scale feature memory F layer by layer as keys through Deformable Cross-Attention. Based on the pose query, features are aggregated according to the bias as values. The Cross-Attention module outputs K deduced joint coordinates as the initial coordinates of the human pose joints. Subsequently, the pose query with the attention feature information of the keys is input into the multi-task prediction head, where the classification head predicts the confidence of each target through a linear mapping layer; the pose regression head uses an MLP with 256 hidden layers to predict the relative position offset of the K deduced points.

[0080] The pose decoder can consist of multiple decoding layers. This differs from other Transformer methods that only use the last decoder layer to predict pose coordinates. PETR's pose decoder estimates pose coordinates layer by layer using all decoder layers, with each layer refining the pose based on the prediction of the previous layer.

[0081] Step S230: Joint Decoder of the PETR Model. The input to this step is the K joint information predicted by the pose decoder in step S220 for each human pose, used as randomly initialized joint queries. The output is further refined joint position and structure information. The joint queries and key feature information are continuously updated through self-attention and deformable cross-attention. Finally, the joint queries are aggregated through bias to further refine the joint position and structure information. Since the joints of each human pose are independent, all poses can be processed in parallel, significantly reducing the time complexity of network prediction and derivation. The structure diagram of the joint decoder is shown below. Figure 4 As shown.

[0082] The keypoint lookup process first involves the interaction of keypoint query features through a self-attention module, i.e., Joint-To-Joint attention. Then, a deformable cross-attention module, i.e., Feature-To-Joint, extracts visual features. Subsequently, the keypoint prediction head uses an MLP to predict the relative displacement ΔJ = (Δx, Δy) between 2D keypoints. Similar to the pose decoder, the keypoint coordinates are also progressively refined.

[0083] The steps S200 to S230 above constitute the construction of the overall framework of the PETR model. The specific steps for training the PETR model are described below.

[0084] Step S240: During training, a ensemble Hungarian loss-based approach is used to force a unique prediction for each true pose, reducing false positives and negative detections at key points. The classification loss function is denoted as L... cls It is used in the classification head. To eliminate the relative error of the prediction results, the PETR model uses OKS loss, which is calculated as the regression loss between the joint coordinates of the model's predicted human pose and the true joint coordinates. L1 loss is denoted as L reg The OKS loss is denoted as L. oks The loss functions used for the pose regression head and joint regression head, respectively, enable the pose decoder and joint decoder to converge better.

[0085] Using heatmap-based regression to assist training adds directional guidance to the model's training. The closer to the target joint, the higher the activation value, allowing the model to quickly approach the target joint according to the guided direction, achieving rapid convergence. Therefore, the PETR model uses a deformable transformer encoder to generate heatmap predictions, and then calculates a variant Focal Loss, denoted as L, between the predicted and actual heatmaps. hm The heatmap branch is only used to assist training and will be removed during the inference phase. In summary, all loss functions of this PETR model can be expressed as follows:

[0086] L = L cls +λ1L reg +λ2L oks +λ3L hm

[0087] Where λ1, λ2, and λ3 represent the corresponding loss weights.

[0088] In step S250, the COCO joint dataset was used as the training set, and ResNet-50 was used as the backbone network. The model was trained for a total of 100 iterations. The learning rate was 2.5e-5, and to improve model convergence, the learning rate was reduced to 2.5e-6 after 80 epochs. The PETR model was then trained and tested. Using the trained PETR model, the coordinates of human joints were detected for each input image, and human pose was generated.

[0089] Step S30, which calculates pose features using a motion amplitude detection algorithm, specifically includes:

[0090] The input for this step is the coordinates of the human joints from step S20, and the output is the pose feature value of the human body in the image. The algorithm takes into account the differences in image depth and human scale, and consists of four detection conditions.

[0091] Step S300: The pose estimation joint connection format of this invention adopts the COCO human joint connection format, such as... Figure 5 As shown, (x i ,y i Let x and y be the x and y coordinates of the i-th joint of the human body. Calculate the coordinates of the midpoint s of the left and right shoulder joints (joint number: 5, 6), the midpoint h of the left and right hip joints (joint number: 11, 12), and the midpoint k of the left and right knee joints (joint number: 13, 14).

[0092] Step S310, Detection Condition 1: Calculate the ratio of the hand opening length to the body center trunk length. The hand opening length is determined by the joints of the left and right wrists (joint numbers: 9, 10), and the body center trunk length is determined by s and h. Based on the joint coordinates, the hand opening length d_hand and the body center trunk length d_body are calculated. When the ratio of the hand opening length to the body center trunk length is greater than or equal to a first threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on experimental results, the first threshold is set to 1.8, and the discrimination formula is as follows:

[0093]

[0094] Step S320, Detection Condition Two: Calculate the body torso tilt angle. The vector x1 composed of s and h, and the vector x2 composed of h and k, represent the body torso tilt angle. When the body torso tilt angle_1 is less than or equal to the second threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on experimental results, the second threshold is set to 150°. The discrimination formula is as follows:

[0095]

[0096] Step S330, Detection Condition 3: Calculate the opening and closing angle between the arm and torso. The opening and closing angle of the left arm is composed of the vector x3 of the left elbow and left shoulder (joint number: 7, 5) and the vector x4 of the left shoulder and left hip (joint number: 5, 11). The opening and closing angle of the right arm is composed of the vector x5 of the right elbow and right shoulder (joint number: 8, 6) and the vector x6 of the right shoulder and right hip (joint number: 6, 12). When the opening and closing angle of the left arm (angle_2) or the opening and closing angle of the right arm (angle_3) is greater than or equal to the third threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on the experimental results, the third threshold is set to 90°. The discrimination formula is as follows:

[0097]

[0098] Step S340, Detection Condition 4: Calculate the opening angle between the legs and torso. The opening angle of the left leg is composed of the vector x7 of the left knee and left hip (joint number: 13, 11) and the vector x8 of the left hip and right hip (joint number: 11, 12). The opening angle of the right leg is composed of the vector x9 of the right knee and right hip (joint number: 14, 12) and the vector x8 of the right hip and left hip (joint number: 11, 12). 10 Composition. When the opening angle of the left leg (angle_4) or the opening angle of the right leg (angle_5) is greater than or equal to the fourth threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on experimental results, the fourth threshold is set to 125°, and the discrimination formula is as follows:

[0099]

[0100] The motion amplitude determination step S40 specifically includes:

[0101] This step takes the human posture feature values ​​from the image output in step S30 as input and outputs the motion amplitude detection result. By calculating the amplitude of the posture feature changes, a threshold method is used to determine whether the motion is large.

[0102] The output results of steps S400 and S30 are compared with the set threshold. If any of the posture feature values ​​among the four detection conditions exceeds the system-set threshold, the performance action is determined to be a large-amplitude action; otherwise, it is a small-amplitude action.

[0103] Based on the judgment result, the intelligent broadcast control system's camera switching step S50 specifically includes:

[0104] The input for this step is the motion amplitude detection result output by step S40. The intelligent broadcasting system will switch the camera to the camera view corresponding to the image with large motion amplitude.

[0105] In step S500, as described in step S100, each image is named according to the camera position information. Therefore, each image represents the current program view of a camera position. For example, 1_01.jpg represents the first frame image of camera number 1. Thus, the intelligent broadcast control system can switch the camera view to the camera corresponding to the image with large-amplitude motion based on the image name and motion amplitude detection results.

[0106] The experimental results of applying this invention are given below.

[0107] In the intelligent broadcasting system of this invention, the interval between keyframe images is approximately T1 = 0.3333s, and the processing time for one keyframe image is T2 = 0.3024s. Because T1 > T2, the system can complete motion amplitude detection when the next image is input, achieving real-time camera switching. Therefore, the detection speed and accuracy of this invention meet the requirements of practical application scenarios.

[0108] Table 1 shows the results of testing using the present invention on 600 performing arts test images.

[0109] Table 1 Results of Performance Movement Range Test

[0110] Image categories quantity accuracy large-scale performance movements 300 88.0 Small movements in performance 300 86.33

[0111] The beneficial effects of this invention are as follows:

[0112] This invention applies a fully end-to-end multi-person pose estimation model to motion amplitude detection, achieving functional and performance improvements, focusing on enhancing the accuracy of human joint localization and motion amplitude detection. Based on joint coordinates, pose features are accurately and quickly calculated, and motion amplitude is determined accordingly, achieving automated or assisted program production, saving significant manpower and resources, and improving work efficiency. Furthermore, this invention also has the following beneficial effects:

[0113] 1. High performance: It can better reduce interference from the background environment, focus on the person itself, eliminate manual operation, does not need to rely on object detection models, reduces a lot of computing resources, and has the advantages of high speed and low memory consumption.

[0114] 2. Fully end-to-end, with a trade-off between speed and accuracy: Compared to two-stage models such as top-down and bottom-up multi-person pose estimation, this model summarizes human pose estimation as a hierarchical set prediction problem, and unifies the processing of human instances and fine-grained human joint coordinates, achieving fully end-to-end multi-person pose estimation. It achieves a high level of speed and accuracy, with a trade-off between the two.

[0115] 3. High Accuracy: Traditional motion amplitude detection mainly relies on the relative displacement of joints in adjacent video frames, resulting in a limited number of motion categories. This invention takes into account the differences in scene depth and character scale, proposing a motion amplitude detection algorithm. It normalizes motion amplitude detection through posture characteristic analysis, leading to better and more accurate detection of performance motion amplitude in entertainment scenarios.

Claims

1. A smart broadcast switching method based on multi-person posture estimation and motion amplitude detection, characterized in that, Includes the following steps: Step 1: Extract images by frame extraction from the performance video captured by the camera; Step 2: Build and train a multi-person pose estimation model; the multi-person pose estimation model, namely the PETR model, includes a backbone network module, a position encoding module, a visual feature encoder module, a pose decoder module, and a joint decoder module; based on the image obtained in Step 1, the PETR model can output the coordinates of human joints in the image and draw the human pose in the image according to the joint association information. Step 3: Input the image from Step 1 into the multi-person pose estimation model in Step 2, and calculate each pose feature using the motion amplitude detection algorithm. The input for this step is the coordinates of the human joints from Step 2, and the output is the pose feature value of the human body in the image. Considering the different depth of field and human scale, the motion amplitude detection algorithm consists of four detection conditions: Detection condition 1: Calculate the ratio of the opening and closing length of the hands to the length of the torso at the center of the body; Detection condition 2: Calculate the torso tilt angle; Detection condition 3: Calculate the opening and closing angle between the arms and torso; Detection condition 4: Calculate the opening and closing angle between the legs and torso. Step 4: Determine the amplitude of movement based on the posture features obtained in Step 3; Step 5: Switch the camera angle based on the judgment of the movement amplitude; Step 4 specifically includes: The input for this step is the human posture feature value in the image output from step 3, and the output is the motion amplitude detection result; by calculating the amplitude of the posture feature change, a threshold method is used to determine whether it is a large-amplitude motion; Step 4.1: Compare the output of Step 3 with the set threshold. If any of the posture feature values ​​among the four detection conditions exceeds the set threshold, the performance action is determined to be a large-amplitude action; otherwise, it is a small-amplitude action. Step 5 specifically includes: The input for this step is the motion amplitude detection result output from step 4. The intelligent broadcasting system controls the camera to switch the lens to the camera screen corresponding to the image with large motion amplitude. Step 5.1, as described in step 1.2, each image is named according to the camera position information, and each image represents the current program screen of a camera position; the intelligent directing system switches the camera to the camera screen corresponding to the image with large-amplitude motion based on the image name and motion amplitude detection results.

2. The intelligent broadcast switching method based on multi-person posture estimation and motion amplitude detection according to claim 1, characterized in that, In step 1, multiple cameras are used to acquire video of the posture and movement amplitude of multiple people, and frames are extracted from this video to obtain a multi-person posture and movement amplitude image; specifically, this includes: Step 1.1: Select video frames of the multi-person posture and movement amplitude video at intervals of n frames, and discard other video frames directly due to information redundancy; where n is a positive integer greater than 1. Step 1.2: Name each video frame according to the camera position information to obtain multiple images, each image representing the current program screen of a camera position.

3. The intelligent broadcast switching method based on multi-person posture estimation and motion amplitude detection according to claim 2, characterized in that, Step 2 specifically includes: Step 2.1: The backbone network module is used to input the image from Step 1 and output a multi-scale feature map. The backbone network module is a ResNet-50, which is a 50-layer residual network used to extract the feature map of the image. The backbone network module is used to extract high-resolution multi-scale feature maps. Step 2.2: The visual feature encoder generates a position code for each pixel based on the multi-scale feature map obtained in step 2.1 and the position encoding module, and generates a multi-scale feature token and pose query. Step 2.3: The pose decoder is based on the multi-scale feature token F obtained in step 2.2 and N randomly initialized pose queries Q. pose ∈R N×D And obtain the poses of N bodies. in Let represent the coordinates of the K key points of the i-th person, and D represent the dimension of the query key. The pose decoder estimates the pose coordinates layer by layer with the help of all decoder layers, and each layer refines the pose based on the prediction of the previous layer. Step 2.4: The joint decoder uses the K joint information of each human pose predicted by the pose decoder in Step 2.3 as a randomly initialized joint query, and refines the joint position information and joint structure information. The joint query and key feature information are continuously updated through self-attention and deformable cross-attention. Finally, the joint query is further refined through deviation aggregation of features to refine the joint position information and joint structure information. Step 2.5, during training, a set Hungarian loss is used, with the classification loss function denoted as L. cls It is used in the classification head; to eliminate the relative error of the prediction results, the PETR model uses OKS loss, which is calculated as the regression loss between the joint coordinates of the human pose predicted by the model and the true joint coordinates; L1 loss is denoted as L reg The OKS loss is denoted as L. oks , which are used as loss functions for the attitude regression head and the joint regression head, respectively; The PETR model uses a deformable transformer encoder to generate heatmap predictions. A variant Focal Loss, denoted as L, is calculated between the predicted and actual heatmaps. hm The heatmap branch is only used to assist training and will be removed during the inference phase; all loss functions of the PETR model are expressed as follows: L=L cls +λ1L reg +λ2L oks +λ3L hm Where λ1, λ2, λ3 represent the corresponding loss weights; Step 2.6: Construct the COCO joint dataset as the training set, and train and test the PETR model. The COCO joint dataset contains a total of 200,000 images, of which there are 64,355 joint images, containing joints of 250,000 people. Each person has 17 labeled joints. The PETR model is trained for a total of 100 iterations. The learning rate is 2.5e-5. In order to make the PETR model converge better, the learning rate is reduced to 2.5e-6 when the training reaches 80 epochs. After training the PETR model, the coordinates of human joints are detected for each input image, and human pose is generated.

4. The intelligent broadcast switching method based on multi-person posture estimation and motion amplitude detection according to claim 3, characterized in that, Step 3 specifically includes: Step 3.1, the pose estimation joint connection format adopts the COCO human joint connection format, (x i ,y i Let x and y be the x and y coordinates of the i-th joint of the human body. Calculate the coordinates of the midpoint s of the left and right shoulder joints, the midpoint h of the left and right hip joints, and the midpoint k of the left and right knee joints. Step 3.2, Detection Condition 1 is as follows: Calculate the hand opening and closing length d_hand and the body center trunk length d_body based on the joint coordinates. When the ratio of the hand opening and closing length to the body center trunk length is greater than or equal to the first threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on experimental results, the first threshold is set to 1.8, and the discrimination formula is as follows: Step 3.3, the second detection condition is as follows: when the body torso tilt angle_1 is less than or equal to the second threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on the experimental results, the second threshold is set to 150°, and the discrimination formula is as follows: Step 3.4, the third detection condition is as follows: when the opening angles of the left arm (angle_2) and the right arm (angle_3) are greater than or equal to the third threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on the experimental results, the third threshold is set to 90°, and the discrimination formula is as follows: Step 3.5, the fourth detection condition is as follows: when the opening angle of the left leg (angle_4) and the opening angle of the right leg (angle_5) are greater than or equal to the fourth threshold, it is judged as a large-amplitude movement; otherwise, it is a small-amplitude movement. Based on the experimental results, the fourth threshold is set to 125°, and the discrimination formula is as follows: true。

Images