Multi-camera automatic switching method based on large model interpretable director intention modeling

Abstract

The application provides a multi-camera automatic switching method based on large model interpretable director intention modeling, relates to the technical field of intelligent director and image processing, and first converts the dancer's body movement, facial expression and mutual interaction into a structured description through multi-agent time sequence semantic coding; then quantifies indexes such as "highlight degree, focus degree, composition coverage, occlusion and rhythm" into reward signals with an interpretable director intention model, and clearly defines the switching basis and interpretability; further, a lightweight and efficient decision learning method such as in-group normalization and commentator-free strategy optimization is adopted to generate stable camera selection under the constraints of low latency and little jitter; finally, the optimal action is mapped into director station control instructions, and a continuous and good-observation main screen stream is output in real time; the application carries the interpretable expression of "director intention" with the semantic understanding of a large model, and realizes the stability, low latency and consistent stylized output of multi-camera automatic switching with a lightweight and efficient strategy optimization.

Description

Technical Field

[0001] This invention relates to the fields of intelligent broadcasting and image processing technology, and in particular to a multi-camera automatic switching method based on large-scale interpretable broadcasting intent modeling. Background Technology

[0002] The booming field of live streaming group dance is facing a very challenging reality: "multiple cameras, high pace, and low tolerance for error." Existing technologies have many obvious shortcomings in addressing this issue.

[0003] Currently, human directors hold a certain position in live group dance broadcasts. However, their limitations become apparent when faced with high-speed switching between 3-5 camera angles. Due to limited human energy and physiological limits to reaction speed, human directors are prone to missing crucial moments under such frequent switching operations. Those fleeting, brilliant dance moves and highly infectious interactions between dancers may not be captured and presented to the audience in time due to director oversight, significantly reducing the broadcast's entertainment value and appeal. Furthermore, the style of human directors largely depends on their individual experience and aesthetic preferences, making it difficult to establish a stable and reusable directing model. Different directors may have significant differences in the timing of switching and camera selection for the same dance broadcast, resulting in inconsistent broadcast quality and hindering a consistent and high-quality viewing experience.

[0004] Traditional solutions based on thresholds / rules or single visual triggers also attempt to address multi-camera switching issues, but these solutions have serious flaws. They are extremely sensitive to factors such as occlusion, stage movement, and lighting changes in the live streaming environment. In actual group dance live streaming scenarios, frequent interactions between dancers inevitably lead to occlusion, and the movement of dancers on stage causes dynamic changes in the image. Furthermore, lighting differences at different times and angles are common. When faced with these complex situations, these traditional solutions often fail to accurately judge and switch camera positions, resulting in problems such as stuttering, discontinuity, or switching errors in the live stream. Moreover, these solutions lack explanatory evidence; when switching errors or unsatisfactory situations occur, it is difficult to analyze the causes of the problems from a technical perspective, making targeted improvements and optimizations impossible.

[0005] Furthermore, while black-box intelligent switching technology can achieve a certain degree of automated camera switching, it also has many problems. Because its internal working mechanism is like a "black box," its switching decision-making process is difficult to explain and understand. In practical applications, we cannot know why it chooses a specific camera position for switching at a certain moment, which is a huge obstacle to pursuing high-quality live broadcasts and precise directing effects. At the same time, black-box intelligent switching technology struggles to meet industrial constraints of low latency and minimal jitter. In live group dance broadcasts, low latency is crucial to ensuring viewers can watch the live stream in real time and smoothly, while minimal jitter guarantees the stability and continuity of the image. However, existing black-box intelligent switching technologies perform poorly in both aspects, frequently exhibiting problems such as image delay and jitter, severely impacting the viewer's experience.

[0006] Therefore, there is an urgent need for a multi-camera automatic switching method based on large-scale interpretable director intent modeling to solve the above problems. Summary of the Invention

[0007] Addressing the real-world pain points of "multi-camera, high-paced, and low-tolerance" in live group dance broadcasts: human directors are prone to missing exciting moments and struggling to consistently reuse styles when switching between 3-5 cameras at high speeds; traditional solutions based on thresholds / rules or single visual triggers are extremely sensitive to occlusion, stage movement, and lighting changes, and lack interpretable evidence; while black-box intelligent switching can automate, it is difficult to interpret and cannot meet the aforementioned technical problems of low latency and minimal jitter in industrial applications. This invention proposes a multi-camera automatic switching method based on interpretable director intent modeling using a large model. The core of this invention is a large model, motivated by the goal of abstracting the actions, expressions, and interactions of multiple people into measurable and reusable "director intent" indicators (such as excitement level, focus, coverage, and switching cost), using interpretable intent modeling to connect perception and decision-making; and without increasing the computational burden, employing lightweight and efficient strategies to optimize low-latency, stable automatic switching, solidifying the aesthetics and rhythm of human directors into transferable process capabilities, ultimately reducing human workload, improving image quality, and achieving consistent output across different scenarios.

[0008] This invention provides a method for automatic multi-camera switching based on large-scale interpretable director intent modeling, comprising the following steps:

[0009] S1. Align the original video stream of each camera position k to the common timeline to obtain a time-aligned frame sequence. For each frame Multiple skeletons were extracted to obtain a set of key points for the skeletons of multiple people. Then, time-aligned frame sequences Set of key points for multiple skeletons After performing skeleton-based face embedding and expression classification, temporal correlation is performed to obtain the face embedding feature sequence. With emoji tag vector sequence And align; ; t is the time step; m is the trajectory index; K is the total number of bits; T is the total number of frames after alignment; The number of subjects detected in the frame;

[0010] S2, Set of skeletal key points After performing affine motion estimation and skeletal residual decomposition, the main skeleton residual is obtained. The main skeleton remains With facial embedding features and emoji tag vectors Concatenate into a subject-level semantic vector Then, HAIC-style packaging is performed to obtain a machine-level semantic vector sequence. Then, the machine-level semantic vector sequence After performing cross-camera semantic aggregation, the cross-camera global state is obtained through mapping. ;

[0011] The HAIC style packaging is a multi-subject time-series packaging. At the same camera position and at the same time, it maintains the consistency of the order of person-by-person, time-by-attribute or interaction during the packaging process, and aggregates key interaction relationships internally.

[0012] S3. Based on the camera position-level semantic vector corresponding to camera position action a Global status across machine positions Construct the action-related feature vector of camera position action a and their corresponding instant rewards And based on the global state across machine positions Generate a set of trajectory reward samples Finally, based on the global status across camera positions With trajectory reward sample set Construct a training sample set ;

[0013] S4. GPG optimization of training sample set based on group policy gradient. Gradient optimization is used to obtain the optimal strategy. ;in, Indicates camera position movement; Indicates the global status across machine positions;

[0014] S5. Calculate the action of any candidate camera position in the current state. Instant rewards After filtering out candidate actions that do not meet the residency constraints, proceed according to the optimal strategy. Select the directing action to get the current time step. Then, the directing action at the current time step. Control vector mapped to cross-point trigger command And according to the control vector Select the corresponding camera position to obtain the director's output stream. .

[0015] Specifically, step S1 includes the following steps:

[0016] S11, All raw video streams from all camera positions Based on the public time index using the time alignment operator Alignment is performed to obtain the time-aligned frame sequence of all camera positions. ;

[0017] S12, Positioning the camera RGB frames at time step t Estimation function for multiple skeletons Obtain the set of skeletal key points corresponding to this frame. ;

[0018] S13. Align the frame sequence according to the time of all camera positions. and skeletal key point set Through face embedding and expression classification functions Obtain face embedding feature sequence With emoji tag vector sequence ;

[0019] S14. Within each camera position k, through time correlation and trajectory generation functions Set up the key points of the skeleton Face embedding feature sequence and emoji tag vector sequence exist Top-aligned to a fixed trajectory index .

[0020] Specifically, step S2 includes the following steps:

[0021] S21. Fit the skeletal points of all subjects in adjacent frames from the same camera position to obtain the affine motion matrix of the camera position. Then, based on the affine motion matrix of the camera position The corrected skeletal residuals of each subject are calculated and motion suppression is performed to obtain the subject's skeletal residuals. ;

[0022] S22, Regarding the residual of the main skeleton After dimensionality compression and normalization, it is then embedded with face features. and emoji tag vectors Concatenate into a subject-level semantic vector Then, all subject-level semantic vectors from the same camera position at the same time are packaged in a HAIC style to obtain a camera-level semantic vector sequence. ;

[0023] S23. At each time step, the seat-level semantic vector sequence is processed in a fixed order according to the seat index. Machine-level semantic vectors of each camera position The joint vector at the machine position level is obtained by concatenation. And input it into the mapping function to obtain the global state across machine positions. .

[0024] Specifically, step S3 includes the following steps:

[0025] S31, Redirect the action of the previous time step Encoded as a one-hot encoded vector Then take action on any candidate camera position. Corresponding machine-level semantic vector and cross-machine global state and one-hot encoded vector Input the feature extraction function together Obtain action-related feature vectors ;

[0026] S32. Introducing a linear weight vector With switching penalty coefficient The station actions are obtained through linear summarization. The instantaneous reward function at time step t :

[0027] ,

[0028] in For indicator functions; Currently executed actions; top right corner Indicates transpose;

[0029] S33, Based on cross-station global state sequence Using temperature sampling strategy At each time step t, an action is sampled to generate a set of trajectory reward samples with discounted rewards. ;

[0030]

[0031] Where GroupSample is the group sampling function; This is a temperature-inclusive sampling strategy; G represents the number of samples. For the action sequence of the i-th sample, The corresponding discount will be given in return;

[0032] S34 is the set of trajectory return samples. Add a strategy log loss term Combined with cross-camera global status Construct a training sample set Where B represents the total number of trajectories in the batch; b represents the index of the total number of trajectories in the batch. The global state across machine positions at time step t; For the action sequence of the i-th sample within the b-th batch trajectory; for Corresponding discount reward.

[0033] Specifically, step S4 includes the following steps:

[0034] First, calculate the mean return within each group of the sample sequence. And define the group's strengths accordingly. Subsequently, the gradient objective of the original policy is... Obtain its unbiased gradient estimate When invalid samples exist in the batch, causing a systematic underestimation of the gradient magnitude, a linear correction for AGE is introduced, and the correction amount is set to... Finally, using the learning rate For broadcast strategy parameters Perform a minimal update:

[0035]

[0036] After several rounds of iteration, the optimal broadcasting strategy parameters were obtained. Then, based on the optimal broadcast strategy parameters Obtain its corresponding optimal strategy Where G is the group size; T is the duration; and M is the number of invalid samples. The logarithmic loss term is the policy term; j is the group index; As a discount return.

[0037] Specifically, step S5 includes the following steps:

[0038] S51. Calculate the action of any candidate camera position in the current state. Action-related feature vectors and their corresponding instant rewards ;

[0039] S52. Based on the threshold mask, candidate actions that do not satisfy the dwell constraints are masked to obtain a set of actionable actions. Then, based on the set of actionable actions, the optimal strategy is applied. Make a decision to obtain the director's action at the current time step. ;

[0040] S53, Set the directing action for the current time step. Control vector mapped to cross-point trigger command And according to the control vector Select the corresponding camera position to obtain the director's output stream. .

[0041] Specifically, step S21 includes:

[0042] The affine motion matrix of the machine position As shown below:

[0043] ,

[0044] Among them, AffineFit() is an affine fitting function, which is used to perform least squares fitting of a two-dimensional affine transformation on a set of paired key points of the same camera position in adjacent frames. Indicates the camera position All human skeletal points at time t; Indicates the camera position All human skeletal points at time t+1; To fit the linear part; To fit the translation vector;

[0045] Based on the affine motion matrix of the machine position The corrected skeletal residuals for each subject are calculated as follows:

[0046] ,

[0047] in, To correct for skeletal defects; For camera position At time t The individual's key skeletal matrix; For camera position At time t+1, the The individual's key skeletal matrix;

[0048] Motion suppression is applied to the residuals using a motion clipping function, while preserving the main skeleton residuals obtained from the same-dimensional representation. :

[0049] ,

[0050] Where ClipMotion is the motion shearing function; J is the number of skeletal keypoints.

[0051] Specifically, step S22 includes:

[0052] For each machine position Each time step Each subject First, through the aggregation operator Residual of the main skeleton Dimensional compression and normalization are performed to obtain a length of The motion description vector is then used in conjunction with the face embedding. Emoji Tags Concatenate the data to construct a subject-level semantic vector. ,in ; The dimension of the subject-level semantic vector; Indicates the output dimension of Pool; The dimension of facial embedding features; Number of expression categories; then, use the same camera position. At the same time step All main semantic vectors The input HAIC-style packing operator is used to pack the data to obtain the machine-level semantic vector. As shown in the following formula:

[0053] ,

[0054] HAICPack is the HAIC-style packing operator; This represents the HAIC-style representation of camera position k at time step t; further, it is derived from the camera position-level semantic vector. Obtain the machine-level semantic vector sequence .

[0055] Specifically, the feature extraction function described in step S31 As shown below:

[0056] ;

[0057] Among them, the first layer mapping matrix Second layer mapping matrix is a trainable linear mapping matrix.

[0058] Specifically, step S33 includes:

[0059] Given a global state sequence across machine positions Under the premise of temperature sampling strategy Group sampling is performed at each time step t to obtain the group sampling results, which serve as the trajectory report sample set. , i.e., candidate broadcast trajectory;

[0060]

[0061] Where GroupSample is the group sampling function; This is a temperature-inclusive sampling strategy; G represents the number of samples. For the action sequence of the i-th sample, The corresponding discount will be given as a reward.

[0062] This invention centers on "a large-scale model that can explain the director's intent," and is implemented in a closed loop of "perception—intent—decision—control." First, 3–5 camera feeds are aligned along a unified timeline, and multiple subjects are detected, key points and facial expressions are recognized, and trajectory association is performed on each feed, encoding multi-source information ("who is where, what did they do") into a stable temporal semantic vector. Then, through camera motion estimation and cancellation, pseudo-motion introduced by translation / zoom is eliminated, retaining only the real human-driven action signals, and aggregating across camera positions to obtain the global state. Based on this state, an interpretable director's intent model is constructed, quantifying "excitement level, focus level, composition coverage, occlusion risk, and rhythmic coherence" into unified reward signals. Simultaneously, constraints such as switching costs and minimum dwell time are introduced to clarify "when and why to switch." In the policy learning phase, a lightweight policy optimization model with intra-group normalization (not relying on a value network or reference policy) is adopted, directly targeting reward improvement for offline / quasi-online training, balancing convergence speed and low latency requirements. During online inference, the strategy distribution and instant rewards are integrated for scoring. Combined with minimum dwell time gating and business rules, the optimal camera position is selected in real time, and the control command from the broadcast control station is issued to complete the switching. At the same time, the "state-action-reward" feedback is used for continuous fine-tuning and effect tracking, and interpretable logs are output to support debriefing and style reuse. The overall solution uses the semantic understanding of a large model to carry the interpretable expression of "director's intent", and uses lightweight and efficient strategy optimization to achieve stability, low latency, and consistent stylistic output for automatic switching of multiple cameras.

[0063] This invention provides a multi-camera automatic switching method based on interpretable director intent modeling using a large model. First, through multi-subject temporal semantic encoding, the dancers' body movements, facial expressions, and interactions are transformed into structured descriptions. Then, using an interpretable director intent model, indicators such as "excitement level, focus, composition coverage, occlusion, and rhythm" are quantified into reward signals, clarifying the switching basis and interpretability. Next, lightweight and efficient decision learning methods, such as in-group normalization and commentator-free strategy optimization, are employed to generate stable camera selections under low latency and low jitter constraints. Finally, the optimal action is mapped to control commands from the control console, outputting a continuous and visually appealing main video stream in real time. In summary, this invention replicates the aesthetics and rhythm of human directors through "interpretable intent modeling using a large model + efficient strategy optimization," achieving intelligent real-time switching of multiple cameras.

[0064] Furthermore, this invention abstracts the actions, expressions, and interactions of multiple people into measurable and reusable "director intent" indicators (such as brilliance, focus, coverage, and switching cost), using interpretable intent modeling to connect perception and decision-making; and without increasing the heavy computational burden, it adopts a lightweight and efficient strategy to optimize and achieve low-latency, stable automatic switching, solidifying the aesthetics and rhythm of human directors into transferable process capabilities, ultimately reducing the human burden, improving picture quality, and achieving consistent output across scenarios;

[0065] Furthermore, this invention uses a large-scale interpretable director intent model to encode the "actions-expressions-interactions" of multiple people into temporal semantics and map them into a set of interpretable indicators (excitement, focus, composition coverage, occlusion risk, rhythm and switching cost), forming a unified reward that supports auditing and style reuse.

[0066] Furthermore, the present invention also provides multi-subject temporal fusion for camera motion cancellation: by eliminating translation / zoom pseudo-motion through affine residuals, the real changes driven by the human body are preserved; and then the semantic vectors of each camera position are fused globally to significantly improve the robustness and stability under occlusion, light variation and complex backgrounds.

[0067] Furthermore, this invention also provides a latency-stable decision engine: it adopts lightweight policy optimization with intra-group normalization (no commentator / no reference policy), and combines minimum dwell time gating with a hybrid scoring of "policy distribution + instant reward" to achieve fast convergence and automatic switching with less jitter, meeting the latency and stability requirements of live streaming. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of a multi-camera automatic switching method based on interpretable director intent modeling provided in an embodiment of the present invention. Detailed Implementation

[0070] The present invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0071] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0072] Example 1

[0073] refer to Figure 1 This embodiment provides a multi-camera automatic switching method based on large-model interpretable director intent modeling, including the following steps:

[0074] S1. Convert the original video stream from each camera position k. Align to a common timeline to obtain a time-aligned frame sequence For each frame Multiple skeletons were extracted to obtain a set of key points for the skeletons of multiple people. Then, time-aligned frame sequences Set of key points for multiple skeletons After performing skeleton-based face embedding and expression classification, temporal correlation is performed to obtain the face embedding feature sequence. With emoji tag vector sequence And generate a set of main sequence at the machine station level. ;

[0075] S11, All raw video streams from all camera positions Based on the public time index using the time alignment operator Alignment is performed to obtain the time-aligned frame sequence of all camera positions. ;

[0076] To ensure a consistent timeline, the original video streams from each of the K camera positions are... Input a predefined time alignment operator Video frames are resampled and interpolated based on timestamps and uniformly mapped to a common time index. Obtain the time-aligned frame sequence for each path. As shown in the following formula:

[0077] ,

[0078] in, This represents the time alignment operator that resamples / interpolates to the common time axis based on timestamps; T is the total number of frames after alignment. Indicates the camera position At time step RGB frames; Each is a camera position The frame height and frame width; K is the total number of bits; K is a positive integer greater than or equal to 3, generally not exceeding 10, and can be set according to actual needs; Refers to the camera position The captured raw video stream (a sequence of frames with timestamps, the main bitstream is taken by default).

[0079] Time-aligned frame sequences for each path Place them in the same sequence to obtain a time-aligned frame sequence for all camera positions. ;

[0080] S12, Positioning the camera At time step RGB frames Estimation function for multiple skeletons Obtain the set of skeletal key points corresponding to this frame. ;

[0081] Each machine position Each time step RGB frames Input multi-person skeleton estimation function This yields the key skeleton matrix corresponding to the (17 homogeneous) skeletal keypoints of all detected human figures within the frame. This yields a set of skeletal key points. As shown in the following formula:

[0082] ,

[0083] This example treats each human body as a subject, where, For camera position In time The Individual key skeleton matrix (each column represents the homogeneous coordinates of a joint); J represents the number of subjects detected in this frame; J represents the number of skeletal keypoints, J=17; providing spatial location information for subsequent face embedding and temporal correlation;

[0084] The multi-person skeleton estimation function A multi-person pose estimation pipeline (human detection → keypoint regression) can be adopted, which can be implemented by OpenPose / HRNet / ViTPose, etc. This embodiment does not limit the specific model;

[0085] S13. Align the frame sequence according to the time of all camera positions. and skeletal key point set Through face embedding and expression classification functions Obtain face embedding feature sequence With emoji tag vector sequence ;

[0086] According to the position At time step RGB frames And the key skeletal matrix of each individual Input is obtained through face embedding and expression classification functions. We obtain the face embedding features and expression label vectors that correspond one-to-one with the human body:

[0087]

[0088] in, Embedded features for human faces; This is an emoji tag vector, which is a one-hot encoded vector; The dimension of facial embedding features; Number of emoji categories;

[0089] The face embedding and expression classification function Determine the face ROI using skeletal head / shoulder keypoints, and then frame-by-frame. The corresponding face region is cropped out from the image, and face detection and alignment are performed on the face region and skeletal information; then, a face representation network is used to obtain the face semantic embedding features. (e.g., 128 / 512 dimensions), then use an expression classifier to obtain the one-hot encoded vector of the expression. ;

[0090] Finally, a set of skeletal key points is obtained. One-to-one face embedding feature sequence With emoji tag vector sequence ; ,

[0091] That is, the facial features and expression tags of each camera position, each time step, and each subject provide semantic and appearance constraints for the association of time trajectories.

[0092] S14. Within each camera position k, through time correlation and trajectory generation functions Set up the key points of the skeleton Face embedding feature sequence and emoji tag vector sequence exist Top-aligned to a fixed trajectory index And obtain the main sequence set at the machine station level. ;

[0093] To obtain stable time series of the same subject, a time correlation and trajectory generation function that maintains identity consistency is used. Will Frame by frame Top-aligned to a fixed trajectory index ( (For the number of trajectories), we obtain the main sequence set at the machine station level. :

[0094] ,

[0095] in For camera position The total number of main trajectories;

[0096] The time correlation and trajectory generation function Using skeletal geometry and face embedding similarity as cost metrics, minimum cost matching is performed between adjacent time steps to connect the detection results of the same real subject at different time steps into a trajectory. Internally, short-term occlusion or missed detection is interpolated or the trajectory is kept in place to ensure the temporal continuity and identity consistency of the trajectory. This also makes it easier to extract different sequences or sets of data from the same subject m at different camera positions k at the same time step t from the same set.

[0097] S2, Set of skeletal key points After performing affine motion estimation and skeletal residual decomposition, the main skeleton residual is obtained. The main skeleton remains With facial embedding features and emoji tag vectors Concatenate into a subject-level semantic vector Then, HAIC-style packaging is performed to obtain a machine-level semantic vector sequence. Then, the machine-level semantic vector sequence After performing cross-camera semantic aggregation, the cross-camera global state is obtained through mapping. ;

[0098] S21. Affine Motion Estimation and Residual Skeleton Decomposition: Fitting the skeleton points of all subjects in adjacent frames of the same camera position to obtain the affine motion matrix. Then, based on the affine motion matrix of the camera position The corrected skeletal residuals of each subject are calculated and motion suppression is performed to obtain the subject's skeletal residuals. ;

[0099] The affine motion matrix of the machine position As shown below:

[0100]

[0101] Among them, AffineFit() is an affine fitting function used to perform least squares fitting of the two-dimensional affine transformation on the "paired key point set" of the same camera position in adjacent frames. Indicates the camera position All human skeletal points at time t; Indicates the camera position All human skeletal points at time t+1; To fit the linear part; To fit the translation vector;

[0102] Based on the affine motion matrix of the machine position The corrected skeletal residuals for each subject are calculated as follows:

[0103] ,

[0104] in, To correct for skeletal defects;

[0105] Calculate corrected skeletal residuals After removing the parts that cannot be explained after the overall camera movement, it can be regarded as the real motion signal driven by the human body;

[0106] Subsequently, to suppress pseudo-small residuals introduced by slight camera jitter or minor registration errors, motion suppression / clipping is performed on the residuals using a motion clipping function, while retaining the main skeleton residuals obtained from the same-dimensional representation. :

[0107] ,

[0108] ClipMotion is a motion clipping function used to suppress / clip residuals while maintaining the same dimension. It is a conventional and robust preprocessing method, such as element-wise soft thresholding, Huber clipping, and quantile clipping. Therefore, it can suppress micro-jitter and registration errors without changing the tensor shape, and it is a mature existing technology.

[0109] S22, Regarding the residual of the main skeleton After dimensionality compression and normalization, it is then embedded with face features. and emoji tag vectors Concatenate into a subject-level semantic vector Then, all subject-level semantic vectors from the same camera position at the same time are packaged in a HAIC style to obtain a camera-level semantic vector sequence. ;

[0110] For each machine position Each time step Each subject First, through the aggregation operator Residual of the main skeleton Dimensional compression and normalization are performed to obtain a length of The motion description vector;

[0111] It is a fixed-dimensional convergence operator, which can be implemented through "vectorization + normalization (such as LayerNorm) + linear projection / attention convergence", with the goal of skeletal residuals. Stable mapping to fixed-dimensional features facilitates subsequent concatenation and learning;

[0112] The vector was then embedded with the face. Emoji Tags Concatenate the data to construct a subject-level semantic vector. ,in ; The dimension of the subject-level semantic vector; Indicates the output dimension of Pool; The dimension of facial embedding features; Number of expression categories; then, use the same camera position. At the same time step All main semantic vectors The input HAIC-style packing operator is used to pack the data to obtain the machine-level semantic vector. As shown in the following formula:

[0113] ,

[0114] HAICPack is the HAIC-style packing operator; This represents the HAIC representation of position k at time step t; The dimension representing the machine-level semantic vector;

[0115] HAIC-style packing is a general multi-subject temporal packing method. During the packing process, it maintains consistency in the order of "person-by-person – by time – including attributes / interactions" at the same camera position and time. Internally, it aggregates key interaction relationships to obtain camera-level vectors. ; Compact coding of machine positions Multi-agent action and emotion semantics at time step t are used for subsequent global state modeling and reward design.

[0116] Furthermore, from the machine-level semantic vector Obtain the machine-level semantic vector sequence .

[0117] S23. At each time step, the seat-level semantic vector sequence is processed in a fixed order according to the seat index. Machine-level semantic vectors of each camera position The joint vector at the machine position level is obtained by concatenation. And input it into the mapping function to obtain the global state across machine positions. ;

[0118] At each time step According to the seat index By concatenating the semantic tokens of each camera position in a fixed order, a camera position-level joint vector is obtained. And input the mapping function to obtain the global state across machine positions:

[0119] ,

[0120] in, For mapping functions, there are state projection functions that can contain linear / nonlinear mappings, used to compress multi-camera local semantics into a unified global state space.

[0121] Preferably, the mapping function in this embodiment is a state projection function of a linear mapping. It can be achieved through linear projection (FC) + normalization or shallow MLP, which can meet the requirements of low latency and ensure shape consistency.

[0122] To avoid interference from pseudo-motions such as camera panning / shaking / zooming, and to focus on the real actions and emotions of the characters, this step first estimates the affine camera motion at adjacent time steps within the camera position, and decomposes the skeletal trajectory into "overall camera motion + residual subject motion". Camera shake is suppressed using residual operators and amplitude clipping functions, retaining only the relative motion of the subject. Then, the subject residual skeleton, face embedding, and expression tags are concatenated at each time step into a HAIC-style multi-subject semantic vector. The HAICPack operator is then used to package the vectors within the camera position according to a consistent order of "person-by-person – time-including attributes / interactions" to obtain camera-level semantic tokens. Finally, the semantic tokens from each camera position are concatenated in a fixed order and mapped using a mapping function. The projection is a global state vector, providing a unified semantic representation for subsequent reward modeling and policy output.

[0123] S3. Based on the camera position-level semantic vector corresponding to camera position action a Global status across machine positions Construct the action-related feature vector of camera position action a and their corresponding instant rewards And based on the global state across machine positions Generate a set of trajectory reward samples Finally, based on the global status across camera positions With trajectory reward sample set Construct a training sample set ;

[0124] S31, Trajectory Sampling: The action from the previous time step... Encoded as a one-hot encoded vector This indicates the "historical dependency" of the current time step t relative to the previous camera position selection; then, actions are taken for any candidate camera position. Corresponding machine-level semantic vector and cross-machine global state and one-hot encoded vector Input the feature extraction function together Obtain action-related feature vectors :

[0125] ,

[0126] Among them, the first layer mapping matrix Second layer mapping matrix It is a trainable linear mapping matrix (i.e., the weights of the two-layer MLP). , , Common activation functions (ReLU / GeLU); feature extraction functions Existing feature extraction / fusion modules can be used to achieve a standard paradigm of "splicing → linear / MLP → nonlinear", but this invention is not limited to any specific one. The dimension of the machine-level semantic vector; The dimension representing the global state across machine positions; The dimension of the action-related feature vector;

[0127] Action-related feature vectors A high-dimensional feature composed of elements such as "sharpness / visibility / expression focus / composition coverage" (some statistics are derived from...). (Decoded)

[0128] S32. Immediate Reward Construction: Introducing a Linear Weight Vector With switching penalty coefficient The station actions are obtained through linear summarization. The instantaneous reward function at time step t :

[0129]

[0130] in This is an indicator function (taking 1 if not equal to), used to compare with the currently executed action. Apply a switching penalty when there is inconsistency.

[0131] Action of any candidate camera position Action-related feature vectors And the corresponding interpretable instant reward function for face-to-face broadcasting intent modeling This provides a foundation for subsequent trajectory reward calculation and GPG training sample construction.

[0132] S33. Trajectory Sampling and Discounted Return Calculation: Based on Cross-Aircraft Station Global State Sequence Using a temperature sampling strategy At each time step t, an action is sampled to generate a set of trajectory reward samples with discounted rewards. ;

[0133] Specifically, given a global state sequence across machine positions Under the premise of temperature sampling strategy Group sampling is performed at each time step t to obtain the group sampling results, which serve as the trajectory report sample set. , i.e., candidate broadcast trajectory;

[0134]

[0135] Where GroupSample is the group sampling function; This is a temperature-inclusive sampling strategy; G represents the number of samples. For the action sequence of the i-th sample, The corresponding discount will be given in return;

[0136] Each of them For a complete sequence of director actions, The corresponding discount reward provides a monitoring signal for subsequent GPG-style strategy optimization.

[0137] This embodiment provides a specific implementation detail of GroupSample, which adopts a conventional strategy gradient / sequence sampling paradigm, equivalent to performing sampling on the same state sequence. Secondary independent rollout: Set temperature With length ;right :according to Stepwise sampling, based on the immediate reward function of step S32 Calculate instant rewards And accumulate discount rewards ;return ,in This set is then used as a baseline for advantage estimation and policy updates based on the within-group mean; This is a discount factor used to balance the contribution of rewards at the current time step and those at future time steps;

[0138] Temperature sampling strategy To perform temperature conditioning on the policy distribution during sampling:

[0139] or ,

[0140] The original strategy; Enhanced exploration (more random); It's more greedy. This allows for sampling diverse action sequences from the same state sequence.

[0141] S34 is the set of trajectory return samples. Add a strategy log loss term Combined with cross-camera global status Construct a training sample set ;

[0142] For trajectory reward sample set Add a strategy log loss term Obtain the probability trajectory reward sample set Then, based on the global state across camera positions With probability trajectory reward sample set Construct a training sample set ;in Here are the parameters for the broadcast control strategy; where B is the total number of trajectories in the batch; and b is the index of the total number of trajectories in the batch. The global state across machine positions at time step t; For the action sequence of the i-th sample within the b-th batch trajectory; for Corresponding discount reward.

[0143] Interpretable Instantaneous Reward Function for Modeling Broadcast Intent and its discount returns This will serve as an evaluation benchmark in the online decision-making process of step S5;

[0144] training sample set Training samples used for Group Policy Gradient Optimization (GPG).

[0145] S4. GPG optimization of training sample set based on group policy gradient. Gradient optimization is performed to obtain the optimal broadcasting strategy parameters. and its corresponding optimal strategy ;in, Indicates camera position movement; Indicates the global status across machine positions;

[0146] Group Policy Gradient Optimization (GPG) is a novel reinforcement learning training method proposed by the team from Alibaba-Gaode Maps. It is designed to reduce the variance of policy gradient estimation through a group reward mechanism, thereby improving training stability and efficiency. It is particularly suitable for optimizing the inference capabilities of large language models (LLMs).

[0147] Training sample set taken from S3 , of which The probability trajectory reward sample set of group sampling of a trajectory sequence is: . The global state across machine positions at time step t; For the first A sequence of actions; ; For corresponding discount rewards; The strategy's logarithmic loss term; director's strategy parameters. Needs optimization.

[0148] Let represent the policy log loss term for the i-th group trajectory of sample sequence b at time step t. Used to characterize the log loss of this action under the current strategy, and within-group advantage. The multiplication results in a policy gradient optimization objective, which in turn optimizes the policy gradient. Perform a minimal update.

[0149] By replacing the value function commentator and reference policy with the within-group normalization advantage of group policy gradient optimization, the optimal policy can be obtained directly under minimum components. ;

[0150] Specifically, this includes: first calculating the mean return within each group of sample sequences. And define the group's strengths accordingly. (For the b-th batch trajectory, the i-th sample, and time step t, the same sequence shares the same...) Subsequently, the gradient objective of the original policy (without commentators / reference policies / KL constraints) is:

[0151]

[0152] The unbiased gradient estimate is obtained as follows:

[0153] ,

[0154] in, For broadcast strategy parameters; when there are in the batch "Invalid samples" (e.g., those with the same score within the same group) When the gradient magnitude is systematically underestimated due to approximation to zero or numerical degradation, a linear correction using AGE (Accurate Gradient Estimation) is introduced to adjust the correction amount. Finally, using the learning rate For broadcast strategy parameters Perform a minimal update:

[0155] ,

[0156] Thus, without introducing a value function, reference strategy, or additional regularization term, the strategy's preference for "high-return switching sequences" is steadily enhanced solely by the intra-group relative return signal.

[0157] Where B is the total number of trajectories in the batch, G is the number of samples, which is also the group size; T is the duration and the reward defined in S3. and returns Maintain consistency; j is the group index;

[0158] After several rounds of iteration, the optimal broadcast control strategy parameters were obtained. This represents the optimal parameters after convergence; then, based on the optimal directing strategy parameters... Obtain its corresponding optimal strategy The optimal strategy It will be directly called by S5 to be in real-time state Choose the action below and combine it with the instant rewards of S3. This forms an online decision-making criterion with the switching penalty; simultaneously, the calculations made during training... and It can also be used for efficient incremental optimization in subsequent rounds (maintaining symbol consistency and reusability).

[0159] S5. Calculate the action of any candidate camera position in the current state. Instant rewards After filtering out candidate actions that do not meet the residency constraints, proceed according to the optimal strategy. Select the directing action at the current time step t. Then, the directing action at the current time step t. Control vector mapped to cross-point trigger command And according to the control vector Select the corresponding camera position to obtain the director's output stream. ;

[0160] S51. Calculate the action of any candidate camera position in the current state. Action-related feature vectors and their corresponding instant rewards :

[0161]

[0162]

[0163] in For feature extraction function; The feature weight vector; To switch the penalty coefficient; As an indicator function (not equal to 1), it quantifies the director's intentions such as "excitement level / focus level / coverage / switching cost".

[0164] S52. Based on the threshold mask, candidate actions that do not satisfy the dwell constraints are masked to obtain a set of actionable actions. Then, based on the set of actionable actions, the optimal strategy is applied. Make a decision to obtain the director's action at the current time step. ;

[0165] To avoid "jittering" switching, a minimum dwell time gating is first introduced: the dwell time count is set to... Let be the cumulative number of frames since the last switch (in frames). For any candidate action, the minimum dwell threshold (in the same unit) is... Define gate mask This allows us to filter out candidate actions that do not meet the residency constraint, thus obtaining a set of actionable actions. As shown below;

[0166] ,

[0167] Subsequently, in order to combine the strategy distribution to discriminate and weight the uncertain scenarios for online scoring:

[0168] ,

[0169] in The policy confidence weighting coefficients (larger values ​​indicate greater dependence on the policy distribution, while smaller values ​​indicate a bias towards immediately interpretable rewards) are used. The policy in the online scoring is the optimal policy in step S4. ;

[0170] Accordingly, the director's action for the current time step is determined by making a decision based on online scoring from the set of possible actions after gating. ;

[0171] ,

[0172] Director's actions at the current time step To satisfy the minimum dwell constraint set of possible actions, the corresponding online scoring, and the selected director actions after fusion judgment, decision results are provided for the coordinated output of the director's station.

[0173] S53, Set the directing action for the current time step. Control vector mapped to cross-point trigger command And according to the control vector Select the corresponding camera position to obtain the director's output stream. ;

[0174] Step S52 Selected directing action Alignment frames corresponding to each camera position and the current global state Previous time step action Resident count .

[0175] To achieve synchronization with the control room, the selected action, i.e., the control action at the current time step, will be selected. Mapped to a one-hot encoded control vector for cross-point triggering instructions. :

[0176] ,

[0177] in, For one-hot encoding mapping functions, the decision result is... Mapped to control vectors;

[0178] Control Vector This refers to the "one-hot" representation of the broadcast control commands: Indicates the first strobe Only one path is input; the rest are 0.

[0179] This allows for the selection of the corresponding camera position on a physical or virtual switcher, thereby obtaining the director's output stream:

[0180] ,

[0181] Finally, to support continuous learning and statistical monitoring, Record it in the online cache and update the dwell count based on the action results. (Keep or Reset):

[0182] .

[0183] This embodiment centers on "a large model that can explain the director's intent," implementing a closed loop of "perception—intent—decision—control." First, it performs unified timeline alignment on 3–5 camera feeds, and on each feed, it detects multiple subjects, identifies key points and facial expressions, and correlates trajectories, encoding multi-source information ("who is where, what did they do") into stable temporal semantic vectors. Then, through camera motion estimation and cancellation, it eliminates pseudo-motion introduced by translation / zoom, retaining only genuine human-driven action signals, and aggregating across camera positions to obtain the global state. Based on this state, it constructs an interpretable director's intent model, quantifying "excitement level, focus, composition coverage, occlusion risk, and rhythmic coherence" into unified reward signals. It also introduces constraints such as switching costs and minimum dwell time to clarify "when and why to switch." In the policy learning phase, it employs lightweight policy optimization with intra-group normalization (not relying on a value network or reference policy), directly targeting reward improvement for offline / quasi-online training, balancing convergence speed and low latency requirements. During online inference, the strategy distribution and instant rewards are integrated for scoring. Combined with minimum dwell time gating and business rules, the optimal camera position is selected in real time, and the control command from the broadcast control station is issued to complete the switching. At the same time, the "state-action-reward" feedback is used for continuous fine-tuning and effect tracking, and interpretable logs are output to support debriefing and style reuse. The overall solution uses the semantic understanding of a large model to carry the interpretable expression of "director's intent", and uses lightweight and efficient strategy optimization to achieve stability, low latency, and consistent stylistic output for automatic switching of multiple cameras.

[0184] This embodiment provides a multi-camera automatic switching method based on interpretable director intent modeling using a large model. First, through multi-subject temporal semantic encoding, the dancers' body movements, facial expressions, and interactions are transformed into structured descriptions. Then, using an interpretable director intent model, indicators such as "excitement level, focus, composition coverage, occlusion, and rhythm" are quantified into reward signals, clarifying the switching basis and interpretability. Next, lightweight and efficient decision learning methods, such as in-group normalization and commentator-free strategy optimization, are employed to generate stable camera selections under low latency and low jitter constraints. Finally, the optimal action is mapped to control commands from the control console, outputting a continuous and visually appealing main video stream in real time. In summary, this invention replicates the aesthetics and rhythm of human directors through "interpretable intent modeling using a large model + efficient strategy optimization," achieving intelligent real-time switching of multiple cameras.

[0185] Furthermore, this embodiment abstracts the actions, expressions, and interactions of multiple people into measurable and reusable "director intent" indicators (such as brilliance, focus, coverage, and switching cost), using interpretable intent modeling to connect perception and decision-making; and without increasing the heavy computational burden, it adopts a lightweight and efficient strategy to optimize and achieve low-latency, stable automatic switching, solidifying the aesthetics and rhythm of human directors into transferable process capabilities, ultimately reducing the human burden, improving picture quality, and achieving consistent output across scenarios;

[0186] Furthermore, this embodiment uses a large-scale interpretable director intent model to encode the "actions-expressions-interactions" of multiple people into temporal semantics and map them into a set of interpretable indicators (excitement, focus, composition coverage, occlusion risk, rhythm and switching cost), forming a unified reward that supports auditing and style reuse.

[0187] Furthermore, this embodiment also provides multi-subject temporal fusion for camera motion cancellation: by eliminating translation / zoom pseudo-motion through affine residuals, the real changes driven by the human body are preserved; and then the semantic vectors of each camera position are fused globally to significantly improve the robustness and stability under occlusion, light variation and complex backgrounds.

[0188] Furthermore, this embodiment also provides a latency-stabilized decision engine: it adopts lightweight policy optimization with intra-group normalization (no commentator / no reference policy), and combines minimum dwell time gating with a hybrid scoring of "policy distribution + instant reward" to achieve fast convergence and automatic switching with less jitter, meeting the latency and stability requirements of live streaming.

[0189] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 A process, multiple processes, and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0192] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A multi-camera automatic switching method based on large-scale interpretable director intent modeling, characterized in that, Includes the following steps: S1. Align the original video stream of each camera position k to the common timeline to obtain a time-aligned frame sequence. For each frame Multiple skeletons were extracted to obtain a set of key points for the skeletons of multiple people. Then, time-aligned frame sequences Set of key points for multiple skeletons After performing skeleton-based face embedding and expression classification, temporal correlation is performed to obtain the face embedding feature sequence. With emoji tag vector sequence And align; ; t is the time step; m is the trajectory index; K is the total number of bits; T is the total number of frames after alignment; The number of subjects detected in the frame; S2, Set of skeletal key points After performing affine motion estimation and skeletal residual decomposition, the main skeleton residual is obtained. The main skeleton remains With facial embedding features and emoji tag vectors Concatenate into a subject-level semantic vector Then, HAIC-style packaging is performed to obtain a machine-level semantic vector sequence. Then, the machine-level semantic vector sequence After performing cross-camera semantic aggregation, the cross-camera global state is obtained through mapping. ; The HAIC style packaging is a multi-subject time-series packaging. At the same camera position and at the same time, it maintains the consistency of the order of person-by-person, time-by-attribute or interaction during the packaging process, and aggregates key interaction relationships internally. S3. Based on the camera position-level semantic vector corresponding to camera position action a Global status across machine positions Construct the action-related feature vector of camera position action a and their corresponding instant rewards And based on the global state across machine positions Generate a set of trajectory reward samples Finally, based on the global status across camera positions With trajectory reward sample set Construct a training sample set ; S4. GPG optimization of training sample set based on group policy gradient. Gradient optimization is used to obtain the optimal strategy. ;in, Indicates camera position movement; Indicates the global status across machine positions; S5. Calculate the action of any candidate camera position in the current state. Instant rewards After filtering out candidate actions that do not meet the residency constraints, proceed according to the optimal strategy. Select the directing action to get the current time step. Then, the directing action at the current time step. Control vector mapped to cross-point trigger command And according to the control vector Select the corresponding camera position to obtain the director's output stream. .

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11, All raw video streams from all camera positions Based on the public time index using the time alignment operator Alignment is performed to obtain the time-aligned frame sequence of all camera positions. ; S12, RGB frames of camera position k at time step t Estimation function based on multiple skeletons Obtain the set of skeletal key points corresponding to this frame. ; S13. Align the frame sequence according to the time of all camera positions. and skeletal key point set Through face embedding and expression classification functions Obtain face embedding feature sequence With emoji tag vector sequence ; S14. Within each camera position k, through time correlation and trajectory generation functions Set of skeletal key points Face embedding feature sequence and emoji tag vector sequence exist Top-aligned to a fixed trajectory index .

3. The method according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Fit the skeletal points of all subjects in adjacent frames from the same camera position to obtain the affine motion matrix of the camera position. Then, based on the affine motion matrix of the camera position The corrected skeletal residuals of each subject are calculated and motion suppression is performed to obtain the subject's skeletal residuals. ; S22, Regarding the residual of the main skeleton After dimensionality compression and normalization, it is then embedded with face features. and emoji tag vectors Concatenate into a subject-level semantic vector Then, all subject-level semantic vectors from the same camera position at the same time are packaged in a HAIC style to obtain a camera-level semantic vector sequence. ; S23. At each time step, the seat-level semantic vector sequence is processed in a fixed order according to the seat index. Machine-level semantic vectors of each camera position The joint vector at the machine position level is obtained by concatenation. And input it into the mapping function to obtain the global state across machine positions. .

4. The method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, Redirect the action of the previous time step Encoded as a one-hot encoded vector Then take action on any candidate camera position. Corresponding machine-level semantic vector and cross-machine global state and one-hot encoded vector Input the feature extraction function together Obtain action-related feature vectors ; S32. Introducing a linear weight vector With switching penalty coefficient The station actions are obtained through linear summarization. The instantaneous reward function at time step t : ， in For indicator functions; Currently executed actions; top right corner Indicates transpose; S33, Based on cross-station global state sequence Using temperature sampling strategy At each time step t, an action is sampled to generate a set of trajectory reward samples with discounted rewards. ; ， Where GroupSample is the group sampling function; This is a temperature-inclusive sampling strategy; G represents the number of samples. For the action sequence of the i-th sample, The corresponding discount will be given in return; S34 is the set of trajectory return samples. Add a strategy log loss term Combined with cross-camera global status Construct a training sample set Where B represents the total number of trajectories in the batch; b represents the index of the total number of trajectories in the batch. For the first A set of probabilistic trajectory return samples for group sampling of a trajectory sequence; The global state across machine positions at time step t; For the action sequence of the i-th sample within the b-th batch trajectory; for Corresponding discount reward.

5. The method according to claim 4, characterized in that, Step S4 specifically includes the following steps: First, calculate the mean return within each group of the sample sequence. And define the group's strengths accordingly. Subsequently, the gradient objective of the original policy is... Obtain its unbiased gradient estimate When invalid samples exist in the batch, causing a systematic underestimation of the gradient magnitude, a linear correction for AGE is introduced, and the correction amount is set to... Finally, using the learning rate For broadcast strategy parameters Perform a minimal update: ， After several rounds of iteration, the optimal broadcasting strategy parameters were obtained. Then, based on the optimal broadcast strategy parameters Obtain its corresponding optimal strategy Where B is the total number of trajectories in the batch; G is the group size; T is the duration; and M is the number of invalid samples. This is the policy logarithmic loss term.

6. The method according to claim 1, characterized in that, Step S5 specifically includes the following steps: S51. Calculate the action of any candidate camera position in the current state. Action-related feature vectors and their corresponding instant rewards ; S52. Based on the threshold mask, candidate actions that do not satisfy the dwell constraints are masked to obtain a set of actionable actions. Then, based on the set of actionable actions, the optimal strategy is applied. Make a decision to obtain the director's action at the current time step. ; S53, Set the directing action for the current time step. Control vector mapped to cross-point trigger command And according to the control vector Select the corresponding camera position to obtain the director's output stream. .

7. The method according to claim 3, characterized in that, Step S21 specifically includes: The affine motion matrix of the machine position As shown below: ， Among them, AffineFit() is an affine fitting function, which is used to perform least squares fitting of a two-dimensional affine transformation on a set of paired key points of the same camera position in adjacent frames. Indicates the camera position All human skeletal points at time t; Indicates the camera position All human skeletal points at time t+1; To fit the linear part; To fit the translation vector; Based on the affine motion matrix of the machine position The corrected skeletal residuals for each subject are calculated as follows: ， in, To correct for skeletal defects; For camera position At time t The individual's key skeletal matrix; For camera position At time t+1, the The individual's key skeletal matrix; Motion suppression is applied to the residuals using a motion clipping function, while preserving the main skeleton residuals obtained from the same-dimensional representation. : ， Where ClipMotion is the motion shearing function; J is the number of skeletal keypoints.

8. The method according to claim 7, characterized in that, Step S22 specifically includes: For each machine position Each time step Each subject First, through the aggregation operator Residual of the main skeleton Dimensional compression and normalization are performed to obtain a length of The motion description vector is then used in conjunction with the face embedding. Emoji Tags Concatenate the data to construct a subject-level semantic vector. ,in ; The dimension of the subject-level semantic vector; Indicates the output dimension of Pool; The dimension of facial embedding features; Number of expression categories; then, use the same camera position. At the same time step All main semantic vectors The input HAIC-style packing operator is used to pack the data to obtain the machine-level semantic vector. As shown in the following formula: ， HAICPack is the HAIC-style packing operator; This represents the HAIC-style representation of camera position k at time step t; further, it is derived from the camera position-level semantic vector. Obtain the machine-level semantic vector sequence .

9. The method according to claim 4, characterized in that, The feature extraction function described in step S31 As shown below: ， Among them, the first layer mapping matrix Second layer mapping matrix is a trainable linear mapping matrix.

10. The method according to claim 4, characterized in that, Step S33 specifically includes: Given a global state sequence across machine positions Under the premise of temperature sampling strategy Group sampling is performed at each time step t to obtain the group sampling results, which serve as the trajectory report sample set. , i.e., candidate broadcast trajectory; ， Where GroupSample is the group sampling function; This is a temperature-inclusive sampling strategy; G represents the number of samples. For the action sequence of the i-th sample, The corresponding discount will be given as a reward.

Images