A multi-position video shooting real-time collaborative scheduling and dynamic composition optimization method

Abstract

The application discloses a multi-position video shooting real-time cooperative scheduling and dynamic composition optimization method, relates to the multi-position shooting technical field, and is applied to a multi-position video shooting system comprising a plurality of camera equipment, an edge computing unit, a cooperative scheduling hub and a real-time picture processing unit. The edge computing unit is used for jointly calibrating the time and space reference of the camera equipment cluster, constructing a global unified time and space coordinate system, and synchronously acquiring the holder, lens and space pose parameters of each camera equipment. Composition score, picture stability and blind area coverage data can be collected in real time, the position scheduling priority and the holder slow motion coefficient can be dynamically adjusted, and unified composition optimization can be autonomously completed. When a director manually adjusts the position, the equipment automatic permission is automatically locked, command conflicts are avoided, scene and artificial adjustment samples are reserved, and after the sample accumulation reaches the standard, the composition evaluation weight and the scheduling strategy are automatically iterated.

Description

Technical Field

[0001] This invention relates to the field of multi-camera shooting technology, and in particular to a method for real-time collaborative scheduling and dynamic composition optimization in multi-camera film and television shooting. Background Technology

[0002] Nowadays, various outdoor variety show recordings and large-scale offline event live broadcasts generally use multiple camera devices to collaboratively complete the shooting of materials. The industry has equipped dedicated director control consoles to realize camera position management. Existing multi-camera intelligent scheduling devices on the market are preset with fixed operating parameters at the factory, and are equipped with independent gimbal control modules and image output modules. Some devices integrate basic image correction and simple camera position switching functions. On the shooting site, there are dynamic environments such as people moving continuously and objects temporarily obscuring the scene. The director will manually adjust the gimbal angle and image cropping boundaries in real time according to his own shooting aesthetics. There is currently no dedicated storage unit to retain this kind of manual operation data.

[0003] The camera position scheduling and composition evaluation parameters of existing shooting equipment are only fixed values ​​at the factory. When faced with dynamic shooting scenarios such as guests moving and obstructing the view, the automatically output image composition and timing of the replacement often do not meet the director's aesthetics. The director needs to continuously manually fine-tune the gimbal and crop the image. Manual adjustment commands are also prone to conflict with the automatic scheduling program, resulting in problems such as image shaking, camera position switching gaps, and cluttered visuals of multi-camera compositions. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies, such as the fact that the camera positioning and composition evaluation parameters of existing shooting equipment are only fixed values ​​at the factory. When facing dynamic shooting scenarios such as guests moving and obstructing view, the automatically output image composition and timing of the replacement often do not meet the director's aesthetic, requiring the director to continuously manually fine-tune the gimbal and crop the image. Furthermore, manual adjustment commands are prone to conflict with the automatic scheduling program, resulting in problems such as image shaking, discontinuous camera switching, and cluttered visuals of multi-camera compositions. Therefore, this invention proposes a method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for real-time collaborative scheduling and dynamic composition optimization in multi-camera film and television shooting, applied to a multi-camera film and television shooting system including multiple camera devices, an edge computing unit, a collaborative scheduling center, and a real-time image processing unit, includes the following steps: S1. The edge computing unit performs joint calibration of the spatiotemporal reference for the camera equipment cluster, constructs a globally unified spatiotemporal coordinate system, and synchronously acquires the gimbal, lens, and spatial pose parameters of each camera equipment. S2. Based on the global unified spatiotemporal coordinate system, the edge computing unit tracks the dynamic shooting object and predicts the motion trend to generate a dynamic shooting blind spot distribution. S3. The collaborative scheduling center performs camera position replacement scheduling based on the dynamic shooting blind spot; The real-time image processing unit uses dense optical flow to compensate for image shift defects caused by camera position and target movement, and smoothly completes the handover of multi-camera perspectives. S4. The real-time image processing unit quantifies and calculates the comprehensive composition score of each camera position's image, optimizes the multi-camera composition based on the score, and sends the composition and image stability data back to the collaborative scheduling center to iteratively update the scheduling strategy.

[0006] The above technical solution further includes: Specifically, in step S1, the edge computing unit integrates a PTP synchronization module and a SLAM calibration unit, and the spatiotemporal calibration process is as follows: The PTP synchronization module is used to correct the timestamps of each camera device by combining static feature points of the scene; frame-level synchronization is achieved. By matching multi-camera features through the SLAM calibration unit, the poses of all camera devices are uniformly mapped to the global unified spatiotemporal coordinate system.

[0007] Specifically, in step S2, the edge computing unit integrates a visual tracking module and a Kalman filter module, and the target tracking prediction process is as follows: The visual tracking module identifies and tracks dynamically captured objects across camera positions, and calculates the target's three-dimensional trajectory using the global unified spatiotemporal coordinate system. The Kalman filter module predicts the short-term movement position of the target and updates the dynamic shooting blind zone distribution simultaneously.

[0008] Specifically, the machine position replacement scheduling in step S3 is as follows: The collaborative scheduling center assigns main shooting and backup roles to each camera device according to the dynamic shooting blind spot, and issues scheduling instructions; The camera equipment's gimbal control unit uses an S-shaped gradual curve to adjust the camera's position and attitude, filling in blind spots during shooting.

[0009] Specifically, the image offset compensation in step S3 is as follows: The real-time image processing unit has a built-in dense optical flow calculation module that calculates the motion vector field of the image frame by frame and performs sub-pixel correction on the image based on the vector field to eliminate jitter and image distortion caused by camera rotation and target movement.

[0010] Specifically, the handover of camera positions in step S3 is as follows: When switching camera positions, the target positions and motion trends in the two images are first aligned by dense optical flow. Then, the image weight fusion unit uses a cosine curve to smoothly transition the output ratio of the two images, eliminating switching gaps and jumps.

[0011] Specifically, in step S4, the real-time image processing unit has a built-in composition evaluation calculation unit; A unified composition quantification dimension is set: subject proportion, golden ratio deviation, line-of-sight margin, and horizon levelness; the composition evaluation calculation unit weights the values ​​of each dimension and outputs the comprehensive composition score corresponding to each camera device.

[0012] Specifically, the composition optimization in step S4 is as follows: Based on the sub-item deviation values ​​corresponding to the comprehensive composition score, the camera angle is slightly adjusted by the gimbal fine-tuning unit, and the image is corrected by the lossless digital cropping module to unify the composition visual effect of all camera positions.

[0013] Specifically, the scheduling strategy iteration in step S4 is as follows: The real-time image processing unit transmits the comprehensive composition score, image stability, and blind spot coverage data to the scheduling parameter update unit of the collaborative scheduling center to dynamically adjust the camera position scheduling priority and the gimbal easing coefficient.

[0014] Specifically, the system is also equipped with a director's interactive control console and a sample storage unit; If the system detects manual adjustments to camera positions or composition parameters on the director's console, it will lock the automatic scheduling privileges for that camera position. Simultaneously, the current scene and manually adjusted parameters are stored in the sample storage unit, and the graph evaluation weights and scheduling strategies are iteratively updated.

[0015] The present invention has the following beneficial effects: In this invention, composition scoring, image stability, and blind spot coverage data can be collected in real time, and the camera position scheduling priority and PTZ easing coefficient can be dynamically adjusted to autonomously complete the unified optimization of composition. When the director manually adjusts the camera position, the device's automatic permissions will be automatically locked to avoid command conflicts. At the same time, scene and manual adjustment samples are retained. After the sample accumulation reaches the standard, the composition evaluation weight and scheduling strategy will be automatically iterated.

[0016] With spatiotemporal calibration, dense optical flow stabilization, and cosine smoothing image switching algorithms, the director no longer needs to manually correct camera positions and crop the footage repeatedly. This completely improves issues such as image shake, transition gaps, and cluttered multi-camera compositions, effectively reducing the intensity of manual operation and continuously outputting visually consistent and smooth shooting footage. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multi-camera film shooting real-time collaborative scheduling and dynamic composition optimization system proposed in this invention. Figure 2 This is the overall hardware interaction flowchart in this invention; Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0019] like Figures 1-2 As shown, the present invention proposes a real-time collaborative scheduling and dynamic composition optimization method for multi-camera film and television shooting, which is applied to a multi-camera film and television shooting system including multiple camera devices, an edge computing unit, a collaborative scheduling hub, and a real-time image processing unit, and includes the following steps: S1. The camera equipment cluster is jointly calibrated using the edge computing unit to construct a globally unified spatiotemporal coordinate system and simultaneously acquire the gimbal, lens and spatial pose parameters of each camera equipment. S2. Relying on a globally unified spatiotemporal coordinate system, the edge computing unit tracks the dynamic shooting object and predicts the motion trend to generate the dynamic shooting blind zone distribution. S3. The collaborative scheduling center performs camera position replacement scheduling based on the dynamic shooting blind spot; The real-time image processing unit uses dense optical flow to compensate for image shift defects caused by camera and target movement, and smoothly completes the handover of multi-camera perspectives. S4, the real-time image processing unit quantifies and calculates the comprehensive composition score of each camera position's image, optimizes the composition of multiple camera positions based on the score, and sends the composition and image stability data back to the collaborative scheduling center to iteratively update the scheduling strategy.

[0020] Furthermore, this four-step workflow operates sequentially and in tandem on a hardware platform consisting of multiple camera devices, edge computing units, collaborative scheduling hubs, and real-time image processing units. The specific operation mode of each step is clear and feasible, forming a data closed loop throughout the process: In the S1 spatiotemporal reference joint calibration stage, the edge computing unit takes the lead in execution. In terms of time dimension, a unified clock reference is first established for each camera device using the PTP precision time protocol. Then, static feature points such as fixed buildings and props common to all camera positions are extracted for cross-camera frame matching. The timestamp deviation of each device is corrected by the least squares method to achieve sub-frame level time synchronization. In the spatial dimension, the scene feature points of each camera position are extracted through a lightweight SLAM algorithm. The external parameter transformation matrix between camera positions is solved by multi-view feature matching. The gimbal rotation angle, lens focal length, and spatial position of each camera device are uniformly mapped to the global world coordinate system. Finally, the global unified spatiotemporal coordinate system and the real-time pose parameters of each camera position are output, providing a unified benchmark for the whole-link calculation. Taking four mobile follow-up cameras for outdoor variety shows as an example, this step does not require manual setting of calibration targets. It can complete the spatiotemporal alignment of all camera positions by relying solely on natural features such as the fixed stage and landscape trees on site, thus eliminating the problems of time difference and coordinate misalignment between camera positions from the root.

[0021] In the S2 target tracking and blind spot prediction stage, which is also executed by the edge computing unit, the real-time video streams of all camera devices are first integrated. The dynamic shooting object is identified through a lightweight target detection and tracking algorithm. Cross-view feature matching is used to achieve full-domain continuous tracking of the same target. Then, combined with a global unified spatiotemporal coordinate system, the three-dimensional spatial coordinates and continuous motion trajectory of the target are calculated through multi-view geometric operations. Subsequently, based on historical trajectory data, the Kalman filter algorithm is used to predict the spatial position and motion direction of the target in the next 1-2 seconds. At the same time, the location, range and occurrence time of the dynamic shooting blind spot are calculated and updated in real time by combining the distribution of scene occlusions and the field of view of each camera position. In response to variety show filming scenarios, the system can track walking guests in real time, predict when they will enter the blind spot behind the landscape pillars, and mark the gap information in advance for use in the scheduling process.

[0022] In the S3 collaborative scheduling and image compensation stage, the collaborative scheduling center and the real-time image processing unit execute in parallel. After receiving dynamic shooting blind spot data, the collaborative scheduling center dynamically assigns main shooting and supplementary shooting roles to each camera device through the task matching algorithm. It only issues high-rise shooting target instructions to the camera position to be adjusted. The specific gimbal rotation and lens zoom parameters are calculated autonomously by the local gimbal control unit of the camera position. The camera position attitude is smoothly adjusted according to the S-shaped easing curve to fill the shooting blind spot. The real-time image processing unit synchronously performs dense optical flow calculations on the moving camera image, calculates the complete motion vector field between the current image and the reference frame frame by frame, and performs sub-pixel-level offset correction on the image based on the vector field to offset the image jitter and local deformation caused by camera rotation and rapid target movement. When the main camera and the backup camera switch perspectives, the target position and overall movement trend of the two images are first aligned by dense optical flow, and then the output ratio of the two images is smoothly transitioned by cosine weight curve, so as to completely eliminate the image jump and splicing discontinuity at the moment of camera switching. In the context of variety show scenes, the replacement camera moves smoothly with no noticeable shaking, and there is no abrupt transition when it hands over to the main camera. It can capture the entire process of the guest being blocked by the pillar completely and smoothly.

[0023] In the S4 composition optimization and strategy iteration phase, the real-time image processing unit leads and coordinates with the central scheduling unit to complete the closed loop. The real-time image processing unit pre-sets the main subject's proportion of the image, the positional deviation between the main subject and the golden ratio point, the proportion of white space, and the horizontality of the horizon as four quantitative composition dimensions. By weighted summation, the comprehensive composition score and the deviation value of each dimension of the image are calculated. Then, based on the deviation data, the image is corrected by a combination of gimbal micro-angle adjustment and lossless digital cropping to ensure that the main subject position, framing size, and white space ratio of all camera positions are consistent, thus ensuring the visual style of the multi-camera images is unified. Meanwhile, the real-time image processing unit sends three types of data—comprehensive composition score, image stability, and blind spot coverage—back to the parameter update unit of the collaborative scheduling center. This dynamically adjusts the scheduling priority and gimbal motion easing coefficient of different camera positions, and iteratively optimizes the scheduling strategy for subsequent similar scenarios.

[0024] In step S1, the edge computing unit integrates the PTP synchronization module and the SLAM calibration unit. The spatiotemporal calibration process is as follows: Frame-level synchronization is achieved by using the PTP synchronization module in conjunction with scene static feature points to correct the timestamps of each camera device. By matching multi-camera features through SLAM calibration units, the poses of all camera devices are uniformly mapped to a global unified spatiotemporal coordinate system.

[0025] Furthermore, the spatiotemporal calibration process described in this section is completed collaboratively by the PTP synchronization module and the SLAM calibration unit built into the edge computing unit. It is the underlying benchmark construction link of the entire multi-camera shooting system. Time calibration and spatial calibration can be executed in parallel. The specific operation mode is as follows: In the time dimension, the PTP synchronization module first issues a unified clock reference to all camera devices through the local area network to complete the initial clock alignment at the hardware level. Then, it extracts the static feature points of the scene such as building outlines, fixed props, and site signs that are common in the images of each camera device. Through feature matching, it locates the corresponding frame of the same feature point captured by all cameras. Using the actual time of the feature point as a reference, the least squares method is used to correct the timestamp deviation of each camera device, refining the hardware-level clock synchronization into precise alignment at the frame level. In the spatial dimension, the SLAM calibration unit synchronously extracts static feature points from the images of each camera device, solves the extrinsic transformation matrix between each camera position through multi-view geometric matching, and then transforms and maps the gimbal rotation angle, lens focal length, and physical spatial pose of each camera device to the same set of globally unified spatiotemporal coordinate system, so that the pose data of all camera positions have a unified spatial reference standard.

[0026] Taking four mobile camera devices used in outdoor variety show shooting as an example, the PTP synchronization module first completes the basic clock alignment of each device through the local area network on site, and then extracts static feature points such as fixed stage pillars and ground marking lines on site, matches the corresponding frame correction timestamps of each camera position, and achieves microsecond-level frame synchronization. Meanwhile, the SLAM calibration unit extracts feature points such as stage edges, landscape trees, and fixed props from the images of each camera position. Through cross-camera matching, it calculates the relative positional relationship between the camera positions and unifies the gimbal angles, zoom parameters, and spatial coordinates of the four devices into a global coordinate system with the center of the stage as the origin. The entire process can be completed without the need for manual placement of special calibration targets, providing a unified and accurate spatiotemporal reference for subsequent target tracking and camera position scheduling.

[0027] In step S2, the edge computing unit integrates the visual tracking module and the Kalman filter module. The target tracking and prediction process is as follows: The visual tracking module identifies and tracks dynamic shooting objects across camera positions, and calculates the target's three-dimensional trajectory by combining a globally unified spatiotemporal coordinate system. The Kalman filter module predicts the short-term movement position of the target and updates the distribution of the dynamic shooting blind spot simultaneously.

[0028] Furthermore, the target tracking and prediction process in this section is carried out by the visual tracking module and Kalman filtering module built into the edge computing unit, which work together in a coordinated manner. The entire process relies on the globally unified spatiotemporal coordinate system constructed in step S1 for calculation. It is the core perception link connecting the underlying spatiotemporal reference calibration and the upper-level machine position scheduling decision. The visual tracking module receives real-time video streams from all camera devices in parallel. It identifies dynamic objects in the video using a lightweight target detection algorithm and then uses a cross-view feature matching algorithm to associate and continuously track the same target across different camera positions, avoiding target loss or identity confusion. At the same time, it combines the pose parameters of each camera position under a globally unified spatiotemporal coordinate system and calculates the real-time three-dimensional spatial coordinates of the target using the principle of multi-view triangulation, generating a continuous and smooth three-dimensional motion trajectory of the target. The Kalman filter module takes the target's 3D trajectory data from the historical period as input, combines the target's speed and directional inertia to predict the target's spatial position and movement trend within the next 1 to 2 seconds. At the same time, it matches the spatial position of fixed obstructions in the scene and the real-time field of view coverage of each camera device to calculate and update the position, range, and duration of the upcoming dynamic shooting blind spot, providing a predictive basis for subsequent camera position replacement scheduling.

[0029] Taking the tracking of walking guests by four mobile cameras in an outdoor variety show shooting scene as an example, the visual tracking module will simultaneously identify the main body of the guest from the four images, complete the target matching across cameras through clothing and body features, and calculate the real-time three-dimensional coordinates and walking trajectory of the guest in the venue by combining the calibrated global coordinate system. The Kalman filter module predicts that the guest will walk behind the landscape pillar in 1.5 seconds based on the guest's walking speed and direction, and simultaneously calculates the specific range and start and end time of the shooting blind spot that the main camera will appear in during this period. Finally, the target prediction trajectory and dynamic blind spot data are output to the collaborative scheduling center to support the subsequent advance replacement scheduling.

[0030] The specific steps of the machine position replacement scheduling in step S3 are as follows: The collaborative scheduling center assigns main shooting and backup roles to each camera device based on dynamic shooting blind spots and issues scheduling instructions. The camera equipment's gimbal control unit uses an S-shaped gradual curve to adjust the camera's position and attitude, filling in blind spots during shooting.

[0031] Furthermore, the camera position replacement scheduling process in this section is executed in a hierarchical and coordinated manner by the collaborative scheduling center and the gimbal control unit of each camera device. It fully receives the dynamic shooting blind spot data output in step S2 and is the core execution link that transforms the predicted shooting gap into the actual shooting action of the camera position. The runtime collaborative scheduling center first obtains the spatial location, duration and shooting priority of the dynamic shooting blind zone. Combining the current spatial pose, field of view coverage and operating status of each camera device, it dynamically assigns the main shooting and supplementary shooting functions to each device. It only issues high-rise shooting task instructions to the supplementary camera position to be adjusted to cover the target blind zone, and does not directly output frame-by-frame control parameters to reduce data transmission latency. After receiving the scheduling command, the gimbal control unit of the corresponding camera equipment uses an S-shaped gradual curve to plan the motion trajectory of the gimbal rotation and lens zoom. This allows the camera to accelerate slowly at the start, adjust at a constant speed in the middle, and decelerate slowly at the end, avoiding sudden changes in camera posture that could cause harsh shaking in the image. The camera is then smoothly adjusted to the target shooting angle and focal length, accurately filling the corresponding shooting blind spots.

[0032] Taking an outdoor variety show with four cameras as an example, when step S2 predicts that the guest will walk behind the landscape pillar in 1.5 seconds, creating a 2-second shooting blind spot, the coordination and scheduling center determines that the No. 1 camera, which is facing the guest, should maintain the main shooting role and be responsible for the front view before the blind spot. The No. 2 camera, which is closest to the blind spot, is assigned as the replacement role and a scheduling instruction is issued to "cover the mid-range view of the guest behind the pillar". The gimbal control unit of the No. 2 camera then generates an S-shaped easing motion curve based on the difference between the current gimbal angle and the target angle, controls the gimbal to rotate smoothly horizontally, and adjusts the lens focus synchronously. The entire picture has no abrupt acceleration and is smoothly adjusted into place before the guest enters the blind spot, completely filling the shooting gap of the main camera.

[0033] The specific steps for image offset compensation in step S3 are as follows: The real-time image processing unit has a built-in dense optical flow calculation module that calculates the motion vector field of the image frame by frame and performs sub-pixel correction on the image based on the vector field to eliminate jitter and image distortion caused by camera rotation and target movement.

[0034] Furthermore, the image offset compensation in this section is performed by the dense optical flow calculation module built into the real-time image processing unit. It adopts the DIS dense optical flow algorithm with gimbal motion prior constraints and runs synchronously and in parallel with the camera position replacement scheduling process. The core of the algorithm includes a three-level progressive calculation formula, which innovatively introduces gimbal motion parameters as prior constraints into the optical flow solution. Compared with the traditional dense optical flow of pure pixel grayscale matching, it significantly compresses the calculation search range and reduces the mismatch rate in high-speed motion scenes: the first level is the global motion prior prediction formula, and the module synchronously reads the yaw angle increment output by the gimbal control unit. Pitch angle increment Combined with lens focal length Sensor pixel size The global pixel offset caused by the rigid body motion is calculated through perspective imaging geometric transformation. The formula is: , , in , The first level provides the estimated global pixel offsets in the horizontal and vertical directions of the image, respectively, which serve as the initial benchmark for optical flow solving, narrowing the pixel matching range from the entire image to a small area near the estimated offset. The second level is the dense optical flow energy functional with prior constraints. A gimbal prior constraint term is added to the variational solution framework of DIS dense optical flow, and the total energy function is:

[0035] In the formula, u(x, y) and v(x, y) are the horizontal and vertical components of the optical flow field at the image coordinates (x, y). These are data items based on the grayscale consistency assumption, used to match the correspondence between pixels in adjacent frames. This is a spatial smoothing term used to constrain the local continuity of the optical flow field. These are prior constraints for the gimbal, used to calculate the optical flow and global prediction. , Deviation penalty , Using preset weighting coefficients, the module iteratively minimizes the total energy function to obtain the sub-pixel precision motion vector field of the entire image; the third level is the image sub-pixel correction formula, which uses a reverse mapping method to resample the original image pixels, and the corrected image satisfies ,in For the original input screen, To compensate for the corrected output image, neighborhood bilinear interpolation is used to fill the edge gaps caused by the correction, ultimately offsetting the global image shake caused by gimbal rotation and lens zoom, as well as the local deformation and motion blur caused by the rapid movement of the shooting target.

[0036] Taking the dynamic following scene of an outdoor variety show as an example, the basic hardware parameter settings are as follows: The supplementary camera has a lens focal length of f=12mm and an image sensor pixel size of p=4.5μm=4.5×10−3mm; the gimbal only rotates horizontally, with a yaw angle increment. =3∘, pitch angle increment =0∘; Optical flow algorithm weight coefficients λ=0.15 (smoothing term weight), γ=0.4 (gimbal prior constraint weight), image resolution 1920×1080, the subject being photographed is a walking guest.

[0037] Step 1: Calculate the estimated global pixel offset: Using the global motion prior prediction formula: , Substituting the values: tan3∘≈0.0524, =0, =4.5×10−312×0.0524≈139.73 pixels, =0 Calculations show that camera rotation will cause an overall horizontal offset of approximately 140 pixels in the image. This value is used as the initial constraint for optical flow calculation, limiting the pixel matching search range to [ -15, The +15 pixel range reduces computation by more than 70% compared to full-image search.

[0038] Step 2: Solving the energy functional of dense optical flow with prior constraints on the gimbal: General energy formula: Taking the center coordinates of the guest (x=960, y=540) in the image, the initial vector (139.73, 0) obtained solely from the prior knowledge of the gimbal has a bias. The guest's own movement causes a local offset, and the initial total energy... =28.6; After iteratively minimizing the total energy function, the precise sub-pixel optical flow component at that point is obtained. =142.16、 =2.74, this value takes into account both the global offset of camera position rotation and the local offset of guest movement. The constraint terms suppress significant deviations between the optical flow results and the gimbal predictions, thus avoiding optical flow matching failures in fast-rotation scenarios.

[0039] Step 3: Subpixel image correction output: Correction mapping formula: Reverse resampling was performed on the pixel at coordinates (960, 540), and the grayscale value of the pixel at the original image position (960−142.16, 540−2.74) was taken as the corrected pixel. The missing areas caused by the offset at the edge of the image were filled by neighborhood bilinear interpolation. After compensation, the inter-frame jitter of the image was stably controlled within 0.3 pixels. There was no lateral jitter or motion blur of the person during the smooth rotation of the camera to follow the guest. There was no deformable discontinuity in the connection between the main camera and the supplementary camera.

[0040] After receiving the replacement scheduling command, the gimbal slowly rotates 3° to align with the guest in the obstructed area. The dense optical flow calculation module synchronously reads the gimbal rotation angle to calculate the global offset reference. Then, through DIS optical flow with prior constraints, it accurately distinguishes between the rigid body motion of the camera position and the autonomous motion of the person. It completes sub-pixel level image correction frame by frame. No additional hardware image stabilization equipment is required. It can eliminate the image defects caused by camera position scheduling by relying solely on the real-time algorithm of the edge unit. It is suitable for low-latency shooting scenarios such as variety show dynamic tracking and live broadcast of large-scale events.

[0041] The handover of camera positions in step S3 is specifically as follows: When switching camera positions, the target positions and motion trends in the two images are first aligned by dense optical flow. Then, the image weight fusion unit uses a cosine curve to smoothly transition the output ratio of the two images, eliminating switching gaps and jumps.

[0042] Furthermore, the camera position handover process in this section is executed collaboratively by the dense optical flow calculation module and the image weight fusion unit in the real-time image processing unit. This is applied to the switching between the main camera and the supplementary camera to eliminate the image jumps and spatial discontinuities caused by traditional hard cuts. During execution, the dense optical flow calculation module first extracts the subject area from both images to be handed over, calculates their respective sub-pixel motion vectors, and adjusts the spatial position, scale, and motion trend of the supplementary camera image through affine transformation. This ensures that the center coordinate deviation of the target subject in the two images is controlled within 1 pixel, and that the direction and speed of movement remain consistent, achieving precise spatial alignment and avoiding sudden changes in target position during the switch. Subsequently, the image weight fusion unit sets a fixed handover time window and uses a cosine square weight curve to control the output ratio of the two images. The weight calculation formula is: , ,in For the current time within the handover window, The total handover time is calculated as follows: during the handover process, the weight of the main camera position smoothly decreases from 1 to 0, while the weight of the replacement camera position smoothly increases from 0 to 1, with no sudden inflection points throughout the process.

[0043] Taking outdoor variety show guest tracking scenes as an example, when the main camera position is about to be blocked by a landscape pillar and needs to be switched to the second replacement camera position, the system sets a 0.8-second handover window. First, it aligns the body position and walking movement trend of the guests in the two images through dense optical flow, so that the guests are in the thirds position of the image in both images and the walking speed is visually consistent. Then, it smoothly transitions the image output weight according to the cosine curve, and finally presents a continuous tracking shot without obvious switching traces, completely eliminating the visual jump and image discontinuity caused by hard cuts.

[0044] In step S4, the real-time image processing unit has a built-in composition evaluation calculation unit; Set unified composition quantification dimensions: subject proportion, golden ratio deviation, line-of-sight margin, and horizon levelness; the composition evaluation calculation unit weights the values ​​of each dimension and outputs the comprehensive composition score for each camera device.

[0045] Furthermore, this step is executed by the composition evaluation calculation unit deployed in the real-time image processing unit. It conducts quantitative composition evaluation on the stable camera images after dense optical flow motion compensation and smooth handover of camera angles. It transforms the traditional composition aesthetic standards that rely on the director's subjective experience into reusable and calculable numerical indicators, providing a clear basis for subsequent unified optimization of multi-camera composition. The calculation unit pre-sets four core quantitative dimensions: subject proportion, golden ratio deviation, eye shadow, and horizon levelness. For each dimension, the original measurement value is first extracted through the corresponding visual algorithm, and then normalized and mapped to the 0~1 range according to the optimal range or deviation threshold set by the general aesthetic standards of film and television shooting to obtain the individual score: The subject proportion dimension obtains the bounding box of the subject through the object detection algorithm, calculates the pixel area ratio of the bounding box to the whole picture as the original value, and sets the optimal proportion range of mid-range scenes to 0.25~0.4. If it falls within the range, it gets full marks of 1. If it deviates from the range, the score decreases linearly according to the degree of deviation. The golden ratio deviation dimension uses the intersection of the four thirds lines of the image as a reference, calculates the Euclidean distance between the center of the subject and the nearest reference point as the deviation value, and sets the maximum deviation threshold at 1 / 2 of the diagonal of the image. The score is linearly normalized according to the rule that the closer the distance, the higher the score. The visual white space dimension first identifies the subject's orientation, calculates the width ratio of the blank space on the facing side as the original value, sets the optimal white space ratio to 0.6, and scores are mapped back based on the absolute deviation between the actual value and the optimal value; the horizon level dimension extracts the horizon reference line through edge detection, calculates the absolute value of the angle between it and the horizontal axis of the image as the deviation value, sets 5° as the maximum allowable deviation, and the smaller the angle, the higher the score.

[0046] The calculation unit is configured with weight coefficients of w1=0.3, w2=0.35, w3=0.2, and w4=0.15 for the four dimensions respectively. The comprehensive composition score in the range of 0 to 1 is calculated by weighted summation formula Score=w1・S1+w2・S2+w3・S3+w4・S4. The higher the score, the better the composition quality. At the same time, the individual scores of each dimension and the original deviation data are saved simultaneously for subsequent accurate positioning of composition deviations and correction.

[0047] Taking a 1920×1080 resolution outdoor variety show guest follow-up shot as an example, the subject's proportion of 0.32 falls within the optimal range, so S1=1. The subject's center distance from the nearest golden section point is 62 pixels, corresponding to a normalized score of S2≈0.944. The deviation of the line-of-sight white space ratio from the optimal value is 0.03, so S3≈0.95. The horizon tilt is 0.8°, so S4≈0.84. Substituting these values ​​into the formula, the overall composition score is approximately 0.93. This objectively quantifies the composition level of a single camera position and provides a standardized numerical reference for the unified adjustment of composition in multi-camera shots.

[0048] The specific composition optimization in step S4 is as follows: Based on the deviation values ​​of the sub-items corresponding to the comprehensive composition score, the camera angle is slightly adjusted by the gimbal fine-tuning unit, and the image is corrected by the lossless digital cropping module to unify the composition visual effect of all camera positions.

[0049] Furthermore, the composition optimization process in this section is executed collaboratively by the gimbal fine-tuning unit and the lossless digital cropping module. It takes over the sub-item deviation data output by the aforementioned comprehensive composition score and adopts a layered correction logic of "physical coarse adjustment + digital fine-tuning" to unify the multi-camera composition standard: during runtime, it first analyzes the sub-item deviation values ​​corresponding to the composition score of each camera position, classifies deviations such as subject position offset and horizon tilt as gimbal physical adjustment, and classifies deviations such as subject proportion and line-of-sight white space ratio as digital cropping correction. The gimbal fine-tuning unit calculates the gimbal adjustment amount based on the pixel deviation of the image and the focal length of the lens. Small rotations of the gimbal complete coarse correction, avoiding large movements that would damage the stability of the image. The lossless digital cropping module performs edge cropping within the original full-resolution image range without pixel interpolation to ensure lossless image quality. It also makes fine adjustments to the composition ratio. Together, the two modules converge the composition parameters of all camera positions to a unified preset standard.

[0050] Taking an outdoor variety show following camera position with a resolution of 1920×1080 as an example, a certain camera position detected that the subject's center was 75 pixels to the left, the horizon was tilted 0.8° to the right, and the subject's proportion of 0.23 was below the optimal range. The gimbal micro-adjustment unit first rotated horizontally to the right by about 0.38° to correct the subject's horizontal position, and simultaneously adjusted the tilt to level the horizon. Then, the cropping module cropped a total of 70 pixels from the left and right edges of the image, raising the subject's proportion of 0.3 to the optimal value. After correction, the subject of this camera position fell at the golden ratio point, the horizon was horizontal, and the subject's proportion of 0.3 met the standard, and the composition parameters were completely consistent with those of other camera positions, achieving a unified visual style for the multi-camera images.

[0051] The scheduling strategy iteration in step S4 is as follows: The real-time image processing unit transmits the comprehensive composition score, image stability, and blind spot coverage data to the scheduling parameter update unit of the collaborative scheduling center to dynamically adjust the camera position scheduling priority and the PTZ easing coefficient.

[0052] Furthermore, the scheduling strategy iteration in this section is the core link of the entire system's self-optimization closed loop. It is executed by the scheduling parameter update unit built into the collaborative scheduling center. It receives three types of quality data from the real-time image processing unit: comprehensive composition score, image stability, and blind spot coverage. It adjusts two core scheduling parameters, namely the camera position scheduling priority and the gimbal easing coefficient, to achieve dynamic adaptive optimization of the scheduling strategy. During runtime, the scheduling parameter update unit will classify and statistically analyze the historical performance of each camera device according to scene type. Camera positions with consistently high comprehensive composition scores and satisfactory image stability will have their scheduling priority increased in the corresponding scene, so that they will be assigned replacement tasks first. For camera positions with unsatisfactory image stability, the S-shaped easing coefficient of their gimbal movement will be increased, and the acceleration and deceleration time will be extended to reduce image jitter. At the same time, the advance amount of scheduling triggering will be adjusted in combination with the timeliness data of blind spot coverage. If the lag time of the replacement camera position reaching the blind spot exceeds the standard, the scheduling instruction will be issued in advance. Taking the scene of filling in the obstruction of the pillar in an outdoor variety show as an example, in this filling, the overall composition score of camera position 2 was 0.93, the image stability met the standard, and the blind spot coverage had zero lag. The scheduling parameter update unit then raised its scheduling priority from 2nd to 1st in the scene of obstruction by the pillar on the left. On the other hand, the image stability of camera position 3 was low during this filling, so its gimbal easing coefficient was adjusted from 0.6 to 0.75 to slow down the gimbal acceleration and deceleration process. After multiple scene iterations, the system's camera position scheduling matching degree and image quality will continue to improve without the need for manual correction of scheduling rules.

[0053] The system is also equipped with a director's interactive control console and a sample storage unit; If the system detects manual adjustments to camera positions or composition parameters on the director's console, it will lock the automatic scheduling privileges for that camera position. Simultaneously, the current scene and manually adjusted parameters are stored in the sample storage unit, and the graph evaluation weights and scheduling strategies are iteratively updated.

[0054] Furthermore, the human-machine collaborative self-optimization mechanism in this section is implemented through the linkage of the director's interactive console, sample storage unit, collaborative scheduling center, and graph evaluation calculation unit. As a supplementary optimization path to the automatic scheduling strategy, it ensures the priority control of manual operation and the stability of system iteration throughout the process. The system polls the director's interactive console for operation commands in real time. When it detects that the director has manually adjusted parameters such as the yaw / pitch angle of the camera, the lens focal length, and the image cropping boundary, it immediately stops the automatic control command issued by the collaborative scheduling center to the camera and locks the automatic scheduling permission of the camera to avoid conflicts between automatic logic and manual operation. During the lock period, the system only continuously collects the image and status data of the camera and does not actively issue adjustment commands. After the director stops manual operation and there are no new commands for 3 seconds, the automatic scheduling can be manually restored through the console to avoid accidental triggering of permission switching. The system will extract the scene feature tags corresponding to this operation, the specific parameter values ​​adjusted manually, and the composition quantification indicators of the stable image after adjustment, and combine them into a valid sample and store it in the sample storage unit. Before storing, the system will verify the stability duration of the adjusted image and only retain valid operation samples with a stability duration of more than 1 second, excluding invalid data such as accidental touches and quick adjustments. When the cumulative number of valid samples in the same scenario reaches the preset threshold, the system triggers a parameter iteration. The optimal standard value and weight coefficient of the corresponding dimension in the composition evaluation system are updated using a weighted average method. At the same time, the preset angle of camera position scheduling and the gimbal easing parameters in the corresponding scenario are adjusted. The magnitude of each parameter adjustment is controlled within 5% of the original value to ensure that the iteration process is smooth and without sudden changes, so that the automatically output composition and scheduling effects gradually approach the director's operating habits.

[0055] Taking the scene of a pillar blocking the view in an outdoor variety show as an example, the director thought that the blank space in the guest's line of sight was too small when the automatic output of the No. 2 replacement camera was used. He manually adjusted the gimbal to the right by 0.2° and expanded the right-side cropping range. The system then locked the automatic scheduling permission of the No. 2 camera and stopped the automatic composition correction command. Once the operation is stable, the scene tags "pillar obstruction and replacement, walking guest mid-shot", the 0.2° gimbal adjustment, and the adjusted white space ratio of 0.62 are stored as valid samples in the unit. When the cumulative number of similar manual adjustment samples in this scene reaches 5, the system will slightly adjust the default value of the optimal white space ratio of this scene from 0.6 to 0.61, and at the same time slightly increase the weight coefficient of the line-of-sight white space dimension. When similar scenes are automatically scheduled in the future, the composition effect will be closer to the director's aesthetic preferences, and the frequency of manual correction can be gradually reduced.

[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time collaborative scheduling and dynamic composition optimization in multi-camera film and television shooting, applied to a multi-camera film and television shooting system including multiple camera devices, an edge computing unit, a collaborative scheduling hub, and a real-time image processing unit, characterized in that, Includes the following steps: S1. The edge computing unit performs joint calibration of the spatiotemporal reference for the camera equipment cluster, constructs a globally unified spatiotemporal coordinate system, and synchronously acquires the gimbal, lens, and spatial pose parameters of each camera equipment. S2. Based on the global unified spatiotemporal coordinate system, the edge computing unit tracks the dynamic shooting object and predicts the motion trend to generate a dynamic shooting blind spot distribution. S3. The collaborative scheduling center performs camera position replacement scheduling based on the dynamic shooting blind spot; The real-time image processing unit uses dense optical flow to compensate for image shift defects caused by camera position and target movement, and smoothly completes the handover of multi-camera perspectives. S4. The real-time image processing unit quantifies and calculates the comprehensive composition score of each camera position's image, optimizes the multi-camera composition based on the score, and sends the composition and image stability data back to the collaborative scheduling center to iteratively update the scheduling strategy.

2. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, In step S1, the edge computing unit integrates a PTP synchronization module and a SLAM calibration unit, and the spatiotemporal calibration process is as follows: The PTP synchronization module is used to correct the timestamps of each camera device by combining static feature points of the scene; frame-level synchronization is achieved. By matching multi-camera features through the SLAM calibration unit, the poses of all camera devices are uniformly mapped to the global unified spatiotemporal coordinate system.

3. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, In step S2, the edge computing unit integrates a visual tracking module and a Kalman filter module, and the target tracking and prediction process is as follows: The visual tracking module identifies and tracks dynamically captured objects across camera positions, and calculates the target's three-dimensional trajectory using the global unified spatiotemporal coordinate system. The Kalman filter module predicts the short-term movement position of the target and updates the dynamic shooting blind zone distribution simultaneously.

4. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, The specific steps of the machine position replacement scheduling in step S3 are as follows: The collaborative scheduling center assigns main shooting and backup roles to each camera device according to the dynamic shooting blind spot, and issues scheduling instructions; The camera equipment's gimbal control unit uses an S-shaped gradual curve to adjust the camera's position and attitude, filling in blind spots during shooting.

5. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, The specific steps for image offset compensation in step S3 are as follows: The real-time image processing unit has a built-in dense optical flow calculation module that calculates the motion vector field of the image frame by frame and performs sub-pixel correction on the image based on the vector field to eliminate jitter and image distortion caused by camera rotation and target movement.

6. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, The handover of camera positions in step S3 is specifically as follows: When switching camera positions, the target positions and motion trends in the two images are first aligned by dense optical flow. Then, the image weight fusion unit uses a cosine curve to smoothly transition the output ratio of the two images, eliminating switching gaps and jumps.

7. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, In step S4, the real-time image processing unit has a built-in composition evaluation calculation unit; A unified composition quantification dimension is set: subject proportion, golden ratio deviation, line-of-sight margin, and horizon levelness; the composition evaluation calculation unit weights the values ​​of each dimension and outputs the comprehensive composition score corresponding to each camera device.

8. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, The specific composition optimization in step S4 is as follows: Based on the sub-item deviation values ​​corresponding to the comprehensive composition score, the camera angle is slightly adjusted by the gimbal fine-tuning unit, and the image is corrected by the lossless digital cropping module to unify the composition visual effect of all camera positions.

9. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, The scheduling strategy iteration in step S4 is as follows: The real-time image processing unit transmits the comprehensive composition score, image stability, and blind spot coverage data to the scheduling parameter update unit of the collaborative scheduling center to dynamically adjust the camera position scheduling priority and the gimbal easing coefficient.

10. The method for real-time collaborative scheduling and dynamic composition optimization of multi-camera film and television shooting according to claim 1, characterized in that, The system is also equipped with a director's interactive console and a sample storage unit; If the system detects manual adjustments to camera positions or composition parameters on the director's console, it will lock the automatic scheduling privileges for that camera position. Simultaneously, the current scene and manually adjusted parameters are stored in the sample storage unit, and the graph evaluation weights and scheduling strategies are iteratively updated.

Images