Intelligent Stage Control Method and System Based on Multi-device Collaboration

Abstract

This invention relates to the interdisciplinary field of intelligent control and constraint programming, and provides a stage intelligent control method and system based on multi-device collaboration. To address the inherent spatiotemporal lag between equipment execution and actor behavior caused by the "post-event response" mode of traditional stage control, the method establishes a unified spatiotemporal reference, captures the continuous motion trajectory flow of actors in real time, and directly encodes it into dynamic spatiotemporal constraints. These constraints are then fused with the physical response characteristics of the stage execution equipment to construct a multi-dimensional fusion constraint space. Feasible state sequences are solved in real time within this multi-dimensional fusion constraint space, generating instruction sets with execution timestamps, which are then distributed to each stage execution device for synchronous execution. Through a constraint satisfaction mechanism of "direct behavior encoding - constraint fusion - real-time solution," real-time and precise collaboration between multiple stage devices and actor behavior is achieved, significantly enhancing the artistic expression of stage performances and the creative freedom of actors.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of intelligent control and constraint programming, and particularly to real-time collaborative control technology based on constraint satisfaction, specifically to an intelligent stage control method and system based on multi-device collaboration. Background Technology

[0002] Multi-device collaborative control of a stage refers to the unified scheduling of stage equipment such as lighting, sound, and mechanical rigging through a central control system, enabling them to synchronize with the actors' performance movements in time and space. It is the core support for presenting immersive effects in modern stage performances. With the rapid development of new performance forms such as large-scale live performances and immersive dramas, higher demands are placed on the freedom of actors' actions, the real-time response of equipment, and the precision of multi-device collaboration.

[0003] Currently, the collaborative control of multiple stage devices mainly adopts the following two technical paths: Path 1: Timeline pre-programmed control, where technicians pre-arrange the status parameters of each device according to the rehearsal process, and control the stage equipment to execute according to the preset timeline playback instructions during the performance; Path 2: Sensor-triggered feedback control, which uses position sensors and motion capture equipment to automatically trigger preset programs to control the execution of stage equipment when an actor is detected to have reached a specific position.

[0004] However, the aforementioned traditional technologies suffer from a lag in control command generation compared to the actor's actual actions, resulting in a temporal and spatial misalignment between the device's execution and the actor's improvisation, thus creating a disconnect in artistic effect. The core reason for this problem lies in the existing technology's "preset-execution" control mode, whose underlying logic is a passive response "after the action occurs." The system must wait for the actor's behavior to be captured and processed before generating commands. In this serial "perception-computation-execution" link, perception delay, computation delay, and communication delay are superimposed, forming an inherent lag. Long-term technological improvements in this field have focused on optimizing perception accuracy and reducing computation delay, attempting to approach real-time performance through "faster response." However, regardless of the improvement in response speed, the "post-response" mode is essentially still a form of lag compensation, unable to fundamentally eliminate the time difference between the actor's behavior and the device's execution, creating a technical dilemma where "the more optimized, the closer to the limit, yet the lag bottleneck remains unbroken."

[0005] Therefore, how to achieve real-time coordination between multiple stage devices and actors' free behavior without relying on behavior prediction has become a pressing technical problem in the field of intelligent stage control. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a stage intelligent control method and system based on multi-device collaboration. By adopting a technical approach of "direct behavior encoding + constraint propagation solution," compared with the traditional "behavior prediction + post-event response" mode, it achieves complete preservation of the actor's behavioral freedom and real-time synchronization of equipment response. It enables real-time and precise collaboration between multiple stage devices and the actor's improvisational performance, eliminating the inherent time and space lag in traditional methods.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a stage intelligent control method based on multi-device collaboration, comprising: establishing a unified spatiotemporal reference, including: establishing a three-dimensional spatial coordinate system as the spatial reference, and synchronizing the local clocks of the actor tracking sensor, the central controller, and each stage execution device to a unified system time axis as the time reference; in the three-dimensional spatial coordinate system, capturing the continuous motion trajectory flow of the actor in real time, and directly encoding the continuous motion trajectory flow into dynamic spatiotemporal constraints that evolve over time, the dynamic spatiotemporal constraints including the actor's spatial position constraints and kinematic parameter constraints at each moment; fusing the dynamic spatiotemporal constraints with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fused constraint space; in the multi-dimensional fused constraint space, using a constraint propagation algorithm with optimized complexity, solving the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control cycle, generating a set of device control instructions with execution timestamps; and distributing the set of device control instructions to each stage execution device and executing them synchronously according to the execution timestamps.

[0008] This invention also provides an intelligent stage control system based on multi-device collaboration, comprising: a first establishment module for establishing a unified spatiotemporal reference, including: establishing a three-dimensional spatial coordinate system as the spatial reference, and synchronizing the local clocks of the actor tracking sensor, the central controller, and each stage execution device to a unified system time axis as the time reference; a first encoding module for capturing the continuous motion trajectory flow of the actor in real time in the three-dimensional spatial coordinate system, and directly encoding the continuous motion trajectory flow into dynamic spatiotemporal constraints that evolve over time, wherein the dynamic spatiotemporal constraints include the actor's spatial position constraints and kinematic parameter constraints at each moment; a first construction module for fusing the dynamic spatiotemporal constraints with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fused constraint space; a first solution module for using a complexity-optimized constraint propagation algorithm in the multi-dimensional fused constraint space to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control cycle, generating a set of device control instructions with execution timestamps; and a first control module for distributing the set of device control instructions to each stage execution device and executing them synchronously according to the execution timestamps.

[0009] Beneficial Effects: The solution implemented in this invention, by establishing a unified spatiotemporal reference, firstly achieves precise spatiotemporal alignment of actor behavior capture and equipment collaboration under a unified spatial and temporal reference, eliminating system errors caused by inconsistencies in coordinates and time asynchrony. Based on this, by directly encoding the actor's continuous motion trajectory flow into dynamic spatiotemporal constraints, it achieves lossless mapping of the actor's free behavior into computable constraint expressions. This differs from the indirect processing method in traditional technologies that first extracts features and then inputs them into the prediction model. Direct encoding skips the feature extraction and intent recognition stages, avoiding information loss due to feature abstraction and delays introduced by model inference. The above describes a multi-dimensional fusion constraint space constructed by integrating dynamic spatiotemporal constraints with equipment physical response characteristic constraints. This establishes a two-way constraint relationship between the actors' freedom of action and the equipment's capability boundaries, transforming the "how to coordinate" problem into a mathematical problem of finding feasible solutions in a unified constraint space. Furthermore, by employing a constraint propagation algorithm with optimized complexity to solve feasible state sequences in real time within a preset control cycle and generate timestamped instruction sets, efficient real-time mapping from the constraint space to feasible collaborative solutions is achieved. Finally, by distributing the instruction sets to each stage execution device according to the execution timestamp and executing them synchronously, precise collaborative startup of multiple devices under a unified time reference is realized.

[0010] By combining the above interconnected effects, a five-in-one real-time collaborative control mechanism was constructed, which integrates "unified spatiotemporal benchmarks, direct behavior encoding, constraint fusion construction, constraint propagation and solution, and timestamp synchronous execution".

[0011] Therefore, unlike the traditional control method that relies on a "post-event response" mode to passively follow the actor's behavior, this invention directly encodes the actor's behavior into dynamic constraints and integrates them with the equipment's capability constraints to solve collaborative solutions in real time. This transforms the traditional passive and lagging "post-event response" mode into an active and real-time "constraint satisfaction" mode, ultimately achieving real-time and precise collaboration between multiple stage devices and the actor's free behavior. This solves the technical problem mentioned in the background technology that "traditional stage control methods, due to the use of a 'post-event response' mode, result in an inherent time and space lag between equipment execution and actor behavior." It significantly improves the artistic expressiveness of stage performances, the actor's creative freedom, and the real-time performance, accuracy, and reliability of multi-device collaboration. Attached Figure Description

[0012] Figure 1 A flowchart illustrating the intelligent stage control method based on multi-device collaboration provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the first sub-process of the intelligent stage control method based on multi-device collaboration provided in an embodiment of the present invention; Figure 3This is a schematic block diagram of a stage intelligent control system based on multi-device collaboration, provided as an embodiment of the present invention. Detailed Implementation

[0013] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0014] This invention provides a stage intelligent control method and system based on multi-device collaboration. It can be executed by a real-time collaborative control engine hosted by a central controller or edge computing nodes, driving stage performance technology applications across multiple dimensions, including but not limited to: First, in immersive performance systems, by directly encoding actors' real-time motion trajectories into dynamic constraints and solving for equipment collaborative states, millisecond-level real-time precise tracking of lighting, sound, machinery, and actors' improvisational performances is achieved, providing audiences with a highly immersive experience. Second, in large-scale live performance management platforms, by constructing a multi-actor-multi-device fusion constraint space, real-time collaborative control of hundreds or thousands of performance devices and multiple actors is supported, providing high-concurrency equipment scheduling capabilities for large-scale performances. Third, in digital performance rehearsal systems, by recording actors' motion trajectories and equipment response sequences, a traceable collaborative control data stream is generated, providing quantitative reference information for directors to evaluate performance effects and optimize stage scheduling schemes. This invention breaks through the inherent time-lag bottleneck of the traditional "post-event response" control paradigm, providing real-time collaborative control capabilities for the intelligent upgrading of the stage performance industry.

[0015] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0016] Example 1, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the intelligent stage control method based on multi-device collaboration provided in an embodiment of the present invention. Figure 1 As shown, in this embodiment, applied to a server, the method includes the following steps S11-S15: S11: Establish a unified spatiotemporal reference, including: establishing a three-dimensional spatial coordinate system as a spatial reference, and synchronizing the local clocks of the actor tracking sensors, the central controller, and each stage execution device to a unified system time axis as a time reference; S12: In the three-dimensional spatial coordinate system, the continuous motion trajectory flow of the actor is captured in real time, and the continuous motion trajectory flow is directly encoded into dynamic spatiotemporal constraints that evolve over time. The dynamic spatiotemporal constraints include the actor's spatial position constraints and kinematic parameter constraints at each moment.

[0017] A unified spatiotemporal reference system refers to a spatiotemporal reference system jointly constituted by a three-dimensional spatial coordinate system and a unified system time axis. The three-dimensional spatial coordinate system serves as a spatial reference, providing a unified positional reference for locating actor movement trajectories and describing the operational areas of various stage performance equipment. The unified system time axis serves as a time reference, synchronizing the actor tracking sensors, the central controller, and the local clocks of each stage performance equipment to the same time axis, providing a precise time reference for the subsequent timestamp appending of trajectory points and the scheduling of equipment command execution. Clock synchronization can be achieved using well-known technologies in the field, such as precise time protocols or time-sensitive networks, and will not be elaborated upon here.

[0018] Continuous motion trajectory stream: refers to a sequence of three-dimensional spatial positions captured in real time by actor tracking sensors and arranged in chronological order. A continuous motion trajectory stream consists of trajectory points, each containing at least three-dimensional spatial coordinates (x, y, z) and a global timestamp t. The time interval between adjacent trajectory points is determined by the sensor sampling frequency (e.g., 60 frames per second). The continuous motion trajectory stream, through temporal and spatial continuity, fully depicts the actor's motion process.

[0019] Dynamic spatiotemporal constraints refer to the set of constraint expressions that evolve over time after the continuous motion trajectory of the actor is directly encoded. Dynamic spatiotemporal constraints include two dimensions: spatial position constraints, which indicate that the actor must be located within a spatial region centered on the trajectory point P(t) and bounded by a preset tolerance radius r at each time t; and kinematic parameter constraints, which indicate that the actor's velocity at each time t must be within the interval [v_min(t), v_max(t)], and the acceleration must be within the interval [a_min(t), a_max(t)].

[0020] The implementation is as follows: First, during system initialization or before the performance begins, preprocessing to establish a unified spatiotemporal reference is performed. Specifically, a three-dimensional spatial coordinate system is established in the stage space as the spatial reference. This system uses a specific location on the stage (such as the center point or a corner) as its origin, the stage plane as the XY plane, and the vertical upward direction as the Z-axis, providing a unified positional standard for actor positioning and equipment operational area description. Furthermore, a precise time protocol is used to synchronize the actor tracking sensors, the central controller, and the local clocks of each stage execution device to a unified system timeline, ensuring complete time consistency across all devices with a synchronization accuracy reaching sub-millisecond levels.

[0021] Secondly, during the performance, the central controller sends a start command to the actor tracking sensors (such as depth vision sensors or optical capture devices) deployed in the stage area. After responding to the command, each sensor begins to work, capturing the continuous motion trajectory of the actors in real time at a preset sampling frequency (such as 60-120 frames / second) in the established three-dimensional spatial coordinate system. Each captured trajectory point contains the three-dimensional spatial coordinates (x, y, z) of the actor's key skeletal points, and is locally appended by the sensor with a global timestamp t based on the unified system time axis.

[0022] Third, the central controller receives trajectory point data streams from multiple sensors in real time and performs direct encoding on the continuous motion trajectory stream. The encoding process includes: for each time t, generating spatial position constraints based on the trajectory point coordinates. Spatial position constraints refer to the limited expression of the spatial region that the actor must be located in at each time t, which can be expressed as the actor must be located in a spatial region centered on coordinate point P(t) and bounded by a preset tolerance radius r at time t; and calculating instantaneous velocity vectors and instantaneous acceleration vectors based on the trajectory point coordinates of adjacent times to generate kinematic parameter constraints, which are expressed as the actor's velocity at time t must be within the interval [v_min(t), v_max(t)] and acceleration must be within the interval [a_min(t), a_max(t)].

[0023] Fourth, the central controller aligns the generated spatial position constraints and kinematic parameter constraints along the time axis to form dynamic spatiotemporal constraints that evolve over time. These dynamic spatiotemporal constraints can be encapsulated using standardized data structures (such as JSON or Protocol Buffers format) and used as inputs for subsequent fusion with the physical response characteristics constraints of the equipment, awaiting further processing.

[0024] For example, taking a large-scale live performance as an example, the system establishes a three-dimensional spatial coordinate system and synchronizes the clocks of multiple devices during initialization. After the performance begins, actor A moves to coordinates (5.2, -2.3, 1.5) at t=15.32 seconds. The sensor captures this trajectory point and reports it with a timestamp. The central controller calculates the instantaneous velocity (2.0 m / s) and acceleration (0.5 m / s²) based on the trajectory points before and after the initial time, generating dynamic spatiotemporal constraints: the spatial position constraint is a spherical region with a radius of 0.1 meters centered at (5.2, -2.3, 1.5); the kinematic parameter constraints are a velocity range of [1.8, 2.2] m / s and an acceleration range of [0.4, 0.6] m / s². Subsequent trajectory points are continuously reported, generating new constraints, which await fusion processing with device constraints.

[0025] Steps S11-S12 establish a unified spatiotemporal benchmark and directly encode the actor's continuous motion trajectory flow into dynamic spatiotemporal constraints, achieving lossless encoding of the actor's free behavior. Any improvisation, rhythm adjustment, or positional deviation by the actor is preserved in the constraints as is, rather than being "corrected" by the prediction model. Thus, unlike the indirect processing method of extracting features before inputting them into the prediction model in traditional techniques, direct encoding skips the feature extraction and intent recognition stages, avoiding information loss due to feature abstraction and delays introduced by model inference. This provides real-time, authentic, and complete behavioral input for subsequently solving the device collaboration scheme in the constraint space, thereby laying the data foundation for the real-time and accurate collaboration of claim 1.

[0026] S13: The dynamic spatiotemporal constraints are fused with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fused constraint space.

[0027] Physical response characteristic constraints refer to the set of constraint expressions pre-stored in the system that describe the physical performance and response characteristics of each stage execution device. These constraints include, but are not limited to, the following dimensions: 1) Motion capability boundaries, such as the device's maximum speed, maximum acceleration, and minimum response delay; 2) Spatial reachability, such as the device's physical motion range, rotation angle limits, and light spot projection area; 3) Motion continuity constraints, such as requirements for position continuity, velocity continuity, and acceleration continuity. These constraints are obtained through offline calibration or manufacturer parameters and are pre-stored in the system database in a standardized format (such as constraint functions, parameter tables, or configuration files) for real-time retrieval during fusion.

[0028] Multidimensional fusion constraint space: refers to the multidimensional constraints generated by fusing the dynamic spatiotemporal constraints generated on the actor side with the pre-stored physical response characteristics constraints on the equipment side within a unified spatiotemporal framework. The multidimensional fusion constraint space can be a Cartesian product space, its dimensions determined by the number of actors and equipment involved in the collaboration, as well as the dimensions of each constraint. In the multidimensional fusion constraint space, each point represents a possible combination of actor-stage execution equipment states, and the set of points satisfying all constraints constitutes the feasible solution space. The construction process of the multidimensional fusion constraint space includes, but is not limited to, three sub-steps: spatial alignment, parameter matching, and constraint coupling, ensuring that the actor's behavioral freedom and the equipment's capability boundaries form a computable constraint relationship within a unified spatiotemporal reference system.

[0029] The implementation is as follows: First, the central controller parses and extracts the spatial position constraint range and the allowable value range of velocity and acceleration of the actors at each moment from the dynamic spatiotemporal constraints; then, it calls the pre-stored physical parameter library of each stage execution device from the system database to obtain the physical response characteristic constraints such as the maximum movement speed, maximum acceleration, minimum response delay, and reachable spatial range of each stage execution device.

[0030] Secondly, the central controller spatially aligns the actor's spatial position constraint range with the reachable spatial range of each stage execution device. That is, in a unified three-dimensional coordinate system, it calculates the spatial intersection between the actor's trajectory points and the operating areas of each stage execution device. Specifically, for each time t, it determines whether there is an overlap between the actor's spatial position constraint range and the reachable spatial range of stage execution device A. If there is an overlap, the stage execution device is recorded as a candidate interactive device, and the spatial boundary of the overlapping area is determined.

[0031] Third, the central controller performs kinematic matching between the actor's allowed speed and acceleration ranges and the maximum speed and acceleration of each candidate interactive device to determine whether the actor's motion requirements are within the motion performance boundaries of the stage execution equipment. If the actor's required speed exceeds the maximum speed of the stage execution equipment, or the required acceleration exceeds the maximum acceleration of the stage execution equipment, the stage execution equipment is excluded from the candidate set; otherwise, the compatibility parameters between the stage execution equipment's motion capabilities and the actor's requirements are recorded.

[0032] Fourth, the central controller performs time synchronization matching between the time constraints in the actor's dynamic spatiotemporal constraints and the minimum response delay of each candidate interactive device, determining whether the device response delay meets the time accuracy requirements of the actor's movement. Specifically, based on the actor's current position and velocity at time t, it predicts the actor's position range at time t+Δt (Δt being the device response delay), and determines whether the stage execution device, after receiving the instruction at time t, can accurately respond to the actor's position requirement at time t+Δt. If the predicted position range matches the operational area of ​​the stage execution device, it confirms that the stage execution device meets the time synchronization requirements.

[0033] Fifth, based on the combined results of spatial alignment, kinematic matching, and temporal synchronization matching, the central controller identifies actor-stage execution device pairs that have interactive relationships within the same spatial region at the same time. For each identified actor-stage execution device pair, a two-way constraint relationship is established: the actor's spatial position constrains the selection of the stage execution device's area of ​​action (i.e., the stage execution device must adjust its area of ​​action to be near the actor's position), and the stage execution device's responsiveness constrains the upper limit of the actor's motion parameters (i.e., the actor's speed and acceleration cannot exceed the capability boundary of the stage execution device). These two-way constraints, along with other independent constraints, are incorporated to generate a multi-dimensional fusion constraint space that includes the actor-stage execution device two-way constraint relationship.

[0034] For example, following the previous example, actor A's dynamic spatiotemporal constraints at t=15.32 seconds are: position (5.2,-2.3,1.5)±0.1 meters, velocity 2.0 m / s, acceleration 0.5 m / s². The following are performed: 1) Spatial alignment: The actor's position simultaneously falls within the coverage area of ​​light 1 (covering the eastern area of ​​the stage) and hoist 2 (X∈[5,10],Y∈[-5,5],Z∈[0,8]). 2) Kinematic matching: The actor's velocity and acceleration are both within the capability range of hoist 2 (maximum velocity 0.8 m / s, maximum acceleration 0.5 m / s²). 3) Temporal synchronization matching: Based on the displacement prediction according to the response delay, light 1 displaces 0.03 meters in 15 ms, and hoist 2 displaces 0.1 meters in 50 ms, both within their effective range. The interaction between actor A and light source 1 and hoisting device 2 is identified, and a two-way constraint is established: the light spot of light source 1 follows the actor's position, the point of action of hoisting device 2 is constrained by the actor's position, and the actor's movement is limited by the velocity and acceleration capability of hoisting device 2. All of these constraints are uniformly incorporated into a multi-dimensional fusion constraint space.

[0035] It should be noted that the specific implementation methods of spatial alignment, kinematic matching, and temporal synchronization matching are not limited to the examples given. For instance, for spatial alignment, in one example, a fast intersection test algorithm based on bounding boxes can be used, which is suitable for real-time scenarios with high computational efficiency requirements; in another example, a precise collision detection algorithm based on distance fields can be used, which is suitable for fine-grained interaction scenarios with high spatial accuracy requirements. Any conventional technical means that can achieve spatial alignment, kinematic matching, and temporal synchronization matching are within the protection scope of this invention.

[0036] In step S13, by constructing a multi-dimensional fusion constraint space, a two-way constraint relationship is established between the actor's freedom of action and the equipment's capability boundaries within a unified spatiotemporal framework—the actor's spatial position constrains the selection of the equipment's effective area, and the equipment's responsiveness constrains the upper limit of the actor's motion parameters. Thus, the complex problem of "how to coordinate" is transformed into a mathematical problem of finding feasible solutions within a unified constraint space. This differs from the unidirectional control mode in traditional technologies where equipment passively responds to actor behavior, laying the foundation for solving the real-time, precise coordination required in claim 1.

[0037] S14: In the multi-dimensional fusion constraint space, a constraint propagation algorithm with optimized complexity is used to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control period, and generate a set of device control instructions with execution timestamps. S15: Distribute the device control instruction set to each stage execution device according to the execution timestamp and execute them synchronously.

[0038] The complexity-optimized constraint propagation algorithm refers to an algorithm specifically optimized for the real-time requirements of stage collaborative control scenarios, based on the classic constraint propagation algorithm. The constraint propagation algorithm involves repeatedly reducing the value range of variables through constraints. When the value range of a variable is reduced to empty, backtracking is triggered until a combination of variable assignments that satisfies all constraints is found. The complexity optimizations in this invention include, but are not limited to: dynamically sorting the variable domains according to the number of constraints, prioritizing the search of the variable domain with the highest number of constraints to improve pruning efficiency; employing a forward checking strategy, immediately checking relevant constraints after each assignment to preemptively delete conflicting values ​​in subsequent variable domains; and limiting the backtracking depth and propagation range to ensure the algorithm completes the solution within a preset control cycle. Specific parameters (such as variable sorting strategy and backtracking constraints) can be configured according to the number of stage equipment and constraint complexity, and will not be elaborated further here.

[0039] Feasible state sequence: refers to the time sequence of the states of each stage execution device that satisfies all constraints in the multi-dimensional fusion constraint space, output after being solved by the constraint propagation algorithm within a preset control period. The feasible state sequence includes the state parameters that each stage execution device should be at each control sub-time (e.g., every 10ms) within a future control period (e.g., 50ms) starting from the current moment. Examples of parameters include the horizontal and vertical rotation angles and brightness values ​​of lighting equipment; and the volume and beam direction of sound equipment. The feasible state sequence is encapsulated in timestamped formatted data (e.g., JSON arrays or ProtocolBuffers messages), and each state entry is strictly aligned with a unified system timeline.

[0040] The implementation is as follows: First, the central controller acquires a multi-dimensional fusion constraint space, which includes the dynamic spatiotemporal constraints of the actors, the physical response characteristic constraints of each stage execution device, and the bidirectional constraint relationship between the actors and the stage execution devices. The central controller initializes the state variable domains of each stage execution device, with each state variable domain representing the set of all possible values ​​that a stage execution device can take (such as the range of values ​​for the light rotation angle and the feasible space for the mechanical position).

[0041] Secondly, the central controller initiates a complexity-optimized constraint propagation algorithm to solve the problem in real time within a preset control cycle. The constraint propagation algorithm execution process includes: dynamically sorting the variables based on the number of constraints associated with each state variable domain, prioritizing the variable domain with the most associated constraints for assignment attempts; after each variable domain is assigned a value, a forward checking strategy is immediately adopted to check all constraints related to that variable, and values ​​in subsequent variable domains that conflict with the current assignment are deleted in advance; if the assignment is detected to cause any variable domain to become an empty set, a backtracking mechanism is triggered to return to the previous assignment point and reselect; through repeated assignment-propagation-backtracking iterations, the value range of each variable domain is gradually narrowed.

[0042] Third, when the algorithm successfully finds a set of variable assignments that satisfy all constraints, it obtains a set of feasible device states, i.e., a feasible state sequence. The central controller combines this set of device states with other states within a preset control cycle to form a feasible state sequence for a future control cycle starting from the current moment. For example, if the control cycle is 50ms and the control substep size is 10ms, a state sequence containing 5 time points is generated. For each state point in the sequence, the central controller adds an execution timestamp to it according to the unified system timeline, indicating at what precise moment the state should be executed by the stage execution equipment. The accuracy of the execution timestamp can reach the microsecond level to ensure the accuracy of the coordination of multiple stage execution equipment.

[0043] Fourth, the central controller distributes the generated equipment control instruction set (containing the state sequence of each stage performance device within a future control cycle and its corresponding execution timestamp) to the local controllers of each stage performance device via a real-time network. The distribution process can employ a time-sensitive network protocol to ensure deterministic transmission delay of the instruction data. Each stage performance device's local controller receives and parses the equipment control instruction set, extracting its own control instructions and execution timestamp. The local controller continuously synchronizes its local clock with the unified system timeline and compares its local time with the received execution timestamp in real time. When the local time reaches the moment indicated by the execution timestamp, the local controller immediately triggers its device to perform the corresponding action (such as rotating lights to a specified angle, adjusting sound volume, or moving machinery to a specified position). Multiple stage performance devices start execution simultaneously according to their respective execution timestamps, achieving precise spatiotemporal synchronization and coordination.

[0044] For example, following the previous example, the central controller initiates the constraint propagation algorithm at t=15.34 seconds to solve the constraint. After multiple iterations, the algorithm finds the state assignments that satisfy all constraints within 30ms: Light 1 rotates horizontally by 125°, vertically by 42°, and reaches 80% brightness at t=15.35 seconds; it rotates horizontally by 130°, vertically by 40°, and reaches 85% brightness at t=15.36 seconds; the position and speed states of the No. 2 suspension system at the corresponding times are also determined. The central controller organizes these states into a feasible state sequence at 10ms intervals, adds execution timestamps, and distributes them to each device via a time-sensitive network. Light 1 and the No. 2 suspension system execute synchronously at the corresponding times, achieving a coordinated effect of precise lighting tracking and synchronous raising and lowering of the suspension system.

[0045] It should be noted that the specific implementation of the constraint propagation algorithm described above is not limited to the examples given. In one example, a conflict-oriented backtracking intelligent backtracking algorithm can be used, which directly skips multiple layers of invalid assignment points by recording the cause of the conflict, and is suitable for scenarios with complex constraint relationships. In another example, a constraint propagation algorithm based on arc consistency maintenance can be used, which improves propagation efficiency by maintaining the support relationship between variable domains, and is suitable for scenarios with large variable domain sizes. Any algorithm that can solve for the device state sequence that satisfies all constraint conditions within a preset control period is within the protection scope of this invention.

[0046] In steps S14-S15, a constraint propagation algorithm with optimized complexity is used to solve for feasible state sequences in real time within a multi-dimensional fusion constraint space. First, the complex problem of "how actors and equipment can coordinate" is transformed into a solvable mathematical problem within a preset control cycle. Optimization methods such as dynamic sorting, forward checking, and backtracking are used to strictly control computation time while ensuring solution quality, ensuring that an effective coordination solution can be output in each control cycle. Based on this, a set of device control instructions with execution timestamps is generated and distributed for synchronous execution according to the timestamps, thereby achieving precise coordination of multiple devices under a unified time base. Each device starts execution at the same time based on the same time base, fundamentally eliminating the problem of execution asynchrony caused by network latency and differences in device response.

[0047] By combining the aforementioned interconnected effects, steps S14-S15 establish a complete technical loop from "actor behavior capture → constraint encoding → constraint fusion → constraint solving → instruction execution" by extending the unified spatiotemporal reference established in S11 to the final execution stage—based on the same coordinate system in space and the same time axis in time. This differs from the inherent time lag caused by the serial "perception-computation-execution" link in traditional technologies. It achieves millisecond-level synchronization between equipment execution and actor behavior in time and micrometer-level tracking in space. This solves the fundamental technical problem raised in the background technology that "traditional control methods, due to their 'post-response' mode, result in inherent spatiotemporal lag between equipment execution and actor behavior," significantly improving the real-time performance, accuracy, and reliability of multi-device collaboration on stage.

[0048] Therefore, steps S14-S15 provide core solution and execution guarantees for the real-time and accurate collaboration of claim 1, and are key links in building a complete technical closed loop from behavior capture to instruction execution.

[0049] In this embodiment, through sequential processing of S11-S15, a series of technical means such as establishing a unified spatiotemporal benchmark, directly encoding dynamic constraints, fusing and constructing a multi-dimensional constraint space, real-time constraint propagation and solution, and timestamp synchronous execution are used to fully realize the intelligent stage control method based on multi-device collaboration as described in claim 1, providing a complete technical implementation solution for real-time and precise collaboration between multiple stage devices and actors' free behavior.

[0050] In one embodiment, a complexity-optimized constraint propagation algorithm is used to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control period, including: Obtain all constraints in the multidimensional fusion constraint space and the state variable domain of each stage execution device; Based on the number of constraints associated with each state variable domain, the state variable domains are dynamically sorted, and the state variable domain with the most associated constraints is given priority in being assigned a value. A forward checking strategy is adopted. After each state variable field is assigned a value, all constraints related to the state variable field are checked immediately, and values ​​that conflict with the current assignment in subsequent state variable fields are deleted in advance. If an assignment is detected to cause any state variable field to become an empty set, a backtracking mechanism is triggered to return to the previous assignment point and select again. At the end of the preset control cycle, at least one set of device state sequences that satisfy all constraints is output as a feasible state sequence.

[0051] The implementation is as follows: First, the central controller obtains all constraints (including actor dynamic spatiotemporal constraints, equipment physical response characteristic constraints, and actor-stage execution equipment bidirectional constraint relationships) from the multi-dimensional fusion constraint space. It then initializes the state variable domain for each stage execution device. The state variable domain refers to the set representing all possible values ​​of each stage execution device maintained during the constraint propagation algorithm's solution process. Its initial range is determined by the device's physical response characteristic constraints, such as the rotation angle range of lighting equipment or the reachable space range of mechanical equipment. For example, for an LED moving head light, its state variable domain can be represented as the Cartesian product of the horizontal rotation angle range [0°, 360°], the vertical rotation angle range [-90°, 90°], and the brightness range [0%, 100%]. As the constraint propagation algorithm progresses, the values ​​in the state variable domain gradually decrease due to assignment or constraint propagation effects, ultimately shrinking to a single definite value upon successful solution or becoming an empty set upon failure.

[0052] Secondly, the central controller initiates the iterative solution process of the constraint propagation algorithm. Before each assignment attempt, the algorithm first counts the number of constraints associated with each state variable field and dynamically sorts the state variable fields according to this number, prioritizing the state variable fields with the most associated constraints. Dynamic sorting refers to the strategy of sorting the variable fields according to the number of constraints associated with each state variable field before each assignment attempt. The algorithm selects the first sorted state variable field and attempts to assign it a feasible value.

[0053] Third, after assigning a value to each state variable field, the algorithm immediately executes a forward checking strategy: it iterates through all constraints related to that state variable field, checking whether the assigned values ​​restrict other unassigned state variable fields. If some values ​​are found to be infeasible due to conflicts with the current assignment, these conflicting values ​​are prematurely removed from the corresponding state variable field. If any state variable field becomes empty during the deletion process, it indicates that the current assignment path will inevitably lead to no solution, and the algorithm immediately triggers a backtracking mechanism. The forward checking strategy can detect potential conflicts early in the search, avoiding wasting search time on subsequent branches that are bound to fail, thus significantly reducing the number of backtracking attempts and computation time.

[0054] Fourth, when the backtracking mechanism is triggered, the algorithm abandons the current assignment attempt, returns to the previous assignment point, and selects other feasible values ​​from the previous assignment point to continue exploring. The algorithm records the reason and position of each backtracking to avoid repeatedly exploring the same invalid paths. Through repeated assignment-forward checking-backtracking iterations, the algorithm gradually narrows the range of each state variable domain until it finds a set of assignment combinations that satisfy all constraints. The backtracking mechanism is the foundation for the constraint propagation algorithm to ensure completeness. By recording the assignment history and conflict reasons, it can systematically explore the solution space until a feasible solution is found or it is proven that no solution exists.

[0055] Fifth, at the end of the preset control cycle, if the algorithm successfully finds at least one set of assignments that satisfy all constraints, the set of assignments is organized into a time series according to the control sub-step (e.g., 10ms) to generate a feasible state sequence containing the device state at multiple time points; if no feasible solution is found within the cycle, the constraint relaxation mechanism is triggered or the historical feasible solution with the smallest deviation from the current constraint is selected as an alternative.

[0056] For example, following the previous example, the central controller starts the constraint propagation algorithm to solve the problem at t=15.34 seconds. The state variable domains are initialized as follows: Light 1 has a horizontal rotation angle domain of [0°, 360°], a vertical rotation angle domain of [-90°, 90°], and a brightness domain of [0%, 100%]; the No. 2 hanging system has a position domain of [5.0, 10.0] × [-5.0, 5.0] × [1.0, 8.0], and a speed domain of [-0.8, 0.8] m / s. The algorithm counts the number of constraints associated with each variable domain: Light 1 has 8 constraints, and the No. 2 hanging system has 6 constraints. Light 1 is assigned a value first. The first assignment selects a horizontal angle of 125°. Forward checking reveals conflicts with certain values ​​of the vertical angle, so 5 conflicting values, from -30° to -20°, are preemptively removed from the vertical angle domain.

[0057] After multiple assignment-propagation-backtracking iterations, the algorithm finds a state assignment that satisfies all constraints within 30ms: Light 1 is at a horizontal angle of 125°, a vertical angle of 42°, and a brightness of 80% at t=15.35 seconds; Hanging object 2 is at a position (5.3, -2.1, 1.6) and moving upwards at a speed of 0.2m / s at t=15.35 seconds. The feasible state sequence is then output at 10ms intervals.

[0058] It should be noted that the specific implementation of the aforementioned backtracking mechanism is not limited to simple backtracking. In one example, conflict-directed backjumping can be used to directly skip multiple layers of invalid assignment points based on the cause of the conflict, which is suitable for scenarios with complex constraint relationships and frequent backtracking. In another example, an algorithm based on intelligent backtracking can be used to further improve backtracking efficiency by recording the conflict set and calculating the backtracking level.

[0059] The specific implementation methods of the above-mentioned dynamic sorting of variable fields are not limited to the examples given. In one example, a weighted sorting based on constraint propagation strength can be used, which considers not only the number of constraints but also the strictness of the constraints (such as equality constraints having a higher weight than inequality constraints), and is suitable for scenarios with higher requirements for solution accuracy. In another example, a dynamic adaptive sorting based on historical backtracking frequency can be used, which assigns higher priority to variable fields that frequently cause backtracking, and is suitable for scenarios with large fluctuations in solution difficulty.

[0060] The specific implementation of the aforementioned forward checking strategy is not limited to the examples given. In one example, a full forward check can be used, which checks all relevant constraints and removes all conflicting values ​​for each new assignment, suitable for scenarios with tight constraints and small variable domains. In another example, a restricted forward check can be used, which only checks key constraints directly related to the current assignment, suitable for real-time scenarios with extremely high computational efficiency requirements.

[0061] Any algorithm optimization method that can efficiently solve the device state sequence that satisfies all constraints within a preset control period is within the protection scope of this invention.

[0062] This invention optimizes the constraint propagation algorithm through a "dynamic sorting-forward checking-backtracking mechanism" algorithm chain: The state variable domain is dynamically sorted according to constraint density, prioritizing the search for key variables with the greatest impact on the solution space to improve pruning efficiency; a forward checking strategy is employed to immediately detect conflicts and delete invalid branches after each assignment; and a backtracking mechanism is used to systematically avoid unsolvable paths and re-explore when an empty set is detected. This significantly improves the solution efficiency and success rate of the constraint propagation algorithm within a preset control period, solving the technical problem of traditional algorithms timeouts or lack of solutions due to excessively large search spaces in complex constraint scenarios, and providing efficient and reliable solution assurance for real-time and precise collaboration between actors and equipment.

[0063] In one embodiment, the method further includes: Real-time monitoring of the actors' rate of change in movement speed and curvature of movement trajectory; The preset control cycle is adjusted jointly based on the rate of change of motion speed and the curvature of motion trajectory: when the rate of change of motion speed is higher than a first threshold and the curvature of motion trajectory is higher than a second threshold, the preset control cycle is shortened to the first cycle value; when the rate of change of motion speed is lower than a third threshold and the curvature of motion trajectory is lower than a fourth threshold, the preset control cycle is extended to the second cycle value. Wherein, the first period value is less than the second period value, and both the first period value and the second period value are less than the minimum time constant of the actor's motion state change.

[0064] The implementation is as follows: First, within each control cycle, the central controller calculates the rate of change of the actor's velocity and the curvature of the trajectory at each moment, based on the real-time captured continuous motion trajectory stream of the actor. The rate of change of velocity refers to the degree of drastic change in the actor's velocity over time, used to quantify the acceleration fluctuations of the actor's motion. It can be obtained by differencing the velocity vectors at consecutive moments, for example, Δv / Δt = |v(t) - v(t-Δt)| / Δt. The curvature of a motion trajectory refers to the degree of bending of an actor's trajectory at a certain point. It is used to quantify the rate of change of the actor's direction of movement and can be calculated from three consecutive trajectory points. Specifically, it is the curvature of an arc determined by three points. For example, for three consecutive trajectory points P(t-Δt), P(t), and P(t+Δt), an arc can be determined by the three points, with curvature κ = 2 * sin(θ) / d, where θ is the angle between vectors P(t-Δt)P(t) and P(t)P(t+Δt), and d is the distance between P(t-Δt) and P(t+Δt). The larger the curvature value, the more severe the trajectory bending.

[0065] Secondly, the central controller compares the calculated rate of change of motion speed and curvature of motion trajectory with four preset thresholds. If the current rate of change of motion speed is detected to be higher than the first threshold and the curvature of motion trajectory is simultaneously higher than the second threshold, it is determined that the actor is currently in a state of violent movement (such as rapid rotation or sudden acceleration), and higher frequency control and following are required.

[0066] Third, when the judgment result indicates a state of intense movement, the central controller will shorten the preset control cycle from its current value to a smaller first cycle value. The first cycle value is the dynamically shortened control cycle value, which is suitable for scenes with intense actor movement. The shortened control cycle means that more constraint solving and instruction updates are performed per unit time, thereby tracking the actor's rapid changes more intensively. The specific value of the first cycle can be set according to the equipment's responsiveness and the type of performance; for example, the first cycle value can be set to 20ms.

[0067] Fourth, if the detected rate of change of current motion speed is lower than the third threshold and the curvature of the motion trajectory is simultaneously lower than the fourth threshold, it is determined that the actor is currently in a smooth motion state (such as slow walking or static performance), and the control frequency can be appropriately reduced. In this case, the central controller extends the preset control cycle to a larger second cycle value. The second cycle value is the dynamically extended control cycle value, suitable for scenarios with smooth actor movement, reducing unnecessary computational overhead and optimizing system resource allocation. The specific value of the second cycle can be set according to the device's responsiveness and the performance type; for example, the second cycle value could be 50ms.

[0068] Fifth, the central controller uses the adjusted control cycle as the runtime parameter for the constraint propagation algorithm, continuously performing subsequent real-time solutions and instruction generation. All adjusted cycle values ​​must satisfy the constraint of being less than the minimum time constant for changes in the actor's motion state. The minimum time constant refers to the smallest time scale at which a significant change in the actor's motion state occurs, used to define the upper limit of the control cycle. This upper limit can be obtained through historical data analysis of the actor's motion or preset according to the performance type (e.g., 50ms for dance performances, 100ms for theatrical performances). Limiting the preset control cycle to a range less than the minimum time constant ensures that the system completes at least one complete perception-solution-execution loop before the actor's motion state changes, meeting the basic requirements for dynamic system control.

[0069] For example, continuing from the previous example, actor A walks slowly from t=15.34 to 15.40 seconds, with a rate of change of speed of 0.1 m / s² and a trajectory curvature of 0.05 m. - ¹, all are below the preset thresholds (0.3m / s², 0.1m). - ¹), the central controller determines that the motion is smooth and extends the control cycle from 50ms to 40ms.

[0070] Starting at t=15.40 seconds, actor A begins to rotate rapidly, with the rate of change of velocity increasing to 1.0 m / s² and the curvature of the trajectory increasing to 0.8 m. - ¹, all are above the threshold (0.8 m / s², 0.5 m). - ¹), the controller immediately shortens the cycle to 20ms. With a 20ms cycle, light number 1 follows the rotation trajectory at a higher frequency, achieving precise synchronization.

[0071] After the actor resumes slow walking at t=15.50 seconds, the cycle returns to 40ms.

[0072] It should be noted that the methods for determining the threshold parameters mentioned above are not limited to the examples given. In one example, a fixed threshold can be used, preset according to the performance type (e.g., a higher threshold for dance performances and a lower threshold for drama performances), which is suitable for scenarios with relatively fixed performance styles. In another example, an adaptive threshold can be used, dynamically updating the threshold parameters based on the actor's historical motion data, which is suitable for scenarios with varied performance styles that require personalized adaptation.

[0073] It should be noted that the specific implementation methods for the dynamic adjustment of the control cycle mentioned above are not limited to the examples given. Based on the continuity of the adjustment strategy, they can be summarized into the following two categories of implementation methods: Discrete adjustment method: Suitable for scenarios where moderate smoothness of adjustment is required. In one example, discrete level adjustment can be used, with multiple preset period values ​​(such as 20ms, 30ms, 40ms, 50ms), switching between levels according to the intensity of the motion; in another example, a rule-based leveling strategy can be used, dividing the rate of change of motion speed and trajectory curvature into several levels, with each level corresponding to a fixed period value.

[0074] Continuous adjustment method: Suitable for fine control scenarios requiring high smoothness of adjustment. In one example, continuous function mapping can be used to map the rate of change of motion speed and the curvature of motion trajectory into continuous control cycle values; in another example, fuzzy logic control can be used to fuzzify motion parameters into multiple levels and then make comprehensive decisions through fuzzy rules to output continuous cycle adjustment amounts.

[0075] It should be noted that the method for determining the minimum time constant is not limited to the examples given. In one example, a statistical method can be used to perform spectral analysis on the actor's historical motion data and take the reciprocal of the frequency component with the highest energy as the minimum time constant; in another example, an empirical preset can be used, directly setting a fixed value based on the performance type and the characteristics of the actor.

[0076] Any technical means that can dynamically adjust the control cycle according to the actor's movement state is within the scope of protection of this invention.

[0077] This invention employs a dual-factor joint monitoring mechanism of "velocity change rate - trajectory curvature" to perceive the intensity of the actor's movement in real time and dynamically adjust the preset control cycle. During intense movement, the cycle is shortened to enhance tracking density, while during gentle movement, the cycle is extended to optimize resource utilization. Simultaneously, the adjusted cycle value is constrained to a range smaller than the minimum time constant of the actor's movement state change, ensuring that the control sampling frequency meets the basic requirements of dynamic system control and avoiding signal distortion due to insufficient sampling. Thus, a dynamic optimization mechanism for control parameters, based on "motion perception - cycle adaptation - sampling guarantee," is constructed. This mechanism links the control cycle to the actor's movement state in real time, enabling on-demand allocation of computational resources in the time dimension. It solves the "dilemma" of fixed control cycles when dealing with dynamically changing movements: too short a cycle leads to wasted computational resources, while too long a cycle results in insufficient tracking accuracy. This significantly enhances the real-time performance and adaptability of the method in complex and ever-changing stage scenarios while ensuring tracking accuracy.

[0078] In one embodiment, the dynamic spatiotemporal constraints are fused with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fused constraint space, including: The dynamic spatiotemporal constraints are analyzed to extract the spatial position constraint interval and the allowable value range of velocity and acceleration of the actor at each moment; Call the pre-stored physical parameter library of each stage execution device to obtain the maximum movement speed, maximum acceleration, minimum response delay, and reachable space range of each stage execution device; The spatial constraints of the actors' positions and the reachable space of each stage execution device are spatially aligned to determine the intersection area between the actors and each stage execution device in space. The permissible range of the actor's speed and acceleration is kinematically matched with the maximum speed and acceleration of the stage equipment to determine the compatibility between the actor's motion requirements and the boundary of the equipment's motion capabilities. The time constraints in the dynamic spatiotemporal constraints of the actors are matched with the minimum response delay of each stage execution device to determine whether the response delay of the stage execution device meets the time accuracy requirements of the actors' movements. Based on spatial intersection, kinematic matching results, and temporal synchronization matching results, actor-stage execution device pairs that have interactive relationships in the same spatial region at the same time are identified. For each identified actor-stage execution device pair, a two-way constraint relationship is established: the actor's spatial position constrains the selection of the stage execution device's operating area, and the stage execution device's response capability constrains the upper limit of the actor's motion parameters, generating a multi-dimensional fusion constraint space that includes the two-way constraint relationship between actor and stage execution device.

[0079] The implementation is as follows: First, the central controller parses and extracts the core constraint information of the actor at each time t from the dynamic spatiotemporal constraints, including: 1) the spatial position constraint interval, which is represented as a spatial region centered on the coordinate point P(t) and bounded by a preset tolerance radius r; 2) kinematic parameter constraints, including the allowable range of velocity values ​​[v_min(t), v_max(t)] and the allowable range of acceleration values ​​[a_min(t), a_max(t)]; 3) the time constraints implicit in the spatiotemporal coordinates, i.e., the position requirements of the actor at time t.

[0080] Secondly, the central controller calls the pre-stored physical parameter library of each stage execution device from the system database to obtain the key physical response characteristic parameters of each stage execution device, including: maximum motion speed V_max (such as light rotation speed, mechanical lifting speed), maximum acceleration A_max, minimum response delay τ (the time interval from receiving the instruction to starting execution), and reachable space range S (such as the physical motion range of the device, rotation angle limit, light spot / sound beam coverage area).

[0081] Third, spatial alignment is performed. Spatial alignment refers to the process of matching and comparing the spatial position constraint intervals of the actors with the reachable spatial ranges of each stage execution device in the same three-dimensional spatial coordinate system. Its core is calculating the intersection of the actor's constraint intervals and the reachable space of the devices. The central controller calculates the spatial intersection of the actor's spatial position constraint interval at each time t with the reachable spatial range S of each stage execution device, determining whether the actor's position P(t) is within the device's reachable space S, or whether there is an overlap between the actor's constraint interval and S. If there is an overlap, the device is recorded as a spatial candidate device, and the spatial boundary parameters of the overlapping area are determined; if there is no overlap, the device is directly excluded, and no further matching is performed.

[0082] Fourth, for the candidate devices selected through spatial alignment, kinematic matching is performed. Kinematic matching refers to the process of comparing the kinematic parameter constraints of the actor with the motion capability boundaries of the device. The central controller compares the actor's kinematic constraints with the device's motion capability boundaries: if v_max(t) ≤ V_max and a_max(t) ≤ A_max, then the device's motion capability meets the actor's motion requirements and is recorded as kinematically compatible; if not, the device is excluded from the candidate set.

[0083] Fifth, for devices that have undergone spatial alignment and kinematic matching, time synchronization matching is performed. Time synchronization matching refers to the process of matching the actor's time constraints with the device's minimum response delay. The core is to determine whether the device, after receiving the instruction at time t and executing it at time t+τ after a response delay τ, can still be synchronized with the actor's actual position at time t+τ. The central controller predicts the actor's position P_pred(t+τ) at time t+τ based on the actor's current motion state. The prediction method can be a linear prediction based on the current velocity: P_pred(t+τ) = P(t) + v(t)·τ. It is then determined whether the predicted position is still within the device's reachable space S. If yes, the device's response delay meets the time accuracy requirements; otherwise, the device is excluded.

[0084] Sixth, by combining spatial intersection, kinematic matching results, and temporal synchronization matching results, the central controller identifies actor-stage execution equipment pairs that pass the three-dimensional screening. An actor-stage execution equipment pair refers to a combination of actors and stage execution equipment confirmed to have an interactive relationship at the same time and in the same spatial area; it is the basic unit for establishing subsequent bidirectional constraint relationships. The matching in the three dimensions has a logical "AND" relationship: only actor-equipment pairs that simultaneously satisfy the three conditions of spatial intersection, kinematic compatibility, and temporal synchronization feasibility are ultimately identified as valid pairs.

[0085] Seventh, establish a two-way constraint relationship for each identified actor-stage execution equipment pair. This two-way constraint relationship refers to the mutual restriction and limitation between the actor and the stage execution equipment, encompassing two directions: Positive constraint: The spatial position of the actor constrains the selection of the area of ​​action of the stage execution equipment. That is, the stage execution equipment must adjust its area of ​​action to be near the actor's position, which can be called "spatial position constraint on the actor side".

[0086] Reverse constraint: The responsiveness of the stage equipment constrains the upper limit of the actor's motion parameters, that is, the actor's speed and acceleration cannot exceed the capability boundary of the stage equipment, which can be called "equipment-side responsiveness constraint".

[0087] Finally, the aforementioned bidirectional constraint relationship, dynamic spatiotemporal constraint conditions, and independent physical response characteristic constraints of stage execution equipment are uniformly incorporated to generate a multidimensional fusion constraint space that includes the bidirectional constraint relationship between actors and stage execution equipment.

[0088] For example, following the previous example, based on the dynamic spatiotemporal constraints of actor A at t=15.34 seconds (position (5.3,-2.1,1.5)±0.1 meters, speed 2.0 m / s), the central controller calls the equipment parameters and performs a three-dimensional screening: Spatial alignment: The actor's position falls within the coverage area of ​​light 1 (X∈[0,20],Y∈[-15,15]) and the coverage area of ​​hanging beam 2 (X∈[5,10],Y∈[-5,5],Z∈[0,8]), but not within the coverage area of ​​sound beam 3, so sound beam 3 is excluded; Kinematic matching: The actor's maximum speed of 2.2 m / s and maximum acceleration of 0.6 m / s² exceed the capabilities of hoisting device #2 (maximum speed 0.8 m / s, maximum acceleration 0.5 m / s²), therefore hoisting device #2 is excluded; Time synchronization matching: Light No. 1 has a response delay of 15ms. It is predicted that the location (5.33, -2.1, 1.5) will still be within the coverage area after 15ms. Matching is successful.

[0089] Ultimately, light number 1 was identified as an interactive device, and a two-way constraint was established: the light spot of light number 1 follows the actor's position, and the actor's movement is limited by the tracking capability of light number 1, which is incorporated into the multi-dimensional fusion constraint space.

[0090] It should be noted that the specific implementation methods of the above spatial alignment are not limited to geometric intersection testing. In one example, a fast indexing method based on spatial hashing can be used to divide the stage space into a three-dimensional grid, pre-calculate the grid cells covered by each device, and realize fast lookup of actor positions to devices, which is suitable for large stage scenes with a large number of devices; in another example, a precise calculation method based on distance field can be used, which is suitable for scenes with extremely high spatial accuracy requirements and where the complex shapes of devices need to be considered.

[0091] The specific implementation of the aforementioned kinematic matching is not limited to strict numerical comparison. In one example, soft constraint matching can be used, which allows for exceeding the capability boundary under certain conditions but performs degradation processing (such as reducing light brightness in exchange for faster response), suitable for scenarios where artistic effects are more important than physical accuracy; in another example, the acceleration and deceleration characteristics of the device can be considered, and acceleration constraints can be introduced for more accurate matching.

[0092] The specific implementation methods of the aforementioned time synchronization matching are not limited to linear prediction. In one example, state prediction based on Kalman filtering can be used, which is suitable for scenarios with strong motion randomness and a need to improve prediction accuracy; in another example, multi-step prediction based on model predictive control can be used, which is suitable for scenarios with complex motion patterns and a need for multi-step planning in advance to compensate for large response delays.

[0093] Any technical means that can achieve three-dimensional matching of actors and equipment is within the scope of protection of this invention.

[0094] This invention, through a three-dimensional fusion matching mechanism of "spatial alignment - kinematic matching - temporal synchronization matching," constructs a complete system for judging the interaction probability of actors and stage execution equipment from three levels: space, motion, and time. This achieves a precise mode of "accurately identifying actor-stage execution equipment pairs that genuinely have interaction needs," significantly reducing the size of the variable domain that the subsequent constraint propagation algorithm needs to process, and significantly improving the efficiency and accuracy of constraint solving. Furthermore, by establishing bidirectional constraint relationships, the traditional one-way "equipment follows actor" mode is transformed into a bidirectional "actor-equipment mutual adaptation" mode, laying a solid constraint foundation for achieving real-time and precise collaboration.

[0095] In one embodiment, the method further includes: Extract the bidirectional constraint relationship for each actor-stage execution device pair from the multidimensional fusion constraint space. The bidirectional constraint relationship includes the spatial position constraint expression of the actor and the response capability constraint expression of the stage execution device. The spatial location constraint expression and the response capability constraint expression are coupled to eliminate coupling variables and generate a joint constraint expression. The joint constraint expression is discretized along the time axis according to the preset control period to generate a sequence of discrete constraint functions that can be solved independently in each preset control period; The discrete constraint function sequence is encapsulated into coupled constraint functions and stored in the dynamic constraint library of the constraint solver, so that the constraint propagation algorithm can call it in real time within the corresponding preset control period to solve for feasible state sequences that satisfy all constraint conditions.

[0096] The implementation is as follows: First, the central controller traverses each identified actor-stage execution device pair from the multi-dimensional fusion constraint space and extracts the corresponding bidirectional constraint relationship. For example, the bidirectional constraint relationship includes the actor's spatial position constraint expression C_actor(P, v, a, t) and the device's response capability constraint expression C_device(P_device, θ,t), where P is the actor's position, v and a are the actor's motion parameters, P_device is the device's position parameter, θ is the device's state parameter (such as the lighting angle), and t is time.

[0097] Secondly, coupling processing is performed on each set of bidirectional constraint relationships. The central controller analyzes the coupling variables (such as time t and actor position P) in the two expressions, eliminates these variables through algebraic transformations or numerical methods, and generates a joint constraint expression C_joint(P, v, a, P_device, θ). This joint expression directly describes the functional relationship that the actor's motion parameters and the equipment state parameters must satisfy under the premise of satisfying the bidirectional constraints, eliminating intermediate coupling variables and making the form more compact. For example, the core of the coupling processing is variable elimination: by simultaneously solving the actor's spatial position constraint P_device(t) = P_actor(t) and the equipment response capability constraint P_actor(t) = P_actor(t-Δt) + v_actor(t)·Δt, we can obtain P_device(t) = P_actor(t-Δt) + v_actor(t)·Δt, eliminating the intermediate variable P_actor(t), and obtaining the direct relationship between the equipment position and the actor's historical position and velocity.

[0098] Third, the generated joint constraint expression is discretized along the time axis according to a preset control cycle. Time axis discretization refers to the process of sampling and segmenting the joint constraint expression defined in the continuous time domain according to the preset control cycle. The central controller, starting from the current time t0, divides the continuous time domain [t0, t0 + N·T] (T is the control cycle, N is the pre-calculated cycle number) into N discrete time intervals. For each interval [t0 + k·T, t0 + (k+1)·T], based on the predicted motion of the actor within that interval, a discrete constraint function C_k(P_actor(t_k), v_actor(t_k), P_device, θ) is generated that is only applicable to that cycle. These discrete functions are independent of each other and can be solved individually within their respective cycles.

[0099] Fourth, the generated sequence of discrete constraint functions is encapsulated according to a standard interface format to form a coupled constraint function module. Each coupled constraint function accepts the current actor's state parameters (such as position and velocity) and the period number as input, and outputs a set of constraints that the device must satisfy within that period. The encapsulated functions adopt a unified calling specification, which facilitates the dynamic loading and use of the constraint propagation algorithm.

[0100] Fifth, the encapsulated coupling constraint functions are stored in the dynamic constraint library. The dynamic constraint library is a runtime data structure stored in memory to store pre-computed coupling constraint functions, organized using actor-stage execution device pair identifiers as keys and corresponding coupling constraint function pointers as values. At the beginning of each control cycle, the constraint propagation algorithm quickly searches for and loads the corresponding coupling constraint function from the dynamic constraint library based on the currently active actor-stage execution device pair identifiers, serving as the constraint input for the current cycle. The dynamic constraint library can be updated using a lazy strategy: the corresponding coupling constraint functions are only recalculated and updated when the actor's motion pattern changes significantly or the device parameters are adjusted.

[0101] Sixth, within each control cycle, the constraint propagation algorithm calls the coupled constraint functions loaded from the dynamic constraint library and, in conjunction with other global constraints, solves for the feasible state sequence of each stage execution device that satisfies all constraints. Since the coupled constraint functions have already undergone coupling processing and discretization, the algorithm does not need to repeatedly perform complex expression simplification and variable elimination during runtime. It can directly perform iterative solutions of assignment-propagation-backtracking, significantly improving solution efficiency.

[0102] For example, continuing from the previous example, the bidirectional constraints between actor A and light number 1 are coupled to obtain joint constraints: the speed of light number 1 is ≤2.5m / s, the acceleration is ≤1.2m / s², and the position is associated with the actor's historical trajectory. The current control period T=40ms, and the discrete constraint functions for the next 5 periods (N=5) are pre-calculated, encapsulated as coupled constraint functions, and stored in the dynamic constraint library.

[0103] At the start of a new cycle at t=15.38 seconds, the constraint propagation algorithm needs to solve for the feasible state of light 1 within the cycle [t=15.38, t=15.42]. The algorithm searches for the coupled constraint function corresponding to actor A-1 light in the dynamic constraint library, and after loading, directly obtains the constraint conditions that light 1 must satisfy within this cycle: P_light(t)=(5.5,-1.8,1.6)+2.0·(t-15.38) and speed≤2.5m / s. Based on these constraints, the algorithm quickly completes the assignment and solution, generating the rotation angle sequence of light 1 within the cycle. This preprocessing avoids repeated analytical coupling discretization in each cycle, significantly improving real-time performance.

[0104] It should be noted that the specific implementation of the above coupling processing is not limited to symbolic computation. In one example, approximate elimination based on numerical optimization can be used, which is suitable for scenarios with complex constraints that are difficult to express analytically; in another example, machine learning-based methods can be used to learn the coupling relationships, which is suitable for scenarios with a large amount of historical data.

[0105] The specific implementation of the aforementioned time axis discretization is not limited to uniform discretization. In one example, adaptive discretization based on motion prediction can be used, which densifies the discrete points when the actor's movement is intense and sparses the discrete points when the movement is gentle, making it suitable for scenarios that require both computational accuracy and efficiency. In another example, event-triggered discretization can be used, which regenerates the discrete constraint function only when the actor's motion state changes significantly, making it suitable for scenarios with relatively fixed motion patterns.

[0106] The storage and invocation methods of the aforementioned dynamic constraint library are not limited to in-memory databases. In one example, distributed cache storage can be used, which is suitable for scenarios with a large number of actor-device pairs that require horizontal scaling; in another example, a pre-compiled static function library can be used, which is suitable for scenarios where the actor-stage execution device pair relationship is fixed and ultimate operational efficiency is pursued.

[0107] Any technical means that can preprocess bidirectional constraint relationships into coupled constraint functions that can be invoked in real time is within the scope of protection of this invention.

[0108] This invention, through a technical logic chain of "extraction-coupling-discretization-encapsulation," preprocesses complex bidirectional constraint relationships into standardized coupled constraint functions, shifting repetitive computational tasks such as expression simplification, variable elimination, and discretization required at runtime to the preprocessing stage. This significantly reduces the computational load of the constraint propagation algorithm in each control cycle, substantially improving solution speed and success rate, and providing efficient and reliable constraint invocation support for real-time and precise collaboration between actors and equipment.

[0109] In one embodiment, please refer to Figure 2 , Figure 2 This is a schematic diagram of the first sub-process of the intelligent stage control method based on multi-device collaboration provided in an embodiment of the present invention. Figure 2 As shown, in this embodiment, the continuous motion trajectory flow is directly encoded into dynamic spatiotemporal constraints that evolve over time, including: S21: Obtain the three-dimensional coordinate sequence of the actor's skeleton points at each moment in the continuous motion trajectory stream; S22: Based on the three-dimensional coordinate sequence of the actor's skeletal points, calculate the actor's instantaneous velocity vector and instantaneous acceleration vector at each moment; S23: Convert the three-dimensional coordinate sequence of the actor's skeletal points into spatial position constraints. The spatial position constraints mean that the actor must be located in a spatial region centered on the coordinate point P(t) and bounded by a preset tolerance radius r at every time t. S24: Convert the instantaneous velocity vector and the instantaneous acceleration vector into kinematic parameter constraints, which are expressed as follows: the actor's velocity at each time t must be within the interval [v_min(t), v_max(t)], and the acceleration must be within the interval [a_min(t), a_max(t)]. S25: Align the spatial position constraints and the kinematic parameter constraints along the time axis to generate corresponding dynamic spatiotemporal constraints.

[0110] The implementation is as follows: First, the central controller receives continuous motion trajectory data from the actor tracking sensors in real time and obtains the three-dimensional coordinate sequence of the actor's skeletal points at each moment. The three-dimensional coordinate sequence of the actor's skeletal points refers to the set of position data of the actor's key skeletal points in the three-dimensional spatial coordinate system arranged in chronological order. Each skeletal point data includes three-dimensional coordinates (x, y, z) and a global timestamp t. The skeletal points can be tracked as a single point (such as the center of the waist) or multiple points according to the performance requirements.

[0111] Secondly, the central controller calculates the instantaneous velocity vector and instantaneous acceleration vector of the actor at each moment based on the three-dimensional coordinate sequence of the actor's skeletal points. The instantaneous velocity vector refers to the first-order kinematic parameter describing the speed and direction of the actor's movement, which can be calculated using the central difference method: v(t) = (P(t+Δt) - P(t-Δt)) / (2Δt); the instantaneous acceleration vector refers to the second-order kinematic parameter describing the rate of change of velocity, which can be calculated using the second-order position difference method: a(t) = (P(t+Δt) - 2P(t)+ P(t-Δt)) / Δt². Outliers can be smoothed using Kalman filtering during the calculation process.

[0112] Third, the central controller converts the three-dimensional coordinate sequence of the actor's skeletal points into spatial position constraints. Specifically, the spatial position constraints can be represented as a spherical region centered on the trajectory point coordinates P(t) and bounded by a preset tolerance radius r: |P_actor(t) - P(t)| ≤ r. The tolerance radius r is preset according to the sensor accuracy and performance requirements, with a typical value of 0.1-0.2 meters.

[0113] Fourth, the central controller converts the velocity and acceleration vectors into kinematic parameter constraints. These constraints limit the range of values ​​for the actor's velocity and acceleration, including the velocity constraint v_min(t) ≤ |v_actor(t)| ≤ v_max(t) and the acceleration constraint a_min(t) ≤ |a_actor(t)| ≤ a_max(t). These ranges can be determined using tolerance coefficients: v_min(t) = |v(t)|·(1-α_v), v_max(t) = |v(t)|·(1+α_v), where α_v is between 0.1 and 0.2. The acceleration range is similar.

[0114] Fifth, the central controller aligns the spatial position constraints and kinematic parameter constraints at the same time t along the time axis to generate dynamic spatiotemporal constraints containing timestamp t, position constraints (P(t), r), velocity interval [v_min, v_max], and acceleration interval [a_min, a_max], which serve as inputs for subsequent constraint fusion steps.

[0115] For example, continuing from the previous example, actor A's trajectory points at t=15.32-15.36 seconds are: (5.2,-2.3,1.5), (5.4,-2.2,1.5), (5.5,-1.9,1.6), (5.3,-1.8,1.6), (5.0,-1.9,1.5), with a sampling interval of 0.01 seconds. The central controller calculates the instantaneous velocity at t=15.34 seconds to be approximately 2.6 m / s and the acceleration to be approximately 0.6 m / s². After smoothing with a Kalman filter, a spatial position constraint of (5.5,-1.9,1.6)±0.1 meters is generated, with a velocity range of [2.10,3.14] m / s and an acceleration range of [0.50,0.74] m / s². After alignment, dynamic spatiotemporal constraints are generated.

[0116] This invention, through a technical logic link of "three-dimensional coordinate sequence of skeletal points - kinematic parameters - spatiotemporal constraints," directly encodes the actor's original motion trajectory into a computable constraint expression, skipping the feature extraction and intent recognition stages in traditional technologies. This avoids information loss caused by feature abstraction and delays introduced by model inference. Thus, it achieves lossless encoding of the actor's free behavior—any improvisation by the actor is preserved intact within the constraints, rather than being "corrected" by the predictive model. This provides the most realistic and complete behavioral input for subsequent constraint fusion and real-time solution, fundamentally ensuring the complete preservation of the actor's degree of freedom and the accuracy of equipment tracking.

[0117] In one embodiment, capturing a continuous motion trajectory stream of an actor in real time includes: The actor's multi-view depth image sequence is acquired synchronously by N depth vision sensors deployed on the stage, where N is a natural number greater than 2; Perform 3D reconstruction on the multi-view depth image sequence and extract the 3D spatial coordinates of the actor's key skeletal points in each frame image; Kalman filtering is applied to the three-dimensional spatial coordinates to remove acquisition noise and instantaneous jump points, generating a smooth and continuous motion trajectory flow; A global timestamp is attached to each trajectory point in the continuous motion trajectory stream, and the global timestamp is strictly synchronized with the unified system timeline.

[0118] The implementation is as follows: First, deploy N depth vision sensors around the stage. Depth vision sensors are imaging devices capable of simultaneously acquiring color images and depth information of the scene, such as depth cameras based on time-of-flight, structured light, or binocular stereo vision principles. All sensors are connected to the central controller's synchronization signal generator via hardware trigger lines, receiving unified trigger pulse control to ensure coverage of the entire performance area. Before the performance begins, all depth vision sensors are calibrated to obtain their intrinsic parameters (such as focal length, principal point coordinates, and distortion coefficients) and extrinsic parameters (such as position and orientation) in a unified three-dimensional coordinate system. Existing related technologies can be referenced for this process, which will not be elaborated upon here.

[0119] Secondly, the central controller sends a synchronization trigger signal at a fixed frequency. All depth vision sensors can acquire a depth image frame at the same moment they receive the trigger signal, forming a multi-view depth image sequence. A multi-view depth image sequence refers to a set of depth images captured simultaneously from N different viewpoints. Each depth vision sensor transmits its image data to the central controller in real time.

[0120] Third, the central controller performs 3D reconstruction processing on the received multi-view depth image sequence. The core of 3D reconstruction is triangulation: for the same skeleton point pixels (u1, v1) and (u2, v2) matched in at least two view images, the projection matrices P1 and P2 of each sensor are combined to solve the equation system P1·X = s1·[u1, v1, 1]^T and P2·X = s2·[u2, v2, 1]^T to obtain the 3D coordinates X. When there are more than two effective viewpoints, the least squares method can be used to solve for the optimal 3D coordinates, improving reconstruction accuracy and robustness.

[0121] Fourth, the central controller can input the continuous time-series skeleton point coordinates obtained from the 3D reconstruction into a Kalman filter for smoothing. The Kalman filter model used in this invention is designed as follows: the state vector x = [p_x, p_y, p_z, v_x, v_y, v_z]^T, containing 3D position and 3D velocity; the state transition model assumes uniform motion, x_k = F·x_{k-1} + w, where F is a 6×6 matrix; the observation model z_k = H·x_k + v, where H is a 3×6 observation matrix, the observation is 3D position, and the process noise covariance matrix Q and the observation noise covariance matrix R can be preset according to sensor characteristics and the intensity of motion. Through prediction-update iterative calculation, the optimal estimate is output, effectively removing acquisition noise and instantaneous jump points caused by occlusion.

[0122] Fifth, the central controller adds a global timestamp to each smoothed trajectory point. The global timestamp is strictly synchronized with the unified system timeline, accurately recording the moment the point was acquired, with a time accuracy down to the microsecond level. For scenarios requiring compensation for transmission delay, a hardware timestamp based on the IEEE 1588 precise time protocol can be used to directly mark the acquisition time at the sensor end.

[0123] For example, following the previous example, taking a large-scale live performance as an example, 16 depth vision sensors are deployed around the stage, covering the entire performance area. The central controller sends a synchronization trigger signal at a frequency of 60Hz, and all sensors simultaneously acquire a depth image frame at t=15.32 seconds. The three-dimensional coordinates of the actor's hip center are reconstructed through triangulation as (5.2, -2.3, 1.5). The coordinate sequence of consecutive moments is input into a Kalman filter, and after smoothing, the output is (5.192, -2.285, 1.502), effectively suppressing measurement noise. Finally, a global timestamp of t=15.32 seconds is added to the trajectory points, generating trajectory point data (5.192, -2.285, 1.502, t=15.32) for subsequent processing.

[0124] It should be noted that the specific implementation methods of the above-mentioned 3D reconstruction are not limited to triangulation. In one example, a voxel-based method can be used to divide the space into a 3D mesh, and the position of the skeleton points can be determined by the projection consistency voting of each viewpoint, which is suitable for scenes with a lot of occlusion; in another example, an end-to-end 3D pose estimation method based on deep learning can be used to directly regress the coordinates of 3D skeleton points from multi-view images, which is suitable for scenes with high requirements for computational efficiency.

[0125] It should be noted that the specific implementation of the Kalman filter described above is not limited to linear Kalman filtering. In one example, extended Kalman filtering can be used to handle nonlinear motion, suitable for scenes with complex actor motion patterns; in another example, particle filtering can be used to handle non-Gaussian noise and multimodal distributions, suitable for scenes with severe occlusion or data jumps. Any filtering algorithm capable of smoothing trajectory sequences and suppressing noise falls within the scope of protection of this invention.

[0126] This invention, through multi-view synchronous acquisition and 3D reconstruction, first achieves precise spatial positioning of the actor's motion trajectory. Based on this, Kalman filtering is used to smooth the 3D coordinates, effectively removing acquisition noise and instantaneous jump points, generating a continuous and smooth trajectory flow. Furthermore, by attaching a global timestamp synchronized with the unified system timeline to each trajectory point, precise alignment of the trajectory data on a unified time reference is achieved. Combining these interconnected effects, a high-quality trajectory capture chain of "multi-view synchronous acquisition - 3D reconstruction - Kalman filtering - timestamp synchronization" is constructed, providing a reliable data foundation for subsequent "direct encoding" and fundamentally ensuring the accuracy and stability of the real-time collaborative control system.

[0127] In one embodiment, when there are multiple actors on stage, the method further includes: Simultaneously capture the continuous motion trajectory stream of each actor and assign a unique identifier to each actor; The continuous motion trajectory flow of each actor is directly encoded into the corresponding dynamic spatiotemporal constraints. When constructing a multi-dimensional fusion constraint space, mutual constraint conditions between actors are added. These mutual constraint conditions include: minimum safe distance constraints between any two actors, relative position constraints in multi-person interaction scenarios, and consistency constraints of group movement. In the multidimensional fusion constraint space, the feasible state sequence of each stage execution device is solved simultaneously to satisfy all dynamic constraints of actors, mutual constraints between actors, and physical constraints of stage execution devices.

[0128] The implementation is as follows: First, the central controller initiates a multi-actor synchronous capture process. Based on the aforementioned multi-view depth vision sensor system, the central controller performs multi-target detection and tracking on each frame of multi-view images. Specifically, the DeepSORT multi-target tracking algorithm can be used to detect multiple actors in each frame, assign a temporary identifier to each detected actor, and maintain identity consistency through cross-frame matching (based on position and appearance features).

[0129] Secondly, a unique identifier is assigned to each actor detected for the first time. The central controller maintains a global actor table, recording the identifiers and states of currently active actors. When a new actor enters the stage, the next available identifier is assigned (e.g., incrementing by an integer 1, 2, 3, ...); when an actor leaves the stage, the identifier is released for subsequent reuse. For scenarios requiring long-term identity retention, a long-term re-identification mechanism based on appearance features can be used, restoring the original identifier when the actor re-enters after a brief absence.

[0130] Third, the continuous motion trajectory stream of each actor is input into the aforementioned direct encoding module to generate corresponding dynamic spatiotemporal constraints for each actor independently. Actor A generates constraint set C_A(t), actor B generates constraint set C_B(t), and so on. Each of these constraint sets contains the actor's own spatial position constraints and kinematic parameter constraints, which is exactly the same as the processing method in the single actor scene.

[0131] Fourth, when constructing the multi-dimensional fusion constraint space, the central controller adds mutual constraint conditions between actors according to performance requirements. These mutual constraint conditions include the following three categories: (1) Minimum safe distance constraint: expressed as |P_i(t) - P_j(t)| ≥ d_safe, used to prevent collisions between actors. Where d_safe is a preset safe distance threshold, which can be adjusted according to the performance type, for example, 0.5 meters for ballet and 1.0 meters for martial arts scenes.

[0132] (2) Relative position constraint: expressed as |P_j(t) - P_i(t) - ΔP_ij(t)| ≤ ε_rel, used to maintain a specific formation or interaction relationship. Where ΔP_ij(t) is the preset relative offset (which can change with time), and ε_rel is the allowed position deviation.

[0133] (3) Group motion consistency constraint: expressed as |v_i(t) - v_G(t)| ≤ ε_v, i∈G is used to ensure the overall coordination of group movements. Here, v_G(t) is the group reference speed (which can be the average value or the speed of the lead dancer), and ε_v is the allowable speed deviation.

[0134] Fifth, the central controller incorporates all the above constraints to construct an extended multi-dimensional fusion constraint space containing multiple actors and multiple devices. The variables include the position coordinates, velocity, and acceleration of each actor, as well as the state parameters of each stage execution device. The constraints include the dynamic constraints of each actor, the bidirectional constraint relationship between actors and devices, and the mutual constraint conditions between actors.

[0135] Sixth, in this extended multi-dimensional fusion constraint space, the central controller performs joint solving using the aforementioned constraint propagation algorithm. The variable domain of the constraint propagation algorithm is extended to the joint space of all actor state variables and all device state variables. The dynamic sorting strategy needs to consider the constraint densities on both the actor side and the device side simultaneously, and preferentially attempt to assign values to the variable domain with the largest number of associated constraints. Forward checking needs to propagate constraints between all variable domains. Since the device-side constraints have been pre-computed as coupled constraint functions as described above, and most of the mutual constraints are linear constraints, the constraint propagation algorithm can still complete the solving within the preset control period. The obtained feasible state sequence includes device collaboration instructions for all actors (such as multiple lights following different actors respectively), and a device resource allocation plan that may need to be switched between different actors.

[0136] Exemplarily, continuing from the previous example, assume that there are three actors performing on the stage simultaneously: Actor A (the protagonist), Actor B (the backup dancer), and Actor C (the prop operator). The central controller synchronously captures the movement trajectories of the three through the multi-view sensor system. At the first detection, Actor A is assigned ID = 1, Actor B is assigned ID = 2, and Actor C is assigned ID = 3. At t = 15.34 seconds, the positions of the three are as follows: Actor A (ID = 1): (5.3, -2.1, 1.5); Actor B (ID = 2): (6.0, -1.5, 1.5); Actor C (ID = 3): (4.0, -3.0, 1.5); The central controller encodes the trajectory streams of the three into independent dynamic spatio-temporal constraint conditions respectively.

[0137] When constructing the multi-dimensional fusion constraint space, the following mutual constraints are added: Minimum safety distance constraint: Preset d_safe = 0.8 meters. The current distance between A and B is 0.92 meters, the distance between A and C is 1.58 meters, and the distance between B and C is 2.5 meters, all of which meet the requirements.

[0138] Relative position constraint: The director requires Actor B to be the backup dancer of Actor A and maintain a relative offset ΔP = (0.5, 0.3, 0) meters, that is, P_B(t) = P_A(t) + (0.5, 0.3, 0).

[0139] Group motion consistency constraint: It is required that the velocity directions of the three are the same, that is, v_A(t)·v_B(t) ≥ 0.9·|v_A(t)|·|v_B(t)| (the included angle of the velocity directions is less than 25 degrees), and similar constraints are applied to other pairs of actors.

[0140] The central controller constructs a multi-dimensional fusion constraint space that includes dynamic constraints for three people, bidirectional constraints between the three people and multiple devices (such as light 1 following actor A, light 2 following actor B, and sound 3 positioning the group center), and the aforementioned mutual constraints.

[0141] The constraint propagation algorithm begins solving the problem. Dynamic sorting reveals that actor A has the most variable-related constraints (self-dynamic constraints + bidirectional constraints with light #1 + relative position constraints with actor B + group consistency constraints), so it is assigned a value first. After assigning actor A's position (5.3, -2.1, 1.5), a forward check is triggered: according to the relative position constraint, actor B's position must be (5.8, -1.8, 1.5); the minimum safe distance constraint is checked, and this position is 2.38 meters away from actor C, which is safe; however, the group consistency constraint requires all three actors to have the same speed direction, requiring further coordination.

[0142] After multiple rounds of assignment-propagation-backtracking, the algorithm solves for the feasible state sequence for the next cycle (15.35-15.39 seconds) within 45ms: Light #1 tracks actor A, in the position sequence (5.3→5.5→5.7...); Light number 2 tracks actor B, in the position sequence (5.8→6.0→6.2...); Light number 3 tracks actor C, in the position sequence (4.0→4.2→4.4...); The three moved at the same speed and direction, moving as a whole to the right and forward.

[0143] The generated timestamped instruction set is distributed to various devices, enabling real-time and precise collaboration in multi-actor scenes. Viewers observed that the lighting for the three actors perfectly followed their respective roles, while the actors maintained a graceful formation.

[0144] It should be noted that the specific implementation methods of the above-mentioned multi-target tracking are not limited to DeepSORT. In one example, a detection-by-detection paradigm can be used, first detecting the actor's position in each frame, and then performing cross-frame association using the Hungarian algorithm, which is suitable for scenarios with a relatively fixed number of actors. In another example, an end-to-end method of joint detection and tracking (such as JDE) can be used, simultaneously outputting detection results and trajectory associations, which is suitable for scenarios with higher real-time requirements. Any technical means that can simultaneously track multiple actors and maintain identity consistency is within the protection scope of this invention.

[0145] The specific implementation of the aforementioned minimum safe distance constraint is not limited to a static fixed threshold. In one example, a dynamic safe distance can be used, which is adjusted according to the actor's current speed: d_safe = d_base + k·|v|. The faster the speed, the larger the safe distance, which is suitable for high-speed motion scenes such as martial arts. In another example, a direction-dependent safe distance can be used, which reserves more space in the direction of movement and is suitable for scenes that require more precise collision avoidance.

[0146] The specific form of the aforementioned relative position constraints is not limited to fixed offsets. In one example, a dynamic offset ΔP_ij(t) that varies over time can be used, driven by trajectory data pre-arranged by the director, suitable for choreography scenarios requiring precise timing control; in another example, an elastic constraint |P_j(t) - P_i(t) - ΔP_ij(t)| ≤ ε_rel can be used, allowing for fluctuations within a certain range, suitable for scenarios in improvisational performances where maintaining a general formation is required.

[0147] The specific implementation of the above-mentioned group motion consistency constraint is not limited to the consistency of velocity direction. In one example, the position consistency constraint |P_i(t) - P_center(t)| ≤ R can be used, requiring the actors to stay within a circle with radius R centered on the group center, which is suitable for choreography performances; in another example, the phase consistency constraint can be used, requiring the actors to move in synchronized rhythms, which is suitable for dance choreography scenarios.

[0148] This invention addresses the technical problems of traditional control methods in multi-actor scenarios, such as collision risks, disorganized formations, and group incoordination, by independently encoding each actor, ensuring individual behavioral freedom, ensuring group coordination through mutual constraints, and achieving overall optimal collaboration through joint solution. This significantly improves the applicability and expressiveness of this method in complex multi-person performance scenarios such as large-scale group dances, martial arts scenes, and group gymnastics.

[0149] In one embodiment, the method further includes: If a feasible state sequence that satisfies all constraints cannot be solved within the preset control period, the strictness of the constraints is relaxed step by step according to the preset priority order. The preset priority order is as follows: first, relax the upper limit of velocity and acceleration in the kinematic parameter constraints; second, relax the tolerance radius of the spatial position constraints; and finally, relax the response time requirements of the stage execution equipment. After relaxing each level of constraint, try to solve the feasible state sequence again; If a solution still cannot be found after all priority relaxations, select the historical feasible solution that deviates the least from the current constraints as an alternative and generate an alarm message.

[0150] The implementation is as follows: First, the central controller performs a no-solution check at the end of each preset control cycle. If the aforementioned constraint propagation algorithm fails to find any feasible state sequence that satisfies all constraints within the preset control cycle (i.e., the algorithm still finds no solution after searching the entire solution space, or the algorithm times out without finding a solution), then the current cycle is determined to have "failed to solve," triggering the constraint relaxation mechanism. The constraint relaxation mechanism refers to the automatic process of gradually relaxing constraints when no feasible solution can be found. By strategically relaxing some non-core constraints, the system maintains the synergistic effect as much as possible while ensuring continuous system operation.

[0151] Secondly, the first-level constraint relaxation is initiated according to a preset priority order. This preset priority order is the pre-defined sequence of constraint relaxations, designed based on the principle of prioritizing constraints with minimal impact on audience perception and actor safety, and finally relaxing constraints that may affect the reliability of equipment response. The specific order could be: First priority: Relax the upper limits of velocity and acceleration in the kinematic parameters. Moderately relaxing these constraints will have a relatively small impact on the audience's visual perception and will not involve equipment safety.

[0152] Second priority: Relax the tolerance radius of spatial position constraints. Moderately relaxing it will result in a slight decrease in following accuracy, but it is still within an acceptable range.

[0153] Third priority: Relaxing the response time requirements of stage execution equipment. Excessive relaxation may lead to accumulated delays and affect the collaborative effect of subsequent cycles. Therefore, it is the last priority.

[0154] Third, perform first-level constraint relaxation and retry the solution. The central controller multiplies the upper limit of velocity v_max(t) in the kinematic parameter constraints of all actors in the current cycle by the relaxation coefficient α_v (α_v>1), and the upper limit of acceleration a_max(t) by the relaxation coefficient α_a (α_a>1), generating new relaxed constraints. The relaxation coefficient can be preset to a fixed increment (e.g., 1.2 times) or dynamically adjusted according to the number of failures (e.g., increasing by 10% each time). At the same time, the lower limits v_min(t) and a_min(t) are adjusted accordingly to maintain the reasonableness of the interval. The system records the current relaxation level as "Level 1" and uses the relaxed constraints to re-call the constraint propagation algorithm for solution. If the solution is successful, the solution result is output as a feasible state sequence, and it is recorded that constraint relaxation was used this time.

[0155] Fourth, if the first level of relaxation still fails to solve the problem, the second level of constraint relaxation is executed. While retaining the results of the first level of relaxation, the central controller multiplies the tolerance radius *r* of the spatial position constraint by a relaxation factor *β* (β>1), for example, relaxing it from 0.1 meters to 0.15 meters, 0.2 meters, etc. The relaxation increment for each level can be preset according to the stage size. The system updates the relaxation level to "Level 2" and calls the constraint propagation algorithm again to attempt a solution.

[0156] Fifth, if the solution still cannot be found after the second level of relaxation, the third level of constraint relaxation is executed. Based on the results of the first two levels of relaxation, the central controller relaxes the stage execution device response time requirement from "must ≤ minimum response delay" to allowing a certain delay deviation. Specifically, the constraint on device response delay is changed from the hard constraint τ_actual ≤ τ_min to the soft constraint τ_actual ≤ τ_min + δ, where δ is the allowed additional delay, which can be gradually increased (e.g., starting from 5ms and increasing by 5ms each time). The system updates the relaxation level to "Level 3" and calls the constraint propagation algorithm again to attempt a solution.

[0157] Sixth, if a solution still cannot be found after all three levels of constraint relaxation, an alternative historical solution is initiated. The historical feasible solution refers to the sequence of feasible states of each stage execution device that has been successfully solved and verified in several consecutive control cycles prior to the current control cycle. These are stored in a circular buffer with a capacity of K (e.g., K=10). When an alternative solution needs to be selected, the system calculates the applicability of each historical solution at the current time: score_k = w1·|P_actor(t) - P_pred_k(t)| + w2·|v_actor(t) - v_pred_k(t)| + w3·device state difference, where P_pred_k(t) is the actor position predicted by historical solution k at time t. The historical solution with the smallest score is selected as the alternative solution, i.e., the solution that deviates the least from the current constraints.

[0158] Seventh, generate alarm information. The alarm information refers to the abnormal notification generated by the central controller to alert technical personnel when the system initiates the constraint relaxation mechanism and still cannot solve the problem after all priority relaxations have been performed. It includes at least: a timestamp, the actor-equipment pair that caused the unsolvable problem, the current constraint status, the relaxation level already executed, the identifier of the adopted alternative solution, and suggested measures. The system will distribute and execute the alternative solution as the equipment control instruction set for this control cycle to ensure uninterrupted performance.

[0159] For example, continuing from the previous example, suppose actor A enters a rapid spinning dance at t=15.42 seconds, with the speed of movement rising sharply to 3.5m / s and the acceleration reaching 1.2m / s².

[0160] The central controller runs the constraint propagation algorithm within the current period [t=15.42, t=15.46], attempting to find a feasible state sequence for LED moving head light No. 1 (maximum tracking speed 2.5m / s) to follow actor A. After searching, the algorithm finds that actor A's speed of 3.5m / s far exceeds the equipment's maximum speed of 2.5m / s, and its acceleration of 1.2m / s² exceeds the equipment's maximum acceleration of 0.8m / s², indicating that no solution exists, thus triggering the constraint relaxation mechanism.

[0161] First-level relaxation: The upper limit of velocity is relaxed to 2.5×1.2=3.0m / s, and the upper limit of acceleration is relaxed to 0.8×1.2=0.96m / s². The solution is solved again, but there is still no solution.

[0162] Second-level relaxation: The tolerance radius is relaxed from 0.1 meters to 0.1 × 1.3 = 0.13 meters, and the solution is solved again, but there is still no solution.

[0163] Level 3 relaxation: The response time requirement is relaxed from "must be within 15ms" to allow an additional 5ms delay (within 20ms), and the solution is recalculated, but there is still no solution.

[0164] Initiate historical alternatives: Query the historical solutions for the 5 most recent successful cycles and calculate their applicability. At t=15.38, the predicted position is (5.8, -1.5, 1.6), which is 0.5 meters away from the actual position (6.2, -1.2, 1.7). At t=15.36, the predicted position is (5.5, -1.8, 1.6), with a distance of 0.8 meters. At t=15.34, the predicted position is (5.2, -2.1, 1.5), with a distance of 1.1 meters. The solution with the smallest distance at t=15.38 was chosen as the alternative and executed.

[0165] An alarm message was generated: "Time 15.42 seconds, Actor A's speed of 3.5 m / s exceeds the capacity of light #1 (maximum 2.5 m / s). After three levels of relaxation, there is still no solution. The historical solution at t=15.38 is used as a substitute. It is recommended to check whether the actor's movement is abnormal or whether the equipment performance has degraded." Although the cycle was not perfectly followed, the performance was not interrupted by the historical alternative. In the next cycle (t=15.46), the velocity of actor A dropped to 2.8 m / s, and the system resumed normal solution.

[0166] It should be noted that the above priority order design is not limited to the examples given. In one example, the priority can be dynamically adjusted according to the performance type, such as making the safety distance constraint the highest priority in an acrobatic performance; in another example, machine learning-based adaptive prioritization can be introduced, learning from historical data which constraints have the least impact on the performance and relaxing them first.

[0167] The selection strategy for the aforementioned historical feasible solutions is not limited to location distance. In one example, a selection based on device state similarity can be used, prioritizing the historical solution closest to the current device state, which is suitable for scenarios where device response characteristics change significantly. In another example, a selection based on motion trend matching can be used, considering the continuity of actor movement, selecting the historical solution most consistent with the current motion trend, which is suitable for performance types with strong motion regularity.

[0168] The specific increment for the aforementioned constraint relaxation is not limited to a fixed coefficient. In one example, an adaptive increment can be used to dynamically adjust the relaxation magnitude based on the degree of solution failure; in another example, a progressive increment can be used, with each relaxation level consisting of multiple sub-steps, suitable for fine control scenarios requiring smooth transitions.

[0169] Any technical means that can achieve orderly relaxation of constraints, selection of alternative solutions, and generation of alarms when there is no solution is within the scope of protection of this invention.

[0170] In this embodiment of the invention, by actively detecting and responding to solution failures and triggering a constraint relaxation mechanism when a solution cannot be found, the historical feasible solution with the smallest deviation from the current constraint conditions can be selected as an alternative, ensuring the continuity of the performance without interruption. Thus, even in extreme cases, basic functions can be maintained through constraint relaxation and historical substitution, avoiding performance interruption, providing a solid technical guarantee for the stable operation of large-scale commercial performances, and significantly improving the robustness and reliability of this method.

[0171] The stage intelligent control method based on multi-device collaboration described in the above embodiments can be recombined with the technical features included in different embodiments as needed to obtain a combined implementation scheme, but all are within the protection scope claimed by this invention.

[0172] In one embodiment, a stage intelligent control system based on multi-device collaboration is provided, which corresponds one-to-one with the stage intelligent control method based on multi-device collaboration described in the above embodiments. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic block diagram of a stage intelligent control system based on multi-device collaboration, provided as an embodiment of the present invention. Figure 3 As shown, the stage intelligent control system 30 based on multi-device collaboration is applied to a server. The stage intelligent control system 30 based on multi-device collaboration includes: a first establishment module 31, a first encoding module 32, a first construction module 33, a first solution module 34, and a first control module 35. The above functional modules are described in detail below: The first module 31 is used to establish a unified spatiotemporal reference, including: establishing a three-dimensional spatial coordinate system as a spatial reference, and synchronizing the local clocks of the actor tracking sensor, the central controller and each stage execution device to a unified system time axis as a time reference. The first encoding module 32 is used to capture the continuous motion trajectory flow of the actor in the three-dimensional spatial coordinate system in real time, and directly encode the continuous motion trajectory flow into dynamic spatiotemporal constraints that evolve over time. The dynamic spatiotemporal constraints include the actor's spatial position constraints and kinematic parameter constraints at each moment. The first construction module 33 is used to integrate the dynamic spatiotemporal constraints with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fusion constraint space. The first solution module 34 is used to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control period in the multi-dimensional fusion constraint space using a constraint propagation algorithm with optimized complexity, and generate a set of device control instructions with execution timestamps. The first control module 35 is used to distribute the set of device control instructions to each stage execution device and execute them synchronously according to the execution timestamp.

[0173] In one embodiment, the first solving module 34 includes: The first acquisition submodule is used to acquire all constraints in the multidimensional fusion constraint space and the state variable domains of each stage execution device; The first assignment submodule is used to dynamically sort the state variable fields according to the number of constraints associated with each state variable field, and to prioritize assigning values ​​to the state variable fields with the most associated constraints. The first checking submodule is used to adopt a forward checking strategy. After each state variable field is assigned a value, it immediately checks all constraints related to the state variable field and deletes values ​​in subsequent state variable fields that conflict with the current assignment in advance. The first triggering submodule is used to trigger the backtracking mechanism and return to the previous assignment point to reselect if the assignment is detected to cause any state variable field to become an empty set. The first output submodule is used to output at least one set of device state sequences that satisfy all constraints as feasible state sequences at the end of the preset control cycle.

[0174] In one embodiment, the stage intelligent control system 30 further includes: The first monitoring module is used to monitor the rate of change of the actor's movement speed and the curvature of the movement trajectory in real time. The first adjustment module is used to jointly adjust the preset control period based on the rate of change of the motion speed and the curvature of the motion trajectory: when the rate of change of the motion speed is higher than a first threshold and the curvature of the motion trajectory is higher than a second threshold, the preset control period is shortened to the first period value; when the rate of change of the motion speed is lower than a third threshold and the curvature of the motion trajectory is lower than a fourth threshold, the preset control period is extended to the second period value. Wherein, the first period value is less than the second period value, and both the first period value and the second period value are less than the minimum time constant of the actor's motion state change.

[0175] In one embodiment, the first building module 33 includes: The first extraction submodule is used to parse the dynamic spatiotemporal constraints and extract the spatial position constraint interval and the allowable value range of velocity and acceleration of the actor at each moment. The second acquisition submodule is used to call the pre-stored physical parameter library of each stage execution device to obtain the maximum motion speed, maximum acceleration, minimum response delay and reachable space range of each stage execution device; The first alignment submodule is used to spatially align the spatial position constraint range of the actors with the reachable spatial range of each stage execution device, and determine the spatial intersection area between the actors and each stage execution device. The first matching submodule is used to perform kinematic matching between the allowable range of the actor's speed and acceleration and the maximum speed and acceleration of the stage execution equipment, and to determine the compatibility between the actor's motion requirements and the boundary of the equipment's motion capabilities. The second matching submodule is used to time-synchronize and match the time constraints in the dynamic spatiotemporal constraints of the actors with the minimum response delay of each stage execution device to determine whether the response delay of the stage execution device meets the time accuracy requirements of the actors' movements. The first identification submodule is used to identify actor-stage execution device pairs that have an interactive relationship in the same spatial area at the same time, based on spatial intersection, kinematic matching results and temporal synchronization matching results. The first submodule is used to establish a two-way constraint relationship for each identified actor-stage execution device pair: the actor's spatial position constrains the selection of the stage execution device's operating area, and the stage execution device's response capability constrains the upper limit of the actor's motion parameters, generating a multi-dimensional fusion constraint space that includes the two-way constraint relationship between actor and stage execution device.

[0176] In one embodiment, the stage intelligent control system 30 further includes: The first extraction module is used to extract the bidirectional constraint relationship of each actor-stage execution device pair from the multidimensional fusion constraint space. The bidirectional constraint relationship includes the spatial position constraint expression of the actor and the response capability constraint expression of the stage execution device. The first generation module is used to couple the spatial location constraint expression and the response capability constraint expression, eliminate coupling variables, and generate a joint constraint expression. The second generation module is used to discretize the joint constraint expression according to the preset control period on the time axis, and generate a sequence of discrete constraint functions that can be solved independently in each preset control period. The first storage module is used to encapsulate the discrete constraint function sequence into coupled constraint functions and store them in the dynamic constraint library of the constraint solver, so that the constraint propagation algorithm can call them in real time within the corresponding preset control period to solve for the feasible state sequence that satisfies all constraint conditions.

[0177] In one embodiment, the first encoding module 32 includes: The third acquisition submodule is used to acquire the three-dimensional coordinate sequence of actor skeleton points at each moment in the continuous motion trajectory stream; The first calculation submodule is used to calculate the instantaneous velocity vector and instantaneous acceleration vector of the actor at each moment based on the three-dimensional coordinate sequence of the actor's skeletal points. The first conversion submodule is used to convert the three-dimensional coordinate sequence of the actor's skeletal points into spatial position constraints. The spatial position constraints mean that the actor must be located in a spatial region centered on the coordinate point P(t) and bounded by a preset tolerance radius r at every time t. The second conversion submodule is used to convert the instantaneous velocity vector and the instantaneous acceleration vector into kinematic parameter constraints. The kinematic parameter constraints are expressed as follows: the actor's velocity at each time t must be within the interval [v_min(t), v_max(t)], and the acceleration must be within the interval [a_min(t), a_max(t)]. The second alignment submodule is used to align the spatial position constraints and the kinematic parameter constraints along the time axis to generate corresponding dynamic spatiotemporal constraints.

[0178] In one embodiment, the first encoding module 32 includes: The first acquisition submodule is used to synchronously acquire multi-view depth image sequences of actors through N depth vision sensors deployed on the stage, where N is a natural number greater than 2. The second extraction submodule is used to perform three-dimensional reconstruction on the multi-view depth image sequence and extract the three-dimensional spatial coordinates of the actor's key skeletal points in each frame image. The first generation submodule is used to perform Kalman filtering on the three-dimensional spatial coordinates to remove acquisition noise and instantaneous jump points, and generate a smooth continuous motion trajectory flow. The first additional submodule is used to attach a global timestamp to each trajectory point in the continuous motion trajectory stream, and the global timestamp is strictly synchronized with the unified system time axis.

[0179] In one embodiment, when there are multiple actors on stage, the stage intelligent control system 30 further includes: The first capture module is used to synchronously capture the continuous motion trajectory stream of each actor and assign a unique identifier to each actor; The second encoding module is used to directly encode the continuous motion trajectory flow of each actor into the corresponding dynamic spatiotemporal constraints. The first addition module is used to add mutual constraint conditions between actors when constructing a multi-dimensional fusion constraint space. The mutual constraint conditions include: minimum safe distance constraint between any two actors, relative position constraint in multi-person interaction scenario, and consistency constraint of group movement. The first solving module 34 is used to simultaneously solve the feasible state sequence of each stage execution device in the multi-dimensional fusion constraint space, which satisfies all dynamic constraints of actors, mutual constraints between actors, and physical constraints of stage execution devices.

[0180] In one embodiment, the stage intelligent control system 30 further includes: The first relaxation module is used to relax the strictness of the constraints step by step according to a preset priority order if a feasible state sequence that satisfies all constraints cannot be solved within the preset control period. The preset priority order is as follows: first relax the upper limit of velocity and acceleration in the kinematic parameter constraints, then relax the tolerance radius of the spatial position constraints, and finally relax the response time requirements of the stage execution equipment. The first attempt module is used to retry solving the feasible state sequence after each level of constraint is relaxed; The first fault-tolerant module is used to select the historical feasible solution that deviates the least from the current constraints as an alternative if the solution still cannot be found after all priority relaxations are performed, and to generate an alarm message.

[0181] Specific limitations regarding the multi-device collaborative stage intelligent control system can be found in the limitations of the multi-device collaborative stage intelligent control method described above, and will not be repeated here. Each module in the aforementioned multi-device collaborative stage intelligent control system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0182] Those skilled in the art will understand that the methods and systems provided in the embodiments of the present invention can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The methods and systems can also be implemented as computer program products stored in one or more computer-readable storage media, including but not limited to: disks, optical disks, read-only memory (ROM), random access memory (RAM), flash memory, etc. When the computer program product is executed by one or more data processing devices (such as computers), the devices perform the steps as described in any of the preceding method embodiments.

[0183] Software tools, components, or models not belonging to this company that appear in the embodiments of this invention are merely illustrative examples and do not represent actual use. The data collection methods used in the embodiments of this invention comply with relevant laws and regulations, such as the "Data Security Law of the People's Republic of China," the "Personal Information Protection Law of the People's Republic of China," GDPR (General Data Protection Regulation of the European Union), or information security standards of other countries and regions.

[0184] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A stage intelligent control method based on multi-device collaboration, characterized in that, include: Establishing a unified spatiotemporal reference includes: establishing a three-dimensional spatial coordinate system as a spatial reference, and synchronizing the local clocks of the actor tracking sensors, the central controller, and each stage execution device to a unified system time axis as a time reference; In the three-dimensional spatial coordinate system, the continuous motion trajectory flow of the actor is captured in real time, and the continuous motion trajectory flow is directly encoded into dynamic spatiotemporal constraints that evolve over time. The dynamic spatiotemporal constraints include the actor's spatial position constraints and kinematic parameter constraints at each moment. The dynamic spatiotemporal constraints are fused with the physical response characteristics constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fused constraint space; In the multidimensional fusion constraint space, a constraint propagation algorithm with optimized complexity is used to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control period, and generate a set of device control instructions with execution timestamps. Based on the execution timestamp, the set of device control instructions is distributed to each stage execution device and executed synchronously.

2. The intelligent stage control method based on multi-device collaboration as described in claim 1, characterized in that, A constraint propagation algorithm with optimized complexity is used to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control period, including: Obtain all constraints in the multidimensional fusion constraint space and the state variable domain of each stage execution device; Based on the number of constraints associated with each state variable domain, the state variable domains are dynamically sorted, and the state variable domain with the most associated constraints is given priority in being assigned a value. A forward checking strategy is adopted. After each state variable field is assigned a value, all constraints related to the state variable field are checked immediately, and values ​​that conflict with the current assignment in subsequent state variable fields are deleted in advance. If an assignment is detected to cause any state variable field to become an empty set, a backtracking mechanism is triggered to return to the previous assignment point and select again. At the end of the preset control cycle, at least one set of device state sequences that satisfy all constraints is output as a feasible state sequence.

3. The intelligent stage control method based on multi-device collaboration as described in claim 2, characterized in that, The method further includes: Real-time monitoring of the actors' rate of change in movement speed and curvature of movement trajectory; The preset control cycle is adjusted jointly based on the rate of change of motion speed and the curvature of motion trajectory: when the rate of change of motion speed is higher than a first threshold and the curvature of motion trajectory is higher than a second threshold, the preset control cycle is shortened to the first cycle value; when the rate of change of motion speed is lower than a third threshold and the curvature of motion trajectory is lower than a fourth threshold, the preset control cycle is extended to the second cycle value. Wherein, the first period value is less than the second period value, and both the first period value and the second period value are less than the minimum time constant of the actor's motion state change.

4. The intelligent stage control method based on multi-device collaboration as described in claim 1, characterized in that, The dynamic spatiotemporal constraints are fused with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fused constraint space, including: The dynamic spatiotemporal constraints are analyzed to extract the spatial position constraint interval and the allowable value range of velocity and acceleration of the actor at each moment; Call the pre-stored physical parameter library of each stage execution device to obtain the maximum movement speed, maximum acceleration, minimum response delay, and reachable space range of each stage execution device; The spatial constraints of the actors' positions and the reachable space of each stage execution device are spatially aligned to determine the intersection area between the actors and each stage execution device in space. The permissible range of the actor's speed and acceleration is kinematically matched with the maximum speed and acceleration of the stage equipment to determine the compatibility between the actor's motion requirements and the boundary of the equipment's motion capabilities. The time constraints in the dynamic spatiotemporal constraints of the actors are matched with the minimum response delay of each stage execution device to determine whether the response delay of the stage execution device meets the time accuracy requirements of the actors' movements. Based on spatial intersection, kinematic matching results, and temporal synchronization matching results, actor-stage execution device pairs that have interactive relationships in the same spatial region at the same time are identified. For each identified actor-stage execution device pair, a two-way constraint relationship is established: the actor's spatial position constrains the selection of the stage execution device's operating area, and the stage execution device's response capability constrains the upper limit of the actor's motion parameters, generating a multi-dimensional fusion constraint space that includes the two-way constraint relationship between actor and stage execution device.

5. The intelligent stage control method based on multi-device collaboration as described in claim 4, characterized in that, The method further includes: Extract the bidirectional constraint relationship for each actor-stage execution device pair from the multidimensional fusion constraint space. The bidirectional constraint relationship includes the spatial position constraint expression of the actor and the response capability constraint expression of the stage execution device. The spatial location constraint expression and the response capability constraint expression are coupled to eliminate coupling variables and generate a joint constraint expression. The joint constraint expression is discretized along the time axis according to the preset control period to generate a sequence of discrete constraint functions that can be solved independently in each preset control period; The discrete constraint function sequence is encapsulated into coupled constraint functions and stored in the dynamic constraint library of the constraint solver, so that the constraint propagation algorithm can call it in real time within the corresponding preset control period to solve for feasible state sequences that satisfy all constraint conditions.

6. The intelligent stage control method based on multi-device collaboration as described in claim 1, characterized in that, The continuous motion trajectory flow is directly encoded into dynamic spatiotemporal constraints that evolve over time, including: Obtain the sequence of three-dimensional coordinates of actor skeleton points at each moment in the continuous motion trajectory stream; Based on the three-dimensional coordinate sequence of the actor's skeletal points, calculate the actor's instantaneous velocity vector and instantaneous acceleration vector at each moment; The three-dimensional coordinate sequence of the actor's skeletal points is converted into spatial position constraints. The spatial position constraints mean that the actor must be located in a spatial region centered on the coordinate point P(t) and bounded by a preset tolerance radius r at every time t. The instantaneous velocity vector and the instantaneous acceleration vector are converted into kinematic parameter constraints, which are expressed as follows: the actor's velocity at each time t must be within the interval [v_min(t), v_max(t)], and the acceleration must be within the interval [a_min(t), a_max(t)]. Align the spatial position constraints and the kinematic parameter constraints along the time axis to generate corresponding dynamic spatiotemporal constraints.

7. The intelligent stage control method based on multi-device collaboration as described in claim 6, characterized in that, Real-time capture of actors' continuous motion trajectory streams, including: The actor's multi-view depth image sequence is acquired synchronously by N depth vision sensors deployed on the stage, where N is a natural number greater than 2; Perform 3D reconstruction on the multi-view depth image sequence and extract the 3D spatial coordinates of the actor's key skeletal points in each frame image; Kalman filtering is applied to the three-dimensional spatial coordinates to remove acquisition noise and instantaneous jump points, generating a smooth and continuous motion trajectory flow; A global timestamp is attached to each trajectory point in the continuous motion trajectory stream, and the global timestamp is strictly synchronized with the unified system timeline.

8. The intelligent stage control method based on multi-device collaboration as described in claim 1, characterized in that, When there are multiple actors on stage, the method further includes: Simultaneously capture the continuous motion trajectory stream of each actor and assign a unique identifier to each actor; The continuous motion trajectory flow of each actor is directly encoded into the corresponding dynamic spatiotemporal constraints. When constructing a multi-dimensional fusion constraint space, mutual constraint conditions between actors are added. These mutual constraint conditions include: minimum safe distance constraints between any two actors, relative position constraints in multi-person interaction scenarios, and consistency constraints of group movement. In the multidimensional fusion constraint space, the feasible state sequence of each stage execution device is solved simultaneously to satisfy all dynamic constraints of actors, mutual constraints between actors, and physical constraints of stage execution devices.

9. The intelligent stage control method based on multi-device collaboration as described in claim 1, characterized in that, The method further includes: If a feasible state sequence that satisfies all constraints cannot be solved within the preset control period, the strictness of the constraints is relaxed step by step according to the preset priority order. The preset priority order is as follows: first, relax the upper limit of velocity and acceleration in the kinematic parameter constraints; second, relax the tolerance radius of the spatial position constraints; and finally, relax the response time requirements of the stage execution equipment. After relaxing each level of constraint, try to solve the feasible state sequence again; If a solution still cannot be found after all priority relaxations, select the historical feasible solution that deviates the least from the current constraints as an alternative and generate an alarm message.

10. A stage intelligent control system based on multi-device collaboration, characterized in that, include: The first module is used to establish a unified spatiotemporal reference, including: establishing a three-dimensional spatial coordinate system as a spatial reference, and synchronizing the local clocks of the actor tracking sensors, the central controller and various stage execution equipment to a unified system time axis as a time reference; The first encoding module is used to capture the actor's continuous motion trajectory flow in the three-dimensional spatial coordinate system in real time, and directly encode the continuous motion trajectory flow into dynamic spatiotemporal constraints that evolve over time. The dynamic spatiotemporal constraints include the actor's spatial position constraints and kinematic parameter constraints at each moment. The first construction module is used to integrate the dynamic spatiotemporal constraints with the physical response characteristic constraints of each stage execution device pre-stored in the system to construct a multi-dimensional fusion constraint space. The first solution module is used to solve the feasible state sequence of each stage execution device that satisfies all constraints in real time within a preset control period in the multi-dimensional fusion constraint space using a constraint propagation algorithm with optimized complexity, and generate a set of device control instructions with execution timestamps. The first control module is used to distribute the set of device control instructions to each stage execution device and execute them synchronously according to the execution timestamp.

Images