Cooperative control of actuators in virtual interaction based on field perception and system

Abstract

This disclosure presents an embodiment of actuator collaborative control and system based on on-site perception in virtual interaction. A specific implementation of the method includes: acquiring user interaction data from heterogeneous sensors and real-time pose data of each actuator, generating alignment data after time synchronization; generating a desired control instruction set based on virtual scene interaction logic, and constructing a spatiotemporal constraint matrix for the actuators accordingly; performing collision detection on the motion envelopes of any two actuators within the same time window; if collision conditions are met, compensating for interference commands according to safety priority to obtain interference-free commands; performing timing compensation on the interference-free commands based on the inherent response delay of the actuators and sensor link delay, and outputting a candidate instruction set for controlling each actuator. This implementation can eliminate spatial conflicts and command timing deviations between actuators, improve the timing accuracy of multi-actuator collaborative control, and avoid the risk of collisions or boundary violations.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to an actuator collaborative control and system in virtual interaction based on on-site perception. Background Technology

[0002] Currently, immersive virtual interactive experiences, such as dark rides, immersive theaters, or large-scale escape rooms, typically require the coordinated control of multiple actuators (e.g., wire motors, moving screens, rising platforms, and special effects spray devices) to complement the virtual storyline. For controlling multiple actuators, the common approach is for a central control system to receive trigger signals from the player, generate corresponding control commands, and directly send them to each actuator to execute the appropriate actions.

[0003] However, when using the above method to control the actuator, the following technical problems often arise: First, because each actuator moves within a limited physical space and the triggering times are relatively concentrated, the motion trajectories of multiple actuators may intersect in space. Existing control methods do not consider the real-time spatial constraints between actuators. When the motion paths of actuators (e.g., two or more actuators) overlap within the same time period, there may be risks of equipment collisions or boundary violations, leading to equipment damage and posing safety hazards.

[0004] Secondly, the types of sensors used for on-site perception are diverse, and the data information obtained varies (e.g., data refresh rate and transmission delay). When asynchronous perception data is directly used to generate control commands, a mismatch may occur between the command timing and the player's actual actions. This results in inaccurate timing matching between virtual scene feedback and actuator actions, affecting the immersive experience and even causing equipment collisions or damage.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide a method, apparatus, electronic device, and computer-readable medium for actuator cooperative control in virtual interaction based on on-site perception, to solve one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide an actuator collaborative control method in virtual interaction based on on-site perception, comprising: acquiring user interaction data collected by heterogeneous sensors deployed on-site, and real-time pose data of each actuator; performing time synchronization alignment processing on the user interaction data and the real-time pose data to obtain aligned user interaction data and aligned actuator pose data; generating desired control instructions for the actuators based on the current interaction logic in the virtual scene and the aligned user interaction data to obtain a desired control instruction set; and constructing the actuator collaborative control method based on the desired control instruction set and the aligned actuator pose data. The spatiotemporal constraint matrix of the actuators is determined. Based on the spatiotemporal constraint matrix, collision interference detection is performed on the motion trajectory envelopes of any two actuators within the same time window to obtain the collision interference detection result. In response to the collision interference detection result satisfying the collision interference condition, the expected control command corresponding to the interfered actuator is compensated based on a preset safety priority list to obtain an interference-free command set. Based on the inherent response delay of each actuator and the data transmission link delay of the heterogeneous sensors, timing and phase compensation processing is performed on each interference-free command in the interference-free command set to obtain a compensated candidate command set for controlling each actuator.

[0009] Secondly, some embodiments of this disclosure provide an actuator collaborative control device for virtual interaction based on on-site perception, comprising: an acquisition unit configured to acquire user interaction data collected by heterogeneous sensors deployed on-site, and real-time pose data of each actuator; a time synchronization and alignment unit configured to perform time synchronization and alignment processing on the aforementioned user interaction data and the aforementioned real-time pose data to obtain aligned user interaction data and aligned actuator pose data; a first generation unit configured to generate desired control instructions for the actuators based on the current interaction logic in the virtual scene and the aforementioned aligned user interaction data to obtain a desired control instruction set; and a matrix construction unit configured to construct a matrix based on the aforementioned desired control instruction set and the aforementioned aligned actuator pose data. The aforementioned actuators have spatiotemporal constraint matrices; a collision detection unit is configured to perform collision interference detection on the motion trajectory envelopes of any two actuators within the same time window based on the aforementioned spatiotemporal constraint matrices, and obtain collision interference detection results; a command compensation unit is configured to compensate for the desired control command corresponding to the interfered actuator based on a preset safety priority list in response to the aforementioned collision interference detection results satisfying the collision interference condition, and obtain an interference-free command set; a compensation control unit is configured to perform timing phase compensation processing on each interference-free command in the aforementioned interference-free command set based on the inherent response delay of the aforementioned actuators and the data transmission link delay of the aforementioned heterogeneous sensors, and obtain a compensated candidate command set for controlling the aforementioned actuators.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0012] The above-described embodiments of this disclosure have the following beneficial effects: Through the actuator collaborative control method in virtual interaction based on on-site perception according to some embodiments of this disclosure, spatial conflicts between actuators can be eliminated, and instruction timing deviations caused by asynchronous and delayed data sources can be eliminated, thereby improving the timing accuracy of multi-actuator collaborative control and avoiding collisions or boundary violations during execution. Specifically, the reasons for the associated multi-actuator spatial collision risks and the mismatch between virtual feedback and physical action time are: in traditional methods, the control commands of each actuator are directly generated and issued based on trigger signals, lacking spatiotemporal joint modeling of the future motion trajectory of the actuators, and lacking unified timing alignment processing for data refresh differences and transmission delays of different sensors. This easily leads to collision risks and instruction timing deviations. Based on this, the actuator collaborative control method in virtual interaction based on on-site perception according to some embodiments of this disclosure first acquires user interaction data collected by heterogeneous sensors deployed on-site, as well as real-time pose data of each actuator. Thus, perception data reflecting player actions and pose data reflecting the current state of physical devices can be obtained, providing a complete data foundation for subsequent analysis. Secondly, the aforementioned user interaction data and real-time pose data are time-synchronized and aligned to obtain aligned user interaction data and aligned actuator pose data. This unifies the perceived data with different refresh rates and transmission delays onto the same time base, eliminating timing deviations introduced by asynchronous data sources and providing time-consistent data for subsequent instruction generation. Next, based on the current interaction logic in the virtual scene and the aligned user interaction data, expected control instructions for the actuators are generated, resulting in an expected control instruction set. Thus, according to the player's real-time behavior and plot progression rules in the virtual scene, the target action that each actuator should currently execute is determined, establishing a control mapping from the virtual layer to the physical layer, and obtaining instructions to control the actuators. Furthermore, based on the expected control instruction set and the aligned actuator pose data, a spatiotemporal constraint matrix for each actuator is constructed. This unifies the motion trajectory and spatial envelope of each actuator within a future preset time period into a unified data structure, allowing the spatiotemporal constraint relationships between multiple actuators to be quickly retrieved and compared within the same data structure (e.g., a matrix), providing an efficient retrieval basis for collision detection. Next, based on the aforementioned spatiotemporal constraint matrix, collision interference detection is performed on the motion trajectory envelopes of any two actuators within the same time window to obtain the collision interference detection results. Thus, by traversing the envelope space parameters of each actuator at the same time sampling point in the matrix, it can be determined whether there is spatial overlap (or the minimum distance is lower than the safe distance threshold). This allows for pre-identification of collision risks before command issuance, reducing the risk of collisions or boundary violations after command execution.Then, in response to the collision interference detection results satisfying the collision interference conditions, the expected control commands corresponding to the interfered actuators are compensated based on a preset safety priority list to obtain a set of interference-free commands. Thus, when a collision risk is detected between actuators, the commands of low-priority actuators can be delayed or rolled back according to their safety priorities, while the commands of high-priority actuators remain unchanged, eliminating spatial conflicts between actuators and maintaining the continuity of critical actuator actions. Finally, based on the inherent response delays of each actuator and the data transmission link delays of the heterogeneous sensors, each interference-free command in the set of interference-free commands undergoes timing and phase compensation to obtain a compensated candidate command set for controlling the actuators. Therefore, considering the differences in electromechanical response hysteresis between different actuators and the differences in link delays between different sensors, the total compensation amount is determined and the timing of each command is adjusted, ensuring that the actual action time of each actuator in physical space aligns with the expected trigger time in the virtual scene. This eliminates asynchronous actions of multiple actuators due to delay differences and avoids sensory fragmentation (e.g., "sound first, water mist later"). Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of an actuator collaborative control method in virtual interaction based on on-site perception, according to the present disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of an actuator collaborative control device in virtual interaction based on on-site perception, according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0017] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0018] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0019] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0020] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of an actuator cooperative control method in virtual interaction based on scene perception, according to the present disclosure. This actuator cooperative control method in virtual interaction based on scene perception includes the following steps: Step 101: Obtain user interaction data collected by heterogeneous sensors deployed on site, as well as real-time pose data of each actuator.

[0022] In some embodiments, the execution subject (e.g., an electronic device) of the above-described actuator collaborative control method based on field perception in virtual interaction can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules to provide distributed services, or as a single software or software module. No specific limitations are made here.

[0023] In some embodiments, the aforementioned actuator can acquire user interaction data collected by heterogeneous sensors deployed on-site, as well as real-time pose data of each actuator. These heterogeneous sensors can be different types of sensing devices deployed on-site. For example, they may include, but are not limited to: motion capture cameras, pressure-sensitive floors, infrared gratings, radar positioning devices, controllers, and haptic wearable devices. The aforementioned user interaction data can be data generated after the player's actions in the physical space are captured by sensors. For example, they may include, but are not limited to: the player's position coordinates, limb movement sequences, step trigger signals, and voice commands. The aforementioned actuators can be electromechanical devices deployed on-site to generate physical feedback. For example, they may include, but are not limited to: wire motors, movable screens, lifting platforms, special effects spray devices, and vibration platforms. The aforementioned real-time pose data can be the position and attitude parameters of the actuators in the physical space at the current moment.

[0024] As an example, the aforementioned execution entity can establish communication connections with each sensor and each actuator through a local area network, continuously receive the perception data streams uploaded by each sensor and the pose status data uploaded by each actuator, and obtain user interaction data and real-time pose data.

[0025] For example, in a dark riding scenario, when a player's vehicle enters a trigger area, radar positioning devices deployed in that area upload the player's real-time position coordinates as user interaction data. Simultaneously, the trigger signals generated by the player stepping on pressure-sensitive flooring are also uploaded. Furthermore, moving screens and special effects spray devices in the scene can upload the current translational position and rotation angle as real-time pose data.

[0026] Step 102: Perform time synchronization and alignment processing on the user interaction data and real-time pose data to obtain aligned user interaction data and aligned actuator pose data.

[0027] In some embodiments, the aforementioned execution entity may perform time synchronization alignment processing on the aforementioned user interaction data and the aforementioned real-time pose data to obtain aligned user interaction data and aligned actuator pose data. The aforementioned time synchronization alignment processing may be a process of unifying different data (e.g., different sources and different timestamps) under the same time reference.

[0028] As an example, the aforementioned execution entity can use a unified global clock source to stamp each data frame (user interaction data and real-time pose data) with a global timestamp. Then, based on a preset synchronization period, it can use a linear interpolation method (or a nearest neighbor sampling method) to remap the data so that the data is aligned on a unified time axis, resulting in aligned user interaction data and aligned actuator pose data.

[0029] For example, in a dark riding scenario, the radar positioning device uploads player position data at a 30Hz refresh rate, while the actuator's encoder uploads pose data at 100Hz. The timestamps of these two types of data are not aligned. In practice, a synchronization period of 10ms can be used to interpolate and align these two types of data separately, ensuring that each 10ms synchronization point has corresponding player position data and actuator pose data.

[0030] Step 103: Based on the current interaction logic in the virtual scene and the aligned user interaction data, generate the expected control instructions for the executor to obtain the expected control instruction set.

[0031] In some embodiments, the aforementioned executing entity can generate expected control instructions for the executors based on the current interaction logic in the virtual scene and the aligned user interaction data, thus obtaining an expected control instruction set. The current interaction logic in the virtual scene can be preset plot progression rules and player behavior response rules in the virtual engine. For example, triggering a corresponding mechanism action when the player reaches a certain coordinate area, or executing corresponding scene feedback when the player makes a specific gesture. The aforementioned expected control instructions can be instructions used to instruct each executor to perform the target action, and may include, but are not limited to, the following physical quantities: expected position, expected velocity, and expected force.

[0032] As an example, user interaction data can be input into the interaction logic module in the virtual engine. Through rule matching (or state machine query), the executor that should be triggered and its target action to be executed can be determined, and the expected control instructions of the executor can be generated.

[0033] For example, in a dark riding scenario, when the aligned user interaction data indicates that the player's vehicle has reached the preset trigger point, the virtual engine determines that the moving screen in front should be moved to the left to display the scene based on the current plot progress. The aforementioned execution entity then generates instructions for the desired position and speed of the motor corresponding to the moving screen and incorporates these instructions into the desired control instruction set.

[0034] Step 104: Based on the desired control instruction set and the aligned actuator pose data, construct the spatiotemporal constraint matrix for each actuator.

[0035] In some embodiments, the execution entity can construct a spatiotemporal constraint matrix for each actuator based on the desired control instruction set and the aligned actuator pose data. The spatiotemporal constraint matrix can be a matrix describing the motion trajectory envelope and spatiotemporal coupling relationship of each actuator within a preset future time period. The rows of the spatiotemporal constraint matrix can be a time sampling sequence, with each row corresponding to a discrete sampling time. The columns of the spatiotemporal constraint matrix can be the spatial pose envelope parameters of each actuator, with each column corresponding to one actuator. The elements in the spatiotemporal constraint matrix can be the spatial occupancy range of the actuator at the corresponding time sampling point.

[0036] For example, in a dark riding scenario, when the player triggers the actions of the moving screen and the special effect spray device, the aforementioned execution entity can determine the motion trajectory within the next 500ms based on the translation command of the moving screen and the spray command of the spray device in the expected control command set. The trajectory is discretized into a pose trajectory sequence with a sampling interval of 10ms. A cubic envelope is added to the moving screen and a cylindrical envelope is added to the spray device, and each is extended with a safety margin of 50mm. Finally, a spatiotemporal constraint matrix is ​​generated with time sampling points as rows and the two actuators as columns.

[0037] In some optional implementations of certain embodiments, the execution entity may construct the spatiotemporal constraint matrix of each actuator based on the desired control instruction set and the aligned actuator pose data, which may include the following steps: The first step is to determine the motion trajectory of each actuator within a preset future time period based on the aforementioned desired control instruction set. This preset future time period can be the length of a preset time window used to predict the actuator's motion state. The motion trajectory can be the path the actuator takes in space as it moves according to the instructions within this time period.

[0038] As an example, the target position or target velocity in the expected control command of each actuator can be used as the initial state, and the kinematic interpolation algorithm (e.g., trapezoidal velocity planning algorithm or fifth-order polynomial interpolation algorithm) can be used to determine the continuous motion path of each actuator in the future preset time period, so as to obtain the motion trajectory.

[0039] The second step is to discretize the above motion trajectory to obtain the expected pose of each actuator at each discrete time sampling point, which is used as the pose trajectory sequence.

[0040] As an example, continuous motion trajectories can be sampled at preset sampling intervals to extract position and attitude parameters at each sampling point, resulting in a discrete pose trajectory sequence.

[0041] Third, for each pose trajectory in the above pose trajectory sequence, a motion trajectory envelope of a preset geometric shape is added at the corresponding time sampling point to obtain a motion trajectory envelope sequence. The motion trajectory envelope represents the spatial occupancy of the actuator at the corresponding time sampling point and includes a preset safety margin. The preset geometric shape can be a bounding volume shape pre-defined according to the actuator's external dimensions. For example, a cylinder, cube, sphere, and polyhedron. The safety margin can be an additional buffer distance beyond the actual size of the actuator.

[0042] As an example, firstly, the corresponding geometric bounding volume can be determined as the envelope shape based on the actuator's external dimensions and motion characteristics. Then, with the actuator's pose at the sampling point as the center, envelope information is added, and a preset safety margin is extended at the outer edge of the envelope to obtain the motion trajectory envelope at that sampling point.

[0043] The fourth step is to construct a spatiotemporal constraint matrix based on the above-mentioned pose trajectory sequence sampling sequence and the above-mentioned motion trajectory envelope sequence. Each element in the above-mentioned spatiotemporal constraint matrix represents the spatial occupancy range of the corresponding actuator at the corresponding time sampling point.

[0044] As an example, the time sampling sequence can be used as the row dimension of the matrix, and each actuator can be used as the column dimension of the matrix. The spatial parameters of the motion trajectory envelope of each actuator at each time sampling point can be filled into the corresponding positions of the matrix to generate a spatiotemporal constraint matrix.

[0045] Step 105: Based on the spatiotemporal constraint matrix, perform collision interference detection on the motion trajectory envelopes of any two actuators within the same time window to obtain the collision interference detection results.

[0046] In some embodiments, the aforementioned execution entity can perform collision interference detection on the motion trajectory envelopes of any two actuators within the same time window based on the aforementioned spatiotemporal constraint matrix, thereby obtaining collision interference detection results. The collision interference detection can be a process of determining whether the motion trajectory envelopes of two actuators at the same moment will spatially overlap (or be too close in distance). The collision interference detection results can be a dataset recording actuator pairs with collision risk (or near-collision risk) and their corresponding collision time windows. The collision time window can be a discrete-time sampling point corresponding to when two actuators collide or are about to collide.

[0047] For example, in a dark riding scenario, the aforementioned execution entity can traverse each time sampling point of the spatiotemporal constraint matrix, sequentially detecting the envelope relationship between the moving screen and the special effects spray device at the same sampling point. Assuming that at the 15th sampling point, the cubic and cylindrical envelopes of both partially overlap in space, this is directly determined as collision interference and recorded. At the 20th sampling point, the envelopes do not overlap, but the minimum distance is 30mm, lower than the preset 50mm safety distance threshold, and is also determined as collision interference and recorded. The detection results from the two sampling points are then combined to obtain the collision interference detection result.

[0048] In some optional implementations of certain embodiments, the aforementioned execution entity can perform collision interference detection on the motion trajectory envelopes of any two actuators within the same time window based on the aforementioned spatiotemporal constraint matrix to obtain the collision interference detection result, which may include the following steps: The first step is to extract the spatial parameters of the motion trajectory envelopes of each actuator at the same time sampling point in the aforementioned spatiotemporal constraint matrix. These same time sampling points constitute the same time window.

[0049] As an example, we can iterate through each row of the spatiotemporal constraint matrix, and for each time sampling point, extract the spatial occupancy range data of the motion trajectory envelope of each actuator in that row as a spatial parameter.

[0050] The second step is to determine the collision time window between the two actuators as the first collision interference detection result, in response to the overlapping region of the motion trajectory envelopes of the two actuators at the same sampling point.

[0051] As an example, a spatial intersection operation can be performed on the motion trajectory envelopes of the two actuators. If the intersection is not empty, it is determined that there is an overlap, and the two actuators and the collision time window are recorded as the first collision interference detection result.

[0052] Third, in response to the fact that the motion trajectory envelopes of the two actuators do not overlap, the following second operation step is performed: The first sub-step involves determining the minimum distance between the motion trajectory envelopes of the two actuators. This minimum distance can be the shortest distance in space between the surfaces of the two envelopes.

[0053] As an example, the minimum spacing value can be obtained by calculating the shortest Euclidean distance between points on the surfaces of the two envelopes.

[0054] The second sub-step involves determining the collision time window between the two actuators as the second collision interference detection result, in response to the minimum spacing value being less than a preset safety distance threshold. The preset safety distance threshold can be a preset minimum allowable spacing used to determine whether the actuators are too close together.

[0055] As an example, the aforementioned executing entity can compare the minimum spacing value with the safe distance threshold. If the minimum spacing value is lower than the threshold, although there is no actual overlap, it has entered the warning range and is also recorded as a collision interference result, serving as the second collision interference detection result.

[0056] The fourth step is to determine the obtained first collision interference detection results and the obtained second collision interference detection results as the collision interference detection results.

[0057] As an example, the collision-causing actuator pairs and corresponding collision time windows at each time sampling point can be deduplicated and merged to obtain the collision interference detection results.

[0058] Step 106: In response to the collision interference detection result satisfying the collision interference condition, based on the preset safety priority list, the expected control command corresponding to the interfered actuator is compensated to obtain an interference-free command set.

[0059] In some embodiments, the execution entity may, in response to the collision interference detection result satisfying the collision interference condition, perform compensation processing on the expected control instructions corresponding to the interfered actuator based on a preset safety priority list to obtain an interference-free instruction set. The collision interference condition may be that there is at least one pair of actuators exhibiting collision interference in the collision interference detection result, i.e., the detection result is not empty. The preset safety priority list may be a list recording the safety response priority order of each actuator within the current control cycle. The interfered actuator may be one of the actuators involved in the collision interference detection result. The compensation processing may be an operation that adjusts the expected control instructions of the interfered actuator to avoid collisions, such as a delay operation.

[0060] For example, in a dark riding scenario, collision interference detection results show that the moving screen and the special effects spray device pose a collision risk at sampling points 15 and 20. The aforementioned execution entities can be based on a safety priority list, where the moving screen, due to its mechanical action and high current load, has a higher safety priority than the spray device. The spray device is then determined to have a lower safety priority. The expected control command for the spray device is then delayed, with a corresponding duration of waiting frames inserted before its command queue. The moving screen's command remains unchanged, replacing the original command to obtain an interference-free command set, allowing the screen to act first and the spray to trigger later, thus eliminating the collision risk.

[0061] In some optional implementations of certain embodiments, the aforementioned execution entity can perform compensation processing on the expected control instructions corresponding to the interfered actuator based on a preset safety priority list to obtain an interference-free instruction set, which may include the following steps: The first step is to obtain the collision interference detection results, which include the actuator pairs that have collision interference and the corresponding collision time windows for each actuator pair.

[0062] The second step is to determine, based on the above collision interference detection results, the actuators involved in the collision among the actuator pairs that have collision interference, and to identify the set of the interfered actuators.

[0063] As an example, the aforementioned execution entity can traverse each actuator pair in the collision interference detection results, remove duplicates from all appearing actuators, and obtain the set of interfered actuators.

[0064] The third step is to determine the actuators with lower safety priority in the above-mentioned actuator set based on the preset safety priority list. The lower safety priority means that in the same pair of actuators that have collision interference, the safety priority of one actuator is lower than that of the other actuator.

[0065] As an example, for each pair of actuators that have collision interference, the safety priority list is queried, and then the two are sorted to determine the actuator with the lower priority that needs to be compensated.

[0066] The fourth step involves performing delay compensation processing on the expected control commands of the interfered actuators with lower safety priority, resulting in compensated expected control commands, while keeping the expected control commands of the interfered actuators with higher safety priority unchanged. This delay compensation processing can involve inserting a waiting period into the instruction queue of the interfered actuator, causing its action start time to be later than the original time, thereby avoiding collision time windows with other actuators.

[0067] As an example, the aforementioned execution entity can determine the required delay duration based on the collision time window and the current movement speed of the actuator. Then, a corresponding number of empty instruction frames are inserted before the original instruction to perform delay compensation processing, resulting in the compensated desired control instruction.

[0068] The fifth step is to replace the original expected control instructions corresponding to the above expected control instruction set with the obtained compensated expected control instructions to obtain the interference-free instruction set.

[0069] As an example, the aforementioned execution entity can traverse the desired control instruction set, replace the original instructions of the actuators involved in compensation with delayed instructions, and leave the instructions of other actuators unchanged, thus obtaining an interference-free instruction set.

[0070] In some optional implementations of certain embodiments, the aforementioned preset safety priority list is a list recording the safety response priority order of each actuator within the current control cycle. Generating the aforementioned safety priority list may include the following steps: The first step is to obtain the task type, real-time load status, and current speed of each actuator in the current control cycle. The task type can be a category based on the function performed by the actuator in the current scenario, and may include, but is not limited to, emergency stop, mechanical motion, and lighting / ambient lighting. The real-time load status can be parameters reflecting the actuator's workload (e.g., current load rate, current, and torque).

[0071] The second step is to determine the initial safety level of each actuator based on the above task types. Among them, the initial safety level of emergency stop actuators is higher than that of mechanical motion actuators, and the initial safety level of mechanical motion actuators is higher than that of lighting and ambient lighting actuators.

[0072] As an example, the aforementioned executor can determine a mapping table between task types and initial safety levels, with emergency stop tasks corresponding to level 1 (highest), mechanical action tasks corresponding to level 2, and lighting / ambience tasks corresponding to level 3.

[0073] The third step involves performing a weighted adjustment on the initial safety level based on the real-time load status and the current movement speed, resulting in a weighted adjusted safety level. This weighted adjustment can be based on the magnitude of the real-time load status and the current movement speed, correcting the initial safety level so that actuators with higher loads or faster speeds receive a greater upward adjustment to achieve a higher protection priority.

[0074] As an example, firstly, the real-time load status and current movement speed can be normalized to obtain normalized load status and movement speed values. Then, the two normalized values ​​are multiplied by their respective preset weighting coefficients to obtain weighted load contribution and speed contribution values. Next, the weighted load contribution and speed contribution values ​​are added together and then superimposed with the initial safety level to obtain the actuator's comprehensive score. Finally, the actuators are sorted from highest to lowest comprehensive score to obtain a weighted adjusted safety level, with higher scores indicating higher priority.

[0075] The fourth step involves sorting the actuators based on the weighted safety levels described above, generating a safety priority list. In practice, the safety priority list can be generated by sorting them from highest to lowest safety level.

[0076] Step 107: Based on the inherent response delay of each actuator and the data transmission link delay of heterogeneous sensors, perform timing phase compensation processing on each non-interference instruction in the non-interference instruction set to obtain a compensated candidate instruction set for controlling each actuator.

[0077] In some embodiments, the execution entity can perform timing and phase compensation processing on each interference-free instruction in the interference-free instruction set based on the inherent response delay of each actuator and the data transmission link delay of the heterogeneous sensors, to obtain a compensated candidate instruction set for controlling each actuator. The inherent response delay can be the time lag between the actuator receiving the instruction and starting its action, and can be determined by the electromechanical characteristics of the actuator (e.g., motor inertia and transmission backlash). The data transmission link delay can be the time consumed by the heterogeneous sensors in transmitting collected user interaction data or pose data to the execution entity.

[0078] As an example, the aforementioned execution entity can store the inherent response latency of each actuator and the data transmission link latency of each sensor in a configuration file. Then, for each interference-free instruction in the interference-free instruction set, the total compensation amount is determined based on the sum of the sensor link latency involved in the interference-free instruction and the response latency of the target actuator. Next, a dynamic sliding time window algorithm is used to adjust the timestamps of the interference-free instructions, so that the timing of the actions of each actuator in the physical world matches the timing of the virtual scene.

[0079] In some optional implementations of certain embodiments, the following steps may also be included: The first step is to obtain the instantaneous communication load rate and the backlog length of the command queue on the centralized control bus. The instantaneous communication load rate can be the ratio of the current data transmission volume to the maximum bandwidth of the bus. The backlog length of the command queue can be the number of command frames that have been generated but not yet sent.

[0080] As an example, the aforementioned execution entity can read the instantaneous communication load rate and the backlog length of the instruction queue from the status register of the bus controller.

[0081] The second step involves prioritizing the communication time slots of each actuator based on the aforementioned instantaneous communication load rate, backlog length, and real-time operating condition levels. The real-time operating condition level can be a comprehensive assessment based on the criticality of the task currently being performed by the actuator, its motion status, or safety level. The communication time slot priority can be determined by the order in which each actuator transmits data on the bus, with higher-priority actuators receiving transmission time slots first.

[0082] As an example, the instantaneous communication load rate and backlog length can be compared with preset thresholds. When either the load rate or backlog length exceeds the threshold, dynamic priority allocation is triggered, assigning different transmission priorities to the actuators based on their real-time operating condition levels. Specifically, for each actuator, its task criticality score, normalized current speed, and safety level score are weighted and summed according to preset weights to obtain a real-time operating condition level value. Then, the actuators are sorted from highest to lowest operating condition level values, and communication time slot priority values ​​are assigned sequentially from smallest to largest. For example, the actuator with the highest operating condition level is assigned priority 1, the next highest is assigned priority 2, and so on.

[0083] The third step is to distribute the compensated candidate instruction set in an orderly manner on the centralized control bus based on the aforementioned communication time slot priority.

[0084] As an example, candidate instructions from the compensated candidate instruction set can be placed into the bus's transmit queue in descending order of communication time slot priority. High-priority executor instructions are sent first, while low-priority instructions wait for the next time slot to be sent, thus alleviating bus congestion.

[0085] In some optional implementations of certain embodiments, after the execution entity has sequentially distributed the compensated candidate instruction set on the centralized control bus based on the communication time slot priority, the following steps may also be included: The first step, for each of the above actuators, is to perform the following first operation step: The first sub-step involves acquiring the real-time output parameters of the actuator in response to the actuator being in an action execution state. These physical output parameters can be parameters actually output by the actuator, such as, but not limited to, actual position, actual speed, and actual torque. In practice, these real-time output parameters can be obtained through feedback elements on the actuator (e.g., encoders, force sensors).

[0086] The second sub-step involves determining the real-time deviation between the aforementioned real-time output parameters and the expected output value corresponding to the actuator, where the expected output value is the output value of the actuator in the compensated candidate instruction set. The real-time deviation can be the difference between the actual output value and the expected output value. For example, the expected output value can be the theoretical output value after the actuator executes the corresponding candidate instruction.

[0087] The third sub-step involves generating a generation weighting coefficient for the desired control command in the next control cycle based on the aforementioned real-time deviation, in response to the real-time deviation exceeding a preset tolerance threshold. The preset tolerance threshold can be the maximum allowable deviation range. The generation weighting coefficient can be a correction factor used to adjust the generation process of the desired control command in the next control cycle.

[0088] As an example, the aforementioned execution entity can use the ratio of the absolute value of the aforementioned real-time deviation to the aforementioned preset tolerance threshold as the deviation ratio, and subtract the aforementioned deviation ratio from 1 as the generated weight coefficient.

[0089] For example, when the preset tolerance threshold is 5mm and the real-time deviation is 3mm, the deviation ratio is 0.6 and the weighting coefficient is 0.4. When constructing the spatiotemporal constraint matrix in the next control cycle, the safety margin of the motion trajectory envelope of the corresponding actuator is divided by this weighting coefficient, so that the actuator with the larger deviation obtains a larger safety margin in the next cycle, thereby triggering earlier intervention in collision detection.

[0090] The second step is to modify the construction of the above-mentioned spatiotemporal constraint matrix based on the obtained generation weight coefficients in the next control cycle.

[0091] As an example, the aforementioned executing entity can, in the third step of step 104 of the next control cycle, divide the safety margin of the motion trajectory envelope of the corresponding actuator by the generation weight coefficient of the actuator to obtain the corrected safety margin. For example, if the generation weight coefficient of an actuator is 0.8 and its preset safety margin is 50mm, then the corrected safety margin is 50mm ÷ 0.8 = 62.5mm. This corrected safety margin is used as the safety margin parameter for the actuator when constructing the motion trajectory envelope in the next control cycle. Thus, for actuators with large actual output deviations, their generation weight coefficients are smaller, and the corrected safety margin is correspondingly increased, thereby triggering earlier intervention in the collision interference detection of the next cycle and automatically compensating for insufficient physical execution force.

[0092] For example, in a dark riding scenario, after receiving the interference-free instruction set, the aforementioned execution entity reads the inherent response latency of the moving screen motor in the configuration as 15ms. The inherent response latency of the special effects spray device is 25ms. Simultaneously, the link latency of the radar positioning device is 5ms. For the moving screen instruction, the compensation is 15+5=20ms. For the spray device instruction, the compensation is 25+5=30ms. Then, by adjusting the instruction timestamps and sending them ahead of time, the actions of the two are aligned on the physical timeline. At this point, the instantaneous load rate of the centralized control bus reaches 85%, and the queue for sending instructions is backed up by 3 frames, exceeding the preset threshold. The aforementioned execution entity can, based on the real-time operating level of each actuator (the moving screen performs mechanical actions and is in motion, with a high real-time operating level; the spray device performs lighting and atmosphere tasks and is in standby mode, with a low real-time operating level), send the moving screen instruction first, followed by the spray device instruction, to avoid congestion. When the movable screen performs its actions, the aforementioned execution entity collects the actual position and finds that the deviation from the expected position is 8mm, exceeding the tolerance threshold of 5mm, and generates a weighting coefficient of 0.8. When constructing the spatiotemporal constraint matrix in the next control cycle, the safety margin of the movable screen is increased from 50mm to 50mm ÷ 0.8 = 62.5mm to increase the safety margin.

[0093] The above-described embodiments of this disclosure have the following beneficial effects: Through the actuator collaborative control method in virtual interaction based on on-site perception according to some embodiments of this disclosure, spatial conflicts between actuators can be eliminated, and instruction timing deviations caused by asynchronous and delayed data sources can be eliminated, thereby improving the timing accuracy of multi-actuator collaborative control and avoiding collisions or boundary violations during execution. Specifically, the reasons for the associated multi-actuator spatial collision risks and the mismatch between virtual feedback and physical action time are: in traditional methods, the control commands of each actuator are directly generated and issued based on trigger signals, lacking spatiotemporal joint modeling of the future motion trajectory of the actuators, and lacking unified timing alignment processing for data refresh differences and transmission delays of different sensors. This easily leads to collision risks and instruction timing deviations. Based on this, the actuator collaborative control method in virtual interaction based on on-site perception according to some embodiments of this disclosure first acquires user interaction data collected by heterogeneous sensors deployed on-site, as well as real-time pose data of each actuator. Thus, perception data reflecting player actions and pose data reflecting the current state of physical devices can be obtained, providing a complete data foundation for subsequent analysis. Secondly, the aforementioned user interaction data and real-time pose data are time-synchronized and aligned to obtain aligned user interaction data and aligned actuator pose data. This unifies the perceived data with different refresh rates and transmission delays onto the same time base, eliminating timing deviations introduced by asynchronous data sources and providing time-consistent data for subsequent instruction generation. Next, based on the current interaction logic in the virtual scene and the aligned user interaction data, expected control instructions for the actuators are generated, resulting in an expected control instruction set. Thus, according to the player's real-time behavior and plot progression rules in the virtual scene, the target action that each actuator should currently execute is determined, establishing a control mapping from the virtual layer to the physical layer, and obtaining instructions to control the actuators. Furthermore, based on the expected control instruction set and the aligned actuator pose data, a spatiotemporal constraint matrix for each actuator is constructed. This unifies the motion trajectory and spatial envelope of each actuator within a future preset time period into a unified data structure, allowing the spatiotemporal constraint relationships between multiple actuators to be quickly retrieved and compared within the same data structure (e.g., a matrix), providing an efficient retrieval basis for collision detection. Next, based on the aforementioned spatiotemporal constraint matrix, collision interference detection is performed on the motion trajectory envelopes of any two actuators within the same time window to obtain the collision interference detection results. Thus, by traversing the envelope space parameters of each actuator at the same time sampling point in the matrix, it can be determined whether there is spatial overlap (or the minimum distance is lower than the safe distance threshold). This allows for pre-identification of collision risks before command issuance, reducing the risk of collisions or boundary violations after command execution.Then, in response to the collision interference detection results satisfying the collision interference conditions, the expected control commands corresponding to the interfered actuators are compensated based on a preset safety priority list to obtain a set of interference-free commands. Thus, when a collision risk is detected between actuators, the commands of low-priority actuators can be delayed or rolled back according to their safety priorities, while the commands of high-priority actuators remain unchanged, eliminating spatial conflicts between actuators and maintaining the continuity of critical actuator actions. Finally, based on the inherent response delays of each actuator and the data transmission link delays of the heterogeneous sensors, each interference-free command in the set of interference-free commands undergoes timing and phase compensation to obtain a compensated candidate command set for controlling the actuators. Therefore, considering the differences in electromechanical response hysteresis between different actuators and the differences in link delays between different sensors, the total compensation amount is determined and the timing of each command is adjusted, ensuring that the actual action time of each actuator in physical space aligns with the expected trigger time in the virtual scene. This eliminates asynchronous actions of multiple actuators due to delay differences and avoids sensory fragmentation (e.g., "sound first, water mist later").

[0094] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an actuator collaborative control device in virtual interaction based on on-site perception. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this actuator collaborative control device based on field perception in virtual interaction can be specifically applied to various electronic devices.

[0095] like Figure 2As shown, an actuator collaborative control device 200 based on on-site perception in virtual interaction includes: an acquisition unit 201, a time synchronization and alignment unit 202, a first generation unit 203, a matrix construction unit 204, a collision detection unit 205, an instruction compensation unit 206, and a compensation control unit 207. The acquisition unit 201 is configured to acquire user interaction data collected by heterogeneous sensors deployed on-site, as well as real-time pose data of each actuator. The time synchronization and alignment unit 202 is configured to perform time synchronization and alignment processing on the aforementioned user interaction data and the aforementioned real-time pose data to obtain aligned user interaction data and aligned actuator pose data. The first generation unit 203 is configured to generate desired control instructions for the actuators based on the current interaction logic in the virtual scene and the aligned user interaction data, obtaining a desired control instruction set. The matrix construction unit 204 is configured to construct a spatiotemporal constraint matrix for each actuator based on the desired control instruction set and the aligned actuator pose data. The collision detection unit 205 is configured to: perform collision interference detection on the motion trajectory envelopes of any two actuators within the same time window based on the aforementioned spatiotemporal constraint matrix, and obtain the collision interference detection result. The instruction compensation unit 206 is configured to: in response to the collision interference detection result satisfying the collision interference condition, perform compensation processing on the desired control instruction corresponding to the interfered actuator based on a preset safety priority list, and obtain a non-interference instruction set. The compensation control unit 207 is configured to: perform timing and phase compensation processing on each non-interference instruction in the non-interference instruction set based on the inherent response delay of each actuator and the data transmission link delay of the heterogeneous sensors, and obtain a compensated candidate instruction set for controlling each actuator.

[0096] It is understandable that the units described in the actuator cooperative control device 200 in this on-site perception-based virtual interaction are related to the reference... Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the actuator collaborative control device 200 and its constituent units in the virtual interaction based on on-site perception, and will not be repeated here.

[0097] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0098] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0099] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0100] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0101] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0102] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0103] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire user interaction data collected by heterogeneous sensors deployed in the field, and real-time pose data of each actuator; perform time synchronization alignment processing on the aforementioned user interaction data and the aforementioned real-time pose data to obtain aligned user interaction data and aligned actuator pose data; generate desired control instructions for the actuators based on the current interaction logic in the virtual scene and the aforementioned aligned user interaction data, to obtain a desired control instruction set; and construct the aforementioned actuator pose data based on the aforementioned desired control instruction set and the aforementioned aligned actuator pose data. The spatiotemporal constraint matrix of each actuator; based on the spatiotemporal constraint matrix, collision interference detection is performed on the motion trajectory envelopes of any two actuators within the same time window to obtain the collision interference detection result; in response to the collision interference detection result satisfying the collision interference condition, the expected control command corresponding to the interfered actuator is compensated based on a preset safety priority list to obtain an interference-free command set; based on the inherent response delay of each actuator and the data transmission link delay of the heterogeneous sensors, each interference-free command in the interference-free command set is subjected to timing phase compensation processing to obtain a compensated candidate command set for controlling each actuator.

[0104] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a time synchronization and alignment unit, a first generation unit, a matrix construction unit, a collision detection unit, an instruction compensation unit, and a compensation control unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as "a unit that acquires user interaction data collected by heterogeneous sensors deployed in the field, as well as real-time pose data of various actuators."

[0107] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0108] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for actuator collaborative control in virtual interaction based on on-site perception, characterized in that, include: Acquire user interaction data collected by heterogeneous sensors deployed on-site, as well as real-time pose data of each actuator; The user interaction data and the real-time pose data are time-synchronized and aligned to obtain aligned user interaction data and aligned actuator pose data. Based on the current interaction logic in the virtual scene and the aligned user interaction data, the expected control instructions of the actuator are generated to obtain the expected control instruction set. Based on the desired control instruction set and the aligned actuator pose data, construct the spatiotemporal constraint matrix of each actuator; Based on the spatiotemporal constraint matrix, collision interference detection is performed on the motion trajectory envelopes of any two actuators within the same time window to obtain the collision interference detection results. In response to the collision interference detection result satisfying the collision interference condition, the expected control command corresponding to the interfered actuator is compensated based on the preset safety priority list to obtain an interference-free command set; Based on the inherent response delay of each actuator and the data transmission link delay of the heterogeneous sensors, timing and phase compensation processing is performed on each interference-free instruction in the interference-free instruction set to obtain a compensated candidate instruction set for controlling each actuator.

2. The actuator collaborative control method in virtual interaction based on on-site perception according to claim 1, characterized in that, The method further includes: Obtain the instantaneous communication load rate and the backlog length of the command queue to be issued on the centralized control bus; Based on the instantaneous communication load rate, the backlog length, and the real-time operating condition level of each actuator, the communication time slot priority of each actuator is determined; Based on the communication time slot priority, the compensated candidate instruction set is sequentially distributed on the centralized control bus.

3. The actuator collaborative control method in virtual interaction based on on-site perception according to claim 2, characterized in that, After the compensated candidate instruction set is sequentially distributed on the centralized control bus based on the communication time slot priority, the method further includes: For each of the aforementioned actuators, the following first operational step is performed: In response to the actuator being in an action execution state, the real-time output parameters of the actuator are collected; Determine the real-time deviation between the real-time output parameter and the expected output value corresponding to the actuator, wherein the expected output value is the output value of the actuator corresponding to the compensated candidate instruction set; In response to the real-time deviation exceeding a preset tolerance threshold, a generation weighting coefficient for the desired control command for the next control cycle is generated based on the real-time deviation. In the next control cycle, the construction of the spatiotemporal constraint matrix is ​​modified based on the obtained generation weight coefficients.

4. The actuator collaborative control method in virtual interaction based on on-site perception according to claim 1, characterized in that, The step of constructing the spatiotemporal constraint matrix of each actuator based on the desired control instruction set and the aligned actuator pose data includes: Based on the desired control instruction set, the motion trajectory of each actuator within a future preset time period is determined; The motion trajectory is discretized to obtain the expected pose of each actuator at each discrete time sampling point, which is used as the pose trajectory sequence. For each pose trajectory in the pose trajectory sequence, a motion trajectory envelope of a preset geometric shape is added at the time sampling point corresponding to the pose trajectory to obtain a motion trajectory envelope sequence. The motion trajectory envelope is the spatial occupancy range of the actuator at the corresponding time sampling point and includes a preset safety margin. Based on the sampling sequence of the pose trajectory sequence and the motion trajectory envelope sequence, a spatiotemporal constraint matrix is ​​constructed, wherein each element in the spatiotemporal constraint matrix represents the spatial occupancy range of the corresponding actuator at the corresponding time sampling point.

5. The actuator collaborative control method in virtual interaction based on on-site perception according to claim 1, characterized in that, The step of performing collision interference detection on the trajectory envelopes of any two actuators within the same time window based on the spatiotemporal constraint matrix to obtain the collision interference detection result includes: Extract the spatial parameters of the motion trajectory envelope of each actuator at the same time sampling point in the spatiotemporal constraint matrix, wherein the same time sampling point constitutes the same time window; For any two actuators at the same sampling point, in response to the overlapping region of the motion trajectory envelopes of the two actuators, the collision time window between the two actuators is determined as the first collision interference detection result. In response to the fact that the motion trajectory envelopes of the two actuators do not overlap, the following second operation step is performed: Determine the minimum spacing value between the motion trajectory envelopes of the two actuators; In response to the minimum spacing value being less than a preset safe distance threshold, the collision time window between the two actuators is determined as the second collision interference detection result; The obtained first collision interference detection results and the obtained second collision interference detection results are defined as collision interference detection results.

6. The actuator collaborative control method in virtual interaction based on on-site perception according to claim 1, characterized in that, The step of compensating the desired control instructions corresponding to the interfered actuator based on a preset safety priority list to obtain an interference-free instruction set includes: Obtain the collision interference detection results, wherein the collision interference detection results include actuator pairs that exhibit collision interference and the collision time windows corresponding to each actuator pair; In response to the presence of at least one pair of actuators exhibiting collision interference in the collision interference detection results, each actuator involved in the collision among the pair of actuators exhibiting collision interference is determined as the set of interfered actuators based on the collision interference detection results; Based on a preset safety priority list, the actuators with lower safety priority in the set of the actuators to be interfered with are determined. The lower safety priority means that in the same pair of actuators that are subject to collision interference, the safety priority of one actuator is lower than that of the other actuator. The expected control commands of the interfered actuators with lower safety priority are subjected to delay compensation processing to obtain the compensated expected control commands, while the expected control commands of the interfered actuators with higher safety priority remain unchanged. Based on the obtained compensated expected control commands, the original expected control commands corresponding to the expected control command set are replaced to obtain an interference-free command set.

7. The actuator collaborative control method in virtual interaction based on on-site perception according to claim 1, characterized in that, The preset safety priority list is a list that records the safety response priority order of each actuator in the current control cycle. The steps for generating the safety priority list include: Obtain the task type, real-time load status, and current motion speed of each actuator in the current control cycle; Based on the task type, determine the initial security level of each executor; Based on the real-time load status and the current movement speed, the initial safety level is weighted and adjusted to obtain the weighted and adjusted safety level. Based on the weighted and adjusted security levels, the actuators are sorted to generate a security priority list.

8. An actuator collaborative control device for virtual interaction based on on-site perception, comprising: The acquisition unit is configured to acquire user interaction data collected by heterogeneous sensors deployed on-site, as well as real-time pose data of each actuator; The time synchronization alignment unit is configured to perform time synchronization alignment processing on the user interaction data and the real-time pose data to obtain aligned user interaction data and aligned actuator pose data. The first generation unit is configured to generate the desired control instructions for the executor based on the current interaction logic in the virtual scene and the aligned user interaction data, thereby obtaining the desired control instruction set. The matrix construction unit is configured to construct the spatiotemporal constraint matrix of each actuator based on the desired control instruction set and the aligned actuator pose data. The collision detection unit is configured to perform collision interference detection on the motion trajectory envelopes of any two actuators within the same time window based on the spatiotemporal constraint matrix, and obtain the collision interference detection result. The instruction compensation unit is configured to compensate the expected control instruction corresponding to the interfered actuator based on a preset safety priority list in response to the collision interference detection result satisfying the collision interference condition, so as to obtain an interference-free instruction set. The compensation control unit is configured to perform timing phase compensation processing on each interference-free instruction in the interference-free instruction set based on the inherent response delay of each actuator and the data transmission link delay of the heterogeneous sensors, so as to obtain a compensated candidate instruction set for controlling each actuator.

9. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the actuator cooperative control method in virtual interaction based on scene perception as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, It stores a computer program, wherein when the program is executed by a processor, it implements the actuator cooperative control method in virtual interaction based on on-site perception as described in any one of claims 1-7.

Images