A large screen interaction method and system based on a handheld device

Abstract

The application relates to the technical field of human-computer interaction, in particular to a large-screen interaction method and system based on a handheld device, which comprises the following steps: initializing the access mode and permission configuration of a master control device, a vice control device and a large screen; adopting a double-anchor positioning recognition algorithm to distinguish and recognize gesture instructions sent to the large screen by multiple users at the same time, calculating the visual space weighted score between each gesture instruction and a candidate user, combining a trajectory integrity performance energy function to filter discontinuous and noise segments, and dynamically and credibly determining the trajectory attribution of the gesture instructions of the multiple users; and using a layer stripping gesture conflict separation algorithm to perform multi-track reconstruction and instruction source stripping determination on the visually overlapped gesture trajectories generated by the same field operation of the multiple users by the large-screen interaction control system. The application solves the problems of order confusion, gesture instruction attribution disorder and gesture trajectory overlap in the human-computer interaction of the large screen by multiple users based on the layer stripping gesture conflict separation algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and particularly relates to a large-screen interaction method and system based on a handheld device. BACKGROUND

[0002] With the large-scale popularization of multi-user human-computer interaction scenes such as education and teaching, exhibition and display, command and dispatch, large-screen interaction technology has become an important means to improve the efficiency of multi-party collaboration. At present, the large-screen interaction methods mainly include touch direct interaction, air gesture recognition and remote control operation using a handheld device. When multiple users simultaneously interact with a large screen, the existing interaction technology cannot accurately identify the trajectory attribution and cannot determine whether the overlapped trajectory is one instruction or a fusion of multiple instructions due to the chaotic multi-user interaction order, disordered gesture instruction attribution and overlapped gesture trajectories. Therefore, a large-screen interaction method and system based on a handheld device are provided.

[0003] A large-screen interaction method and system based on a handheld device constructs a dual-anchoring positioning recognition algorithm based on a visual anchor path similarity function and a spatial layer positioning score function. The first visual recognition determines the action origin of the user gesture, and the second spatial positioning determines the layer to which the gesture instruction trajectory point belongs, thereby realizing the operation instruction decoupling and identity separation of different users under the condition of gesture image overlap.

[0004] A large-screen interaction method and system based on a handheld device can realize permission constraints between master control and vice control, spatial level gesture stripping and multi-instruction conflict scheduling determination mechanism in a large-screen interaction scene with multiple handheld devices, so as to ensure the display stability and operation order of the large-screen system. SUMMARY

[0005] The present application aims to provide a large-screen interaction method and system based on a handheld device to solve the problem of inaccurate trajectory attribution recognition and determination of whether an overlapped trajectory is one instruction or a fusion of multiple instructions due to chaotic multi-user interaction order, disordered gesture instruction attribution and overlapped gesture trajectories.

[0006] To achieve the above-mentioned purpose, the technical solution of the present application is as follows: a large-screen interaction method based on a handheld device, comprising:

[0007] S1, using an identity token-based decentralized interaction access structure, initializing the access mode and permission configuration of the master control device, the vice control device and the large screen;

[0008] S2, using a double anchor positioning recognition algorithm to distinguish and identify multiple users simultaneously issuing gesture instructions to the large screen, and calculating the visual spatial weighted score between each gesture instruction and the candidate user, combining the trajectory integrity performance energy function to filter discontinuous and noise fragments, and dynamically determining the attribution of the gesture instruction trajectory of multiple users;

[0009] S3, for the visual overlapping gesture trajectory generated by the operation of multiple users in the same domain, the large screen interaction control system uses a layer stripping gesture conflict separation algorithm to perform multi-track reconstruction and instruction source stripping determination on the visual overlapping gesture trajectory;

[0010] S4, for the machine instruction conflict generated by the operation of multiple users in the same domain, the large screen adopts a master-slave control priority mechanism and a master intervention arbitration mechanism to schedule and execute the machine instruction conflict.

[0011] As a further improvement of the technical solution, in S1, the access mode and permission configuration of the master device, the slave control device and the large screen are initialized, specifically as follows:

[0012] S1.1, when the master device accesses the large screen, the system automatically generates a unique identity identifier and a session token for the master device, which is used to identify and authorize the access of the subsequent slave control device;

[0013] S1.2, the identity token distributed by the master device verifies and binds the access request of the slave control device, and the system assigns a unique device identifier to the slave control device based on the verification success result, and records its role type;

[0014] S1.3, the master device sets the interaction area of each slave control device with the large screen, the authorized start and end time, and the operation permission level.

[0015] As a further improvement of the technical solution, in S2, the double anchor positioning recognition algorithm is realized based on a visual anchor point path similarity function and a spatial layer positioning score function, which is used to determine the attribution of the gesture instruction trajectory of multiple users in combination with the trajectory integrity performance energy function.

[0016] As a further improvement of the technical solution, the visual anchor point path similarity function is realized by visually recognizing the gesture instruction trajectory of the ordinary user holding the slave control device and the visual anchor point coordinates and calculating the visual matching score to establish the association between the ordinary user's hand movement and the slave control device;

[0017] Wherein, the association between the ordinary user's hand movement and the slave control device is based on the gesture instruction trajectory and the visual anchor point coordinates The path behavior is mapped to the identity of the secondary control device based on the path similarity between them;

[0018] The spatial layer positioning information scoring function is based on the spatial coordinates of each secondary control device. Construct a spatial layer L for each secondary control device (i) And calculate the trajectory of the gesture command. The proportion of the space layer in the spatial hierarchy determines the spatial affiliation score.

[0019] As a further improvement to this technical solution, the trajectory integrity energy function is based on the gesture command trajectory. The trajectory integrity value A for each gesture command trajectory is calculated by incorporating a noise penalty term. (i) And combined with the visual spatial weighted score S of each gesture command trajectory (i) Construct a user trajectory attribution determination function U * Dynamically and reliably determine the attribution of gesture command trajectories for multiple users;

[0020] Wherein, the visual spatial weighted score S (i) Based on visual matching score Spatial Affiliation Rating We obtain the result by weighting.

[0021] Wherein, the user trajectory attribution determination function U * Specifically as follows:

[0022]

[0023] Among them, U * Here, S is the function for determining user trajectory attribution; i is the index of the secondary control device and the ordinary user; (i) Weighted scoring of visual space; A (i) The value represents the trajectory integrity; ∈ represents the minimum reliable energy threshold; I [] This is an indicator function.

[0024] As a further improvement to this technical solution, in S3, the layer-stripping gesture conflict separation algorithm is implemented based on the multi-track spatiotemporal clustering recognition algorithm and the spatial layer attribution mapping mechanism, and is used to decompose and reconstruct visually overlapping gesture trajectories in multiple tracks and strip the layers.

[0025] As a further improvement to this technical solution, the multi-trajectory spatiotemporal clustering recognition algorithm will collect the trajectory set T of all gesture command trajectories at a certain time. raw Perform spatiotemporal clustering to separate the trajectory set T raw All trajectories construct a trajectory category cluster C = {C1, C2, ..., C} j ,…,C K}; j is the cluster index, and K is the total number of clusters after separation;

[0026] The spatial layer attribution mapping mechanism traverses each cluster C j All trajectory points of the corresponding gesture command trajectory fall into the spatial layer L of each secondary control device. (i) The spatial layer to which the trajectory points fall most frequently is the spatial layer to which the gesture command trajectory belongs. Ultimately, each ordinary user will obtain their own set of trajectory points. and the set of trajectory points Perform multi-track reconfiguration;

[0027] Where m is the trajectory point index; Let t be the m-th trajectory point belonging to the i-th user; m This represents the time at the m-th trajectory point belonging to the i-th user.

[0028] As a further improvement to this technical solution, the set of trajectory points is... Perform multi-track reconfiguration as follows:

[0029] Set of trajectory points All trajectory segments are sorted according to their time sequence, and each pair of adjacent trajectory segments is iterated to determine whether they are temporally close and spatially continuous. If they are both temporally close and spatially continuous, the two trajectory segments are merged into one, resulting in the final set of trajectory points for the ordinary user.

[0030] For the final set of trajectory points of this ordinary user The trajectory integrity energy function eliminates incomplete and jittery trajectory segments.

[0031] As a further improvement to this technical solution, in S4, for machine instruction conflicts arising from multi-user operations in the same domain, the large screen uses a master-slave control priority mechanism and a master control intervention and adjudication mechanism to schedule and execute the machine instruction conflicts, as follows:

[0032] S4.1 The master-slave priority mechanism assigns an instruction priority weight to each machine instruction when it is generated.

[0033] S4.2 The master control intervention and adjudication mechanism is used to adjudicate machine instruction conflicts between slave control devices;

[0034] S4.3 The primary and secondary control priority mechanism and the primary control intervention and adjudication mechanism are executed in parallel.

[0035] On the other hand, the present invention provides a large-screen interactive system based on a handheld device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the large-screen interactive method based on the handheld device described above.

[0036] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects:

[0037] 1. The large-screen interaction method and system based on handheld devices adopts a dual anchoring positioning and recognition algorithm that integrates visual anchor point path similarity function and spatial layer positioning confidence score function. Combined with trajectory integrity energy function to dynamically filter invalid trajectories, it realizes accurate attribution and reliable determination of multi-user gesture command trajectory, and solves the problems of gesture command attribution disorder and gesture trajectory overlap in multi-user collaborative operation.

[0038] 2. In this large-screen interaction method and system based on handheld devices, a layer-peeling gesture conflict separation algorithm is adopted, which is composed of a multi-track spatiotemporal clustering recognition algorithm and a spatial layer attribution mapping mechanism. This algorithm realizes the multi-track separation and reconstruction of visually overlapping gesture trajectories and the separation and attribution of command sources, thus solving the problem of trajectory adhesion caused by the overlap of multiple gesture images. Attached Figure Description

[0039] Figure 1 This is a flowchart of one embodiment of the present invention. Detailed Implementation

[0040] Example 1, as Figure 1 As shown, the present invention proposes a large-screen interaction method based on a handheld device, and its specific implementation steps are as follows:

[0041] S1. Use an identity token-based hierarchical interactive access structure to initialize the access methods and permission configurations of the main control device, secondary control device, and large screen;

[0042] In step S1, the access methods and permission configurations of the main control device, the secondary control device, and the large screen are initialized, as follows:

[0043] S1.1 When the main control device is connected to the large screen, the system automatically generates a unique identifier and session token for the main control device to identify and authorize the access of subsequent secondary control devices.

[0044] S1.2 The identity token distributed by the main control device verifies and binds the access request of the secondary control device. Based on the successful verification result, the system assigns a unique device identifier to the secondary control device and records its role type.

[0045] S1.3 The main control device sets the interaction area between each secondary control device and the large screen, the authorized start and end time, and the operation permission level.

[0046] In this embodiment S1, the main control device is handheld by the administrator for interacting with the large screen and controlling the secondary control device; the secondary control device is handheld by ordinary users and can only interact with the large screen after being authorized by the main control device.

[0047] When the administrator's handheld main control device (which may include a teacher's tablet, a meeting host's mobile phone, or other professional devices for administrators) is connected to the large screen interactive system, the permissions granted to the main control device by the large screen interactive control system include global control over the large screen content, approval of permissions for all secondary control devices, priority execution of conflicting instructions, and the right to divide the large screen interactive area.

[0048] Ordinary users' handheld secondary control devices (such as student mobile phones and audience equipment) request to access the large screen interactive control system through scanning codes, near-field communication, or device discovery mechanisms. Each secondary control device needs to submit a request when connecting, and the main control device needs to confirm and authorize it, and establish an internal permission mapping record.

[0049] An interaction permission control mapping table P is constructed and maintained by the large-screen interactive control system. 权限 It is used to manage the operating range and authorized time period of each secondary control device, and its structure is as follows:

[0050]

[0051] in, R is the unique identifier for the i-th secondary control device; 交互区 The large screen interactive area that is allowed to be operated is set by the main control device; This represents the start time when the i-th secondary control device is authorized to perform interactive operations; This is the end time when the i-th secondary control device is authorized to perform interactive operations;

[0052] Among them, the i-th secondary control device also corresponds to the i-th ordinary user;

[0053] When any secondary control device sends machine commands or gesture command binding commands to the large screen, the system interaction control module will call the permission verification function. The system will only execute the command if the following three conditions are met: the device ID of the command source exists in the interaction permission control mapping table P. 权限 The current time is within the authorized time range and the interaction area involved in the instruction falls within the R owned by the device. 交互区 ;

[0054] During the interaction, the master control device can change the authorization status of any slave control device at any time, enabling administrators to achieve fine-grained control over the slave control devices according to the class progress, experiment stage, or presentation process.

[0055] S2. A dual-anchor positioning and recognition algorithm is used to distinguish and identify multiple users simultaneously sending gesture commands to the large screen. The visual spatial weighted score between each gesture command and the candidate user is calculated. The trajectory integrity energy function is combined to filter discontinuities and noise segments, and the trajectory of multiple users' gesture commands is dynamically and reliably determined.

[0056] In this embodiment S2, the dual anchoring positioning and recognition algorithm is implemented based on the visual anchor point path similarity function and the spatial layer positioning information scoring function, and is used to determine the trajectory attribution of gesture commands of multiple users by combining the trajectory integrity energy function.

[0057] The visual anchor point path similarity function uses visual recognition to identify the gesture command trajectory of ordinary users of handheld sub-control devices. and visual anchor coordinates And calculate the visual matching score. Establish the correlation between ordinary users' hand gestures and their secondary control devices;

[0058] The correlation between the user's hand gestures and the secondary control device is based on gesture command trajectories. With visual anchor point coordinates The path behavior is mapped to the identity of the secondary control device based on the path similarity between them;

[0059] In this embodiment, the gesture command trajectory of an ordinary user The starting point and trajectory in the image space have a stable spatial proximity with the extended area of ​​the user's secondary control device, and the visual anchor point coordinates... The visual anchor point is the starting point of the hand within a certain range above the secondary control device. When a user operates the device, their gesture trajectory is usually emitted from the device towards the screen. In the visual path, this is represented by starting from the anchor point, with the trajectory path consistent with the center direction of the screen. Furthermore, when the secondary control device is activated, its gesture is emitted from this anchor point towards the large screen within a very short time window. Therefore, the visual anchor point is a trigger source, resulting in a visual matching score. This can initially indicate the user's affiliation to the gesture trajectory, thus achieving the first level of visual verification;

[0060] The spatial layer positioning information scoring function is based on the spatial coordinates of each secondary control device. Construct a spatial layer L for each secondary control device (i) And calculate the trajectory of the gesture command. The proportion of the space layer in the spatial hierarchy determines the spatial affiliation score.

[0061] In this embodiment, spatial layer L (i) Each secondary control device has an operable layer area in the space. If a gesture falls within this layer, it may belong to that user. Note this spatial layer L. (i) It is a three-dimensional spatial layer, distinct from a two-dimensional planar layer; it maps the spatial position of the secondary control device to a three-dimensional spatial layer, enabling a trajectory spatial attribution determination method that is not limited to visual methods, ultimately yielding a spatial attribution score. This can demonstrate the user's affiliation to the gesture trajectory, thus achieving a second layer of spatial verification;

[0062] The trajectory integrity energy function is based on gesture command trajectory. The trajectory integrity value A for each gesture command trajectory is calculated by incorporating a noise penalty term. (i) This is used to evaluate whether each gesture command trajectory is complete, and combines the visual-spatial weighted score S of each gesture command trajectory. (i) Construct a user trajectory attribution determination function U * Dynamically and reliably determine the attribution of gesture command trajectories for multiple users;

[0063] Wherein, the visual spatial weighted score S (i) Based on visual matching score Spatial Affiliation Rating We obtain the result by weighting.

[0064] Wherein, the user trajectory attribution determination function U * Specifically as follows:

[0065]

[0066] Among them, U * Here, S is the function for determining user trajectory attribution; i is the index of the secondary control device and the ordinary user; (i) Weighted scoring of visual space; A (i) The value represents the trajectory integrity; ∈ represents the minimum reliable energy threshold; I [] This is an indicator function.

[0067] In this embodiment, based on the user trajectory attribution determination function U * The final determination of the attribution of gesture command trajectories for multiple users is as follows:

[0068] When multiple users (secondary control devices) simultaneously send gestures to the large screen, the system must determine which user (secondary control device) each gesture command trajectory belongs to, and this attribution needs to be accurate, reliable, and dynamic; the user trajectory attribution determination function U * Visual spatial weighted score S(i) This function is used to determine whether a gesture command trajectory was issued by the user; the higher the score, the more likely it belongs to that user. The user trajectory attribution function U... * Trajectory integrity value A (i) Used to determine trajectory integrity, ∈ is the minimum reliable energy threshold preset by the system; clear all A (i) For ordinary users ≤∈, select S from the reserved users. (i) The largest one is designated as the owner.

[0069] S3. For visually overlapping gesture trajectories generated by multi-user operations in the same domain, the large-screen interactive control system uses a layer-peeling gesture conflict separation algorithm to reconstruct the multi-track visually overlapping gesture trajectories and determine the instruction source.

[0070] In this embodiment S3, the layer-stripping gesture conflict separation algorithm is implemented based on the multi-track spatiotemporal clustering recognition algorithm and the spatial layer attribution mapping mechanism, and is used to decompose and reconstruct visually overlapping gesture trajectories in multiple tracks and strip the layers.

[0071] The multi-track spatiotemporal clustering recognition algorithm will use the trajectory set T of all gesture command trajectories at a certain time to... raw Perform spatiotemporal clustering to separate the trajectory set T raw All trajectories construct a trajectory category cluster C = {C1, C2, ..., C} j ,…,C K}; j is the cluster index, and K is the total number of clusters after separation;

[0072] The spatial layer attribution mapping mechanism traverses each cluster C j All trajectory points of the corresponding gesture command trajectory fall into the spatial layer L of each secondary control device. (i) The spatial layer to which the trajectory points fall most frequently is the spatial layer to which the gesture command trajectory belongs. Ultimately, each ordinary user will obtain their own set of trajectory points. and the set of trajectory points Perform multi-track reconfiguration;

[0073] Where m is the trajectory point index; Let t be the m-th trajectory point belonging to the i-th user; m This represents the time at the m-th trajectory point belonging to the i-th user.

[0074] In this embodiment, the set of trajectory points is calculated as follows:

[0075]

[0076] in, Set of trajectory points; k is the index of each trajectory; The trajectory of the k-th gesture command at time t;

[0077] The set of trajectory points Perform multi-track reconfiguration as follows:

[0078] Set of trajectory points All trajectory segments are sorted according to their time sequence, and each pair of adjacent trajectory segments is iterated to determine whether they are temporally close and spatially continuous. If they are both temporally close and spatially continuous, the two trajectory segments are merged into one, resulting in the final set of trajectory points for the ordinary user.

[0079] For the final set of trajectory points of this ordinary user The trajectory integrity energy function eliminates incomplete and jittery trajectory segments.

[0080] In this embodiment, when multiple users stand in front of the screen and perform gesture operations, the gesture trajectories of multiple users will overlap in the image plane, and from the visual image, they form a mixed overlapping trajectory; when three users simultaneously draw circles in the center area of ​​the screen with their hands in front of the large screen: from the image, the large screen may perceive it as a cluster of overlapping curves, and from the trajectory recording, some trajectory points come from different people, and some may overlap.

[0081] At this time, the user trajectory attribution determination function U * If an accurate judgment cannot be made, a layer-peeling gesture conflict separation algorithm combining a multi-trajectory spatiotemporal clustering recognition algorithm with a spatial layer attribution mapping mechanism is used for judgment:

[0082] The system collects a set of all gesture trajectory points extracted from the visual image at each moment. These trajectories are not separated and may come from different users, or some trajectory segments may overlap. The system uses a multi-trajectory spatiotemporal clustering recognition algorithm to determine how many independent trajectories are contained in the current overlapping trajectory and which user each belongs to. Each user has their own spatial layer, and the system uses a spatial layer attribution mapping mechanism to attach the trajectory segments of each trajectory sub-cluster to these layers. The layer that is attached to the most is the gesture command that may belong to the user of that layer.

[0083] Furthermore, since the initial trajectories overlap, they are then clustered and segmented, and then the trajectory segments belonging to user i are filtered out again based on the layers to obtain the set of trajectory points. Then the set of trajectory points It is fragmented, which means that gestures may be broken into several trajectories. Therefore, it is necessary to: splice these trajectory segments back into a continuous trajectory or turn them into multiple valid trajectories and perform trajectory quality assessment.

[0084] The determination of the source of the instruction is based on the fact that each trajectory segment belongs to a specific user and is complete. In this case, the operation instruction generated for that trajectory segment is bound to the operation method of that user.

[0085] S4. For machine instruction conflicts caused by multi-user operations in the same domain, the large screen uses a master-slave control priority mechanism and a master control intervention and adjudication mechanism to schedule and execute the machine instruction conflicts.

[0086] In S4, for machine instruction conflicts arising from multi-user operations in the same domain, the large screen uses a master-slave control priority mechanism and a master control intervention and adjudication mechanism to schedule and execute the machine instruction conflicts, as follows:

[0087] S4.1 The master-slave priority mechanism assigns an instruction priority weight to each machine instruction when it is generated.

[0088] S4.2 The master control intervention and adjudication mechanism is used to adjudicate machine instruction conflicts between slave control devices;

[0089] S4.3 The primary and secondary control priority mechanism and the primary control intervention and adjudication mechanism are executed in parallel.

[0090] In this embodiment, the instruction priority weight is determined by the device identity. The master device always has the highest execution right, and the priority of the secondary device can be configured by the master device to be 0.5 by default. When a conflict is detected between the secondary devices and the instructions have a small priority difference / close time / high overlap in area, the system will initiate an "arbitration request" and hand over the judgment right to the master device for immediate decision.

[0091] Example 2: The present invention proposes a large-screen interaction system based on a handheld device, which is applied to the large-screen interaction method based on a handheld device proposed in Example 1. It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the large-screen interaction method based on a handheld device as described above.

[0092] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A large-screen interaction method based on a handheld device, characterized in that, Includes the following steps: S1. Use an identity token-based hierarchical interactive access structure to initialize the access methods and permission configurations of the main control device, secondary control device, and large screen; S2. A dual-anchor positioning and recognition algorithm is used to distinguish and identify multiple users issuing gesture commands to the large screen at the same time. The visual spatial weighted score between each gesture command and the candidate user is calculated. The trajectory integrity energy function is combined to filter discontinuities and noise segments, and the trajectory of multiple users' gesture commands is dynamically and reliably determined. S3. For visually overlapping gesture trajectories generated by multi-user operations in the same domain, the large-screen interactive control system uses a layer-peeling gesture conflict separation algorithm to reconstruct the multi-track visually overlapping gesture trajectories and determine the instruction source. S4. For machine instruction conflicts caused by multi-user operations in the same domain, the large screen uses a master-slave control priority mechanism and a master control intervention and adjudication mechanism to schedule and execute the machine instruction conflicts. In S2, the dual anchoring positioning and recognition algorithm is based on the visual anchor point path similarity function and the spatial layer positioning information scoring function, and is used to determine the trajectory attribution of gesture commands of multiple users by combining the trajectory integrity energy function. The visual anchor point path similarity function uses visual recognition to identify the gesture command trajectory of ordinary users of handheld sub-control devices. and visual anchor coordinates And calculate the visual matching score. Establish a correlation between ordinary user hand gestures and their secondary control devices; wherein, the correlation between ordinary user hand gestures and their secondary control devices is based on gesture command trajectories. With visual anchor point coordinates The path behavior is mapped to the identity of the secondary control device based on the path similarity between them; The spatial layer positioning information scoring function is based on the spatial coordinates of each secondary control device. Construct a spatial layer for each secondary control device. And calculate the trajectory of the gesture command. The proportion of the space layer in the spatial hierarchy determines the spatial affiliation score. ; In S3, the layer-stripping gesture conflict separation algorithm is implemented based on the multi-track spatiotemporal clustering recognition algorithm and the spatial layer attribution mapping mechanism, and is used to decompose and reconstruct visually overlapping gesture trajectories in multiple tracks and strip the layers. The multi-track spatiotemporal clustering recognition algorithm sets the trajectory of all gesture command trajectories at a certain time. Perform spatiotemporal clustering to separate trajectory sets All trajectories construct trajectory category clusters ; For cluster indexing, The total number of clusters after separation. For the first A cluster, For the first One cluster; The spatial layer attribution mapping mechanism traverses each cluster. All trajectory points of the corresponding gesture command trajectory fall into the spatial layer of each secondary control device. The spatial layer to which the trajectory points fall most frequently is the spatial layer to which the gesture command trajectory belongs. Ultimately, each ordinary user will obtain their own set of trajectory points. and the set of trajectory points Multi-track reconfiguration is performed; among them, Index for trajectory points; For belonging to the first The first user's A trajectory point; For belonging to the first The first user's The time at which each trajectory point is located.

2. The large-screen interaction method based on a handheld device according to claim 1, characterized in that: In step S1, the access methods and permission configurations of the main control device, the secondary control device, and the large screen are initialized, as follows: S1.1 When the main control device is connected to the large screen, the system automatically generates a unique identifier and session token for the main control device to identify and authorize the access of subsequent secondary control devices. S1.2 The identity token distributed by the main control device verifies and binds the access request of the secondary control device. Based on the successful verification result, the system assigns a unique device identifier to the secondary control device and records its role type. S1.3 The main control device sets the interaction area between each secondary control device and the large screen, the authorized start and end time, and the operation permission level.

3. The large-screen interaction method based on a handheld device according to claim 2, characterized in that: The trajectory integrity energy function is based on gesture command trajectory. The trajectory integrity value of each gesture command trajectory is calculated by incorporating a noise penalty term. And combine the visual spatial weighted score of each gesture command trajectory Construct a user trajectory attribution determination function Dynamically and reliably determine the attribution of gesture command trajectories for multiple users; Among them, the visual spatial weighted score Based on visual matching score Spatial belonging score We obtain the result by weighting. Wherein, the user trajectory attribution determination function Specifically as follows: ； in, This is a function for determining user trajectory attribution. Indexes for secondary control devices and ordinary users; Weighted scoring of visual space; This is the trajectory integrity value; The minimum reliable energy threshold; This is an indicator function.

4. The large-screen interaction method based on a handheld device according to claim 3, characterized in that: The set of trajectory points Perform multi-track reconfiguration as follows: Set of trajectory points All trajectory segments are sorted according to their time sequence, and each pair of adjacent trajectory segments is iterated to determine whether they are temporally close and spatially continuous. If they are both temporally close and spatially continuous, the two trajectory segments are merged into one, resulting in the final set of trajectory points for the ordinary user. ; For the final set of trajectory points of this ordinary user The trajectory integrity energy function is used to eliminate incomplete and jittery trajectory segments.

5. The large-screen interaction method based on a handheld device according to claim 4, characterized in that: In S4, for machine instruction conflicts arising from multi-user operations in the same domain, the large screen uses a master-slave control priority mechanism and a master control intervention and adjudication mechanism to schedule and execute the machine instruction conflicts, as follows: S4.1 The master-slave priority mechanism assigns an instruction priority weight to each machine instruction when it is generated. S4.2 The master control intervention and adjudication mechanism is used to adjudicate machine instruction conflicts between slave control devices; S4.3 The primary and secondary control priority mechanism and the primary control intervention and adjudication mechanism are executed in parallel.

6. A large-screen interactive system based on a handheld device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes a computer program to implement the large-screen interaction method based on a handheld device as described in any one of claims 1-5.

Images