Intelligent synchronization system based on multi-end interaction

Abstract

The present application relates to the field of intelligent interaction, especially to a kind of intelligent synchronization system based on multi-terminal interaction, the terminal analysis module is carried out motion analysis to the coordinate information of each action node of instruction terminal, analyzes the action trend of action node, instruction transmission module adaptively selects transmission strategy based on action trend and transmits the action node corresponding coordinate information of instruction terminal to cloud, positioning module verifies the coordinate time sequence matching of action cycle, interactive synchronization module responds to the verification result of positioning module, adaptively selects transmission strategy and transmits the action node corresponding coordinate information of instruction terminal to cloud, model construction module restores all coordinate information in disassembled unit, constructs corresponding three-dimensional model, the present application can guarantee the readability of coordinate data when facing a large number of terminal transmission data by adaptively selecting transmission strategy based on action trend, complete modeling function and save transmission bandwidth, improve multi-terminal data interaction efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction, and more particularly to an intelligent synchronization system based on multi-terminal interaction. Background Technology

[0002] With the rapid development of computer and internet technologies, online multi-terminal interactive technology has gradually matured. It is now possible to realize the transmission of motion data between multiple terminals and the corresponding 3D modeling presentation. Especially in the field of online interaction, a single terminal can project a 3D model into a 3D space in various forms for other terminals to observe. At the same time, the terminal can also collect feedback information from other terminals, realizing online multi-terminal interaction and thus being applied in multiple fields.

[0003] For example, Chinese Patent Publication No. CN114779942A discloses a virtual reality immersive interactive system. The system includes: a virtual reality simulation module for modeling a virtual teaching scene, wherein the virtual teaching scene is a complete scene of UAV manufacturing; a virtual teaching scene selection module for selecting various sub-scenes of the complete UAV manufacturing scene; a recognition and positioning module for collecting feature information of a person in the virtual teaching scene through multiple sensors and using the feature information to perform pose recognition; wherein the pose includes at least one of the poses of the hands, feet, and head; and a display module for displaying the screen of the virtual teaching scene after human interaction.

[0004] However, the following problems still exist in the existing technology. In existing technologies, during multi-terminal interaction, multiple terminals need to exchange data streams or aggregate data streams to the cloud for processing. Data transmission from multiple terminals consumes a large amount of network bandwidth resources, and in weak network environments, it can easily lead to transmission delays or data packet loss. Summary of the Invention

[0005] To address this, the present invention provides an intelligent synchronization system based on multi-terminal interaction, which overcomes the problems in the prior art where multiple terminals need to exchange data streams or aggregate data streams to the cloud for processing during multi-terminal interaction. This results in data transmission from multiple terminals consuming a large amount of network bandwidth resources, and in weak network environments, it can easily lead to transmission delays or data packet loss.

[0006] To achieve the above objectives, the present invention provides an intelligent synchronization system based on multi-terminal interaction, comprising: The terminal analysis module is used to acquire the coordinate information of each action node of the command terminal in real time, and to perform motion analysis on each action node every action cycle, including determining the spatiotemporal action characteristics based on the coordinate information and analyzing the action trend of the action node based on the changes in the spatiotemporal action characteristics. The instruction transmission module is used to select a transmission strategy based on the action trend and transmit the coordinate information corresponding to the action node of the instruction terminal to the cloud. The positioning module is used to obtain the coordinate information of each action node of the response terminal, locate the action cycle to which it belongs based on the similarity between the coordinate information and the coordinates of the command terminal in each action cycle, and continuously verify the coordinate timing matching of subsequent action cycles. The interactive synchronization module responds to the verification result of the positioning module and uses it to select a transmission strategy based on the action trend to transmit the coordinate information corresponding to the action node of the responding terminal to the cloud. The model building module is used to restore all the corresponding coordinate information in the disassembly unit based on the remaining coordinate information in the disassembly unit, and to build the corresponding three-dimensional model based on the coordinate information. The transmission strategy includes decomposing coordinate information into several decomposition units based on spatiotemporal action characteristics in the temporal dimension, extracting and transmitting a portion of the coordinate information of each decomposition unit, or transmitting all the coordinate information.

[0007] Furthermore, the terminal analysis module analyzes the action trends of action nodes based on changes in spatiotemporal action characteristics, including: Used to determine the spatiotemporal action characteristics of action nodes within an action cycle, obtain the changes in spatiotemporal action characteristics of several sub-cycles relative to adjacent sub-cycles, and solve for the stability corresponding to the action node. If the stability is greater than or equal to the predetermined stability threshold, the action node is determined to be a steady-state action trend. If the stability is less than the predetermined stability threshold, the action node is determined to be an unsteady action trend. The spatiotemporal motion features include acceleration values, motion direction change angles, and movement speed values ​​at each moment, and the stability is determined based on the statistical variance of the changes in the spatiotemporal motion features.

[0008] Furthermore, the instruction transmission module transmits coordinate information to the cloud, including: For action nodes with steady-state action trends, select several decomposition units by decomposing the coordinate information in the time domain based on spatiotemporal action characteristics, and extract part of the coordinate information of each decomposition unit and transmit it to the cloud. For action nodes with non-steady-state action trends, we choose to transmit all coordinate information to the cloud.

[0009] Furthermore, the instruction transmission module and the interactive synchronization module perform time-domain dimension decomposition to determine several decomposition units, including: Used to perform temporal segmentation based on the action segmentation time, and the segmented time period is determined as the decomposition unit; The action splitting moment is captured on the created timeline based on spatiotemporal action features.

[0010] Furthermore, the instruction transmission module and the interactive synchronization module determine the moment on the time axis that meets the action segmentation condition as the action segmentation moment; The motion segmentation conditions include: the moving speed value is equal to a predetermined speed threshold, or the acceleration value is not within the acceleration value threshold range, or the change angle of the motion direction is greater than a predetermined change angle threshold.

[0011] Furthermore, the positioning module determines the corresponding action cycle based on the similarity between the coordinate information and the coordinates of the command terminal within each action cycle. The positioning module compares the coordinate information of each action node with the coordinate information of the corresponding action node in several action cycles of the command terminal to determine the action cycle corresponding to the maximum coordinate similarity. If the coordinate similarity corresponding to the action cycle is greater than a predetermined similarity threshold, then the located action node belongs to the action cycle; The coordinate similarity is determined based on the single coordinate similarity of each action node.

[0012] Furthermore, the positioning module continuously verifies the coordinate timing matching of the action cycle, including: The coordinate information of the corresponding action node of the response terminal in the subsequent action cycle is extracted and compared with the coordinate information of the corresponding action node of the instruction terminal in the corresponding action cycle to determine the coordinate similarity. Specifically, only a portion of the coordinate information within a predetermined proportion of the time period after the start of the action cycle is extracted. If the coordinate similarity is greater than a predetermined temporal matching threshold, the existence of coordinate temporal matching in the action cycle is verified.

[0013] Furthermore, the interactive synchronization module responds to the positioning module's verification of the temporal matching of the coordinate information, wherein, For action nodes with steady-state action trends, select several decomposition units by decomposing the coordinate information in the time domain based on spatiotemporal action characteristics, and extract part of the coordinate information of each decomposition unit and transmit it to the cloud. For action nodes with non-steady-state action trends, we choose to transmit all coordinate information to the cloud.

[0014] Furthermore, the model building module restores the decomposition unit containing all coordinate information, including: In the time axis, the disassembly unit is calibrated to identify two types of moments: one containing coordinate information and the other not containing coordinate information. Time groups are formed based on temporal sequence, and the coordinate information corresponding to the second type of time is predicted using the coordinate information corresponding to the first type of time in the time group. The time group includes adjacent first-class and second-class times.

[0015] Furthermore, the coordinate information corresponding to the predicted second-type time points includes, Read the coordinate information of the first type of time and determine the momentum characteristics, including the velocity, direction of motion, and time interval between the second type of time and the first type of time; Based on the coordinate information corresponding to the first type of time, the coordinate information of the second type of time after the corresponding time interval is predicted according to the momentum characteristics.

[0016] Compared with existing technologies, this invention uses a terminal analysis module to perform motion analysis on the coordinate information of each action node of the command terminal, analyzes the action trend of the action nodes, and uses an adaptive transmission strategy based on the action trend to transmit the coordinate information corresponding to the action nodes of the command terminal to the cloud. A positioning module verifies the coordinate timing matching of the action cycle, and an interactive synchronization module responds to the verification results of the positioning module, adaptively selecting a transmission strategy to transmit the coordinate information corresponding to the action nodes of the command terminal to the cloud. Finally, a model building module reconstructs all the coordinate information within the decomposition unit and constructs the corresponding 3D model. This invention, through its action trend-adaptive transmission strategy, ensures the readability of coordinate data when facing a large amount of terminal data transmission, completes the modeling function, saves transmission bandwidth, and improves the efficiency of multi-terminal data interaction.

[0017] In particular, this invention performs motion analysis on each action node every action cycle. In practice, the command terminal is the input end of the action example, and the motion trends of the actual action nodes are diverse. Some action nodes have a single motion trend with strong regularity, such as linear motion or motion with a specific curvature. For these types of action nodes, the backend has strong predictability using local coordinate information. Conversely, some action nodes have complex motion trends with weak regularity, and the backend has weak predictability using local coordinate information. Based on this, this invention obtains spatiotemporal motion features, which include multiple dimensions such as acceleration, motion direction, and motion speed. By analyzing the changes in spatiotemporal motion features, the stability and motion trend of the motion process of the action node are reflected, which characterizes the predictability of the backend for the motion form of the action node. This facilitates the subsequent selection of transmission strategies based on motion trend adaptability, ensuring the readability of coordinate data when facing a large amount of terminal transmission data, completing the modeling function, saving transmission bandwidth, and improving the efficiency of multi-terminal data interaction.

[0018] In particular, the instruction transmission module of this invention transmits the coordinate information corresponding to the action nodes of the instruction terminal to the cloud based on the action trend selection transmission strategy. For steady-state action trends, the coordinate information is decomposed based on spatiotemporal action characteristics to determine several decomposition units. Based on this, the motion of the action node is further decomposed using action segmentation conditions to lock the potential action breakpoints during the motion of the action node. The action segmentation conditions reflect the time when the motion node stops, the time when the acceleration changes significantly, and the time when the motion direction changes significantly. As a result, the motion form corresponding to the action node in a single decomposition unit is relatively simple. Under this simple motion form, it is easier to predict the remaining coordinate points using local coordinate points, and the simple motion form can better ensure the accuracy of the prediction. Subsequently, the coordinate information of the action node corresponding to each decomposition unit can be restored separately, and then the coordinate information can be integrated. By extracting part of the coordinate information of the decomposition unit for transmission, and then restoring all the coordinate information corresponding to the decomposition unit based on the transmitted coordinate information, the amount of data transmission is greatly reduced, saving bandwidth. Moreover, the backend uses local coordinate information to restore the remaining coordinate information, which is reliable and ensures the readability of the coordinate data, thus completing the modeling function.

[0019] In particular, this invention sets up a positioning module to locate the action cycle to which the response terminal belongs. In practice, after the instruction transmission module transmits the coordinate information corresponding to the taught action to the cloud, modeling can be performed. After the response terminal downloads the modeling data, it can follow and practice according to the taught action corresponding to the modeling data. Since there is a certain delay in the transmission process, it is necessary to locate the coordinate information of each action node of the response terminal relative to the action cycle to which the instruction terminal belongs. After the positioning is completed, the characteristics of action following can be used to verify the coordinate timing matching of subsequent action cycles using only local coordinate information in each cycle. This reflects whether the coordinate information of the response terminal corresponds to that of the instruction terminal in each action cycle. Based on this, the computing power required for verification is saved while ensuring reliability. Moreover, when the action cycle has coordinate timing matching, the transmission strategy corresponding to the action node of the instruction terminal is adaptively selected to transmit the coordinate information of the action node of the response terminal by using the already determined action trend of the action node of the instruction terminal. When facing a large number of terminals transmitting data, the readability of the coordinate data can be guaranteed, the transmission bandwidth can be saved, and the efficiency of multi-terminal data interaction can be improved. Attached Figure Description

[0020] Figure 1 A simplified structural diagram of an intelligent synchronization system based on multi-terminal interaction, as an embodiment of the invention; Figure 2 A logic block diagram for analyzing the action trends of action nodes in an embodiment of the invention; Figure 3 This is a logic block diagram for determining the action segmentation time in an embodiment of the invention. Figure 4This is a logic block diagram for verifying the coordinate timing matching of the action cycle in an embodiment of the invention. Detailed Implementation

[0021] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0022] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0023] Please see Figure 1 The diagram shown is a simplified structural diagram of an intelligent synchronization system based on multi-terminal interaction according to an embodiment of the invention. The intelligent synchronization system based on multi-terminal interaction according to an embodiment of the invention includes: The terminal analysis module is used to acquire the coordinate information of each action node of the command terminal in real time, and to perform motion analysis on each action node every action cycle, including determining the spatiotemporal action characteristics based on the coordinate information and analyzing the action trend of the action node based on the changes in the spatiotemporal action characteristics. The instruction transmission module is used to select a transmission strategy based on the action trend and transmit the coordinate information corresponding to the action node of the instruction terminal to the cloud. The positioning module is used to obtain the coordinate information of each action node of the response terminal, locate the action cycle to which it belongs based on the similarity between the coordinate information and the coordinates of the command terminal in each action cycle, and continuously verify the coordinate timing matching of subsequent action cycles. The interactive synchronization module responds to the verification result of the positioning module and uses it to select a transmission strategy based on the action trend to transmit the coordinate information corresponding to the action node of the responding terminal to the cloud. The model building module is used to restore all the corresponding coordinate information in the disassembly unit based on the remaining coordinate information in the disassembly unit, and to build the corresponding three-dimensional model based on the coordinate information. The transmission strategy includes decomposing coordinate information into several decomposition units based on spatiotemporal action characteristics in the temporal dimension, extracting and transmitting a portion of the coordinate information of each decomposition unit, or transmitting all the coordinate information.

[0024] Specifically, there are no restrictions on the type of command terminal and response terminal. They can be mobile communication devices with visual acquisition capabilities that acquire image data and then detect the coordinate information of human motion nodes. In practice, there are no restrictions on the method of detecting motion nodes and their corresponding coordinate information. For example, MediaPipe Pose can be used to extract the coordinate information of motion nodes in video data. Of course, other methods can also be used, which are existing technologies and will not be elaborated further.

[0025] Specifically, the terminal analysis module, command transmission module, positioning module, and interactive synchronization module are deployed on the command terminal and response terminal, while the model building module is deployed in the cloud. Each module can be a computer program module that can perform the corresponding function. A computer program module refers to computer program code stored in a computer-readable storage medium, which performs the corresponding function when executed by a processor.

[0026] Specifically, the teaching user uses the instruction terminal to obtain the coordinate information of each action node corresponding to the teaching action, and then uploads it to the cloud for 3D modeling. The follow-up user uses the response terminal. After the model building module restores all the corresponding coordinate information in the decomposed unit, it can build the 3D model of the instruction terminal in the cloud. After the response terminal obtains the 3D model from the cloud, it can perform synchronous follow-up training based on the 3D model and facilitate multi-angle observation of the teaching action. The response terminal can also upload the coordinate information of each action node corresponding to the follow-up action for 3D modeling, which makes it easier for the teacher to observe the accuracy of the follow-up action from multiple angles. In some possible applications, the established 3D model or coordinate information of the teacher can also be used to analyze the accuracy of the action, which will not be elaborated further.

[0027] Specifically, there is no limitation on the specific method of constructing a three-dimensional model using three-dimensional coordinates. The coordinate information is uploaded to the cloud with a timestamp and the corresponding action node. In practice, any existing technology that can reconstruct a three-dimensional model based on the three-dimensional coordinate information of the action node can be applied to this invention. For example, a preset standardized three-dimensional human body template can be directly driven using three-dimensional coordinates. Of course, other methods can also be used, which will not be elaborated here.

[0028] Specifically, please refer to Figure 2 As shown, it is a logical block diagram of the action trend analysis of action nodes according to an embodiment of the invention. The terminal analysis module analyzes the action trend of action nodes based on changes in spatiotemporal action characteristics, including: Used to determine the spatiotemporal action characteristics of action nodes within an action cycle, obtain the changes in spatiotemporal action characteristics of several sub-cycles relative to adjacent sub-cycles, and solve for the stability corresponding to the action node. If the stability is greater than or equal to the predetermined stability threshold, the action node is determined to be a steady-state action trend. If the stability is less than the predetermined stability threshold, the action node is determined to be an unsteady action trend. The spatiotemporal motion features include acceleration values, motion direction change angles, and movement speed values ​​corresponding to each moment. The stability is determined based on the statistical variance of the changes in the spatiotemporal motion features, and the motion direction change angle is the angle between the motion direction at the current moment and the motion direction at the previous moment.

[0029] Specifically, a shorter action cycle is used to consider the coordinate information of the action node in order to avoid including multiple actual actions within a single action cycle. Therefore, the action cycle is selected within the range [0.5s, 1.5s], preferably 1s, and the sub-cycle is set to 0.1 times the action cycle.

[0030] The spatiotemporal characteristic changes include the changes in acceleration value, the changes in the angle of change of motion direction, and the changes in the value of movement speed. When calculating, the mean of the spatiotemporal motion characteristics in the sub-period and the adjacent previous sub-period can be calculated separately and then the difference can be taken to obtain the spatiotemporal characteristic changes. When calculating stability, the statistical variances of several spatiotemporal characteristic changes are first calculated, including the statistical variances corresponding to the changes in acceleration values, the statistical variances corresponding to the changes in the angle of change of the direction of motion, and the statistical variances corresponding to the changes in the speed values. After normalizing each statistical variance to the interval [0.1], the mean is calculated, and the reciprocal of the mean is used as the stability.

[0031] Specifically, the purpose of setting a stability threshold is to characterize situations where the motion of the action nodes is relatively stable within the action cycle, making it easier to predict the remaining coordinates using local coordinates. When the stability threshold is determined, The coordinate information of each action node in several action cycles is pre-recorded as a sample. At the same time, some coordinate information corresponding to each action node in the action cycle is extracted. The remaining coordinate information is used to predict all coordinate information. Then, the prediction accuracy for each action node in each action cycle is statistically analyzed. The coordinate information of the action node in the action cycle when the prediction accuracy is greater than 90% is recorded. The stability is calculated using the coordinate information. Then, the average stability value is calculated and set as the stability threshold to reflect the overall level of stability of the coordinate information when the accuracy is high. When calculating the prediction accuracy of action nodes, the accuracy is determined by comparing the total number of predicted coordinates within the action cycle with the actual coordinates. The ratio of the number of accurately predicted coordinates to the total number of predicted coordinates is used to determine the prediction accuracy. When determining whether the predicted coordinate information is accurate with the actual coordinate information, the distance between the predicted coordinates and the actual coordinates is determined. If the distance is greater than the threshold distance, the prediction is determined to be inaccurate. The threshold distance is the distance between 5 and 15 pixels in the image data, preferably the distance between 10 pixels. Taking a 720P image as an example, this distance is within the acceptable error range for the human eye.

[0032] This invention performs motion analysis on each action node every action cycle. In practice, the command terminal is the input end of the action example. The motion trends of actual action nodes are diverse. Some action nodes have a single motion trend with strong regularity, such as linear motion or motion with a specific curvature. For these types of action nodes, the backend has strong predictability using local coordinate information. Conversely, some action nodes have complex motion trends with weak regularity, and the backend has weak predictability using local coordinate information. Based on this, this invention obtains spatiotemporal motion features, which include multiple dimensions such as acceleration, motion direction, and motion speed. By analyzing the changes in spatiotemporal motion features, the stability and motion trend of the motion process of the action node are reflected, which characterizes the predictability of the backend for the motion form of the action node. This facilitates the subsequent selection of transmission strategies based on motion trend adaptation, ensuring the readability of coordinate data when facing a large amount of terminal transmission data, completing the modeling function, saving transmission bandwidth, and improving the efficiency of multi-terminal data interaction.

[0033] Specifically, the instruction transmission module transmits coordinate information to the cloud, including: For action nodes with steady-state action trends, select several decomposition units by decomposing the coordinate information in the time domain based on spatiotemporal action characteristics, and extract part of the coordinate information of each decomposition unit and transmit it to the cloud. For action nodes with non-steady-state action trends, we choose to transmit all coordinate information to the cloud.

[0034] The instruction transmission module of this invention transmits the coordinate information corresponding to the action nodes of the instruction terminal to the cloud based on the action trend selection transmission strategy. For steady-state action trends, the coordinate information is decomposed based on spatiotemporal action characteristics to determine several decomposition units. Based on this, the motion of the action node is further decomposed using action segmentation conditions to lock the potential action breakpoints during the motion of the action node. The action segmentation conditions reflect the time when the motion node stops, the time when the acceleration changes significantly, and the time when the motion direction changes significantly. Therefore, the motion form corresponding to the action node in a single decomposition unit is relatively simple. Under this simple motion form, it is easier to predict the remaining coordinate points using local coordinate points, and the simple motion form can better ensure the accuracy of the prediction. Subsequently, the coordinate information of the action node corresponding to each decomposition unit can be restored separately, and then the coordinate information can be integrated. By extracting part of the coordinate information of the decomposition unit for transmission, and then restoring all the coordinate information corresponding to the decomposition unit based on the transmitted coordinate information, the amount of data transmission is greatly reduced, saving bandwidth. Moreover, the backend uses local coordinate information to restore the remaining coordinate information, which is reliable and ensures the readability of the coordinate data, thus completing the modeling function.

[0035] Specifically, there is no limitation on the method of extracting the corresponding coordinate information within the disassembly unit. For example, coordinate information can be extracted at intervals according to the time sequence, such as extracting a single coordinate information at intervals. Of course, the number of intervals and the number of extractions can be adjusted, which will affect the accuracy of subsequent coordinate prediction. Those skilled in the art can make adjustments according to their needs.

[0036] It is understandable that when the command terminal obtains the coordinate information of the action node, it can cache it on the command terminal for motion parsing. Since motion parsing consumes low computing power and can be completed quickly, the subsequent transmission strategy can be adaptively selected based on the results of motion parsing to transmit the coordinate information to the cloud.

[0037] Specifically, the instruction transmission module and the interactive synchronization module perform time-domain dimension decomposition to determine several decomposition units, including: Used to perform temporal segmentation based on the action segmentation time, and the segmented time period is determined as the decomposition unit; The action splitting moment is captured on the created timeline based on spatiotemporal action features.

[0038] It is understandable that the action segmentation divides the timeline into several time periods. The timeline is created based on the corresponding action cycle, which will not be elaborated further.

[0039] Specifically, please refer to Figure 3 As shown, Figure 3 The following is a logic block diagram for determining the action segmentation time in an embodiment of the invention. The instruction transmission module and the interactive synchronization module determine the moment on the time axis that meets the action segmentation conditions as the action segmentation time. The motion segmentation conditions include: the moving speed value is equal to a predetermined speed threshold, or the acceleration value is not within the acceleration value threshold range, or the change angle of the motion direction is greater than a predetermined change angle threshold.

[0040] The purpose of setting a predetermined speed threshold is to observe the moment when the motion node pauses. Therefore, the predetermined speed threshold is 0 m / s. The purpose of setting an acceleration value threshold range is to observe the moment when the acceleration of the motion node fluctuates greatly. The acceleration value threshold range changes in real time and is determined based on the acceleration value of the adjacent previous moment. It is essentially a closed interval. The upper limit of the interval is 1.25 times the acceleration value, and the lower limit of the interval is 0.75 times the acceleration value. The purpose of setting a change angle threshold is to observe the situation where the turning amplitude of the motion node suddenly increases. When the change angle of the motion direction of the motion node is greater than 30°, the change amplitude of the motion node is observed to be large by the naked eye. In practice, the change angle threshold is set to 30°.

[0041] Specifically, the positioning module determines the corresponding action cycle based on the similarity between coordinate information and the coordinates of the command terminal within each action cycle. The positioning module compares the coordinate information of each action node with the coordinate information of the corresponding action node in several action cycles of the command terminal to determine the action cycle corresponding to the maximum coordinate similarity. If the coordinate similarity corresponding to the action cycle is greater than a predetermined similarity threshold, then the located action node belongs to the action cycle; The coordinate similarity is determined based on the single coordinate similarity of each action node.

[0042] It is understandable that the teaching action of the instruction terminal is generated in time sequence before the follow-up action of the response terminal. Therefore, it is necessary to compare the coordinate information of the response terminal with the coordinate information of the corresponding action node within several action cycles of the instruction terminal to determine the action cycle corresponding to the maximum coordinate similarity in order to perform positioning. It is understandable that there are multiple action nodes involved in the comparison within the action cycle. Therefore, the single coordinate similarity is the cosine similarity between a single action node and the corresponding trajectory of the corresponding action node in the instruction terminal within the action cycle. Coordinate similarity is the average of the individual coordinate similarities of each action node within the action cycle.

[0043] The predetermined similarity threshold was obtained from the experiment. A teacher was selected to demonstrate the teaching action and the coordinate data of the action node was collected through the command terminal. Several learners imitated the teaching action and the coordinate data of the action node was collected through the response terminal and the video data was recorded synchronously. The teacher selected the coordinate data of several action cycles corresponding to the learners who achieved the learning standard. Furthermore, the coordinate similarity between the coordinate data and the coordinate data of the action node collected by the command terminal is calculated separately, and the mean coordinate similarity is further calculated. In order to allow a certain error, the product of the mean coordinate similarity and the error adjustment coefficient is set as a predetermined similarity threshold. The error adjustment coefficient is selected in the range [0.75, 0.95] to appropriately reduce the mean coordinate similarity and allow a certain error. In practice, 0.85 is preferred.

[0044] Specifically, Figure 4 The following is a logic block diagram illustrating the verification of the coordinate timing matching of an action cycle according to an embodiment of the invention. The positioning module continuously verifies the coordinate timing matching of the action cycle, including: The coordinate information of the corresponding action node of the response terminal in the subsequent action cycle is extracted and compared with the coordinate information of the corresponding action node of the instruction terminal in the corresponding action cycle to determine the coordinate similarity. Specifically, only a portion of the coordinate information within a predetermined proportion of the time period after the start of the action cycle is extracted. If the coordinate similarity is greater than a predetermined temporal matching threshold, the existence of coordinate temporal matching in the action cycle is verified.

[0045] Understandably, after locating the corresponding action cycle, it is only necessary to select the action cycle of the instruction terminal in sequence according to the timing order, extract the partial coordinate information of each action node and perform coordinate timing matching verification with the partial coordinate information of the action node in the current action cycle of the response terminal.

[0046] Specifically, the predetermined ratio is set to 30% to avoid insufficient data representation. The temporal matching threshold is set based on the similarity threshold. Since only a portion of the coordinate data at the beginning of the action cycle is extracted for verification, a more lenient verification boundary can be given. The temporal matching threshold is set to 0.85 times the similarity threshold.

[0047] It is understandable that the method of calculating coordinate similarity remains unchanged. Instead, the form of calculating coordinate similarity by extracting partial coordinate information of action nodes within the action cycle is essentially the same as calculating coordinate similarity using a shorter action cycle, which will not be elaborated further.

[0048] Specifically, the interactive synchronization module responds to the positioning module's verification of the temporal matching of coordinate information, wherein... For action nodes with steady-state action trends, select several decomposition units by decomposing the coordinate information in the time domain based on spatiotemporal action characteristics, and extract part of the coordinate information of each decomposition unit and transmit it to the cloud. For action nodes with non-steady-state action trends, we choose to transmit all coordinate information to the cloud.

[0049] This invention sets up a positioning module to determine the action cycle to which the response terminal belongs. In practice, after the instruction transmission module transmits the coordinate information corresponding to the taught action to the cloud, modeling can be performed. After downloading the modeling data, the response terminal can follow along with the taught action based on the modeling data. Due to the certain delay in the transmission process, it is necessary to determine the coordinate information of each action node of the response terminal relative to the action cycle to which the instruction terminal belongs. After completing the positioning, the characteristics of action following can be utilized to verify the coordinate timing matching of subsequent action cycles using only local coordinate information in each cycle. This reflects whether the coordinate information of the response terminal corresponds to that of the instruction terminal in each action cycle. Based on this, the computational power required for verification is saved while ensuring reliability. Furthermore, when the action cycle has coordinate timing matching, the transmission strategy is adaptively selected to transmit the coordinate information corresponding to the action node of the response terminal by utilizing the already determined action trend of the corresponding action node of the instruction terminal. When facing a large number of terminals transmitting data, the readability of the coordinate data can be guaranteed, transmission bandwidth can be saved, and the efficiency of multi-terminal data interaction can be improved.

[0050] Specifically, the model building module restores the decomposition unit containing all coordinate information, including: In the time axis, the disassembly unit is calibrated to identify two types of moments: one containing coordinate information and the other not containing coordinate information. Time groups are formed based on temporal sequence, and the coordinate information corresponding to the second type of time is predicted using the coordinate information corresponding to the first type of time in the time group. The time group includes adjacent first-class and second-class times.

[0051] Specifically, the coordinate information for predicting the second type of time includes: Read the coordinate information of the first type of time and determine the momentum characteristics, including the velocity, direction of motion, and time interval between the second type of time and the first type of time; Based on the coordinate information corresponding to the first type of time, the coordinate information of the second type of time after the corresponding time interval is predicted according to the momentum characteristics.

[0052] In practice, knowing the starting point of the action node, the coordinate information of the action node after moving at the corresponding speed and direction of motion for the corresponding time interval is predicted. Of course, there are many ways to predict coordinates, and those skilled in the art can also use other coordinate prediction methods, which will not be elaborated here.

[0053] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A multi-end interactive based intelligent synchronization system, characterized in that, include: The terminal analysis module is used to acquire the coordinate information of each action node of the command terminal in real time, and to perform motion analysis on each action node every action cycle, including determining the spatiotemporal action characteristics based on the coordinate information and analyzing the action trend of the action node based on the changes in the spatiotemporal action characteristics. The instruction transmission module is used to select a transmission strategy based on the action trend and transmit the coordinate information corresponding to the action node of the instruction terminal to the cloud. The positioning module is used to obtain the coordinate information of each action node of the response terminal, locate the action cycle to which it belongs based on the coordinate information and the coordinate similarity of the command terminal in each action cycle, and continuously verify the coordinate timing matching of subsequent action cycles, including extracting part of the coordinate information of the corresponding action node of the response terminal in subsequent action cycles and comparing it with part of the coordinate information of the action node of the command terminal in the corresponding action cycle to determine the coordinate similarity. Specifically, only a portion of the coordinate information within a predetermined proportion of the time period after the start of the action cycle is extracted. If the coordinate similarity is greater than the predetermined temporal matching threshold, the existence of coordinate temporal matching in the action cycle is verified. The interactive synchronization module responds to the location module's verification of coordinate timing matching, and uses this to select a transmission strategy based on action trends to transmit the coordinate information corresponding to the action node of the responding terminal to the cloud. The action trend-based transmission strategy includes, For action nodes with steady-state action trends, select several decomposition units by decomposing the coordinate information in the time domain based on spatiotemporal action characteristics, and extract part of the coordinate information of each decomposition unit and transmit it to the cloud. For action nodes with non-steady-state action trends, choose to transmit all coordinate information to the cloud; The model building module is used to restore all the corresponding coordinate information in the disassembly unit based on the remaining coordinate information in the disassembly unit, and to build the corresponding three-dimensional model based on the coordinate information. The model building module restores all the coordinate information corresponding to the disassembly unit, including... In the time axis, the disassembly unit is calibrated to identify two types of moments: one containing coordinate information and the other not containing coordinate information. Time groups are formed based on temporal sequence, and the coordinate information corresponding to the second type of time is predicted using the coordinate information corresponding to the first type of time in the time group. The time group includes adjacent first-class and second-class times.

2. The multi-end interaction based intelligent synchronization system of claim 1, wherein, The terminal analysis module analyzes the action trends of action nodes based on changes in spatiotemporal action characteristics, including... Determine the spatiotemporal action characteristics of the action node within the action cycle, obtain the changes in spatiotemporal action characteristics of several sub-cycles relative to adjacent sub-cycles, and solve for the stability corresponding to the action node. If the stability is greater than or equal to the predetermined stability threshold, the action node is determined to be a steady-state action trend. If the stability is less than the predetermined stability threshold, the action node is determined to be an unsteady action trend. The spatiotemporal motion features include acceleration values, motion direction change angles, and movement speed values ​​at each moment, and the stability is determined based on the statistical variance of the changes in the spatiotemporal motion features.

3. The multi-end interaction based intelligent synchronization system of claim 1, wherein, The instruction transmission module and the interactive synchronization module perform time-domain dimension decomposition to determine several decomposition units, including... Temporal segmentation is performed based on the action segmentation time, and the segmented time period is determined as the decomposition unit; The action splitting moment is captured on the created timeline based on spatiotemporal action features.

4. The multi-end interaction based intelligent synchronization system of claim 3, wherein, The instruction transmission module and the interactive synchronization module determine the moment on the time axis that meets the action segmentation condition as the action segmentation moment. The motion segmentation conditions include: the moving speed value is equal to a predetermined speed threshold, or the acceleration value is not within the acceleration value threshold range, or the change angle of the motion direction is greater than a predetermined change angle threshold.

5. The multi-end interaction based intelligent synchronization system of claim 1, wherein, The positioning module determines the corresponding action cycle based on the similarity between coordinate information and the coordinates of the command terminal within each action cycle. The positioning module compares the coordinate information of each action node with the coordinate information of the corresponding action node in several action cycles of the command terminal to determine the action cycle corresponding to the maximum coordinate similarity. If the coordinate similarity corresponding to the action cycle is greater than a predetermined similarity threshold, then the located action node belongs to the action cycle; The coordinate similarity is determined based on the single coordinate similarity of each action node.

6. The multi-end interaction based intelligent synchronization system of claim 1, wherein, The coordinate information for predicting the second type of time includes, Read the coordinate information of the first type of time and determine the momentum characteristics, including the velocity, direction of motion, and time interval between the second type of time and the first type of time; Based on the coordinate information corresponding to the first type of time, the coordinate information of the second type of time after the corresponding time interval is predicted according to the momentum characteristics.

Images