Visual interaction system for control and real-time feedback of digital twin cross-platform equipment

Abstract

The invention relates to the field of interaction systems, in particular to a visual interaction system for digital twin cross-platform equipment control and real-time feedback, which is characterized in that through cooperative work of a pre-processing module, a data analysis module, a sampling scheduling module and a twin construction module, sampling data and videos are acquired based on the pre-processing module; the data analysis module calibrates a target contour and a sub-contour contained in the target contour and analyzes a motion trend consistency parameter, and then the sampling scheduling module controls the sampling unit to sample based on the motion trend consistency parameter, and sets a sampling label for the obtained sampling data; and finally, the twinning construction module constructs the digital twinning model in different modes according to the sampling labels. When the digital twinning model is constructed, the computing power can be saved, the resource utilization rate can be improved, and the construction effect of the digital twinning model can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of interactive systems, and in particular to a visual interactive system for digital twin cross-platform device control and real-time feedback. Background Art

[0002] With the deep integration of information technology and industrial systems, digital twin technology, as an innovative virtualization method, plays an important role in industrial automation, intelligent monitoring and other fields. By adopting the sampling method, relevant sampling data of equipment controlled by various control platforms can be obtained, and then a twin model can be constructed. Digital twin technology can not only dynamically reflect the real-time status of equipment and systems, but also optimize the decision-making process through simulation and prediction. The application of digital twin technology provides a solid foundation for various industries to move towards the intelligent era.

[0003] Chinese patent publication number: CN112462722A, discloses a real-time digital twin factory system for control and display, including a two-dimensional module, a three-dimensional module and an equipment module that establish communication connections with each other, the equipment module includes a controlled object and a control unit, the two-dimensional module is used to display the controlled object in a two-dimensional manner so that the operator can control its state; the three-dimensional module is used to display the controlled object in a three-dimensional manner so that the operator can display the controlled object currently controlled or viewed, the control unit is used to send the attribute information of the controlled object to the two-dimensional module and / or the three-dimensional module, so as to control the controlled object in response to the instruction information of the two-dimensional module and / or the three-dimensional module, the three-dimensional graphic component of the controlled object establishes a mapping association relationship with the two-dimensional graphic component of the controlled object, so that the operator can locate the two-dimensional graphic component in the three-dimensional view, and the attributes of the controlled object can be displayed in parallel in two-dimensional and three-dimensional manners in response to the operation of the operator, providing a light-load, real-time, cross-platform system.

[0004] However, there are still the following problems in the prior art: In actual situations, as the number of targets that need to build digital twin models continues to increase, the overall amount of computing increases exponentially, increasing the computing power burden. At the same time, some targets may contain multiple components that can operate independently. Since the movement trends of the targets and the components contained in the targets have different requirements for twin model construction and different sampling requirements, the existing system lacks differentiated processing for the movement trends of these independent components and uses the same sampling method and model construction method, resulting in high computing power requirements, waste of computing power, and reduced resource utilization efficiency and response speed of the system. Summary of the invention

[0005] To this end, the present invention provides a visual interactive system for cross-platform device control and real-time feedback of digital twins to overcome the problem in the prior art that as the number of targets that need to build digital twin models continues to increase, the overall amount of calculation increases exponentially, increasing the computing power burden; at the same time, since the movement trends of targets and components included in the targets have different requirements for twin model construction and different sampling requirements, the same sampling method and model construction method are used, which leads to high computing power requirements, waste of computing power, and reduced resource utilization efficiency and response speed of the system.

[0006] To achieve the above objectives, the present invention provides a visual interactive system for cross-platform digital twin device control and real-time feedback, which includes: A pre-processing module, comprising a sampling unit arranged in a sampling area for collecting a number of sampling data and a visual perception unit for collecting a number of target videos; A data analysis module, connected to the pre-processing module, for calibrating a target contour and each sub-contour within the target contour in a target video within a predetermined time domain segment, and analyzing a motion trend consistency parameter based on motion trajectories of key points of the target contour and each sub-contour; A sampling scheduling module, which is connected to the pre-processing module and the data analysis module respectively, and is used to control the sampling unit to perform sampling based on the motion trend consistency parameter and set a label for the sampled data; A twin construction module, which is connected to the pre-processing module, the data analysis module and the sampling scheduling module respectively, is used to receive the sampling data acquired by the sampling unit and construct a digital twin model based on the label of the sampling data, including: A digital twin model is constructed in a virtual space based on the sampling data corresponding to the first sampling time group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampling data corresponding to the adjacent sampling time group; Or, determine the time domain variation curve of the image parameters within the target contour based on the target video, identify the periodic time domain segment, update the digital twin model in real time based on the sampling data for the non-periodic time domain segment, and replicate and construct the digital twin model for the periodic time domain segment; The sampling time group includes a number of continuous sampling times, and controlling the sampling unit to perform sampling includes performing sampling based on the sampling time group at each sampling interval, or performing real-time sampling.

[0007] Furthermore, the data analysis module is also used to analyze the motion trend consistency parameters based on the motion trajectory of the target contour and the key points of each sub-contour, including: To respectively extract the target contour and the center coordinates of each sub-contour inside the target contour as key points; To determine the motion trajectory of each of the key points; To extract the velocity characteristics and direction vector characteristics of each of the said key points moving along the motion trajectory; To solve the average similarity of the velocity characteristics corresponding to the key points of the target contour at each moment and the velocity characteristics corresponding to the key points of each sub - contour as the velocity trend consistency characteristic factor; To solve the average similarity of the direction vector characteristics corresponding to the key points of the target contour at each moment and the direction vector characteristics corresponding to the key points of each sub - contour as the direction trend consistency characteristic factor; To perform a weighted sum of the velocity trend consistency characteristic factor and the direction trend consistency characteristic factor to obtain the motion trend consistency parameter.

[0008] Furthermore, the sampling scheduling module is used to control the sampling unit to perform sampling based on the motion trend consistency parameter, and set labels for the sampling data, including If the motion trend consistency parameter is greater than or equal to the preset standard motion trend consistency parameter threshold, then control the sampling unit to sample at each sampling interval based on the sampling time group, and set a motion trend consistency label for the sampling data; If the motion trend consistency parameter is less than the preset standard motion trend consistency parameter threshold, then control the sampling unit to sample in real - time, and set a motion trend non - consistency label for the sampling data.

[0009] Furthermore, it is characterized in that the twin construction module is used to construct a digital twin model based on the labels of the sampling data, including If the sampling data has a motion trend consistency label, then construct a digital twin model in the virtual space based on the sampling data corresponding to the first sampling time group, determine the movement index, control the movement of the digital twin model with the corresponding movement index, and update the movement index based on the sampling data corresponding to the adjacent sampling time group; If the sampling data has a motion trend non - consistency label, then determine the time - domain change curve of the image parameters within the target contour based on the target video, identify the periodic time - domain segments, update the digital twin model in real - time based on the sampling data for the non - periodic time - domain segments, and construct a digital twin model by replication for the periodic time - domain segments.

[0010] Furthermore, the twin construction module is used to construct a digital twin model in the virtual space based on the sampling data corresponding to the first sampling time group, and determine the movement index, including To call the sampling data corresponding to any sampling moment of the first sampling time group and construct a digital twin model based on the sampling data; To call the sampling data corresponding to at least two sampling moments in the sampling time group and determine the position information of the target at each sampling moment based on the sampling data; For determining a movement metric based on location information, the movement metric including speed and direction.

[0011] Further, the twin construction module is used to update the movement metric based on the sampling data corresponding to the adjacent sampling time groups, including For controlling the digital twin model to move in the virtual space according to the movement metric corresponding to the first sampling time group; For determining a movement metric based on the adjacent sampling time groups at each sampling interval, and controlling the digital twin model to move according to the newly determined movement metric.

[0012] Further, the twin construction module is used to determine the time-domain change curve of the image parameters within the target contour based on the target video, including For determining the chroma and saturation of the target contour in each video frame of the target video; For constructing a chroma time-domain change curve and a saturation time-domain change curve.

[0013] Further, the twin construction module is used to identify a periodic time segment, including For determining the corresponding periodic time segment in the chroma time-domain change curve; For determining the corresponding periodic time segment in the saturation time-domain change curve; For taking the intersection of the corresponding periodic time segment in the chroma time-domain change curve and the corresponding periodic time segment in the saturation time-domain change curve as the identified periodic time segment.

[0014] Further, the twin construction module is used to copy and construct a digital twin model for the periodic time segment, including For screening out the first period corresponding to the periodic time segment; For calling the sampling data corresponding to the first period to update the digital twin model in real time and storing the update record; For determining the remaining periods, and updating the digital twin model in each of the remaining periods according to the stored update record.

[0015] Further, it further includes a visualization module, which is connected to the twin construction module and is used to display the virtual scene and the digital twin model within the virtual scene.

[0016] Compared with the prior art, the present invention sets up the collaborative work of a pre-processing module, a data analysis module, a sampling scheduling module and a twin construction module. The pre-processing module collects sampling data and video, the data analysis module calibrates the target contour and the sub-contours contained in the target contour and analyzes the motion trend consistency parameters, and then the sampling scheduling module controls the sampling unit to perform sampling based on the motion trend consistency parameters, sets sampling labels for the obtained sampling data, and finally the twin construction module constructs a digital twin model in different ways according to the sampling labels. The present invention can save computing power and improve resource utilization when constructing a digital twin model, and ensure the construction effect of the digital twin model.

[0017] In particular, the present invention calibrates the target contour and the target contour contains sub-contours based on the data analysis module and analyzes the motion trend consistency parameters. In actual situations, if there are many targets that need to build twin models, the system computing power burden will be increased. Among them, the motion manifestations of various types of equipment are different. For example, the components of some equipment move in the same trend as a whole, and the demand for building a twin model based on local details is not high. Similarly, the demand for sampling is not high. Some equipment may contain multiple components that can operate independently, and the movement of these components may be more random or complex. In this case, this type of equipment has a high demand for building a twin model based on local details, and the sampling demand will also be high. Based on this, the present invention considers the motion trajectory of the key points of the target contour and the sub-contour, and then calculates the motion trend consistency parameters to characterize the overall motion trend of the equipment and the complexity of the local motion, so as to provide data support for the subsequent setting of sampling labels and adaptively build a digital twin model, thereby improving the resource utilization efficiency when building a digital twin model on the basis of ensuring accuracy.

[0018] In particular, the present invention sets sampling labels based on motion trend consistency parameters. In actual situations, the motion trends of devices that need to build twin models are complex and diverse. In the face of this complexity, if real-time sampling modeling is uniformly adopted for all targets, it will cause a waste of computing power. For example, for devices that move with the same overall trend and have no local motion details, they maintain a certain form during the movement process. During the entire movement process, their sampling data may be basically the same. Therefore, there is no need to obtain the sampling data for the device at each moment during sampling, nor is there any need to perform modeling based on real-time sampling data. Based on this, the present invention adaptively performs sampling and sets labels for the sampling data. For sampling data with consistent motion trends, the sampling data corresponding to the first sampling moment group is used to construct a digital twin model in the virtual space. The digital twin model is subsequently controlled according to the movement index, and the form of updating the digital twin model based on the sampling data in real time is not adopted, thereby saving computing power while ensuring the expressiveness of the digital twin model.

[0019] In particular, for the sampled data with inconsistent motion trends, the present invention determines the time-domain change curve of the image parameters within the target contour based on the target video, identifies the periodic time-domain segments to analyze whether the device has periodic motion. Furthermore, for the periodic time-domain segments, the digital twin model is constructed by replicating using their periodic characteristics, which can ensure the accuracy of the model and avoid unnecessary computational overhead. For the non-periodic time-domain segments, real-time updating of the digital twin model can ensure that it always remains consistent with the actual state of the physical entity, thereby improving the credibility and practicality of the model. Moreover, when constructing the digital twin model, computing power can be saved, resource utilization rate can be improved, and the construction effect of the digital twin model can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic structural diagram of a visual interaction system for digital twin cross-platform device control and real-time feedback according to an embodiment of the invention; Figure 2 It is a logical decision diagram for sampling based on the motion trend consistency parameter control sampling unit according to an embodiment of the present invention; Figure 3 It is a logical decision diagram for setting labels for sampling data according to an embodiment of the present invention; Figure 4 It is a logical block diagram for constructing a digital twin model based on the labels of sampling data according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] In order to make the objectives and advantages of the present invention clearer and more understandable, the present invention will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and do not limit the protection scope of the present invention.

[0023] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and defined, the term "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0024] Please refer to Figure 1As shown in the figure, it is a schematic structural diagram of a visual interaction system for digital twin cross-platform device control and real-time feedback according to an embodiment of the present invention. The visual interaction system for digital twin cross-platform device control and real-time feedback according to an embodiment of the present invention includes: A preprocessing module, which includes a sampling unit arranged in a sampling area for collecting a plurality of sampling data and a visual perception unit for collecting a plurality of target videos; A data analysis module, which is connected to the preprocessing module, and is used for calibrating the target contour in the target video within a predetermined time domain segment and each sub-contour inside the target contour, and analyzing the motion trend consistency parameter based on the motion trajectories of the key points of the target contour and each sub-contour. The predetermined time domain segment is selected within the interval [2s, 3s]; A sampling scheduling module, which is respectively connected to the preprocessing module and the data analysis module, and is used for controlling the sampling unit to perform sampling based on the motion trend consistency parameter and setting labels for the sampling data; A twin construction module, which is respectively connected to the preprocessing module, the data analysis module and the sampling scheduling module, and is used for receiving the sampling data obtained by the sampling unit and constructing a digital twin model based on the label of the sampling data, including, Constructing a digital twin model in the virtual space based on the sampling data corresponding to the first sampling moment group, determining a movement index, controlling the movement of the digital twin model with the corresponding movement index, and updating the movement index based on the sampling data corresponding to the adjacent sampling moment group; Or, determining a time-domain change curve of the image parameters inside the target contour based on the target video, identifying a periodic time domain segment, updating the digital twin model in real time based on the sampling data for the non-periodic time domain segment, and replicating and constructing the digital twin model for the periodic time domain segment; Wherein, the sampling moment group includes a plurality of consecutive sampling times. Controlling the sampling unit to perform sampling includes sampling based on the sampling moment group at every sampling interval, or real-time sampling.

[0025] Specifically, the structures of the data analysis module, the sampling scheduling module and the twin construction module are not limited, and they can be composed of logic components or combinations of logic components. The logic components include field programmable processors, computers or microprocessors in computers.

[0026] Specifically, the specific structure of the preprocessing module is not limited. The visual perception unit can be a photographic device, as long as it can obtain the target video corresponding to the target. The sampling unit can be a laser scanner or a depth photographic device, as long as it can obtain the scan data of the target. The scan data is the point cloud of the target, which is used for subsequent construction of the digital twin model. In practice, the target is a device, which will not be elaborated here.

[0027] Specifically, the present invention does not limit the method of constructing a digital twin model for the twin building blocks based on the labeled sampling data. The sampling data is point cloud, and a corresponding digital twin model can be constructed based on the point cloud. This is prior art and will not be elaborated here.

[0028] Specifically, the data analysis module is further configured to analyze the motion trend consistency parameters based on the target contour and the motion trajectories of the key points of each sub - contour, including extracting the central coordinates of the target contour and each sub - contour inside the target contour as key points respectively; determining the motion trajectories of each of the key points; extracting the speed characteristics and direction vector characteristics of each key point moving along the motion trajectory; solving the similarity mean value between the speed characteristics corresponding to the key points of the target contour at each moment and the speed characteristics corresponding to the key points of each sub - contour as the speed trend consistency characteristic factor; In implementation, when calculating the similarity mean value of the speed characteristics, first calculate the similarity between the direction vector characteristics corresponding to the key points of the target contour at each moment and the direction vector characteristics corresponding to the key points of each sub - contour respectively, and then solve the similarity mean value of the speed characteristics solving the similarity mean value between the direction vector characteristics corresponding to the key points of the target contour at each moment and the direction vector characteristics corresponding to the key points of each sub - contour as the direction trend consistency characteristic factor; In implementation, when calculating the similarity mean value of the direction vector characteristics, first calculate the similarity between the direction vector characteristics corresponding to the key points of the target contour at each moment and the direction vector characteristics corresponding to the key points of each sub - contour respectively, and then solve the similarity mean value of the direction vector characteristics; weighted - summing the speed trend consistency characteristic factor and the direction trend consistency characteristic factor to obtain the motion trend consistency parameter; In implementation, solve the speed difference ratio of the speed characteristics corresponding to the key points of the target contour and the key points of each sub - contour at each moment, and use the speed difference ratio as the similarity of the speed characteristics; In implementation, solve the angle difference ratio of the direction vector characteristics corresponding to the key points of the target contour and the key points of each sub - contour at each moment, and use the angle difference ratio as the similarity of the direction vector characteristics; It can be understood that the difference ratio of two values is the ratio of the absolute value of the difference between the two values to the mean value of the two values, which will not be elaborated here.

[0029] In implementation, when weighted - summing the speed trend consistency characteristic factor and the direction trend consistency characteristic factor, the weight of the speed trend consistency characteristic factor is 0.55, and the weight of the direction trend consistency characteristic factor is 0.45.

[0030] Specifically, there is no limitation on the method for identifying the target contour and each sub - contour inside the target contour. For example, existing image segmentation algorithms can be used, or other methods can be adopted, which will not be elaborated here.

[0031] Based on the data analysis module, the present invention calibrates the target contour and the sub - contours included in the target contour and analyzes the motion trend consistency parameter. In actual situations, if there are many targets for which twin models need to be constructed, it will increase the computing power burden of the system. Among them, the motion performance forms of various devices are different. For example, for some devices, all components move in the same trend as a whole, and their need to construct a twin model based on local details is not high. Similarly, the need for sampling is not high. Some devices may contain multiple independently operable components, and the motion of these components may be relatively random or complex. Then, the need for such devices to construct a twin model based on local details is relatively high, and the sampling requirement will also be relatively high. Based on this, the present invention calculates the motion trend consistency parameter by considering the motion trajectories of the key points of the target contour and the sub - contours, thereby characterizing the overall motion trend of the device and the complexity of local motion, facilitating providing data support for subsequent setting of sampling labels, and adaptively constructing a digital twin model, so as to improve the resource utilization efficiency when constructing the digital twin model on the basis of ensuring accuracy.

[0032] Please refer to Figure 2 and Figure 3 as shown in Figure 2 is the logical decision diagram for controlling the sampling unit to perform sampling based on the motion trend consistency parameter in the embodiment of the present invention. Figure 3 is the logical decision diagram for setting labels for sampling data in the embodiment of the present invention. Specifically, the sampling scheduling module is used to control the sampling unit to perform sampling based on the motion trend consistency parameter. Setting labels for sampling data includes If the motion trend consistency parameter is greater than or equal to the preset standard motion trend consistency parameter threshold, then control the sampling unit to perform sampling at each sampling interval based on the sampling time group, and set the motion trend consistency label for the sampling data; If the motion trend consistency parameter is less than the preset standard motion trend consistency parameter threshold, then control the sampling unit to perform real - time sampling, and set the motion trend non - consistency label for the sampling data.

[0033] Specifically, the purpose of setting the standard motion trend consistency parameter threshold is to characterize the situation where there is a large difference in the motion trends between the target contour and the corresponding sub - contour. In implementation, it is selected within the interval [0.15, 0.35].

[0034] Specifically, the sampling interval should not be too long to ensure continuous sampling of the entire motion process. In implementation, the sampling interval is selected within the interval [0.5s, 1s].

[0035] Please refer to Figure 4 shown in the figure, which is a logic block diagram of constructing a digital twin model based on tags of sampling data in an embodiment of the present invention. Specifically, the twin construction module for constructing a digital twin model based on tags of sampling data includes if the sampling data is a motion trend consistency tag, then construct a digital twin model in the virtual space based on the sampling data corresponding to the first sampling time group, determine the movement index, control the movement of the digital twin model with the corresponding movement index, and update the movement index based on the sampling data corresponding to the adjacent sampling time group; if the sampling data is a motion trend non-consistency tag, then determine the time-domain change curve of the image parameters within the target contour based on the target video, identify the periodic time-domain segments, and update the digital twin model in real time based on the sampling data for the non-periodic time-domain segments, and copy and construct the digital twin model for the periodic time-domain segments.

[0036] The present invention sets sampling tags based on motion trend consistency parameters. In actual situations, the motion trends of devices that need to construct twin models are complex and diverse. Facing this complexity, if real-time sampling and modeling are uniformly adopted for all targets, it will cause waste of computing power. For example, for a device that moves in the same trend as a whole and has no local motion details, it maintains a certain shape during the movement process. During the entire movement process, its sampling data may be basically the same. Therefore, when sampling, it is not necessary to obtain the sampling data for the device at each moment, nor to model based on real-time sampling data. Based on this, the present invention adaptively samples and sets tags for the sampling data. For the sampling data with consistent motion trends, the sampling data corresponding to the first sampling time group is used to construct a digital twin model in the virtual space, and then the digital twin model is controlled according to the movement index, instead of using the form of updating the digital twin model based on real-time sampling data, thereby saving computing power while ensuring the performance of the digital twin model.

[0037] Specifically, the twin construction module constructs a digital twin model in the virtual space based on the sampling data corresponding to the first sampling time group and determines the movement index, including being used to call the sampling data corresponding to any sampling moment in the first sampling time group and construct a digital twin model based on the sampling data; being used to call the sampling data corresponding to at least two sampling moments in the sampling time group and determine the position information of the target at each sampling moment based on the sampling data; being used to determine the movement index based on the position information, and the movement index includes speed and direction.

[0038] In implementation, the position information is the center coordinate of the target. Based on the center coordinates of two moments, the speed and direction of the target's movement can be determined, which will not be elaborated here.

[0039] In implementation, the present invention does not limit the manner of determining the position information of the target at each sampling moment. For example, if the sampling data is point cloud, a digital twin model can be constructed based on the point cloud, and then the center coordinates can be determined based on the digital twin model, which will not be elaborated here.

[0040] Specifically, the twin construction module is further configured to update the movement index based on the sampling data corresponding to the adjacent sampling moment groups, including being configured to control the movement of the digital twin model in the virtual space according to the movement index corresponding to the first sampling moment group; being configured to determine the movement index based on the adjacent sampling moment groups at each sampling interval, and control the digital twin model to move according to the newly determined movement index.

[0041] It can be understood that for the already constructed digital twin model in the virtual space, the speed and direction of movement can be set for it, so that the digital twin model moves correspondingly; In implementation, whenever the movement index is determined, the set speed and direction can be updated, so that the digital twin model moves at the corresponding speed and direction, which will not be elaborated here.

[0042] Specifically, the twin construction module is further configured to determine the time-domain change curve of the image parameters within the target contour based on the target video, including being configured to determine the chrominance and saturation of the target contour in each video frame of the target video; being configured to construct the time-domain change curve of chrominance and the time-domain change curve of saturation.

[0043] It can be understood that if the device operates periodically, the change of the corresponding image parameters in the time domain shows periodicity when reflected in the image, and the image parameters are convenient for quick extraction and analysis. Therefore, the periodic time segments are determined by constructing the time-domain change curve of chrominance and the time-domain change curve of saturation.

[0044] Specifically, the twin construction module is further configured to identify the periodic time segments, including being configured to determine the corresponding periodic time segments in the time-domain change curve of chrominance; being configured to determine the corresponding periodic time segments in the time-domain change curve of saturation; being configured to use the intersection of the corresponding periodic time segments in the time-domain change curve of chrominance and the corresponding periodic time segments in the time-domain change curve of saturation as the identified periodic time segments.

[0045] In implementation, there is no limitation on the method of analyzing periodicity. For example, the period segment in the time-domain variation curve can be determined through the autocorrelation function (ACF). Among them, the autocorrelation function of the periodic signal will have significant peaks after a specific time delay. The positions of these significant peaks correspond to the period of the signal, and thus the time period with periodicity can be identified. Of course, those skilled in the art can also use other methods to determine the periodic time domain segment, which will not be elaborated here.

[0046] Specifically, the twin construction module is used to copy and construct a digital twin model for the periodic time domain segment, including being used to screen out the first period corresponding to the periodic time domain segment; being used to call the sampling data corresponding within the first period to update the digital twin model in real time and store the update record; being used to determine the remaining periods and update the digital twin models within each of the remaining periods according to the stored update records.

[0047] Specifically, there is no limitation on the method of updating the digital twin model. For example, a corresponding digital twin model can be constructed in real time based on the sampling data, and the newly constructed digital twin model is used to replace the digital twin model at the previous moment to achieve the update. The stored update records are the digital twin models constructed at each moment. Thus, within the remaining periods, the digital twin model can be updated based on the time domain order, which is equivalent to replicating the entire update process of the digital twin model within the first period, and this will not be elaborated here.

[0048] The present invention aims at sampling data with inconsistent motion trends, determines the time-domain variation curve of the image parameters within the target contour based on the target video, identifies the periodic time domain segment to analyze whether the device has periodic motion. Furthermore, for the periodic time domain segment, its periodic characteristics are used to copy and construct a digital twin model, which can ensure the accuracy of the model and avoid unnecessary computational overhead. For the non-periodic time domain segment, real-time updating of the digital twin model can ensure that it always keeps consistent with the actual state of the physical entity, thereby improving the credibility and practicality of the model. Moreover, when constructing the digital twin model, it can save computing power, improve resource utilization rate, and ensure the construction effect of the digital twin model.

[0049] Specifically, it further includes a visualization module, which is connected to the twin construction module and is used to display the virtual scene and the digital twin model within the virtual scene.

[0050] Specifically, the visualization module can be a display device, as long as it can display the digital twin model within the virtual scene, which will not be elaborated here.

[0051] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims

1. A visual interaction system for digital twin cross-platform device control and real-time feedback, characterized in that Including: A pre - processing module, which includes a sampling unit arranged in a sampling area for collecting a number of sampling data and a visual perception unit for collecting a number of target videos; A data analysis module, which is connected to the pre - processing module, for calibrating the target contour in the target video within a predetermined time domain segment and each sub - contour inside the target contour, and analyzing the motion trend consistency parameter based on the motion trajectories of the key points of the target contour and each sub - contour; A sampling scheduling module, which is respectively connected to the pre - processing module and the data analysis module, for controlling the sampling unit to perform sampling based on the motion trend consistency parameter and setting labels for the sampling data; A digital twin construction module, which is respectively connected to the pre - processing module, the data analysis module and the sampling scheduling module, for receiving the sampling data obtained by the sampling unit and constructing a digital twin model based on the labels of the sampling data, including, Constructing a digital twin model in the virtual space based on the sampling data corresponding to the first sampling time group, determining a movement index, controlling the movement of the digital twin model according to the corresponding movement index, and updating the movement index based on the sampling data corresponding to the adjacent sampling time group; Or, determining the time - domain change curve of the image parameters inside the target contour based on the target video, identifying the periodic time domain segment, updating the digital twin model in real - time based on the sampling data for the non - periodic time domain segment, and replicating and constructing the digital twin model for the periodic time domain segment; Wherein, the sampling time group includes a number of consecutive sampling times, and controlling the sampling unit to perform sampling includes sampling based on the sampling time group at every sampling interval or real - time sampling.

2. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 1, characterized in that The data analysis module is also used to analyze the motion trend consistency parameter based on the motion trajectories of the key points of the target contour and each sub - contour, including, Extracting the central coordinates of the target contour and each sub - contour inside the target contour as key points respectively; Determining the motion trajectories of each key point; Extracting the speed feature and direction vector feature of each key point moving along the motion trajectory; Solving the average similarity of the speed features corresponding to the key points of the target contour at each moment and the speed features corresponding to the key points of each sub - contour as the speed trend consistency feature factor; Solving the average similarity of the direction vector features corresponding to the key points of the target contour at each moment and the direction vector features corresponding to the key points of each sub - contour as the direction trend consistency feature factor; Weighted - summing the speed trend consistency feature factor and the direction trend consistency feature factor to obtain the motion trend consistency parameter.

3. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 2, characterized in that, The sampling scheduling module is used to control the sampling unit to perform sampling based on the motion trend consistency parameter and set labels for the sampling data, including, If the motion trend consistency parameter is greater than or equal to the preset standard motion trend consistency parameter threshold, controlling the sampling unit to sample based on the sampling time group at every sampling interval and setting a motion trend consistency label for the sampling data; If the motion trend consistency parameter is less than the preset standard motion trend consistency parameter threshold, controlling the sampling unit to perform real - time sampling and setting a motion trend non - consistency label for the sampling data.

4. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 3, characterized in that, The twin building module is used to build a digital twin model based on the labels of the sampled data, including if the sampled data is a label of consistent motion trend, building a digital twin model in the virtual space based on the sampled data corresponding to the first sampled time group, determining a movement index, controlling the movement of the digital twin model according to the corresponding movement index, and updating the movement index based on the sampled data corresponding to the adjacent sampled time group; if the sampled data is a label of inconsistent motion trend, determining a time-domain change curve of the image parameters within the target contour based on the target video, identifying a periodic time segment, and updating the digital twin model in real time based on the sampled data for the non-periodic time segment, and building a digital twin model by replicating for the periodic time segment.

5. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 1, wherein The twin building module is used to build a digital twin model in the virtual space based on the sampled data corresponding to the first sampled time group, and determine a movement index, including calling the sampled data corresponding to any sampled time in the first sampled time group and building a digital twin model based on the sampled data; calling the sampled data corresponding to at least two sampled times in the sampled time group and determining the position information of the targets at each sampled time based on the sampled data; determining a movement index based on the position information, where the movement index includes speed and direction.

6. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 5, characterized in that, The twin building module is used to update the movement index based on the sampled data corresponding to the adjacent sampled time group, including controlling the digital twin model to move in the virtual space according to the movement index corresponding to the first sampled time group; determining a movement index based on the adjacent sampled time group every sampling interval and controlling the digital twin model to move according to the newly determined movement index.

7. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 1, wherein The twin building module is used to determine a time-domain change curve of the image parameters within the target contour based on the target video, including determining the chroma and saturation of the target contour in each video frame of the target video; building a time-domain change curve of chroma and a time-domain change curve of saturation.

8. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 7, characterized in that The twin building module is used to identify a periodic time segment, including determining the corresponding periodic time segment in the time-domain change curve of chroma; determining the corresponding periodic time segment in the time-domain change curve of saturation; using the intersection of the corresponding periodic time segment in the time-domain change curve of chroma and the corresponding periodic time segment in the time-domain change curve of saturation as the identified periodic time segment.

9. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 1, characterized in that, The twin building module is used to build a digital twin model by replicating for the periodic time segment, including screening out the first period corresponding to the periodic time segment; calling the sampled data corresponding to the first period and updating the digital twin model in real time, and storing the update record; determining the remaining periods and updating the digital twin models in each of the remaining periods according to the stored update record.

10. The visual interaction system for digital twin cross-platform device control and real-time feedback according to claim 1, characterized in that, It further includes a visualization module, which is connected to the twin building module and is used to display the virtual scene and the digital twin model within the virtual scene.

Images