Digital twin cross-platform device control and real-time feedback visual interactive system

Abstract

The present invention relates to the field of interactive systems, and in particular to a visual interactive system for cross-platform device control and real-time feedback of digital twins. The present invention sets up a pre-processing module, a data analysis module, a sampling scheduling module and a twin construction module for collaborative work. The pre-processing module collects sampling data and video, the data analysis module calibrates the target contour and the sub-contours contained in the target contour and analyzes the motion trend consistency parameters, and then the sampling scheduling module controls the sampling unit to perform sampling based on the motion trend consistency parameters, sets sampling labels for the obtained sampling data, and finally the twin construction module constructs a digital twin model in different ways according to the sampling labels. The present invention can save computing power, improve resource utilization, and ensure the construction effect of the digital twin model when constructing the digital twin model.

Description

Technical Field

[0001] The present invention relates to the field of interactive systems, and in particular to a visual interactive system for digital twin cross-platform device control and real-time feedback. Background Art

[0002] With the deep integration of information technology and industrial systems, digital twin technology, as an innovative virtualization method, plays an important role in industrial automation, intelligent monitoring and other fields. By adopting the sampling method, relevant sampling data of equipment controlled by various control platforms can be obtained, and then a twin model can be constructed. Digital twin technology can not only dynamically reflect the real-time status of equipment and systems, but also optimize the decision-making process through simulation and prediction. The application of digital twin technology provides a solid foundation for various industries to move towards the intelligent era.

[0003] Chinese patent publication number: CN112462722A, discloses a real-time digital twin factory system for control and display, including a two-dimensional module, a three-dimensional module and an equipment module that establish communication connections with each other. The equipment module includes a controlled object and a control unit. The two-dimensional module is used to display the controlled object in two dimensions so that the operator can control its status; the three-dimensional module is used to display the controlled object in three dimensions so that the operator can display the controlled object currently controlled or viewed. The control unit is used to send attribute information of the controlled object to the two-dimensional module and / or the three-dimensional module so as to control the controlled object in response to the instruction information of the two-dimensional module and / or the three-dimensional module. The three-dimensional graphic component of the controlled object establishes a mapping association relationship with the two-dimensional graphic component of the controlled object, so that the operator can locate the two-dimensional graphic component in the three-dimensional view. The attributes of the controlled object can be displayed in two and three dimensions in parallel in response to the operator's operation, providing a light-load, real-time, cross-platform system.

[0004] However, the prior art still has the following problems:

[0005] In actual situations, as the number of targets that need to build digital twin models continues to increase, the overall computing power increases exponentially, increasing the computing power burden. At the same time, some targets may contain multiple components that can operate independently. Since the movement trends of the targets and the components contained in the targets have different requirements for twin model construction and different sampling requirements, the existing system lacks differentiated processing for the movement trends of these independent components and uses the same sampling method and model construction method, resulting in high computing power requirements, waste of computing power, and reduced resource utilization efficiency and response speed of the system. Summary of the Invention

[0006] To this end, the present invention provides a visual interactive system for digital twin cross-platform device control and real-time feedback to overcome the problem in the prior art that as the number of targets that need to build digital twin models continues to increase, the overall computing power increases exponentially, increasing the computing power burden. At the same time, since the movement trends of the targets and the components contained in the targets have different requirements for twin model construction and different sampling requirements, the same sampling method and model construction method are used, resulting in high computing power requirements, waste of computing power, and reduced resource utilization efficiency and response speed of the system.

[0007] To achieve the above objectives, the present invention provides a visual interactive system for digital twin cross-platform device control and real-time feedback, which includes:

[0008] A pre-processing module, comprising a sampling unit arranged in a sampling area for collecting a plurality of sampled data and a visual perception unit for collecting a plurality of target videos;

[0009] a data analysis module connected to the pre-processing module, for calibrating a target contour and each sub-contour within the target contour in a target video within a predetermined time domain segment, and analyzing motion trend consistency parameters based on motion trajectories of key points of the target contour and each sub-contour;

[0010] a sampling scheduling module, connected to the pre-processing module and the data analysis module respectively, for controlling the sampling unit to perform sampling based on the motion trend consistency parameter and setting labels for the sampled data;

[0011] A twin construction module, which is connected to the pre-processing module, the data analysis module and the sampling scheduling module respectively, is used to receive the sampling data obtained by the sampling unit and build a digital twin model based on the labels of the sampling data, including:

[0012] A digital twin model is constructed in a virtual space based on the sampling data corresponding to the first sampling moment group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampling data corresponding to the adjacent sampling moment group;

[0013] Alternatively, the time domain variation curve of the image parameters within the target contour is determined based on the target video, the periodic time domain segments are identified, the digital twin model is updated in real time based on the sampled data for the non-periodic time domain segments, and the digital twin model is replicated and constructed for the periodic time domain segments;

[0014] The sampling time group includes a number of continuous sampling times, and controlling the sampling unit to perform sampling includes performing sampling based on the sampling time group at every sampling interval, or performing real-time sampling.

[0015] Furthermore, the data analysis module is further configured to analyze the motion trend consistency parameters based on the motion trajectory of the target contour and the key points of each sub-contour, including:

[0016] To extract the target contour and the center coordinates of each sub-contour inside the target contour as key points;

[0017] To determine the motion trajectory of each key point;

[0018] for extracting the velocity characteristics and direction vector characteristics of each key point along the motion trajectory;

[0019] The speed trend consistency factor is used to calculate the similarity average of the speed characteristics corresponding to the key points of the target contour at each moment and the speed characteristics corresponding to the key points of each sub-contour;

[0020] The mean similarity between the directional vector features corresponding to the key points of the target contour at each moment and the directional vector features corresponding to the key points of each sub-contour is used as the directional trend consistency feature factor;

[0021] It is used to perform weighted summation of the speed trend consistency characteristic factor and the direction trend consistency characteristic factor to obtain the motion trend consistency parameter.

[0022] Furthermore, the sampling scheduling module is used to control the sampling unit to perform sampling based on the motion trend consistency parameter, and setting labels for the sampled data includes:

[0023] If the motion trend consistency parameter is greater than or equal to a preset standard motion trend consistency parameter threshold, the sampling unit is controlled to perform sampling based on the sampling time group at each sampling interval, and a motion trend consistency label is set for the sampled data;

[0024] If the motion trend consistency parameter is less than a preset standard motion trend consistency parameter threshold, the sampling unit is controlled to perform real-time sampling, and a motion trend inconsistency label is set for the sampled data.

[0025] Furthermore, it is characterized in that the twin construction module is used to construct a digital twin model based on the labels of the sampled data, including:

[0026] If the sampled data is a motion trend consistency label, a digital twin model is constructed in the virtual space based on the sampled data corresponding to the first sampling moment group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampled data corresponding to the adjacent sampling moment group;

[0027] If the sampling data is a motion trend inconsistent label, the time domain change curve of the image parameters within the target contour is determined based on the target video, the periodic time domain segments are identified, the digital twin model is updated in real time based on the sampling data for the non-periodic time domain segments, and the digital twin model is copied and constructed for the periodic time domain segments.

[0028] Furthermore, the twin construction module is used to construct a digital twin model in the virtual space based on the sampling data corresponding to the first sampling moment group and determine the movement index, including:

[0029] Used to call the sampling data corresponding to any sampling moment in the first sampling moment group and build a digital twin model based on the sampling data;

[0030] for calling sampling data corresponding to at least two sampling moments in the sampling moment group, and determining position information of a target at each sampling moment based on the sampling data;

[0031] The movement index is used to determine the movement index based on the location information, and the movement index includes speed and direction.

[0032] Furthermore, the twin construction module is used to update the movement index based on the sampling data corresponding to the adjacent sampling time group, including:

[0033] for controlling the movement of the digital twin model in the virtual space according to the movement index corresponding to the first sampling time group;

[0034] Used to determine a movement index based on the adjacent sampling moment group at each sampling interval, and control the digital twin model to move according to the newly determined movement index.

[0035] Furthermore, the twin construction module is used to determine the time domain variation curve of the image parameters within the target contour based on the target video, including:

[0036] Used to determine the hue and saturation of the target outline in each video frame of the target video;

[0037] Used to construct the chroma time domain change curve and the saturation time domain change curve.

[0038] Furthermore, the twin construction module is used to identify periodic time domain segments including:

[0039] To determine the corresponding periodic time domain segment in the chromaticity time domain variation curve;

[0040] To determine the corresponding periodic time domain segment in the saturation time domain variation curve;

[0041] The intersection of the corresponding periodic time domain segment in the chroma time domain variation curve and the corresponding periodic time domain segment in the saturation time domain variation curve is used as the identified periodic time domain segment.

[0042] Furthermore, the twin construction module is used to replicate and construct a digital twin model for periodic time domain segments, including:

[0043] Used to filter out the first period corresponding to the periodic time domain segment;

[0044] Used to call the corresponding sampling data in the first cycle to update the digital twin model in real time and store the update records;

[0045] Used to determine the remaining periods, and update the digital twin models within each of the remaining periods according to the stored update records.

[0046] Furthermore, it also includes a visualization module, which is connected to the twin construction module to display the virtual scene and the digital twin model in the virtual scene.

[0047] Compared with the existing technology, the present invention sets up the collaborative work of the pre-processing module, the data analysis module, the sampling scheduling module and the twin construction module. The pre-processing module collects sampling data and video, the data analysis module calibrates the target contour and the sub-contours contained in the target contour and analyzes the motion trend consistency parameters, and then the sampling scheduling module controls the sampling unit to perform sampling based on the motion trend consistency parameters, sets the sampling labels for the obtained sampling data, and finally the twin construction module constructs the digital twin model in different ways according to the sampling labels. The present invention can save computing power, improve resource utilization and ensure the construction effect of the digital twin model when constructing the digital twin model.

[0048] In particular, the present invention calibrates the target contour and the target contour contains sub-contours based on the data analysis module and analyzes the motion trend consistency parameters. In actual situations, if there are many targets that need to be built into the twin model, the computing power burden of the system will be increased. Among them, the motion manifestations of various types of equipment are different. For example, the components of some equipment move in the same trend as a whole. The demand for building a twin model based on local details is not high. Similarly, the demand for sampling is not high. Some equipment may contain multiple independently operable components, and the movement of these components may be more random or complex. In this case, such equipment has a high demand for building a twin model based on local details, and the sampling demand will also be high. Based on this, the present invention considers the motion trajectory of the key points of the target contour and the sub-contour, and then calculates the motion trend consistency parameters to characterize the overall motion trend of the equipment and the complexity of the local motion, so as to provide data support for the subsequent setting of sampling labels and adaptively build digital twin models, thereby improving the resource utilization efficiency when building digital twin models while ensuring accuracy.

[0049] In particular, the present invention sets sampling labels based on motion trend consistency parameters. In actual situations, the motion trends of devices for which twin models need to be constructed are complex and diverse. Faced with this complexity, if real-time sampling modeling is uniformly adopted for all targets, it will cause a waste of computing power. For example, for devices that move with the same overall trend and have no local motion details, they maintain a certain form during the movement process. During the entire movement process, their sampling data may be basically the same. Therefore, there is no need to obtain sampling data for the device at each moment during sampling, nor is there any need to perform modeling based on real-time sampling data. Based on this, the present invention adaptively performs sampling and sets labels for the sampling data. For sampling data with consistent motion trends, the sampling data corresponding to the first sampling moment group is used to construct a digital twin model in the virtual space. The digital twin model is subsequently controlled according to the movement index, and the form of updating the digital twin model based on the sampling data in real time is not adopted, thereby saving computing power while ensuring the expressiveness of the digital twin model.

[0050] In particular, the present invention targets sampling data with inconsistent motion trends, determines the time domain change curve of image parameters within the target contour based on the target video, identifies periodic time domain segments, and analyzes whether the device has periodic motion. Then, for the periodic time domain segments, the periodic characteristics are used to replicate and construct a digital twin model, which can ensure the accuracy of the model while avoiding unnecessary computational overhead. For non-periodic time domain segments, real-time updating of the digital twin model can ensure that it always remains consistent with the actual state of the physical entity, thereby improving the credibility and practicality of the model, and thus saving computing power when constructing the digital twin model, improving resource utilization, and ensuring the construction effect of the digital twin model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the structure of a visual interactive system based on digital twin cross-platform device control and real-time feedback according to an embodiment of the invention;

[0052] Figure 2 A logic decision diagram for controlling sampling by a sampling unit based on the motion trend consistency parameter according to an embodiment of the present invention;

[0053] Figure 3 A logic decision diagram for setting labels for sampled data according to an embodiment of the present invention;

[0054] Figure 4 This is a logical block diagram of building a digital twin model based on the labels of sampled data in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0056] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0057] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the term "connection" should be understood in a broad sense. For example, it can mean a fixed connection, a detachable connection, or an integral connection; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0058] See also Figure 1 As shown in FIG, which is a structural diagram of a visual interaction system based on digital twin cross-platform device control and real-time feedback according to an embodiment of the present invention, the visual interaction system based on digital twin cross-platform device control and real-time feedback according to an embodiment of the present invention includes:

[0059] A pre-processing module, comprising a sampling unit arranged in a sampling area for collecting a plurality of sampled data and a visual perception unit for collecting a plurality of target videos;

[0060] a data analysis module connected to the pre-processing module, configured to calibrate a target contour and sub-contours within the target contour in a target video within a predetermined time domain segment, and analyze motion trend consistency parameters based on motion trajectories of key points of the target contour and sub-contours, wherein the predetermined time domain segment is selected within the interval [2s, 3s];

[0061] a sampling scheduling module, connected to the pre-processing module and the data analysis module respectively, for controlling the sampling unit to perform sampling based on the motion trend consistency parameter and setting labels for the sampled data;

[0062] A twin construction module, which is connected to the pre-processing module, the data analysis module and the sampling scheduling module respectively, is used to receive the sampling data obtained by the sampling unit and build a digital twin model based on the labels of the sampling data, including:

[0063] A digital twin model is constructed in a virtual space based on the sampling data corresponding to the first sampling moment group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampling data corresponding to the adjacent sampling moment group;

[0064] Alternatively, the time domain variation curve of the image parameters within the target contour is determined based on the target video, the periodic time domain segments are identified, the digital twin model is updated in real time based on the sampled data for the non-periodic time domain segments, and the digital twin model is replicated and constructed for the periodic time domain segments;

[0065] The sampling time group includes a number of continuous sampling times, and controlling the sampling unit to perform sampling includes performing sampling based on the sampling time group at every sampling interval, or performing real-time sampling.

[0066] Specifically, the structures of the data analysis module, sampling scheduling module and twin construction module are not limited, and they can be composed of logic components or a combination of logic components, and the logic components include field programmable processors, computers or microprocessors in computers.

[0067] Specifically, there is no limitation on the specific structure of the pre-processing module. The visual perception unit can be a photographic device, which only needs to be able to obtain the target video corresponding to the target. The sampling unit can be a laser scanner or a depth photography device, which only needs to be able to obtain the scanning data of the target. The scanning data is the point cloud of the target, which is used to subsequently construct the digital twin model. In implementation, the target is the device, which will not be repeated here.

[0068] Specifically, the present invention does not limit the way in which the twin construction module constructs a digital twin model based on the labels of the sampling data. The sampling data is a point cloud, and the corresponding digital twin model can be constructed based on the point cloud. This is an existing technology and will not be repeated here.

[0069] Specifically, the data analysis module is further configured to analyze the motion trend consistency parameters based on the motion trajectory of the target contour and the key points of each sub-contour, including:

[0070] To extract the target contour and the center coordinates of each sub-contour inside the target contour as key points;

[0071] To determine the motion trajectory of each key point;

[0072] for extracting the velocity characteristics and direction vector characteristics of each key point along the motion trajectory;

[0073] The speed trend consistency factor is used to calculate the similarity average of the speed characteristics corresponding to the key points of the target contour at each moment and the speed characteristics corresponding to the key points of each sub-contour;

[0074] In the implementation, when calculating the similarity mean of the speed feature, the similarity of the corresponding direction vector feature of the key point of the target contour at each moment and the corresponding direction vector feature of the key point of each sub-contour is first calculated, and then the similarity mean of the speed feature is solved.

[0075] The mean similarity between the directional vector features corresponding to the key points of the target contour at each moment and the directional vector features corresponding to the key points of each sub-contour is used as the directional trend consistency feature factor;

[0076] In the implementation, when calculating the similarity mean of the directional vector features, the similarity between the directional vector features corresponding to the key points of the target contour at each moment and the directional vector features corresponding to the key points of each sub-contour is first calculated, and then the similarity mean of the directional vector features is solved;

[0077] for performing weighted summation of the speed trend consistency characteristic factor and the direction trend consistency characteristic factor to obtain the motion trend consistency parameter;

[0078] In the implementation, the speed difference ratio of the corresponding speed features of the key points of the target contour and the key points of each sub-contour at each moment is solved, and the speed difference ratio is used as the similarity of the speed features;

[0079] In the implementation, the angle difference ratio of the corresponding direction vector features between the key points of the target contour and the key points of each sub-contour at each moment is solved, and the angle difference ratio is used as the similarity of the direction vector features;

[0080] It can be understood that the difference ratio of two values ​​is the ratio of the absolute value of the difference between the two values ​​to the mean of the two values, which will not be elaborated here.

[0081] In implementation, when the speed trend consistency characteristic factor and the direction trend consistency characteristic factor are weighted and summed, the weight of the speed trend consistency characteristic factor is 0.55, and the weight of the direction trend consistency characteristic factor is 0.45.

[0082] Specifically, there is no limitation on the method for identifying the target contour and the sub-contours within the target contour. For example, an existing image segmentation algorithm may be used, or other methods may be used, which will not be described in detail.

[0083] The present invention calibrates the target contour and the target contour contains sub-contours based on the data analysis module and analyzes the motion trend consistency parameters. In actual situations, if there are many targets that need to be built into the twin model, the computing power burden of the system will be increased. Among them, the motion manifestations of various types of equipment are different. For example, the components of some equipment move in the same trend as a whole. The demand for building a twin model based on local details is not high. Similarly, the demand for sampling is not high. Some equipment may contain multiple independently operable components, and the movement of these components may be more random or complex. In this case, such equipment has a high demand for building a twin model based on local details, and the sampling demand will also be high. Based on this, the present invention considers the motion trajectory of the key points of the target contour and the sub-contour, and then calculates the motion trend consistency parameters to characterize the overall motion trend of the equipment and the complexity of the local motion, so as to provide data support for the subsequent setting of sampling labels and adaptively build digital twin models, thereby improving the resource utilization efficiency when building digital twin models while ensuring accuracy.

[0084] See also Figure 2 and Figure 3 As shown, Figure 2 This is a logic decision diagram for controlling sampling by the sampling unit based on the motion trend consistency parameter according to an embodiment of the present invention. Figure 3 This is a logic decision diagram for setting labels for sampled data according to an embodiment of the present invention. Specifically, the sampling scheduling module is used to control the sampling unit to perform sampling based on the motion trend consistency parameter. Setting labels for sampled data includes:

[0085] If the motion trend consistency parameter is greater than or equal to a preset standard motion trend consistency parameter threshold, the sampling unit is controlled to perform sampling based on the sampling time group at each sampling interval, and a motion trend consistency label is set for the sampled data;

[0086] If the motion trend consistency parameter is less than a preset standard motion trend consistency parameter threshold, the sampling unit is controlled to perform real-time sampling, and a motion trend inconsistency label is set for the sampled data.

[0087] Specifically, the purpose of setting the standard motion trend consistency parameter threshold is to characterize the situation where there is a large difference in the motion trend between the target contour and the corresponding sub-contour. During implementation, it is selected within the interval [0.15, 0.35].

[0088] Specifically, the sampling interval should not be too long to ensure continuous sampling of the entire motion process. During implementation, the sampling interval is selected within the interval [0.5s, 1s].

[0089] See also Figure 4As shown, it is a logical block diagram of building a digital twin model based on the labels of sampled data according to an embodiment of the present invention. Specifically, the twin construction module is used to build a digital twin model based on the labels of sampled data, including:

[0090] If the sampled data is a motion trend consistency label, a digital twin model is constructed in the virtual space based on the sampled data corresponding to the first sampling moment group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampled data corresponding to the adjacent sampling moment group;

[0091] If the sampling data is a motion trend inconsistent label, the time domain change curve of the image parameters within the target contour is determined based on the target video, the periodic time domain segments are identified, the digital twin model is updated in real time based on the sampling data for the non-periodic time domain segments, and the digital twin model is copied and constructed for the periodic time domain segments.

[0092] The present invention sets sampling labels based on motion trend consistency parameters. In actual situations, the motion trends of devices for which twin models need to be constructed are complex and diverse. Faced with this complexity, if real-time sampling modeling is uniformly adopted for all targets, it will cause a waste of computing power. For example, for devices that move with the same overall trend and have no local motion details, they maintain a certain form during the movement process. During the entire movement process, their sampling data may be basically the same. Therefore, there is no need to obtain the sampling data for the device at each moment during sampling, nor is there any need to perform modeling based on real-time sampling data. Based on this, the present invention adaptively performs sampling and sets labels for the sampling data. For sampling data with consistent motion trends, the sampling data corresponding to the first sampling moment group is used to construct a digital twin model in the virtual space. The digital twin model is subsequently controlled according to the movement index, and the form of updating the digital twin model based on the sampling data in real time is not adopted, thereby saving computing power while ensuring the expressiveness of the digital twin model.

[0093] Specifically, the twin construction module constructs a digital twin model in the virtual space based on the sampling data corresponding to the first sampling moment group and determines the movement index, including:

[0094] Used to call the sampling data corresponding to any sampling moment in the first sampling moment group and build a digital twin model based on the sampling data;

[0095] for calling sampling data corresponding to at least two sampling moments in the sampling moment group, and determining position information of a target at each sampling moment based on the sampling data;

[0096] The movement index is used to determine the movement index based on the location information, and the movement index includes speed and direction.

[0097] In practice, the position information is the center coordinates of the target. The speed and direction of the target movement can be determined based on the center coordinates at two moments, which will not be elaborated here.

[0098] In implementation, the present invention does not limit the method of determining the location information of the target at each sampling moment. For example, the sampling data is a point cloud. A digital twin model can be constructed based on the point cloud, and then the center coordinates can be determined based on the digital twin model. This will not be repeated here.

[0099] Specifically, the twin construction module is further configured to update the movement index based on the sampling data corresponding to the adjacent sampling time group, including:

[0100] for controlling the movement of the digital twin model in the virtual space according to the movement index corresponding to the first sampling time group;

[0101] Used to determine a movement index based on the adjacent sampling moment group at each sampling interval, and control the digital twin model to move according to the newly determined movement index.

[0102] It can be understood that in the virtual space, for the constructed digital twin model, the speed and direction of movement can be set for it, thereby making the digital twin model move accordingly;

[0103] In implementation, whenever the movement index is determined, the set speed and direction can be updated, so that the digital twin model moves at the corresponding speed and direction, which will not be repeated here.

[0104] Specifically, the twin construction module is also used to determine the time domain variation curve of the image parameters within the target contour based on the target video, including:

[0105] Used to determine the hue and saturation of the target outline in each video frame of the target video;

[0106] Used to construct the chroma time domain change curve and the saturation time domain change curve.

[0107] It can be understood that if the device operates periodically, the corresponding image parameters will change periodically in the time domain when reflected in the image, and the image parameters are easy to extract and analyze quickly. Therefore, the periodic time domain segment is determined by constructing a chromaticity time domain change curve and a saturation time domain change curve.

[0108] Specifically, the twin building module is also used to identify periodic time domain segments including:

[0109] To determine the corresponding periodic time domain segment in the chromaticity time domain variation curve;

[0110] To determine the corresponding periodic time domain segment in the saturation time domain variation curve;

[0111] The intersection of the corresponding periodic time domain segment in the chroma time domain variation curve and the corresponding periodic time domain segment in the saturation time domain variation curve is used as the identified periodic time domain segment.

[0112] In implementation, there is no limitation on the method of analyzing periodicity. For example, the periodic segments in the time domain variation curve can be determined by the autocorrelation function (ACF), wherein the autocorrelation function of the periodic signal will have significant peaks after a specific time delay, and the positions of these significant peaks correspond to the period of the signal, thereby identifying the time period where the periodicity exists. Of course, those skilled in the art can also use other methods to determine the periodic time domain segments, which will not be repeated here.

[0113] Specifically, the twin construction module is used to replicate and construct a digital twin model for periodic time domain segments, including:

[0114] Used to filter out the first period corresponding to the periodic time domain segment;

[0115] Used to call the corresponding sampling data in the first cycle to update the digital twin model in real time and store the update records;

[0116] Used to determine the remaining periods, and update the digital twin models within each of the remaining periods according to the stored update records.

[0117] Specifically, there is no limitation on the method of updating the digital twin model. For example, the corresponding digital twin model can be constructed in real time based on the sampling data, and the newly constructed digital twin model can replace the digital twin model of the previous moment to achieve the update. The stored update records are the digital twin models constructed at each moment. Then, in the remaining cycles, the digital twin model can be updated based on the time domain sequence, which is equivalent to copying the entire update process of the digital twin model in the first cycle. This will not be repeated here.

[0118] The present invention targets sampling data with inconsistent motion trends, determines the time domain change curve of image parameters within the target contour based on the target video, identifies periodic time domain segments, and analyzes whether the device has periodic motion. Then, for the periodic time domain segments, the periodic characteristics are used to replicate and construct a digital twin model, which can ensure the accuracy of the model while avoiding unnecessary computational overhead. For non-periodic time domain segments, real-time updating of the digital twin model can ensure that it always remains consistent with the actual state of the physical entity, thereby improving the credibility and practicality of the model, and thus saving computing power when constructing the digital twin model, improving resource utilization, and ensuring the construction effect of the digital twin model.

[0119] Specifically, it also includes a visualization module, which is connected to the twin construction module to display the virtual scene and the digital twin model in the virtual scene.

[0120] Specifically, the visualization module can be a display device, which only needs to be able to display the digital twin model in the virtual scene, and will not be elaborated here.

[0121] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A visual interactive system for digital twin cross-platform device control and real-time feedback, characterized by: include: A pre-processing module, comprising a sampling unit arranged in a sampling area for collecting a plurality of sampled data and a visual perception unit for collecting a plurality of target videos; a data analysis module connected to the pre-processing module, for calibrating a target contour and each sub-contour within the target contour in a target video within a predetermined time domain segment, and analyzing motion trend consistency parameters based on motion trajectories of key points of the target contour and each sub-contour; a sampling scheduling module, connected to the pre-processing module and the data analysis module respectively, for controlling the sampling unit to perform sampling based on the motion trend consistency parameter and setting labels for the sampled data; A twin construction module, which is connected to the pre-processing module, the data analysis module and the sampling scheduling module respectively, is used to receive the sampling data obtained by the sampling unit and build a digital twin model based on the labels of the sampling data, including: A digital twin model is constructed in a virtual space based on the sampling data corresponding to the first sampling moment group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampling data corresponding to the adjacent sampling moment group; Alternatively, the time domain variation curve of the image parameters within the target contour is determined based on the target video, the periodic time domain segments are identified, the digital twin model is updated in real time based on the sampled data for the non-periodic time domain segments, and the digital twin model is replicated and constructed for the periodic time domain segments; The sampling time group includes a plurality of continuous sampling times, and controlling the sampling unit to perform sampling includes performing sampling based on the sampling time group at every sampling interval, or performing real-time sampling; The data analysis module is further configured to analyze the motion trend consistency parameters based on the motion trajectory of the target contour and the key points of each sub-contour, including: To extract the target contour and the center coordinates of each sub-contour inside the target contour as key points; To determine the motion trajectory of each key point; for extracting the velocity characteristics and direction vector characteristics of each key point along the motion trajectory; The speed trend consistency factor is used to calculate the similarity average of the speed characteristics corresponding to the key points of the target contour at each moment and the speed characteristics corresponding to the key points of each sub-contour; The mean similarity between the directional vector features corresponding to the key points of the target contour at each moment and the directional vector features corresponding to the key points of each sub-contour is used as the directional trend consistency feature factor; It is used to perform weighted summation of the speed trend consistency characteristic factor and the direction trend consistency characteristic factor to obtain the motion trend consistency parameter.

2. The visual interactive system for digital twin cross-platform device control and real-time feedback according to claim 1 is characterized in that: The sampling scheduling module is used to control the sampling unit to perform sampling based on the motion trend consistency parameter, and setting labels for the sampled data includes: If the motion trend consistency parameter is greater than or equal to a preset standard motion trend consistency parameter threshold, the sampling unit is controlled to perform sampling based on the sampling time group at each sampling interval, and a motion trend consistency label is set for the sampled data; If the motion trend consistency parameter is less than a preset standard motion trend consistency parameter threshold, the sampling unit is controlled to perform real-time sampling and a motion trend inconsistency label is set for the sampled data.

3. The visual interactive system for digital twin cross-platform device control and real-time feedback according to claim 2 is characterized in that: The twin construction module is used to construct a digital twin model based on the labels of the sampled data, including: If the sampled data is a motion trend consistency label, a digital twin model is constructed in the virtual space based on the sampled data corresponding to the first sampling moment group, a movement index is determined, the movement of the digital twin model is controlled by the corresponding movement index, and the movement index is updated based on the sampled data corresponding to the adjacent sampling moment group; If the sampling data is a motion trend inconsistent label, the time domain change curve of the image parameters within the target contour is determined based on the target video, the periodic time domain segments are identified, the digital twin model is updated in real time based on the sampling data for the non-periodic time domain segments, and the digital twin model is copied and constructed for the periodic time domain segments.

4. The digital twin cross-platform device control and real-time feedback visualization interactive system according to claim 1 is characterized in that: The twin construction module is used to construct a digital twin model in the virtual space based on the sampling data corresponding to the first sampling moment group and determine the movement index, including: Used to call the sampling data corresponding to any sampling moment in the first sampling moment group and build a digital twin model based on the sampling data; for calling sampling data corresponding to at least two sampling moments in the sampling moment group, and determining position information of a target at each sampling moment based on the sampling data; The movement index is used to determine the movement index based on the location information, and the movement index includes speed and direction.

5. The visual interactive system for digital twin cross-platform device control and real-time feedback according to claim 4 is characterized in that: The twin construction module is used to update the movement index based on the sampling data corresponding to the adjacent sampling time group, including: for controlling the movement of the digital twin model in the virtual space according to the movement index corresponding to the first sampling time group; Used to determine a movement index based on the adjacent sampling moment group at each sampling interval, and control the digital twin model to move according to the newly determined movement index.

6. The digital twin cross-platform device control and real-time feedback visualization interactive system according to claim 1 is characterized in that: The twin construction module is used to determine the time domain variation curve of the image parameters within the target contour based on the target video, including: Used to determine the hue and saturation of the target outline in each video frame of the target video; Used to construct the chroma time domain change curve and the saturation time domain change curve.

7. The digital twin cross-platform device control and real-time feedback visualization interactive system according to claim 6 is characterized in that: The twin building module is used to identify periodic time domain segments including: To determine the corresponding periodic time domain segment in the chromaticity time domain variation curve; To determine the corresponding periodic time domain segment in the saturation time domain variation curve; The intersection of the corresponding periodic time domain segment in the chroma time domain variation curve and the corresponding periodic time domain segment in the saturation time domain variation curve is used as the identified periodic time domain segment.

8. The digital twin cross-platform device control and real-time feedback visualization interactive system according to claim 1 is characterized in that: The twin construction module is used to copy and construct a digital twin model for periodic time domain segments, including: Used to filter out the first period corresponding to the periodic time domain segment; Used to call the corresponding sampling data in the first cycle to update the digital twin model in real time and store the update records; Used to determine the remaining periods, and update the digital twin models within each of the remaining periods according to the stored update records.

9. The digital twin cross-platform device control and real-time feedback visualization interactive system according to claim 1 is characterized in that: It also includes a visualization module, which is connected to the twin construction module and is used to display the virtual scene and the digital twin model in the virtual scene.

Images