Twin data and perception data fusion method and system and storage medium

Abstract

The embodiment of the invention discloses a twin data and perception data fusion method and system and a storage medium, and the method comprises the steps: obtaining twin data and perception data of a target region at different moments, the twinborn data is virtual data, predicted through a digital twinborn model, of sensing devices located at different spatial positions, and the sensing data is real data actually monitored by the sensing devices located at different spatial positions in the target area; carrying out time alignment on the twinborn data and the sensing data at different moments; aligning the twin data of the sensing devices at different spatial positions with the spatial position of the sensing data to realize dynamic consistency fusion of the sensing data and the twin simulation data in time and space dimensions; and fusing the aligned twin data and perception data through a data fusion model, thereby improving the precision and efficiency of data fusion.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to a method, system and storage medium for fusing twin data and perception data. Background Art

[0002] With the continuous advancement of digital twin technology, its widespread application in fields such as smart cities, the Industrial Internet, intelligent manufacturing, and energy management continues to unleash its potential. Digital twins rely on virtual models to construct digital representations of physical objects, enabling real-time monitoring, analysis, and optimization of physical systems. However, to fully realize the advantages of digital twins, it is urgent to integrate large amounts of sensory data from physical entities with simulation data generated by virtual models to improve the accuracy and practical application of twin scenarios.

[0003] In practical applications, sensing devices (such as sensors, cameras, and IoT terminals) collect large amounts of real-time physical data, while digital twin models generate virtual data based on historical data, physical laws, and simulation algorithms. The effective integration of these two types of data to support dynamic analysis and decision-making optimization in complex scenarios has become a key technical bottleneck in the development of digital twin technology. Summary of the Invention

[0004] Based on this, it is necessary to propose a twin data and perception data fusion method, system and storage medium to address the above problems.

[0005] A method for fusing twin data and perception data, the method comprising:

[0006] Acquire twin data and perception data of the target area at different times, where the target area includes perception devices. The twin data is virtual data of perception devices at different spatial locations predicted by the digital twin model, and the perception data is real data actually monitored by perception devices at different spatial locations in the target area.

[0007] Temporally aligning the twin data and the perception data at different moments;

[0008] Aligning the twin data of the sensing devices at different spatial positions with the spatial positions of the sensing data;

[0009] The aligned twin data is fused with the perception data through a data fusion model.

[0010] The step of aligning the twin data at different moments with the time series of the perception data specifically includes:

[0011] Construct a time alignment model, the time alignment model is: L time (π)=w(st1 ,s t2 )·||D s (s t1 )-D t (s t2 )||, where L time is the time alignment function, π is the alignment path between twin data and perception data in the time dimension, w(s t1 ,s t2 ) is from time point s t1 To time point s t2 The cost weight, Ds(s t1 ) is s t1 The perception data at the moment, Dt(s t2 ) is s t2 Twin data time series at each moment;

[0012] According to the time alignment model, the perception data corresponding to the twin data is obtained when the time alignment function is minimum, and the twin data and the perception data are time-aligned.

[0013] After the time series of the twin data and the perception data are time-aligned, the method further includes:

[0014] Determining a residual sequence between the twin data and the perception data after time alignment;

[0015] determining a local residual fluctuation amplitude according to the residual sequence;

[0016] If it is determined that the fluctuation amplitude of the local residual is greater than a preset threshold, then there is local drift, and the twin data and the perception data are time-aligned again within the time interval where the local drift exists:

[0017] If it is determined that the fluctuation amplitude of the local residual is less than or equal to the preset threshold, then there is no local drift.

[0018] The step of aligning the twin data of the sensing devices at different spatial positions with the spatial positions of the sensing data specifically includes:

[0019] Construct a spatial alignment model, the spatial alignment model is: Among them, L adj is the spatial alignment function, M is the alignment path mapping the perceptual data to the twin data in the spatial dimension, and D t is twin data, i s ,j s is the number of the spatial location of the perception data, E s is a numbered set of perception data;

[0020] According to the spatial alignment model, the perception data corresponding to the twin data is obtained when the spatial alignment function is minimized, and the twin data and the perception data are spatially aligned.

[0021] Wherein, when the spatial alignment function is minimum according to the spatial alignment model, the perception data corresponding to the twin data is obtained, and after spatially aligning the twin data with the perception data, the method further includes:

[0022] Obtaining a topological similarity score between the spatially aligned perception data and the twin data;

[0023] If it is determined that the topological similarity score is less than a preset score, the twin data and the perception data are spatially aligned again within the neighborhood of the perception data.

[0024] After aligning the twin data and the perception data at the time and space levels, the method further includes:

[0025] Extracting feature embedding vectors of the twin data and the perception data after alignment in time and space dimensions respectively;

[0026] Assign weights to the feature embedding vectors of the perception data and the twin data according to a feature alignment model; the feature alignment model is: Among them, L is the feature alignment function, l is the number of layers of feature extraction, is the perceptual data embedding vector of the lth layer, is the twin data embedding vector of layer l, λ l The weight coefficient for fusion matching of features at different levels;

[0027] Obtain the perception data embedding vector corresponding to the twin data embedding vector when the feature alignment function is minimum, and perform feature alignment between the perception data and the twin data.

[0028] The step of obtaining twin data of the target area at different times specifically includes:

[0029] Extract three-dimensional point cloud data of the target area at different times through lidar;

[0030] Modeling is performed based on the three-dimensional point cloud data to construct a digital twin model;

[0031] Distributed sensing nodes of sensing devices in the target area are set in the digital twin model, and data at the distributed sensing nodes are predicted according to the digital twin model to obtain twin data.

[0032] Wherein, the data fusion model is constructed based on the aligned twin data and the perception data, and before the twin data and the perception data are fused through the data fusion model, the method further includes:

[0033] The operating data of the digital twin model is corrected according to the perception data, and the perception data is compensated according to the prediction error of the digital twin model.

[0034] Wherein, the data fusion model is constructed based on the aligned twin data and the perception data, and after fusing the twin data and the perception data through the data fusion model, the method further includes:

[0035] Model optimization is performed by adjusting the weight parameters of the data fusion model.

[0036] A twin data and perception data fusion system, the system comprising:

[0037] A data acquisition module is used to acquire twin data and perception data of a target area at different times. The target area includes perception devices. The twin data is virtual data of perception devices at different spatial positions predicted by the digital twin model, and the perception data is real data actually monitored by perception devices at different spatial positions in the target area.

[0038] A time alignment module, used to time-align the twin data and the perception data at different times;

[0039] A spatial alignment module, configured to align the twin data of sensing devices at different spatial positions with the spatial positions of the sensing data;

[0040] A data fusion model construction module is used to fuse the aligned twin data with the perception data through a data fusion model.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0042] The embodiments of the present invention have the following beneficial effects:

[0043] The present invention ensures data consistency by temporally aligning the twin data with the perception data at different moments, and aligning the twin data of perception devices at different spatial positions with the spatial positions of the perception data, thereby realizing dynamic consistent fusion of perception data and twin simulation data in time and space dimensions; further, the aligned twin data is fused with the perception data through a data fusion model to improve the accuracy and efficiency of data fusion, and significantly enhance the data accuracy and intelligence level of the twin scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] in:

[0046] Figure 1 A flowchart of an embodiment of a method for fusing twin data and perception data provided by the present invention;

[0047] Figure 2 A flowchart of an embodiment of a method for fusing twin data and perception data provided by the present invention;

[0048] Figure 3 This is a structural diagram of an embodiment of a twin data and perception data fusion system provided by the present invention;

[0049] Figure 4 This is a schematic structural diagram of an embodiment of the storage medium provided by the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] like Figure 1 As shown, Figure 1 A flow chart of an embodiment of a method for fusing twin data and perception data provided by the present invention. A method for fusing twin data and perception data, the method comprising:

[0052] S101: Acquire twin data and perception data of the target area at different times. The target area includes perception devices. The twin data is the virtual data of the perception devices at different spatial positions predicted by the digital twin model. The perception data is the real data actually monitored by the perception devices at different spatial positions in the target area.

[0053] Exemplarily, the target area includes sensing devices, and three-dimensional point cloud data of the target area at different times are extracted by lidar, and modeling is performed based on the three-dimensional point cloud data to construct a digital twin model; distributed sensing nodes of the sensing devices in the target area are set in the digital twin model, and the data at the distributed sensing nodes are predicted based on the digital twin model to obtain twin data.

[0054] S102: Time-align the twin data and perception data at different times.

[0055] Exemplarily, a time alignment model is constructed, and the time alignment model is:

[0056] L time (π)=w(s t1 ,s t2 )·||D s (s t1 )-D t (s t2 )||;

[0057] Among them, L time is the time alignment function, π is the alignment path between twin data and perception data in the time dimension, w(s t1 ,s t2 ) is from time point s t1 To time point s t2 The cost weight, Ds(s t1 ) is s t1 The perception data at the moment, Dt(s t2 ) is s t2 Twin data time series at each moment.

[0058] According to the time alignment model, the perception data corresponding to the twin data is obtained when the time alignment function is minimum, and the twin data and the perception data are time-aligned.

[0059] S103: Align the twin data of the perception devices at different spatial positions with the spatial positions of the perception data.

[0060] Exemplarily, a spatial alignment model is constructed, and the spatial alignment model is:

[0061]

[0062] Among them, L adj is the spatial alignment function, M is the alignment path mapping the perceptual data to the twin data in the spatial dimension, and D t is twin data, i s ,j s is the number of the spatial location of the perception data, E s is a numbered set of perception data;

[0063] According to the spatial alignment model, the perception data corresponding to the twin data is obtained when the spatial alignment function is minimized, and the twin data and the perception data are spatially aligned.

[0064] S104: Fuse the aligned twin data with the perception data through a data fusion model.

[0065] For example, a data fusion model is constructed, and the fusion model is:

[0066] The fusion formula is expressed as follows:

[0067] P(D f |D s ',D t ')=∫ θ P(D f |D s ',D t ',θ)·P(θ|D s ',D t ')dθ;

[0068] Among them, D′ s is the aligned perception data, D′ t is the aligned twin data, D f is the fused output result, θ is the model hyperparameter, which is used to describe the correlation and uncertainty between the perception data and the twin data, P(θ|D s ,D t ) is the model uncertainty distribution under the condition.

[0069] Considering the differences in the confidence levels of perception data and twin data in different application environments, the weight parameters are dynamically adjusted to optimize the fusion effect, as shown in the following formula:

[0070] D f =w s ·F(D s ')+w t ·F(D t ');

[0071] Among them, F(·) is the data source feature enhancement function, which is used to enhance the contribution of multidimensional features such as historical trends and local fluctuations, w s and w t Represent the dynamic confidence weights of perception data and twin data, respectively, satisfying w s +w t =1, the calculation formula for the dynamic confidence weight of perception data and twin data is as follows:

[0072] w t =1-w s ;

[0073] Among them, I(D x ) is the data source D x The information gain, D x ∈Ds ′、D t ′, its calculation formula is as follows;

[0074]

[0075] Among them, p i Data source D x The information probability distribution at the i-th observation point. By introducing information entropy evaluation, this method can dynamically reflect the effective information contribution of each data source and achieve accurate weighting of multi-source heterogeneous data.

[0076] From the above description, it can be seen that the present invention ensures data consistency by temporally aligning the twin data with the perception data at different moments, and aligning the twin data of perception devices at different spatial positions with the spatial positions of the perception data, thereby realizing dynamic consistent fusion of perception data and twin simulation data in time and space dimensions; further, the aligned twin data is fused with the perception data through a data fusion model to improve the accuracy and efficiency of data fusion, and significantly improve the data accuracy and intelligence level of the twin scene.

[0077] like Figure 2 As shown, Figure 2 A flow chart of an embodiment of a method for fusing twin data and perception data provided by the present invention. A method for fusing twin data and perception data, the method comprising:

[0078] S201: Obtain twin data and perception data of the target area at different times. The target area includes perception devices. The twin data is the virtual data of the perception devices at different spatial positions predicted by the digital twin model, and the perception data is the real data actually monitored by the perception devices at different spatial positions in the target area.

[0079] For example, a target area includes sensing devices. LiDAR is used to extract 3D point cloud data of the target area at different times. Modeling is performed based on the 3D point cloud data to construct a digital twin model. Distributed sensing nodes are set up within the digital twin model for the sensing devices within the target area. Data at the distributed sensing nodes is predicted based on the digital twin model to obtain twin data. The sensing data is monitored by sensing devices at different spatial locations within the target area.

[0080] S202: Time-align the twin data and perception data at different times.

[0081] It should be noted that step S202 Figure 1 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0082] S203: Determine the residual sequence between the time-aligned twin data and the perception data.

[0083] For example, the residual sequence r(i t ), the difference between the perception data and its mapped twin data points is represented by the residual sequence, as shown in the following formula:

[0084] r(s t )=D s ′(s t )-D t ′(π(s t ));

[0085] Among them, r(s t ) is the residual sequence, s t is the perception data moment, π(s t ) is the corresponding twin data moment, D s ′ is the aligned perception data, D t ′ is the aligned twin data.

[0086] S204: Determine the local residual fluctuation amplitude according to the residual sequence.

[0087] For example, in order to correct the local dynamic consistency error, a local residual adaptive detection and compensation mechanism is introduced. Specifically, the local residual fluctuation amplitude is first calculated according to the residual sequence, as shown in the following formula:

[0088]

[0089] in, is the local residual fluctuation amplitude at the current moment, s t is the sensing data moment, and k is the local window radius.

[0090] S205: If it is determined that the local residual fluctuation amplitude is greater than a preset threshold, then local drift exists, and the twin data and the perception data are time-aligned again within the time interval where the local drift exists.

[0091] S206: If it is determined that the fluctuation amplitude of the local residual is less than or equal to a preset threshold, then there is no local drift.

[0092] For example, if the fluctuation amplitude of the local residual is less than or equal to a preset threshold, then there is no local drift. If the fluctuation amplitude of the local residual is greater than a preset threshold, then there is local drift. t ) to align the twin data with the perception data again in time and reselect the optimal matching node, as shown in the following formula:

[0093]

[0094] Among them, π′(s t ) is the twin data moment after realignment, W(s t ) is the local window, Ds(s t1 ) is s t1 The perception data at the moment, Dt(s t2 ) is s t2 Twin data time series at each moment.

[0095] S207: Align the twin data of the perception devices at different spatial positions with the spatial positions of the perception data.

[0096] It should be noted that step S207 Figure 1 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0097] S208: Obtain the topological similarity score of the spatially aligned perception data and the twin data.

[0098] For example, in order to enhance the local structure preservation, a local topological similarity evaluation mechanism is introduced to calculate the topological similarity score Sim(is,it) for each number pair (is,it) of the spatial position of the aligned perception data and the twin data:

[0099] Sim(i s ,i t )=α·DegreeSim(i s ,i t )+β·ClusteringSim(i s ,i t )+γ·NeighborDensitySim(i s ,i t );

[0100] Among them, DegreeSim(i s ,i t ) is the node degree similarity, ClusteringSim(is,it) is the local clustering coefficient similarity, NeighborDensitySim(i s ,i t ) represents the second-order neighbor density similarity, α, β, γ are the weighting coefficients of each topological feature, i s is the number of the spatial position of the perception data, i t It is the number of the spatial location of the twin data.

[0101] S209: If it is determined that the topological similarity score is less than a preset score, the twin data and the perception data are spatially aligned again within the neighborhood of the perception data.

[0102] For example, when the topological similarity score Sim(i s ,i t ) is lower than the preset score, triggering local dynamic correction, that is, in the neighborhood N(i s ) and spatially align the twin data with the perception data again, as shown in the following formula:

[0103]

[0104] Among them, i′ t is the spatial position number of the realigned twin data, i s is the number of the spatial position of the perception data, k is the index of the number of the spatial position of each candidate twin data in the neighborhood, and the candidate number that matches i will be found among these candidate numbers. s The spatial position of the twin data with the most similar topological structure is numbered to determine the realigned twin data to dynamically correct the mapping relationship.

[0105] Furthermore, the feature embedding vectors of the twin data and the perception data after alignment in time and space dimensions are extracted respectively, and weights are assigned to the feature embedding vectors of the perception data and the twin data according to the feature alignment model. The feature alignment model is:

[0106]

[0107] Among them, L is the feature alignment function, l is the number of layers of feature extraction, is the perceptual data embedding vector of the lth layer, is the twin data embedding vector of layer l, λ l is the weight coefficient for fusion matching of features at different levels.

[0108] Obtain the perception data embedding vector corresponding to the twin data embedding vector when the feature alignment function is minimum, and perform feature alignment between the perception data and the twin data.

[0109] S210: Correcting the operating data of the digital twin model according to the perception data, and compensating the perception data according to the prediction error of the digital twin model.

[0110] For example, the digital twin model is continuously iterated, corrected and updated through a dynamic feedback mechanism. Specifically, at the data processing level, the operating data of the digital twin model is corrected in real time using sensory data, and the consistency between the operation of the digital twin model and the real equipment is reflected through transportation data. The correction model is shown as follows:

[0111] D” t =D t '+F(ΔDs ,t);

[0112] Where ΔD s is the error term between the aligned perception data and the twin data, D′ t is the aligned twin data, D″ t is the corrected operating data of the digital twin model, F(ΔD s ,t) is the time dynamic correction function, which is calculated as follows:

[0113] F(ΔD s ,t)=α(t)·tanh(ΔD s )·e -λt ;

[0114] Among them, α(t) is the dynamic adjustment factor, tanh(ΔD s ) is a nonlinear mapping that enhances sensitivity to large errors or abnormal fluctuations, e -λt is a time decay term that can prevent historical errors from excessively affecting the current state.

[0115] In order to make up for the deficiencies in twin model prediction caused by missing perception data or equipment failure, a perception data compensation method is proposed. The compensation model is:

[0116] D” s =D s '+G(ΔD t ,σ);

[0117] Among them, D′ s is the aligned perception data, D″ s is the compensated perception data, ΔD t is the twin model prediction error, G(ΔD t ,σ) is the uncertainty weighted compensation function, which is calculated as follows:

[0118] G(ΔD t ,σ)=β(σ)·ΔD t +γ·Var(ΔD t );

[0119] Among them, β(σ) is the compensation coefficient that is dynamically adjusted according to the uncertainty level, Var(ΔD t ) is the historical variance of the twin model's prediction error, which is used to improve its responsiveness to volatility changes. γ is a tuning factor that can be dynamically tuned based on the historical error fluctuations. This method effectively improves the integrity and robustness of perception data.

[0120] S211: Fuse the aligned twin data with the perception data through the data fusion model.

[0121] It should be noted that step S210 Figure 1 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0122] S212: Optimize the model by adjusting the weight parameters of the data fusion model.

[0123] For example, the fusion effect is optimized by dynamically adjusting the weight parameters, as shown in the following formula:

[0124] D f =w s ·F(D s ')+w t ·F(D t ');

[0125] Where F(·) is the data source feature enhancement function, which is used to strengthen the contribution of multidimensional features such as historical trends and local fluctuations. ws and wt represent the dynamic confidence weights of the perception data and twin data, respectively, satisfying ws + wt = 1. The dynamic confidence weights of the perception data and twin data are calculated as follows:

[0126]

[0127] Among them, I(D x ) is the data source D x The information gain, D x ∈D s ′、D t ′, its calculation formula is as follows;

[0128]

[0129] Among them, p i Data source D x The information probability distribution at the i-th observation point. By introducing information entropy evaluation, this method can dynamically reflect the effective information contribution of each data source and achieve accurate weighting of multi-source heterogeneous data.

[0130] As can be seen from the above description, the present invention can effectively solve the problem of dynamic consistency fusion processing of multi-source heterogeneous data by aligning the twin data with the sensor data at different moments in time, and aligning the twin data of sensory devices at different spatial locations with the spatial position of the sensor data. This provides strong support for real-time monitoring and optimization decision-making of complex systems, and is particularly suitable for application needs in scenarios such as smart cities and the Industrial Internet. In addition, through a dynamic feedback mechanism, the accuracy of fused data, computational efficiency, and dynamic response capabilities are improved.

[0131] like Figure 3 As shown, Figure 3This is a schematic diagram of the structure of an embodiment of a twin data and perception data fusion system provided by the present invention. A twin data and perception data fusion system 10 includes:

[0132] The data acquisition module 11 is used to obtain the twin data and perception data of the target area at different times. The target area includes perception devices. The twin data is the virtual data of the perception devices at different spatial positions predicted by the digital twin model, and the perception data is the real data actually monitored by the perception devices at different spatial positions in the target area.

[0133] The time alignment module 12 is used to time align the twin data and the perception data at different times.

[0134] The spatial alignment module 13 is used to align the twin data of the perception devices at different spatial positions with the spatial positions of the perception data.

[0135] The data fusion model construction module 14 is used to fuse the aligned twin data with the perception data through the data fusion model.

[0136] Exemplarily, in the data acquisition module 11, the twin data and perception data of the target area at different times are obtained. The target area includes perception devices. The twin data is the virtual data of the perception devices at different spatial positions predicted by the digital twin model, and the perception data is the real data actually monitored by the perception devices at different spatial positions in the target area. Specifically, the three-dimensional point cloud data of the target area at different times is extracted by the laser radar, and a model is built based on the three-dimensional point cloud data to construct a digital twin model. Distributed perception nodes of the perception devices in the target area are set in the digital twin model, and the data at the distributed perception nodes are predicted based on the digital twin model to obtain twin data. In the time alignment module 12, a time alignment model is constructed. The time alignment model is: L time (π)=w(s t1 ,s t2 )·||D s (s t1 )-D t (s t2 )||, where L time is the time alignment function, π is the alignment path between twin data and perception data in the time dimension, w(s t1 ,s t2 ) is from time point s t1 To time point s t2 The cost weight, Ds(s t1 ,) is s t1 The perception data at the moment, Dt(s t2 ) is s t2The twin data time series at the moment; according to the time alignment model, the perception data corresponding to the twin data is obtained when the time alignment function is minimum, and the twin data and the perception data are time-aligned. In the spatial alignment module 13, a spatial alignment model is constructed, and the spatial alignment model is: Among them, L adj is the spatial alignment function, M is the alignment path mapping the perceptual data to the twin data in the spatial dimension, and D t is twin data, i s ,j s is the number of the spatial location of the perception data, E s is a numbered set of sensory data; the sensory data corresponding to the twin data when the spatial alignment function is minimized is obtained according to the spatial alignment model, and the twin data and the sensory data are spatially aligned. In the data fusion model construction module 14, the operating data of the digital twin model is first corrected according to the sensory data, and the sensory data is compensated according to the prediction error of the digital twin model. Furthermore, the aligned twin data and the sensory data are fused through the data fusion model, and the model is optimized by adjusting the weight parameters of the data fusion model.

[0137] like Figure 4 As shown, Figure 4 The storage medium 20 stores at least one computer program 31, which is executed by the processor 22 to implement the following. Figure 1 and Figure 2 In one embodiment, the storage medium 20 may be a memory chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server.

[0138] Additionally, the processes depicted in the accompanying figures do not necessarily have to be performed in the particular order shown, or sequential order, to achieve desired results. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.

[0139] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer-readable storage medium embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, refer to the description of the method embodiments.

[0140] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other. Therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0141] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0142] For the convenience of description, when describing the above device, various units are divided into functions and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in the same one or more software and / or hardware. It should be understood by those skilled in the art that this specification embodiment can be provided as a method, system, or computer program product. Therefore, this specification embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0147] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0148] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0150] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0151] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For relevant parts, refer to the description of the method embodiments.

[0152] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for fusing twin data and perception data, characterized in that: The method comprises: Acquire twin data and perception data of the target area at different times, where the target area includes perception devices. The twin data is virtual data of perception devices at different spatial locations predicted by the digital twin model, and the perception data is real data actually monitored by perception devices at different spatial locations in the target area. Temporally aligning the twin data and the perception data at different moments; Aligning the twin data of the sensing devices at different spatial positions with the spatial positions of the sensing data; The aligned twin data is fused with the perception data through a data fusion model.

2. The method for fusing twin data and perception data according to claim 1, characterized in that: The aligning of the twin data at different moments with the time series of the perception data specifically includes: Construct a time alignment model, the time alignment model is: L time (π)=w(s t1 ,s t2 )·||D s (s t1 )-D t (s t2 )||, where L time is the time alignment function, π is the alignment path between twin data and perception data in the time dimension, w(s t1 ,s t2 ) is from time point s t1 To time point s t2 The cost weight, Ds(s t1 ) is s t1 The perception data at the moment, Dt(s t2 ) is s t2 Twin data time series at each moment; According to the time alignment model, the perception data corresponding to the twin data is obtained when the time alignment function is minimum, and the twin data and the perception data are time-aligned.

3. The method for fusing twin data and perception data according to claim 2, characterized in that: After the time series of the twin data and the perception data are time-aligned, the method further includes: Determining a residual sequence between the twin data and the perception data after time alignment; determining a local residual fluctuation amplitude according to the residual sequence; If it is determined that the fluctuation amplitude of the local residual is greater than a preset threshold, then there is local drift, and the twin data and the perception data are time-aligned again within the time interval where the local drift exists: If it is determined that the fluctuation amplitude of the local residual is less than or equal to the preset threshold, then there is no local drift.

4. The method for fusing twin data and perception data according to claim 1, characterized in that: The aligning of the twin data of the sensing devices at different spatial positions with the spatial positions of the sensing data specifically includes: Construct a spatial alignment model, the spatial alignment model is: Among them, L adj is the spatial alignment function, M is the alignment path mapping the perceptual data to the twin data in the spatial dimension, and D t is twin data, i s ,j s is the number of the spatial location of the perception data, E s is a numbered set of perception data; According to the spatial alignment model, the perception data corresponding to the twin data is obtained when the spatial alignment function is minimized, and the twin data and the perception data are spatially aligned.

5. The method for fusing twin data and perception data according to claim 4, characterized in that: The method further includes: obtaining the perception data corresponding to the twin data when the spatial alignment function is minimum according to the spatial alignment model, and spatially aligning the twin data with the perception data; Obtaining a topological similarity score between the spatially aligned perception data and the twin data; If it is determined that the topological similarity score is less than a preset score, the twin data and the perception data are spatially aligned again within the neighborhood of the perception data.

6. The method for fusing twin data and perception data according to claim 1, characterized in that: After aligning the twin data and the perception data at the time and space levels, the method further includes: Extracting feature embedding vectors of the twin data and the perception data after alignment in time and space dimensions respectively; Assign weights to the feature embedding vectors of the perception data and the twin data according to a feature alignment model; the feature alignment model is: Among them, L is the feature alignment function, l is the number of layers of feature extraction, is the perceptual data embedding vector of the lth layer, is the twin data embedding vector of layer l, λ l The weight coefficient for fusion matching of features at different levels; Obtain the perception data embedding vector corresponding to the twin data embedding vector when the feature alignment function is minimum, and perform feature alignment between the perception data and the twin data.

7. The method for fusing twin data and perception data according to claim 1, characterized in that: The obtaining of twin data of the target area at different times specifically includes: Extract three-dimensional point cloud data of the target area at different times through lidar; Modeling is performed based on the three-dimensional point cloud data to construct a digital twin model; Distributed sensing nodes of sensing devices in the target area are set in the digital twin model, and data at the distributed sensing nodes are predicted according to the digital twin model to obtain twin data.

8. The method for fusing twin data and perception data according to claim 7, characterized in that: The method further includes: building a data fusion model based on the aligned twin data and the perception data, and fusing the twin data with the perception data through the data fusion model; The operating data of the digital twin model is corrected according to the perception data, and the perception data is compensated according to the prediction error of the digital twin model.

9. The method for fusing twin data and perception data according to claim 1, characterized in that: The method further includes: building a data fusion model based on the aligned twin data and the perception data, and fusing the twin data and the perception data through the data fusion model; Model optimization is performed by adjusting the weight parameters of the data fusion model.

10. A twin data and perception data fusion system, characterized in that: The system comprises: A data acquisition module is used to acquire twin data and perception data of a target area at different times. The target area includes perception devices. The twin data is virtual data of perception devices at different spatial positions predicted by the digital twin model, and the perception data is real data actually monitored by perception devices at different spatial positions in the target area. A time alignment module, used to time-align the twin data and the perception data at different times; A spatial alignment module, configured to align the twin data of sensing devices at different spatial positions with the spatial positions of the sensing data; The data fusion model construction module is used to fuse the aligned twin data with the perception data through the data fusion model.

11. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 9.

Images