Method and system for motion correlation between virtual objects and digital objects in a virtual scene

Abstract

This invention discloses a method for motion association between virtual objects and digital objects in a virtual scene, comprising: constructing a virtual object model in the virtual scene; obtaining connection information of the constructed virtual object model, splitting it according to the connection information to obtain virtual object sub-models, and constructing a restricted range of motion between the virtual object sub-models according to the connection information; obtaining the feature information matrix of the virtual object and combining it with the three-dimensional spatial structure to obtain a digital object model; obtaining a digital object sub-model V according to the three-dimensional spatial structure of the virtual object sub-models; calculating the motion information matrix V1 of each virtual object sub-model; and performing a matrix multiplication operation between the digital object sub-model V and the motion information matrix V1 to obtain the moved digital object sub-model V' = V × V1. This method enables the digital object model to obtain motion information, facilitating subsequent real-time operation of the digital object model. Through matrix operations, the computational load is small and the real-time performance is high.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality technology and digital twins, and specifically to a method and system for motion correlation between virtual objects and digital objects in a virtual scene. Background Technology

[0002] A digital twin is a virtual entity constructed in virtual space that represents the real-time operational state of a physical entity. It possesses comprehensive functions integrating geometric modeling, simulation, and data analysis, playing a role in comprehensive analysis and decision-making. Taking the digital human body as an example, a digital human body model, or simply a digital person, is a human body model that reflects three-dimensional anatomical structure, constructed using medical imaging data such as CT / MRI / color photographs through computer technology. It is primarily used in fields such as nuclear medicine and radiation protection.

[0003] An ideal medical imaging digital twin should meet three requirements: First, the external visual appearance and the simulated internal mechanisms must be organically integrated, interactive, and interconnected, not two separate, mechanically integrated components. Second, the operation process of the digital twin system must be consistent with standard clinical procedures, with high image fidelity and real-time operation essentially identical to the real machine. Third, there must be data interaction between the real system and the twin system. The first requirement means that all movements and operations in the virtual scene must be reflected in the final virtual scan results. Specifically, the movements of virtual devices in the virtual scene (such as tilt angles), the different positioning and orientation of the virtual human model, and the different parameter settings in the virtual scan must all be scientifically and accurately reflected in the virtual scan image; that is, the external virtual scene and the internal mechanism model need to be linked.

[0004] Patent CN 102722908 B discloses a method for spatial positioning of objects in a 3D virtual reality scene. This method acquires driving data generated by an external device that provides three-axis driving data, and then converts this driving data into motion offsets, including translational and rotational offsets. This allows for accurate spatial positioning of objects in a 3D virtual reality scene solely within a 3D view. However, this method only achieves spatial positioning and does not demonstrate the interaction between the external virtual scene and the internal mechanistic model. Summary of the Invention

[0005] To address the aforementioned technical problems, the purpose of this invention is to provide a method and system for motion correlation between virtual objects and digital objects in a virtual scene. This method can link the model operation of the virtual scene with the motion of the digital object model in the operating mechanism model, thereby enabling the digital object model to obtain motion information. This facilitates subsequent real-time operation of the digital object model. Through matrix operations, the computational load is small and the real-time performance is high. For example, it can be used as a virtual scanning object for mechanism simulation, so that the virtual scanned image is completely consistent with the parts, positions, and orientation of the virtual human body model in the virtual scene, achieving a highly realistic virtual scanning effect.

[0006] The technical solution of this invention is:

[0007] A method for motion correlation between virtual objects and digital objects in a virtual scene includes the following steps:

[0008] S01: Construct virtual object models in a virtual scene;

[0009] S02: Obtain the connection information of the constructed virtual object model, split the constructed virtual object model according to the connection information to obtain virtual object sub-models, and construct a restricted range of motion between the virtual object sub-models according to the connection information.

[0010] S03: Obtain the feature information matrix of the virtual object and combine it with the three-dimensional spatial structure of the virtual object model to obtain the digital object model; obtain the digital object sub-model V based on the three-dimensional spatial structure of the virtual object sub-model.

[0011] S04: Control the movement of virtual objects in the virtual scene and calculate the motion information matrix V1 of each virtual object sub-model;

[0012] S05: Perform matrix multiplication on the digital object sub-model V and the motion information matrix V1 to obtain the motiond digital object sub-model V' = V × V1.

[0013] In a preferred embodiment, in step S02, the connection information of the virtual object model is obtained based on the real object. The connection information includes connection type, degree of freedom, and degree of freedom range. The degree of freedom represents the movement in a certain cross-sectional direction, including the sagittal plane S, the coronal plane C, and the transverse plane T.

[0014] In a preferred embodiment, the feature information matrix in step S03 includes physical information, chemical information, and physiological information.

[0015] In a preferred embodiment, the motion information matrix V1 of each sub-model of the virtual object calculated in step S04 includes:

[0016] S41: Construct the geometric information attributes of each virtual object model;

[0017] S42: The geometric information attribute is associated with the world coordinate system matrix sub-attribute;

[0018] S43: Construct sub-attributes of the world coordinate system matrix, including position P(x,y,z) in the world coordinate system, where the values ​​of x,y,z represent the translation values ​​in the X, Y, and Z directions, respectively; scaling S(s1,s2,s3) in the world coordinate system, where the values ​​of s1,s2,s3 represent the scaling ratio values ​​in the X, Y, and Z directions, respectively; and rotation R(r1,r2,r3) in the world coordinate system, where the values ​​of r1,r2,r3 represent the rotation angle values ​​in the X, Y, and Z directions, respectively.

[0019] S44: Change the geometric information properties of each constructed virtual object model to calculate the translation matrix, scaling matrix, and rotation matrix;

[0020] S45: Multiply the translation and scaling rotation matrices along the X, Y, and Z axes respectively to obtain the motion information matrix V1 of each virtual object sub-model.

[0021] In a preferred embodiment, step S05 is followed by:

[0022] S61: Obtain the spatial information matrix Vs of the region of interest of the virtual object;

[0023] S62: Perform a matrix dot product operation between the obtained motiond digital object sub-model V' and the region of interest Vs to obtain the motiond digital object sub-model V0 = V'·Vs within the region of interest;

[0024] S63: If the region of interest includes multiple virtual object sub-models, then after performing matrix concatenation on each virtual object sub-model, calculate the motiond digital object sub-model V0 within the region of interest.

[0025] This invention also discloses a medical imaging digital twin system, comprising:

[0026] The virtual scene module is used to construct virtual scenes.

[0027] The virtual device module constructs virtual devices, including motion functionalization, interfaces, and motion control.

[0028] Virtual human body module: Constructs a virtual human body, and enables the functionalization of movement through bone connections in the virtual human body;

[0029] The virtual scanning simulation module obtains V0 by using the motion correlation method between virtual objects and digital objects in the virtual scene. V0 includes the parts, positions, and orientation information of the virtual human body model, as well as FOV information. It serves as the virtual scanning object of the mechanism simulation model, performs simulated data acquisition and image reconstruction, completes the imaging simulation of the digital twin of medical images, and obtains virtual scanned medical images.

[0030] In a preferred technical solution, the method for selecting the region of interest includes:

[0031] Obtain the relevant parameters for medical image imaging set in the virtual scene imaging operation;

[0032] The coordinates of the three-dimensional center of the frame cavity are taken as V. s The center point;

[0033] V is determined based on the scan field of view (FOV). s Height and width;

[0034] V is obtained based on the scanning range. s Length;

[0035] Determine V s Inclination angles in three directions.

[0036] In the preferred technical solution, V in CT and MRI modes s The height and width are the FOVx and FOVy of the scan parameters, respectively. In DR mode, V s The height and width are the opening and closing range of the beam; V in CT and MRI modes s The length is equal to (layer thickness + interlayer spacing) × number of layers, V in DR mode s The length is the physical digital thickness of the human body; V s The tilt angle in the three directions is determined by the gantry tilt angle in CT mode, by the tilt angle of the imaging section in MRI scanning mode, and by the tilt angle of the beam in DR mode.

[0037] This invention further discloses a motion correlation system between virtual objects and digital objects in a virtual scene, comprising:

[0038] The virtual object model building module constructs virtual object models in a virtual scene.

[0039] The virtual object sub-model construction module obtains the connection information of the constructed virtual object model, splits the constructed virtual object model according to the connection information to obtain virtual object sub-models, and constructs a restricted range of movement between virtual object sub-models according to the connection information.

[0040] The digital object sub-model construction module obtains the feature information matrix of the virtual object and combines it with the three-dimensional spatial structure of the virtual object model to obtain the digital object model; based on the three-dimensional spatial structure of the virtual object sub-model, the digital object sub-model V is obtained.

[0041] The virtual object sub-model motion information matrix calculation module controls the movement of virtual objects in the virtual scene and calculates the motion information matrix V1 of each virtual object sub-model.

[0042] The motion association module performs matrix multiplication on the digital object sub-model V and the motion information matrix V1 to obtain the motiond digital object sub-model V' = V × V1.

[0043] The preferred technical solution also includes a region of interest information output module, comprising:

[0044] S61: Obtain the spatial information matrix Vs of the region of interest of the virtual object;

[0045] S62: Perform a matrix dot product operation between the obtained motiond digital object sub-model V' and the region of interest Vs to obtain the motiond digital object sub-model V0 = V'·Vs within the region of interest;

[0046] S63: If the region of interest includes multiple virtual object sub-models, then after performing matrix concatenation on each virtual object sub-model, calculate the motiond digital object sub-model V0 within the region of interest.

[0047] Compared with the prior art, the advantages of the present invention are:

[0048] 1. The operation of the virtual scene model can be linked with the motion of the digital object model in the runtime mechanism model, thereby enabling the digital object model to obtain motion information, facilitating subsequent real-time operation of the digital object model. Constructing the model matrix and motion information matrix only requires matrix operations, resulting in low computational load and high real-time performance. The virtual object model is split into multiple sub-entities based on connection information, and motion information is calculated for each sub-entity before being correlated, significantly improving realism.

[0049] 2. Motion-associated digital objects can be used as virtual scanning objects for mechanism simulation. The rotation information obtained from the arbitrary positioning and changes in body position of virtual devices and virtual human models in the virtual scene can be used as a bridge to perform the same coordinate mapping transformation on the internal physical digital human matrix associated with the virtual human model. This yields the physical digital human matrix information within the scanning field of view, which is then used as the object of virtual scanning for mechanism simulation. This ensures that the virtual scanned image is completely consistent with the parts, body positions, and positioning of the virtual human model in the virtual scene, achieving a highly realistic virtual scanning effect.

[0050] 3. This method forms the basis for integrating the external visual model and the internal mechanism model of medical image digital twins. It is key to creating highly realistic medical imaging systems with digital twins and is an important component for the subsequent development of metaverse hospitals. Medical image digital twins can be implemented without expensive and bulky hardware, avoiding radiation damage. They can be widely used in practical training and standardized training for medical imaging technicians, physicians, and engineers. They can also be used for virtual scanning of medical images and artificial intelligence sample augmentation of medical images. Attached Figure Description

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0052] Figure 1 This is a flowchart of the motion association method between virtual objects and digital objects in the virtual scene of this embodiment;

[0053] Figure 2 A flowchart illustrating a preferred embodiment of a method for linking virtual and real scenes;

[0054] Figure 3 This is a schematic diagram illustrating the motion relationship between virtual and digital objects in the virtual scene of this embodiment.

[0055] Figure 4 This is a flowchart of the workflow of the medical imaging digital twin system in this embodiment;

[0056] Figure 5 This example illustrates a CT digital twin of medical imaging digital twins.

[0057] Figure 6 This is a flowchart of the 3D modeling method for the motion functionality of virtual devices and virtual humans in this embodiment;

[0058] Figure 7 This is a diagram illustrating the head-raising and head-lowering movements of the head sub-body caused by the occipital-column connection in this embodiment;

[0059] Figure 8 This is a schematic diagram of the movements of the left and right sides of the head sub-body caused by the connection of the occipital ring in this embodiment;

[0060] Figure 9 This is a schematic diagram illustrating the leftward and rightward head turning movements of the head sub-body caused by the occipital-orbicularis oculi connection in this embodiment;

[0061] Figure 10 This embodiment demonstrates the virtual human body positioning and posture effects in CT / MRI digital twins.

[0062] Figure 11 This embodiment simulates the fusion effect of CT localization imaging mechanism.

[0063] Figure 12This embodiment demonstrates the fusion effect of CT tomographic imaging mechanism simulation.

[0064] Figure 13 This embodiment demonstrates the fusion effect of arbitrary MR tomography and mechanism simulation.

[0065] Figure 14 This is a schematic diagram of the operation of the head-mounted display and remote control of the medical imaging digital twin system in this embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0067] Example 1:

[0068] like Figure 1 As shown, a method for motion association between virtual objects and digital objects in a virtual scene includes the following steps:

[0069] S01: Construct virtual object models in a virtual scene;

[0070] S02: Obtain the connection information of the constructed virtual object model, split the constructed virtual object model according to the connection information to obtain virtual object sub-models, and construct a restricted range of motion between the virtual object sub-models according to the connection information.

[0071] S03: Obtain the feature information matrix of the virtual object and combine it with the three-dimensional spatial structure of the virtual object model to obtain the digital object model; obtain the digital object sub-model V based on the three-dimensional spatial structure of the virtual object sub-model.

[0072] S04: Control the movement of virtual objects in the virtual scene and calculate the motion information matrix V1 of each virtual object sub-model;

[0073] S05: Perform matrix multiplication on the digital object sub-model V and the motion information matrix V1 to obtain the motiond digital object sub-model V' = V × V1.

[0074] The virtual objects in step S01 can be virtual human bodies, virtual plants and animals, virtual devices, virtual man-made objects, etc.

[0075] Virtual object models in virtual scenes can be constructed using 3D modeling software such as Unity, U3D, and Unreal Engine, based on the appearance of real medical imaging equipment. Virtual human bodies can also be modeled using 3D modeling software.

[0076] However, the preferred virtual human body modeling method in this embodiment can be obtained by surface reconstruction from images captured by volumetric imaging equipment from medical imaging devices. Alternatively, it can be obtained by processing 3D rotated video recordings.

[0077] In a preferred embodiment, in step S02, the connection information of the virtual object model is obtained based on the real object. The connection information includes the connection type, degree of freedom, and degree of freedom range. The degree of freedom represents the movement in a certain cross-sectional direction, including the sagittal plane S, the coronal plane C, and the transverse plane T.

[0078] Virtual objects possess motion capabilities, including three-dimensional translation, three-dimensional rotation, and three-dimensional scaling motion in virtual space.

[0079] Virtual objects can be connected via joints, ropes, hinges, springs, or other forms of connection, or even through non-physical connections where the sub-objects can move. The specific connection method depends on the specific virtual object.

[0080] Depending on the application scenario, a virtual object can be broken down into multiple related sub-objects, and these sub-objects can be configured with restricted 3D translation, scaling, and rotation functions. The restrictions here refer to limitations on the degrees of freedom and the range of those degrees of freedom.

[0081] In a preferred embodiment, the feature information matrix in step S03 includes physical information, chemical information, and physiological information.

[0082] The characteristic information of digital objects can include various physical information such as density, proton density, electron density, relaxation time, and diffusion coefficient, as well as chemical information such as chemical shift and J-coupling strength, and physiological information such as metabolic coefficient.

[0083] A digital object model is the physical information matrix inherent in a virtual object. A specific construction method can be adopted using the patent application number 201710361588.7, entitled "Construction Method and Device for Digital MRI Atlas." Additionally, a DR / CT digital object construction method and system with application number 202011525280X can be used to construct a CT / DR digital human body. Of course, other methods can also be used; this is not limited to these methods.

[0084] Based on the sub-body splitting of the virtual object, the corresponding digital object physical information matrix is ​​split into multiple sub-body physical information matrices.

[0085] In a preferred embodiment, the motion information matrix V1 of each sub-model of the virtual object calculated in step S04 includes:

[0086] S41: Construct the geometric information attributes of each virtual object model;

[0087] S42: The geometric information attribute is associated with the world coordinate system matrix sub-attribute;

[0088] S43: Construct sub-attributes of the world coordinate system matrix, including position P(x,y,z) in the world coordinate system, where the values ​​of x,y,z represent the translation values ​​in the X, Y, and Z directions, respectively; scaling S(s1,s2,s3) in the world coordinate system, where the values ​​of s1,s2,s3 represent the scaling ratio values ​​in the X, Y, and Z directions, respectively; and rotation R(r1,r2,r3) in the world coordinate system, where the values ​​of r1,r2,r3 represent the rotation angle values ​​in the X, Y, and Z directions, respectively.

[0089] S44: Change the geometric information properties of each constructed virtual object model to calculate the translation matrix, scaling matrix, and rotation matrix;

[0090] S45: Multiply the translation and scaling rotation matrices along the X, Y, and Z axes respectively to obtain the motion information matrix V1 of each virtual object sub-model.

[0091] In a preferred embodiment, such as Figure 2 As shown, after step S05, the following steps are also included:

[0092] S61: Obtain the spatial information matrix Vs of the region of interest of the virtual object;

[0093] S62: Perform a matrix dot product operation between the obtained motiond digital object sub-model V' and the region of interest Vs to obtain the motiond digital object sub-model V0 = V'·Vs of the region of interest;

[0094] S63: If the region of interest includes multiple virtual object sub-models, then after performing matrix concatenation on each virtual object sub-model, calculate the digital object sub-model V0 after motion of the region of interest.

[0095] In another embodiment, such as Figure 3 As shown, a motion correlation system between virtual objects and digital objects in a virtual scene includes:

[0096] Virtual object model building module 10, which builds virtual object models in a virtual scene;

[0097] The virtual object sub-model construction module 20 obtains the connection information of the constructed virtual object model, splits the constructed virtual object model according to the connection information to obtain virtual object sub-models, and constructs a restricted range of activity between the virtual object sub-models according to the connection information.

[0098] The digital object sub-model construction module 30 obtains the feature information matrix of the virtual object and combines it with the three-dimensional spatial structure of the virtual object model to obtain the digital object model; and obtains the digital object sub-model V based on the three-dimensional spatial structure of the virtual object sub-model.

[0099] The virtual object sub-model motion information matrix calculation module 40 controls the movement of virtual objects in the virtual scene and calculates the motion information matrix V1 of each virtual object sub-model.

[0100] The motion association module 50 performs matrix multiplication on the digital object sub-model V and the motion information matrix V1 to obtain the motiond digital object sub-model V' = V × V1.

[0101] In one embodiment, the system further includes a region of interest information output module 60, comprising:

[0102] S61: Obtain the spatial information matrix Vs of the region of interest of the virtual object;

[0103] S62: Perform a matrix dot product operation between the obtained motiond digital object sub-model V' and the region of interest Vs to obtain the motiond digital object sub-model V0 = V'·Vs within the region of interest;

[0104] S63: If the region of interest includes multiple virtual object sub-models, then after performing matrix concatenation on each virtual object sub-model, calculate the digital object sub-model V0 after motion of the region of interest.

[0105] In another embodiment, a medical imaging digital twin system includes:

[0106] The virtual scene module is used to construct virtual scenes.

[0107] The virtual device module constructs virtual devices, including motion functionalization, interfaces, and motion control.

[0108] Virtual human body module: Constructs a virtual human body and enables the functional movement of the virtual human body's bones and joints;

[0109] The virtual scanning simulation module obtains V0 by using the motion correlation method between virtual objects and digital objects in the virtual scene. V0 includes the parts, positions, and orientation information of the virtual human body model, as well as FOV information. It serves as the virtual scanning object of the mechanism simulation model, performs simulated data acquisition and image reconstruction, completes the imaging simulation of the digital twin of medical images, and obtains virtual scanned medical images.

[0110] The following describes a preferred embodiment of a medical imaging digital twin system in detail, using a digital human body as an example:

[0111] The overall workflow diagram of the medical imaging digital twin system is as follows: Figure 4 As shown.

[0112] The construction of a typical medical imaging digital twin involves the creation of a virtual digital human body model and a virtual medical imaging equipment model. The virtual digital human body as a whole possesses three-dimensional translation and rotational motion, but does not include three-dimensional scaling. Three-dimensional translation is typically achieved by the virtual digital human body model's movement up and down and into and out of a virtual examination table. The virtual equipment is generally also stationary. For example... Figure 5 As shown.

[0113] In the construction of digital twins for medical imaging, the virtual digital human body model is divided into 15 sub-body parts based on the positioning requirements of routine medical imaging examinations: head, neck, trunk (chest, abdomen, and pelvis), upper arm ×2, forearm ×2, palm ×2, thigh ×2, lower leg ×2, and foot ×2. The sub-body parts are then configured with 3D degrees of freedom of motion based on joint function, including joints such as the atlanto-occipital joint, cervical spine joint, thoracic spine joint, lumbar spine joint, shoulder joint, elbow joint, wrist joint, hip joint, knee joint, and ankle joint.

[0114] Virtual device and virtual human motion functional 3D modeling methods, such as Figure 6 As shown, the joint movement functions of a virtual human are constructed based on the joint movement functions of a normal human body. Table 1 shows the joint types, degrees of freedom (DOF) dimension, and range of DOF for each joint in the virtual human body. A degree of freedom represents movement in a specific cross-sectional direction. 3D indicates movement is possible in three cross-sections, where S is the sagittal plane, C is the coronal plane, and T is the transverse plane. Considering the joint movement requirements for clinical examination, the functionalization of the temporomandibular joint, finger joints, and toe joints can be disregarded.

[0115] Table 1. Types and degrees of freedom of joints in the virtual human body.

[0116]

[0117]

[0118] The above sub-body splitting is illustrated using the atlanto-occipital joint as an example. Using the atlanto-occipital joint as the boundary, the head of the virtual digital human body can be separated into a single sub-body. The atlanto-occipital joint is a ball-and-socket joint, therefore the head sub-body can perform restricted rotational movements within three cross-sections. For example, in the anterior-posterior sagittal plane S, the rotation angle range is [-45°, 55°]; in the left-right coronal plane C, the head swing angle range is [-30°, 30°]; and in the transverse cross-section T perpendicular to the long axis of the human body, the head rotation range is [-60°, 60°].

[0119] An example of head movement caused by the atlanto-occipital joint: Selecting the head will display operation rings showing the rotation of the atlanto-occipital joint in three sections, represented by red, blue, and green circles with arrows. Dragging the arrows on the corresponding circles will cause the head to rotate in the corresponding direction. At this point, the head is in its original state, with all three degrees of freedom of the atlanto-occipital joint at 0.

[0120] Adjusting the red circle will cause the head to rotate back and forth in the sagittal plane, i.e., head tilting and head lowering movements, as shown. Figure 7 As shown.

[0121] Adjusting the blue circle will cause the head to rotate within the coronal plane, i.e., move the left and right sides of the head, as shown. Figure 8 As shown.

[0122] Adjusting the green circle will cause the head to rotate within the cross-section, i.e., turn left and turn right. For example... Figure 9 As shown.

[0123] Similarly, all other sub-body parts are decomposed through joints and their rotational degrees of freedom.

[0124] Step 3: Construct a digital object model (the physical information matrix inherent in the virtual object) corresponding to the virtual object's 3D model, and divide it into multiple sub-physical information matrices V.

[0125] The digital objects corresponding to virtual objects are physical information matrices inherent in virtual physics, and their construction methods vary depending on the different physical information.

[0126] Based on the sub-body splitting of the virtual object, the corresponding digital human body information matrix is ​​split into multiple sub-body physical information matrices, denoted as V.

[0127] In a typical example, in the construction of a digital twin for medical imaging, the physical digital human body matrix in this step refers to the medical imaging physical information model database of the internal tissues of the human body corresponding to the virtual human body model. Medical imaging physical information refers to the physical information of human tissues that can affect medical imaging signals, including electron density, effective atomic number, proton density, spin-spin relaxation time, spin lattice relaxation time, chemical shift, diffusion coefficient, etc. The specific construction method can be found in the applicant's existing patents.

[0128] Corresponding to the virtual human body segmentation, the digital human body matrix is ​​divided into 15 sub-matrices, namely head, neck, torso (chest, abdomen and pelvis), upper arm ×2, forearm ×2, palm ×2, thigh ×2, calf ×2, and foot ×2, denoted as V.

[0129] Similarly, taking the atlanto-occipital joint as an example, the physical digital human body database is divided into head sub-body data, with the atlanto-occipital joint as the boundary, and denoted as V.

[0130] The remaining sub-entities are split in the same way.

[0131] Step 4: For the overall motion and internal sub-motion of virtual objects in the 3D virtual scene, the position, scaling and rotation information of each sub-body of the virtual object can be obtained based on the WebGL 3D and other drawing tool protocols of the 3D virtual scene. Then, the motion information matrix V1 is calculated, which includes the translation, scaling and rotation information of each sub-body relative to the original state.

[0132] In a 3D virtual scene, each object model has its own geometric information attribute. Under the geometric attribute, there is a world coordinate system matrix sub-attribute, which has three fields: position, scale, and rotation.

[0133] Position represents the position (x, y, z) of the object in the world coordinate system. Changes in this field indicate three-dimensional translational motion, or any translational motion of the object in three-dimensional virtual space will cause a change in the value of this field. The values ​​of x, y, and z represent the translation values ​​in the X, Y, and Z directions, respectively.

[0134] Scale represents the scaling of the sub-body in the world coordinate system (s1, s2, s3). Changes in Scale indicate that the sub-body has been shrunk or enlarged, or in other words, any scaling of the sub-body in the 3D virtual space will cause a change in the value of this field. The values ​​of s1, s2, and s3 represent the scaling ratios in the X, Y, and Z directions, respectively.

[0135] Rotation represents the rotation of the sub-body in the world coordinate system (r1, r2, r3). ​​Its change indicates that the sub-body has rotational motion, or in other words, any rotational motion of the sub-body in the three-dimensional virtual space will cause a change in the value of this field. The values ​​of r1, r2, and r3 represent the rotation angle values ​​in the X, Y, and Z directions, respectively.

[0136] The translation, scaling, and rotation matrices are calculated from the obtained position, scale, and rotation field values. These three matrices are then multiplied together to obtain the overall motion information matrix V1. The specific translation, scaling, and rotation matrices are as follows:

[0137] (1) The translation matrix is:

[0138] The translation matrix is ​​obtained by assigning the scaling Position field values ​​(three values) of each sub-body to Tx, Ty, and Tz respectively and giving the matrix form.

[0139] (2) The scaling matrix is:

[0140] The scaling matrix is ​​obtained by assigning the scaling field values ​​(three values) of each sub-entity to S1, S2, and S3 respectively and presenting the matrix form.

[0141] The rotation matrix along the single x-axis is:

[0142] The first value of the rotation field of each sub-body is taken as θ, and the matrix calculated by substituting it into the matrix is ​​the rotation matrix along the x-axis.

[0143] (3) The rotation matrix along the single y-axis is:

[0144] The second value of the rotation field of each sub-body is used as β. Substituting this value into the matrix yields the rotation matrix along the y-axis.

[0145] (4) The rotation matrix along the single z-axis is:

[0146] The third value of the rotation field of each sub-body is used as α. Substituting this value into the matrix yields the rotation matrix along the y-axis.

[0147] Multiplying the translation matrix, scaling matrix, and rotation matrix along the X, Y, and Z axes together yields the overall motion information matrix V1 of the subbody.

[0148] In a typical example, in the construction of digital twins for medical images, the virtual digital human body model in the virtual space only includes translation and rotation movements, and does not include scaling movements.

[0149] In medical imaging digital twins, the overall translational movement of the digital human body is mainly achieved through the movement of the human body itself, and the translational movement of the virtual human body on the examination bed as the bed rises, falls, and moves in and out of the bed. The movement of the examination bed can be achieved through motion control operations on virtual devices within the virtual scene.

[0150] Virtual human positioning and posture effects in CT / MRI digital twins, such as Figure 10 As shown.

[0151] In medical imaging digital twins, the overall rotational movement of the digital human body is mainly controlled by the positioning operation of CT and MR, including eight types: supine head-first (HFS), supine feet-first (FFS), prone head-first (HFP), prone feet-first (FFP), right lateral head-first (HFDR), right lateral foot-first (FFDR), left lateral head-first (HFDL), and left lateral foot-first (FFDL). For DR examinations, there are three basic positions: standing, lying, and sitting.

[0152] The internal sub-body of a virtual digital human body generally does not contain individual translational movements; all movements are formed through joint rotation.

[0153] The internal sub-body rotation movement of the virtual digital human body is driven by the three-dimensional restricted rotation movement (human operation) of each joint, which in turn drives the movement of each sub-body of the virtual digital human body.

[0154] Step 5: Perform matrix multiplication on each sub-matrix V of the digital object and the motion information matrix V1 to obtain the information matrix V' (V' = V × V1) of each sub-matrix of the digital object after motion, thus realizing the motion association between the virtual object and the digital object.

[0155] Typically, in the construction of a digital twin for medical imaging, the information matrix V of each sub-body of the physical digital human body is multiplied with the motion matrix V1 of the virtual digital human body to obtain the information matrix V' of each sub-body of the physical digital human body after motion. This V' is the physical information matrix after each sub-body of the digital human body has undergone motion corresponding to that of the virtual human body.

[0156] In a digital twin of medical images, the positions of the information matrices of each sub-matrix will not overlap during the actual image placement process. In very special cases, when the rotation of the sub-matrix results in matrix position overlap, the matrix information at the interface can be flexibly contracted.

[0157] Step 6: For the digital object information V' after motion in the virtual scene, select either the entire virtual space or a specific region (region of interest) Vs, and perform matrix dot product operation to obtain the digital object information V0 (V0 = V'.Vs) within the region of interest as the application information output. If the region of interest includes multiple sub-body parts, concatenate the V0 matrices of each sub-body part.

[0158] The region of interest can be the entire virtual 3D space or a specific region of interest.

[0159] Typically, in the construction of a digital twin of medical images, the selection method for the region of interest (ROI) can be to calculate the ROI information space matrix V based on the relevant parameters of medical image imaging set by the user in the virtual scene imaging operation (such as slice thickness, interslice spacing, number of slices, FOV, gantry tilt angle, arbitrary tomographic angle, etc.). s V s The center point is assumed to be the coordinates of the three-dimensional center of the rack cavity. V s The height and width are determined by the scan field of view (FOV), which are FOVx and FOVy in CT and MRI modes, respectively, and the opening and closing range of the beam in DR mode; V sThe length of the scan range is determined by three parameters in CT and MRI modes: slice thickness, interslice spacing, and number of slices (equal to (slice thickness + interslice spacing) multiplied by the number of slices). In DR mode, it is directly the physical digital body thickness. V s In terms of tilt angles in three directions, the CT mode is determined by the gantry tilt angle, the MRI scan mode is determined by the tilt angle of the imaging section, and the DR mode is determined by the tilt angle of the beam.

[0160] The region of interest matrix V s Performing a matrix dot product operation with V' yields V0, which is the physical digital human body information matrix of each sub-body within the region of interest in the virtual space after motion association. When there are physical information matrices of multiple sub-bodies within the region of interest matrix, the physical information matrices of each sub-body can be concatenated before performing the dot product operation.

[0161] The V0 matrix carries information about the body parts, position, and orientation of the virtual human body model, as well as FOV information. It serves as a virtual scanning object for the mechanism simulation model, performing simulated data acquisition and image reconstruction to complete the imaging simulation of the digital twin of medical images. Its simulation effect is related to the overall movement, joint motion, equipment movement, and FOV settings of the virtual human body model in the virtual scene.

[0162] CT localization imaging mechanism simulation fusion effect such as Figure 11 As shown, the simulation fusion effect of CT tomographic imaging mechanism is as follows: Figure 12 As shown, the fusion effect of MR arbitrary tomography and mechanism simulation is as follows: Figure 13 As shown.

[0163] Medical imaging digital twin systems can also be operated with a head-mounted display and remote control, enhancing the immersive experience, such as... Figure 14 As shown.

[0164] This invention establishes a method that uses rotational information obtained from arbitrary positioning and changes in body position of a virtual human body model as a bridge to perform the same coordinate mapping transformation on the internal physical digital human body matrix associated with the virtual human body model. This yields the physical digital human body matrix information within the scanning field of view, which is then used as the object of virtual scanning for mechanism simulation. This ensures that the parts, positions, and orientations of the virtual human body model in the virtual scene are completely consistent with those of the virtual scanned image. This method provides the foundation for fusing the external visual model and the internal mechanism model of medical image digital twins, is key to a highly realistic medical imaging system, and is an important component of the subsequent development of the Metaverse Hospital.

[0165] Digital twins for medical imaging can eliminate the need for expensive and bulky hardware and avoid radiation damage. They can be widely used in practical training and standardized training for medical imaging technicians, physicians, and engineers. They can also be used for virtual scanning of medical images and artificial intelligence sample augmentation of medical images.

[0166] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for motion correlation between virtual objects and digital objects in a virtual scene, characterized in that, Includes the following steps: S01: Construct virtual object models in a virtual scene; S02: Obtain the connection information of the constructed virtual object model, split the constructed virtual object model according to the connection information to obtain virtual object sub-models, and construct a restricted range of motion between the virtual object sub-models according to the connection information. S03: Obtain the feature information matrix of the virtual object and combine it with the three-dimensional spatial structure of the virtual object model to obtain the digital object model; obtain the digital object sub-model V based on the three-dimensional spatial structure of the virtual object sub-model. S04: Control the movement of virtual objects in the virtual scene and calculate the motion information matrix V1 of each virtual object sub-model; The calculated motion information matrix V1 of each sub-model of the virtual object includes: S41: Construct the geometric information attributes of each virtual object model; S42: The geometric information attribute is associated with the world coordinate system matrix sub-attribute; S43: Construct sub-attributes of the world coordinate system matrix, including the position P(x,y,z) in the world coordinate system, where the values ​​of x,y,z represent the translation values ​​in the X, Y, and Z directions, respectively; the scaling S(s1,s2,s3) in the world coordinate system, where the values ​​of s1,s2,s3 represent the scaling ratio values ​​in the X, Y, and Z directions, respectively; and the rotation R(r1,r2,r3) in the world coordinate system, where the values ​​of r1,r2,r3 represent the rotation angle values ​​in the X, Y, and Z directions, respectively. S44: Change the geometric information properties of each constructed virtual object model to calculate the translation matrix, scaling matrix, and rotation matrix; S45: Multiply the translation matrix, scaling matrix, and rotation matrix along the X, Y, Z axes to obtain the motion information matrix V1 of each virtual object sub-model; S05: Perform matrix multiplication on the digital object sub-model V and the motion information matrix V1 to obtain the moved digital object sub-model V'=V×V1, realizing the motion association between the virtual object and the digital object. When the rotation of the sub-model V forms the matrix position superposition, the matrix information at the interface is elastically contracted.

2. The method of claim 1, wherein, In step S02, the connection information of the virtual object model is obtained based on the real object. The connection information includes connection type, degree of freedom and degree of freedom range. The degree of freedom represents the movement in a certain cross-sectional direction, including the sagittal plane S, the coronal plane C and the transverse plane T.

3. The method of claim 1, wherein: The feature information matrix in step S03 includes physical information, chemical information, and physiological information.

4. The method of claim 1-3, wherein, Following step S05, the following is also included: S61: Obtain the spatial information matrix Vs of the region of interest of the virtual object; S62: Perform a matrix dot product operation between the obtained motiond digital object sub-model V' and the region of interest Vs to obtain the motiond digital object sub-model V0=V'·Vs within the region of interest; S63: If the region of interest includes multiple virtual object sub-models, then after performing matrix concatenation on each virtual object sub-model, calculate the digital object sub-model V0 after motion of the region of interest.

5. A medical image digital twin system, characterized by, include: The virtual scene module is used to construct virtual scenes. The virtual device module constructs virtual devices, including motion functionalization, interfaces, and motion control. Virtual human body module: Constructs a virtual human body, and enables the functionalization of movement through bone connections in the virtual human body; The virtual scanning simulation module obtains V0 from the motion association method between virtual objects and digital objects in the virtual scene described in claim 4. V0 includes the parts, positions, and orientation information of the virtual human body model, as well as FOV information. It serves as the virtual scanning object of the mechanism simulation model, performs simulated data acquisition and image reconstruction, completes the imaging simulation of the digital twin of medical images, and obtains virtual scanned medical images.

6. The medical image digital twin system of claim 5, wherein, The method for selecting the region of interest includes: Obtain the relevant parameters for medical image imaging set in the virtual scene imaging operation; The coordinates of the three-dimensional center of the frame cavity are taken as the center point of Vs; Determining V from scan field FOV s height and width; According to the scanning range, V s the length of the V Determination V s Tilt angles in three orientations.

7. The medical image digital twin system of claim 6, wherein, V s The height and width of V s The height and width of V s The length of V s The length of V s The inclination angle of V 8. A kinematic relationship system between a virtual object and a digital object in a virtual scene, characterized in that, include: The virtual object model building module constructs virtual object models in a virtual scene. The virtual object sub-model construction module obtains the connection information of the constructed virtual object model, splits the constructed virtual object model according to the connection information to obtain virtual object sub-models, and constructs a restricted range of movement between virtual object sub-models according to the connection information. The digital object sub-model construction module obtains the feature information matrix of the virtual object and combines it with the three-dimensional spatial structure of the virtual object model to obtain the digital object model; based on the three-dimensional spatial structure of the virtual object sub-model, the digital object sub-model V is obtained. The virtual object sub-model motion information matrix calculation module controls the movement of virtual objects in the virtual scene and calculates the motion information matrix V1 of each virtual object sub-model. The calculated motion information matrix V1 of each sub-model of the virtual object includes: S41: Construct the geometric information attributes of each virtual object model; S42: The geometric information attribute is associated with the world coordinate system matrix sub-attribute; S43: Construct sub-attributes of the world coordinate system matrix, including the position P(x,y,z) in the world coordinate system, where the values ​​of x,y,z represent the translation values ​​in the X, Y, and Z directions, respectively; the scaling S(s1,s2,s3) in the world coordinate system, where the values ​​of s1,s2,s3 represent the scaling ratio values ​​in the X, Y, and Z directions, respectively; and the rotation R(r1,r2,r3) in the world coordinate system, where the values ​​of r1,r2,r3 represent the rotation angle values ​​in the X, Y, and Z directions, respectively. S44: Change the geometric information properties of each constructed virtual object model to calculate the translation matrix, scaling matrix, and rotation matrix; S45: Multiply the translation matrix, scaling matrix, and rotation matrix along the X, Y, Z axes to obtain the motion information matrix V1 of each virtual object sub-model; The motion association module performs matrix multiplication on the digital object sub-model V and the motion information matrix V1 to obtain the moved digital object sub-model V'=V×V1, thus realizing the motion association between the virtual object and the digital object. When the rotation of the sub-model V forms the matrix position superposition, the matrix information at the interface is elastically contracted.

9. The motion linkage system between a virtual object and a digital object in a virtual scene according to claim 8, wherein, It also includes a region of interest information output module, including: S61: Obtain the spatial information matrix Vs of the region of interest of the virtual object; S62: Perform matrix dot product operation on the obtained digital object sub-volume model V' after movement and the region of interest Vs to obtain the digital object sub-volume model V0 in the region of interest after movement V0=V'·Vs; S63: If the region of interest includes multiple virtual object sub-volume models, perform matrix splicing on each virtual object sub-volume model, and then calculate the digital object sub-volume model V0 in the region of interest after movement.

Images