Digital twinning driven virtual and real scene synchronization method and system

Abstract

The invention provides a digital twinning driven virtual and real scene synchronization method and system, relates to the technical field of virtual reality, can realize digital twinning driven virtual and real scene synchronization, and can realize virtual and real scene synchronization by acquiring user position and head posture information in real time, dynamically calculating an analog value and adjusting interactive presentation. And the presentation proportion is adjusted according to the scene target parameter change, and scene rendering is optimized, so that high-precision and immersive virtual and real scene dynamic synchronization is finally realized, and the user interaction experience and the system operation efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a method and system for synchronizing virtual and real scenes driven by digital twins. Background Art

[0002] With the rapid development of digital technology, digital twin technology has broad application prospects in many fields. Take maritime scenario simulation as an example. Whether it is ship driving training, maritime emergency drills, or marine engineering planning, they all rely on the virtual scene to accurately replicate the real environment and operation process, and interact with it in real time.

[0003] However, traditional virtual scene simulation technology faces numerous challenges. For example, during maritime emergency drill simulations, the virtual scene struggles to reflect the impact of user posture changes in real time, leading to a disconnect between the drill data and actual conditions. Furthermore, information such as the user's position and posture within the virtual maritime scene cannot be correlated and reflected with elements such as the ship and equipment within the scene, resulting in reduced training quality.

[0004] Therefore, how to achieve dynamic synchronization of virtual and real scene data based on the user's posture has become an urgent problem that needs to be solved. Summary of the Invention

[0005] The present invention provides a digital twin-driven virtual-real scene synchronization method and system, which can realize dynamic synchronization of virtual-real scene data in combination with the user's posture.

[0006] A first aspect of the present invention provides a method for synchronizing virtual and real scenes driven by a digital twin, comprising: Obtaining the user's real-time location, and dynamically calculating the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target; Determining spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offsetting the simulated values ​​based on the adjustment parameters to obtain offset values ​​of each receiving unit; When the simulation target is greater than a reference constant, the simulation ratio is dynamically adjusted according to the real-time distance.

[0007] Optionally, in a possible implementation of the first aspect, obtaining the real-time location of the user, and dynamically calculating the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target, includes: When the real-time distance is less than the effective propagation distance of the simulated target, determining an attenuation coefficient corresponding to the real-time distance, and obtaining a simulated value at the real-time position according to the product of the initial volume of the simulated target and the attenuation coefficient; Each of the simulated targets is provided with a plurality of distance intervals, and each distance interval is provided with a corresponding attenuation coefficient.

[0008] Optionally, in a possible implementation of the first aspect, determining spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation based on the user's head posture information, and offsetting the simulated values ​​based on an adjustment parameter to obtain a bias value for each receiving unit includes: Determining a head orientation based on the head posture information, and determining a position of the user at each of the spatial orientations according to the head orientation, the spatial orientations including a left ear orientation and a right ear orientation; Obtaining an offset angle between each of the position points and the simulated target, and determining an adjustment parameter corresponding to each spatial orientation based on a comparison result of the offset angle difference and a reference angle difference threshold, wherein the spatial position parameter includes the offset angle; The bias value of each receiving unit is obtained according to the product of the adjustment parameter and the analog value.

[0009] Optionally, in a possible implementation of the first aspect, determining a head orientation based on the head posture information, and determining a position of the user in each of the spatial orientations according to the head orientation, where the spatial orientations include a left ear orientation and a right ear orientation, includes: Determine the head outline of the user in a top-down perspective based on the head posture information, and determine the direction from the center point of the head outline to the key part point as the head orientation; The left side of the head is determined to be the left ear position, the right side is determined to be the right ear position, and the center point of the ear contour is determined to be the position point of each of the spatial positions.

[0010] Optionally, in a possible implementation of the first aspect, in the process of determining the direction from the center point of the head contour to the key part point as the head orientation, the process further includes: If the key part is not identified, obtaining the user's real-time trajectory and determining the trajectory direction of the real-time trajectory; Obtaining a line connecting the ear contours, and generating a vertical line perpendicular to a center point of the line; Select candidate directions from the center point of the line to the endpoints on both sides of the vertical line, obtain the angle between each candidate direction and the trajectory direction, and determine the candidate direction with the smallest angle as the head direction.

[0011] Optionally, in a possible implementation of the first aspect, an offset angle between each of the position points and the simulated target is obtained, and an adjustment parameter corresponding to each spatial orientation is determined based on a comparison result of the offset angle difference and a reference angle difference threshold, where the spatial position parameter includes the offset angle, including: Determining a connection direction between each of the position points and the simulated target, and obtaining the offset angle according to an angle between each of the connection directions and a reference direction; When the offset angle difference is less than the reference angle difference, determining the adjustment parameter to be a reference constant; When the offset angle difference is greater than or equal to the reference angle difference, the adjustment parameters of each of the spatial orientations are determined according to the distance difference between each of the position points and the simulated target.

[0012] Optionally, in a possible implementation of the first aspect, when the offset angle difference is greater than or equal to the reference angle difference, determining the adjustment parameters of each spatial orientation through the following steps includes: Retrieving an increase coefficient table, and determining an offset coefficient of a spatial orientation corresponding to a smaller spacing according to a correspondence between each preset spacing difference and an offset coefficient in the increase coefficient table; Retrieving a reduction coefficient table, and determining the offset coefficient of the spatial orientation corresponding to the larger spacing according to the correspondence between each preset spacing difference and the offset coefficient in the reduction coefficient table; The adjustment parameter includes an offset coefficient, the offset coefficient in the increase coefficient table is greater than a reference constant, and the offset coefficient in the decrease coefficient table is less than the reference constant.

[0013] Optionally, in a possible implementation of the first aspect, when the simulated target is greater than a reference constant, dynamically adjusting the simulated amount ratio according to the real-time distance includes: Calculating the distance ratio of the reciprocal of the real-time distance of each simulated target to the sum of the total reciprocal; Obtain the reference ratio corresponding to each of the simulated targets at the real-time distance. For each of the simulated targets, multiply its reference ratio by the sum of the reference constant and the distance ratio difference between the simulated target and the remaining simulated targets to obtain the analog quantity ratio corresponding to the simulated target.

[0014] Optionally, in a possible implementation of the first aspect, the process of calculating the ratio of the real-time distance of each simulated target to the total real-time distance further includes: The simulated targets whose priority is less than the reference priority and whose real-time distance is greater than the scaled distance are removed.

[0015] A second aspect of the present invention provides a digital twin-driven virtual-real scene synchronization system, comprising: A positioning module is used to obtain the real-time location of the user and dynamically calculate the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target; An offset module is used to determine the spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offset the simulated values ​​based on the adjustment parameters to obtain the offset value of each receiving unit; The adjustment module is used to dynamically adjust the analog quantity ratio according to the real-time distance when the simulation target is greater than a reference constant.

[0016] The beneficial effects of the present invention are as follows: 1. The present invention can dynamically synchronize virtual and real scene data based on the user's position. Specifically, by acquiring the user's position in real time, combining the effective propagation distance of the simulated target, the distance interval, and the attenuation coefficient, and dynamically calculating the simulation value, the sound attenuation process with distance can be accurately simulated. At the same time, based on the user's head posture information, the spatial orientation and the simulated target position parameters are determined. By adjusting the parameter offset simulation value, the sound direction simulation based on the position of both ears can be realized, which improves the realism and immersion of the simulated scene.

[0017] 2. This invention uses a real-time distance reciprocal ratio combined with a baseline ratio and difference calculation to dynamically adjust the simulated target ratio, improving simulation accuracy. Furthermore, by setting a baseline priority and proportional adjustment distance, low-priority and distant simulated targets are automatically removed, improving operational efficiency while ensuring that trainees focus on key sound information, enhancing training effectiveness.

[0018] 3. The present invention uses image acquisition equipment to obtain head posture information, and provides a reliable basis for sound direction simulation through precise head orientation judgment and binaural position point determination methods. Whether through key points of the head contour or using real-time trajectory and ear contour information when key point recognition fails, the head orientation can be accurately determined. On this basis, the offset angle between the position point and the simulated target is calculated, and the adjustment parameters are flexibly determined based on the offset angle difference and the reference angle difference threshold to achieve precise offset of the left and right channel volume of the headphones. When the trainee turns his head to view different directions in the virtual scene, he can respond quickly and adjust the sound direction, so that sound perception is closely integrated with head movements, enhancing interactivity and allowing trainees to obtain a more natural and smooth training experience, which can improve the authenticity and practicality of VR simulation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a method for synchronizing virtual and real scenes driven by digital twins provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of a simulated target located directly in front of a user provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of a simulated target not being located directly in front of a user, provided by an embodiment of the present invention; Figure 4 It is a structural diagram of a digital twin-driven virtual-reality scene synchronization system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] See also Figure 1 , is a flow chart of a digital twin-driven virtual-real scene synchronization method provided by an embodiment of the present invention, Figure 1 The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, the user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDAs) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S1 to S3, as follows: S1, obtaining the real-time location of the user, and dynamically calculating the simulation value at the real-time location according to the real-time distance between the real-time location and each simulation target.

[0022] This embodiment can realize the synchronization of virtual and real scenes at sea driven by digital twins. By acquiring user position and head posture information in real time, dynamically calculating simulation values, adjusting interactive presentation, and adjusting the presentation ratio according to changes in scene target parameters, the scene rendering is optimized, and ultimately achieving high-precision, immersive dynamic synchronization of virtual and real scenes, thereby improving user interactive experience and system operation efficiency.

[0023] To achieve more realistic and accurate sound simulation effects in VR-based maritime sound simulation training, the system obtains the user's real-time position and head posture information, dynamically calculates the simulation value, and adjusts the simulation ratio. This allows the user to perceive different sound effects based on their position and posture, enhancing the immersion and interactivity of the simulation scene. Furthermore, the system can switch sound sources in real time based on the movement of people in the scene and environmental changes. For example, when a simulated ship encounters a sudden malfunction, the sound of normal navigation can be immediately switched to the sound of fault alarms and abnormal operation of the corresponding equipment. The sound volume and multi-source ratio can also be adjusted synchronously in real time, improving the authenticity and effectiveness of VR maritime simulation training.

[0024] In practical applications, the user's real-time location can be tracked in real time by high-precision positioning systems integrated into VR devices, such as optical and inertial positioning technologies. This represents the user's precise position within the virtual maritime scene's coordinate system, expressed as three-dimensional spatial coordinates (x, y, z). Real-time distance can be calculated using mathematical distance calculation formulas, such as the distance formula between two points in three-dimensional space, based on the user's real-time location coordinates and the preset coordinates of each simulated target.

[0025] The simulated target is the source of various sounds in the maritime simulation scene. Each simulated target has unique acoustic characteristics, such as the stable low-frequency characteristics of the engine running sound and the sharp high-frequency and periodic characteristics of the alarm sound. Different simulated targets have different setting parameters such as setting position and setting volume.

[0026] The simulated value is the volume of a specific simulated target perceived at the user's current location, calculated based on real-time distance. It's understandable that as users move through the virtual scene, the perceived volume of sound changes with distance. For example, when approaching simulated targets such as engines and equipment, the sound increases significantly; as they move further away, the sound gradually decreases. Therefore, to conform to the laws of real-world sound propagation, greatly enhance the realism of the simulated scene, improve the user's immersive experience, and help users better integrate into the virtual offshore operating environment, the perceived volume data can be adjusted in real time based on the user's movement.

[0027] In some embodiments, the simulation value at the real-time position can be dynamically calculated by the following steps: When the real-time distance is less than the effective propagation distance of the simulated target, an attenuation coefficient corresponding to the real-time distance is determined, and a simulated value at the real-time position is obtained according to the product of the initial volume of the simulated target and the attenuation coefficient.

[0028] Each of the simulated targets is provided with a plurality of distance intervals, and each distance interval is provided with a corresponding attenuation coefficient.

[0029] It's understandable that based on the real-time distance between the user's real-time location and the simulated target, the attenuation of sound during propagation can be accurately simulated, thereby constructing a highly realistic virtual sound environment. By setting multiple distance intervals and corresponding attenuation coefficients, the degree of sound attenuation at different distances can be more detailed and accurately reflected, so that when trainees move in the virtual scene, the changes in sound volume they perceive are consistent with real-world laws. This not only greatly enhances the immersiveness of simulation training, but also allows trainees to accumulate real sound perception experience in a virtual environment, improving their ability to judge and process sound information in actual maritime operations.

[0030] The effective propagation distance of a simulated target is the maximum distance over which each simulated target can effectively propagate and be perceived by the user in the virtual scene. Beyond this distance, the sound is attenuated to the point where it is barely perceptible to the user. The attenuation coefficient quantifies the proportionality of the sound volume attenuation as distance increases during propagation. Different simulated targets and different distance intervals correspond to different attenuation coefficients. Their values ​​reflect the rate of sound attenuation within a specific distance interval and are key parameters for calculating the simulated value at the real-time location. The initial volume is the base volume level determined during the configuration of each simulated target, representing the volume at the simulated target's starting position (distance 0). The distance interval divides the effective propagation distance of a simulated target into multiple continuous distance ranges, each of which is a distance interval. For example, different distance intervals can be set for 0-10 meters, 10-20 meters, and so on, with each interval corresponding to a specific attenuation coefficient to achieve refined simulation of sound attenuation.

[0031] By calculating simulated values ​​based on real-time distance and the corresponding attenuation coefficient, trainees can realistically experience the sound volume changing with distance as they move through the virtual scene. The volume increases as they approach the simulated target and decreases as they move further away. This realistic sound change significantly enhances the realism of VR simulation training, immersing trainees in a realistic ship operating environment. Furthermore, during training, changes in sound volume serve as important clues for trainees to determine their position and surroundings. For example, hearing the engine sound gradually increase in volume can alert trainees to their approach, assisting them with spatial positioning and operational decision-making within the virtual scene, enhancing the practicality and effectiveness of training.

[0032] S2, determining the spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offsetting the simulated values ​​based on the adjustment parameters to obtain the offset values ​​of each receiving unit.

[0033] Head posture information refers to data captured by an image capture device (such as a camera) within the VR simulation training space, capturing the user's head orientation. Spatial orientation is defined relative to the user's ears and is categorized as left-ear orientation or right-ear orientation. The receiving unit, the headphones in the VR device, precisely adjusts the volume of the left and right channels to simulate the effect of sound coming from different directions, allowing the user to perceive the spatial orientation of the sound. For example, when the simulated target is in front of the user's right, the volume of the right channel of the headphones will increase, giving the user the illusion that the sound is coming from the front right.

[0034] Understandably, in real life, humans rely on the binaural effect to determine the direction of sound. When a simulated target isn't directly in front of them, there are subtle differences in the volume of the sound received by each ear. For example, if a sound comes from the right, the right ear will receive a louder sound than the left. Therefore, in VR simulation training, by combining head orientation to offset the simulated values ​​of the left and right earphone channels, this real-life auditory experience can be accurately simulated. For example, in a ship's cockpit scene, if an alarm sounds from the left rear, the system determines the left and right ear positions and the spatial position parameters of the simulated target based on head orientation, and adjusts the volume of the left and right earphone channels, with the left ear volume increasing relatively. This allows the user to accurately perceive the alarm as coming from the left rear, greatly enhancing the sense of immersion and making it feel like they are in a real ship environment.

[0035] The spatial position parameter is used to quantitatively describe the spatial position of the simulated target relative to the headphone, such as the angle between the simulated target and the left and right channels of the headphone. The adjustment parameter is the spatial position parameter of the simulated target and the headphone, which determines the coefficient for offsetting the simulated value. The offset value is the volume value received by the left and right ear channels of the headphone after applying the adjustment parameter to the initial simulated value. The difference in offset value simulates the difference in volume received by the left and right channels of the headphone when the sound comes from different directions, helping users to accurately perceive the direction of the sound.

[0036] Based on the above embodiment, the specific implementation of step S2 may be: S21, determining a head orientation based on the head posture information, and determining a position point of the user in each of the spatial orientations according to the head orientation, where the spatial orientations include a left ear orientation and a right ear orientation.

[0037] In practice, humans perceive sound direction by turning their heads, and different head orientations affect the volume of sound received by both ears. Based on the user's head posture information, the head orientation is accurately determined, and thus the location of both ears in virtual space. This provides a precise spatial basis for simulating sound direction, ensuring that the sound heard by trainees matches their real-world auditory perception when turning their heads, enhancing the immersion and interactivity of simulation training.

[0038] Head orientation refers to the positive direction of the user's head, derived from head posture analysis and used to determine the user's viewing direction in virtual space. Location points refer to the coordinates of the left and right ears in virtual space.

[0039] Specifically, in a VR-simulated ship cockpit scenario, users wear VR equipment and headphones for training. Image acquisition equipment captures images of the user's head and, through image recognition technology, determines the user's head orientation. Based on this orientation, the left and right ear positions are determined in virtual space.

[0040] Accurately determining head orientation and binaural position points lays the foundation for subsequent simulation of sound direction. When the trainee turns their head in the virtual scene, the binaural position points can be re-determined based on the new head orientation, thereby accurately simulating the sound direction and enhancing the realism of the training. For example, when the trainee turns their head to look behind them, the system determines the change in binaural position points based on the new head orientation and adjusts the sound output in the left and right channels of the headphones, making the trainee feel that the direction of the sound has also changed.

[0041] In some embodiments, the position points of each spatial orientation may be determined by the following steps: Based on the head posture information, the head outline of the user in a bird's-eye view is determined, and the direction from the center point of the head outline to the key part is determined as the head orientation; the left side of the head orientation is determined to be the left ear orientation, and the right side is determined to be the right ear orientation, and the center point of the ear outline is determined to be the position point of each of the spatial orientations.

[0042] It's understandable that the reason spatial orientation is determined by the positive direction is that it establishes a clear spatial reference for determining left and right ear orientation. In a virtual environment, if the positive direction is unclear, left and right ear orientation will lose reference and become confused. Only by determining the positive direction can it be used as a basis to clearly define the left ear orientation (to the left) and the right ear orientation (to the right). For example, in a ship cockpit simulation, if the trainee's head is facing forward, the port side can be accurately identified as the left ear orientation and the starboard side as the right ear orientation, thus providing a basis for differentiated sound simulation in the left and right ears. Furthermore, in a virtual environment, without a clear positive direction as a reference, relying solely on other information to determine left and right ear orientation can easily lead to confusion. For example, if a trainee is in a complex cabin environment with a constantly changing head posture, without a positive direction as a reference, the left ear orientation may be mistakenly identified as the right ear orientation, resulting in the subsequent sound output in the headphones being misdirected. After determining the correct direction, the left ear and right ear positions can be clearly and accurately distinguished, thereby preventing sound adjustment errors caused by incorrect direction judgment and ensuring the correctness of sound direction simulation.

[0043] Moreover, the adjustment of sound based on head posture, such as calculating the offset angle between the simulated target and the two ears, and determining the adjustment parameters, all rely on the accurate orientation of the left and right ears. Once the left and right ears are misjudged, the calculation of these key parameters will deviate, making the volume adjustment of the left and right channels of the headphones inconsistent with the actual situation. For example, when an alarm sound comes from the left, if the left and right ears are misaligned, the volume of the left ear channel, which should have been increased, may be reduced, while the volume of the right ear channel may be mistakenly increased, causing the trainee to be unable to correctly judge the direction of the alarm sound. The accurate positive direction can ensure the precise determination of the left and right ear orientations, providing a reliable premise for subsequent sound adjustments, so that the sound can be accurately presented in the headphones according to the real direction and distance, thereby improving the authenticity and effectiveness of the training.

[0044] Key points are representative features on the head outline, such as the nose tip, and are used to assist in determining the head's positive orientation. After obtaining the head outline, its center point is connected to key points, such as the nose tip, to determine the direction from the center point to the nose tip as the positive direction.

[0045] To extract the head outline, the captured image can be preprocessed with grayscale conversion and Gaussian filtering to remove noise. Edge detection is performed using the Canny algorithm to obtain edge information of the head. The OpenCV library's findContours function is then used to find contours and filter the head outline based on area thresholds and shape features. To identify key points, a deep learning object detection algorithm can be used to identify key points such as the tip of the nose. When determining the location points, the head orientation can be used as a reference, with the left side representing the left ear and the right side representing the right ear. The ear outline is identified in the overhead image, and its center point is calculated, which serves as the location point for the left and right ears in virtual space.

[0046] Through overhead image acquisition and head contour analysis, the user's head orientation can be accurately determined, providing more reliable basic data for sound direction simulation. The left and right ear positions are determined based on head orientation, and the spatial position point is determined by the center point of the ear contour. This makes subsequent calculations of parameters such as the offset angle between the simulated target and the ears more accurate, thereby achieving more precise sound direction simulation.

[0047] In addition, in the process of determining the direction from the center point of the head contour to the key part point as the head orientation, this solution also includes the following embodiments: If the key part point is not identified, the user's real-time trajectory is obtained and the trajectory direction of the real-time trajectory is determined; the connecting line of the ear contour is obtained, and a vertical line perpendicular to the center point of the connecting line is generated; candidate directions from the center point of the connecting line to the endpoints on both sides of the vertical line are selected, and the angle between each candidate direction and the trajectory direction is obtained, and the candidate direction with the smaller angle is determined as the head orientation.

[0048] It is understandable that during the training process, due to factors such as occlusion and light changes, it may not be possible to successfully identify key points on the head contour, and thus it is impossible to determine the head orientation through key points. Accurate head orientation is an important basis for simulating the direction of sound and directly affects the trainer's judgment of the sound source. Therefore, when key point recognition fails, the user's real-time motion trajectory can be obtained, combined with ear contour information, to infer the head orientation, ensuring that accurate head posture data can be provided for sound simulation in various complex situations, maintaining the authenticity and effectiveness of simulation training, allowing trainees to obtain reliable sound direction perception in virtual scenes, and improving the training experience.

[0049] The real-time trajectory is the path formed by the user's continuous position changes over a period of time in a VR simulation scene, recording the user's movement process and direction information. The trajectory direction is the extension direction of the user's real-time trajectory, reflecting the user's movement trend in the simulation scene. The ear contour line is the line segment connecting the center points of the left ear contour and the right ear contour. The vertical line is a line perpendicular to the center points of the ear contour line. The relationship between this vertical line and the ear contour line is used to determine the candidate direction of the head orientation. The candidate directions are two directions starting from the center point of the ear contour line to the endpoints on both sides of the vertical line. The final head orientation is determined by comparing the angle between these two directions and the trajectory direction.

[0050] It's understandable that during actual movement, people typically face in the direction of motion, and the direction of the motion trajectory is highly consistent with the head orientation. Selecting a candidate direction with a small angle to the trajectory direction as the head orientation aligns with the inertial characteristics and visual habits of the human body during movement. In a VR maritime simulation of a ship deck training scenario, when a trainee moves along a certain direction on the deck, their head is likely to face in the direction of motion. In this case, a candidate direction with a small angle better matches the actual head orientation, more realistically simulating the trainee's movement in the scenario.

[0051] S22, obtaining the offset angle between each of the position points and the simulated target, and determining the adjustment parameters corresponding to each spatial orientation based on the comparison result of the offset angle difference and the reference angle difference threshold, wherein the spatial position parameters include the offset angle.

[0052] The angle of the simulated target relative to the user's ears can cause differences in the volume of sound received by each ear. By calculating the offset angle between the location point and the simulated target and comparing the offset angle difference with a baseline angle difference threshold to determine adjustment parameters, we can simulate the sound changes caused by this angle difference. This allows trainees to hear sounds through headphones that conform to real auditory logic and more accurately determine the direction of sounds.

[0053] The offset angle is the angle formed by the line between the position point of the left ear orientation or the right ear orientation and the simulated target, and the reference direction, which is used to quantify the angular position of the simulated target relative to the two ears. The offset angle difference is the difference between the offset angle between the left ear orientation and the simulated target and the offset angle between the right ear orientation and the simulated target, reflecting the degree of angular difference between the simulated target in the left and right ear directions. The reference angle difference threshold is a pre-set angle difference standard used to judge the size of the offset angle difference. The adjustment parameter is a coefficient determined based on the comparison result of the offset angle difference and the reference angle difference threshold, which is used to adjust the simulation value to achieve volume adjustment of the sound between the two ears.

[0054] In some embodiments, step S22 may be implemented by the following steps: Determine the connection direction between each of the position points and the simulated target, and obtain the offset angle based on the angle between each of the connection directions and the reference direction; when the offset angle difference is less than the reference angle difference, determine the adjustment parameter as a reference constant; when the offset angle difference is greater than or equal to the reference angle difference, determine the adjustment parameter of each of the spatial orientations based on the distance difference between each of the position points and the simulated target.

[0055] See also Figure 2 , a schematic diagram of a simulated target located directly in front of a user, provided by an embodiment of the present invention. As can be seen from the figure, when the simulated target is located near the front, the difference in offset angles between the left and right ears is small. In this case, the simulated target is relatively symmetrical relative to both ears, and the path lengths of sound propagation to both ears are nearly equal. With the same sound propagation path length, the difference in time and volume between the two ears receiving the sound is minimal, so the simulated value can be left unchanged and the adjustment parameter can be set to 1.

[0056] See also Figure 3 The following diagram illustrates an embodiment of the present invention, showing a simulated target that is not directly in front of the user. As can be seen from the figure, when the simulated target is positioned to the side, the difference in offset angles from the left and right ears increases significantly. Due to the different distances of the simulated target from the two ears, the path lengths required for sound to reach the two ears differ significantly. The ear closer to the simulated target receives the sound first and at a higher volume. For example, on a ship's deck, when a side alarm sounds, the ear closer to the alarm will hear it earlier and at a higher volume than the ear on the other side, allowing trainees to quickly identify the sound's lateral origin. In VR simulation, when the offset angle difference is greater than or equal to the baseline angle difference, an adjustment parameter is determined based on the distance difference between the location point and the simulated target, resulting in a significant difference in volume between the left and right earphone channels. This simulates the sound changes caused by the lateral angle difference, allowing trainees to accurately determine the direction of the sound source in a virtual environment based on the sound differences, enhancing the realism and practicality of the training.

[0057] The reference direction is a pre-set reference direction, typically set directly in front of the simulated target, used to calculate the offset angle. The reference angle difference is a pre-set angle difference standard used as the basis for determining the adjustment parameter.

[0058] In some embodiments, when the offset angle difference is greater than or equal to the reference angle difference, the adjustment parameters of each spatial orientation can be determined by the following steps: Recall the increase coefficient table, and determine the offset coefficient of the spatial orientation corresponding to the small spacing according to the correspondence between each preset spacing difference and the offset coefficient in the increase coefficient table; call the reduction coefficient table, and determine the offset coefficient of the spatial orientation corresponding to the large spacing according to the correspondence between each preset spacing difference and the offset coefficient in the reduction coefficient table; the adjustment parameter includes the offset coefficient, the offset coefficient in the increase coefficient table is greater than the reference constant, and the offset coefficient in the reduction coefficient table is less than the reference constant.

[0059] When the simulated target is positioned to the side, meaning the offset angle difference is greater than or equal to the baseline angle difference, fine-tuning the volume of the left and right earphone channels is necessary to more accurately simulate the differences in human ear perception of sounds from different directions. The distance between the simulated target and the ears is a key factor influencing these differences. By accessing the increase and decrease coefficient tables and determining the offset coefficient based on the distance difference between the simulated target and the ears, the changes in sound volume received by the ears at different distances can be simulated, making the adjustment parameters more consistent with the actual auditory experience. Ears closer to the simulated target should receive a higher volume, while ears farther away should receive a lower volume. This restores realistic sound direction perception in the virtual scene, improves the trainee's accuracy in determining sound direction, and enhances the immersion and practicality of VR training.

[0060] The preset distance difference is a numerical range set in the increase and decrease coefficient tables to measure the distance difference between the simulated target and the ear. It divides the distance difference between the simulated target and the ear into different levels, each corresponding to a specific offset coefficient.

[0061] The amplification factor table is a pre-compiled table that records the correspondence between preset distance differences and offset factors. The offset factors in the table are all greater than 1 and are suitable for situations where the simulated target is relatively close to one ear. When the distance between the simulated target and one ear is determined to fall within a preset distance difference range in the amplification factor table, the corresponding offset factor is used to increase the volume of the headphone channel on that side.

[0062] The reduction coefficient table is also a pre-set table that maps preset distance differences to offset factors. Unlike the increase coefficient table, the offset factors are all less than 1, making it suitable for situations where the simulated target is farther from one ear. When the distance between the simulated target and one ear falls within the preset distance difference range in the reduction coefficient table, the corresponding offset factor is obtained and used to reduce the volume of the headphone channel on that side.

[0063] Smaller spacing values ​​indicate the distance to the ear that is closer to the simulated target. This corresponds to the increase coefficient table. Larger spacing values ​​indicate the distance to the ear that is farther from the simulated target. This corresponds to the decrease coefficient table.

[0064] S23, obtaining the offset value of each receiving unit according to the product of the adjustment parameter and the analog value.

[0065] After determining the adjustment parameters and simulation values, the bias value of the receiving unit is obtained by multiplying the two. The previously calculated sound direction-related parameters can be converted into the actual left and right channel volume output of the headphones, allowing trainees to hear sounds that conform to the sound direction simulation through the headphones, ultimately achieving accurate sound direction simulation effects and enhancing the immersion and authenticity of VR simulation training.

[0066] S3: When the simulation target is greater than a reference constant, dynamically adjust the simulation ratio according to the real-time distance.

[0067] It is understood that when the intensity of a specific simulated target exceeds a baseline constant, the simulated amount ratio is dynamically adjusted. This ensures that key sound information is clearly captured by the user in a complex multi-source sound environment, comprehensively enhancing the realism and interactive experience of simulation training and helping users more effectively adapt to the complex acoustic environment of offshore operations. The baseline constant is 1.

[0068] The analog scale determines the contribution of each simulated target's volume to the overall sound mix at that location. This is intended to reflect the relative contribution of different simulated targets to the overall sound mix at a specific location.

[0069] Based on the above embodiment, the specific implementation of step S3 may be: Calculate the distance ratio of the reciprocal of the real-time distance of each simulated target and the sum of the total reciprocals; obtain the benchmark ratio corresponding to each simulated target at the real-time distance, and for each simulated target, multiply its benchmark ratio by the sum of the difference between the benchmark constant and the distance ratio of the simulated target and the remaining simulated targets to obtain the analog quantity ratio corresponding to the simulated target.

[0070] It is understandable that when multiple simulated targets exist at the same time and the distance between the trainee and each simulated target is constantly changing, the volume ratio of different simulated targets can be dynamically adjusted according to their relative distance to the trainee. By calculating the distance ratio of the simulated target and adjusting the simulated volume ratio based on the baseline ratio and the distance ratio difference, it is possible to simulate the intensity changes caused by distance during sound propagation, as well as the human ear's perception of simulated targets at different distances. For example, when the trainee is close to the engine, the engine sound should be more prominent; and when close to the waves, the proportion of the wave sound needs to be increased accordingly. In this way, the simulated sound environment is closer to reality, helping trainees to obtain a more realistic auditory experience in virtual scenes, improving their ability to judge and process sound information, and enhancing the practicality and immersion of training.

[0071] Among them, the sum of the total reciprocals is the sum obtained by adding the reciprocals of the real-time distances of all simulated targets. The distance ratio is the ratio of the reciprocal of the real-time distance of each simulated target to the sum of the total reciprocals, reflecting the relative importance of the simulated target in the distance dimension. The larger the ratio, the closer the simulated target is to the trainee. The baseline ratio refers to the volume ratio parameter pre-set for each simulated target at different real-time distances. It reflects the analog quantity ratio that the simulated target should have at a specific distance when the influence of other simulated targets is not considered, and is the starting basis for adjusting the analog quantity ratio. The distance ratio difference is the difference between the distance ratio of a certain simulated target and the distance ratio of the remaining simulated targets. This difference can clearly reflect the relative distance relationship between the simulated targets, and then be used to reasonably adjust the analog quantity ratio to highlight the simulated targets that are closer to the trainee.

[0072] For example, if the real-time distance of the engine simulation target is 8 meters, the real-time distance of the wave simulation target is 12 meters, and the real-time distance of the alarm simulation target is 16 meters, the total reciprocal sum is 0.2705 meters. The engine simulation target distance ratio is 0.462, the wave simulation target distance ratio is 0.307, and the alarm simulation target distance ratio is 0.231. Given a base constant of 1, the base ratio of the engine simulation target at the current distance is 35%, the wave simulation target base ratio is 30%, and the alarm simulation target base ratio is 35%. For example, to calculate the engine simulation target analog ratio, we can use: = 35% * (1 + (0.462 - 0.307) + (0.462 - 0.231) = 48.51%. Similarly, the wave simulation ratio is 27.63%, and the alarm simulation ratio is 24.255%.

[0073] This approach accurately simulates the relationship between distance and intensity in sound propagation. As trainees approach the engine, the simulated volume ratio increases significantly, making the sound louder. As simulated targets move further away, the simulated volume ratio decreases accordingly, ensuring the simulated sound environment conforms to the real-world attenuation of sound with distance, significantly enhancing training realism. Furthermore, the simulated volume ratio adjustment, consistent with human ear perception, enhances the sense of layering. The simulated volume ratios for each simulated target are rationally distributed based on distance, preventing sound clutter. Trainees can clearly distinguish the direction and distance of different simulated targets, improving their ability to process sound information and making better decisions based on sound during training, thus enhancing the practicality of the training.

[0074] Furthermore, based on the above embodiment, this solution will also remove simulated targets whose priority is lower than the reference priority and whose real-time distance is greater than the proportional adjustment distance.

[0075] Understandably, complex VR maritime simulations involve a vast array of different simulated targets, such as engine roars, crashing waves, crew shouts, and equipment alarms. Not all simulated targets are equally valuable to trainees at all times. Some simulated targets have lower priority (such as the faint whistle of a distant ship). When they are farther away from trainees (beyond the scaled distance), their sound intensity is so weak in reality that they are barely perceptible. Retaining these simulated targets during the simulation and calculating and outputting their volume would not only increase the system's computational burden but also create audio clutter, interfering with trainees' ability to discern and interpret critical sounds (such as proximity alarms and operational commands). Therefore, by setting a baseline priority and scaled distance, and removing simulated targets that simultaneously meet a lower priority than the baseline and are farther away than the scaled distance, we can streamline the sound simulation calculation process, improve system efficiency, and create a clearer, more focused sound environment for trainees, while ensuring accurate transmission of core sound information. This enhances training effectiveness.

[0076] The priority of a simulated target refers to the importance index pre-set for various simulated targets in a VR simulation scene. A higher priority simulated target generally represents a greater importance during training, such as a device failure alarm having a higher priority than ordinary ambient sound effects. The baseline priority is a fixed priority value that serves as an important reference standard for determining whether a simulated target should be retained. When the priority of a simulated target is lower than this baseline value, the simulated target may be removed, depending on the real-time distance. The proportional adjustment distance is a pre-set distance threshold. When the real-time distance of a simulated target exceeds this threshold, the intensity of the sound in reality has been significantly attenuated. Combined with the simulation target priority judgment, if the priority is lower than the baseline priority, the simulated target will be removed.

[0077] See also Figure 4 , is a structural diagram of a digital twin-driven virtual-real scene synchronization system provided by an embodiment of the present invention, the digital twin-driven virtual-real scene synchronization system comprising: A positioning module is used to obtain the real-time location of the user and dynamically calculate the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target; An offset module is used to determine the spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offset the simulated values ​​based on the adjustment parameters to obtain the offset value of each receiving unit; The adjustment module is used to dynamically adjust the analog quantity ratio according to the real-time distance when the simulation target is greater than a reference constant.

[0078] Figure 4 The apparatus of the embodiment shown can be used to perform Figure 1 The implementation principles and technical effects of the steps in the method embodiment shown are similar and will not be repeated here.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital twin-driven virtual-real scene synchronization method, characterized in that: include: Obtaining the user's real-time location, and dynamically calculating the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target; Determining spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offsetting the simulated values ​​based on the adjustment parameters to obtain offset values ​​of each receiving unit; When the simulation target is greater than a reference constant, the simulation ratio is dynamically adjusted according to the real-time distance.

2. The method according to claim 1, characterized in that Obtaining the user's real-time location and dynamically calculating the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target includes: When the real-time distance is less than the effective propagation distance of the simulated target, determining an attenuation coefficient corresponding to the real-time distance, and obtaining a simulated value at the real-time position according to the product of the initial volume of the simulated target and the attenuation coefficient; Each of the simulated targets is provided with a plurality of distance intervals, and each distance interval is provided with a corresponding attenuation coefficient.

3. The method according to claim 1, characterized in that Determining spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offsetting the simulated values ​​based on the adjustment parameters to obtain offset values ​​of each receiving unit, including: Determining a head orientation based on the head posture information, and determining a position of the user at each of the spatial orientations according to the head orientation, the spatial orientations including a left ear orientation and a right ear orientation; Obtaining an offset angle between each of the position points and the simulated target, and determining an adjustment parameter corresponding to each spatial orientation based on a comparison result of the offset angle difference and a reference angle difference threshold, wherein the spatial position parameter includes the offset angle; The bias value of each receiving unit is obtained according to the product of the adjustment parameter and the analog value.

4. The method according to claim 3, characterized in that Determining a head orientation based on the head posture information, and determining a position of the user at each of the spatial orientations according to the head orientation, wherein the spatial orientations include a left ear orientation and a right ear orientation, comprising: Determine the head outline of the user in a top-down perspective based on the head posture information, and determine the direction from the center point of the head outline to the key part point as the head orientation; The left side of the head is determined to be the left ear position, the right side is determined to be the right ear position, and the center point of the ear contour is determined to be the position point of each of the spatial positions.

5. The method according to claim 4, characterized in that In the process of determining the direction from the center point of the head contour to the key part point as the head orientation, the process further includes: If the key part is not identified, obtaining the user's real-time trajectory and determining the trajectory direction of the real-time trajectory; Obtaining a line connecting the ear contours, and generating a vertical line perpendicular to a center point of the line; Select candidate directions from the center point of the line to the endpoints on both sides of the vertical line, obtain the angle between each candidate direction and the trajectory direction, and determine the candidate direction with the smallest angle as the head direction.

6. The method according to claim 3, characterized in that Obtain the offset angle between each position point and the simulated target, and determine the adjustment parameters corresponding to each spatial orientation based on the comparison result of the offset angle difference and the reference angle difference threshold, wherein the spatial position parameters include the offset angle, including: Determining a connection direction between each of the position points and the simulated target, and obtaining the offset angle according to an angle between each of the connection directions and a reference direction; When the offset angle difference is less than the reference angle difference, determining the adjustment parameter to be a reference constant; When the offset angle difference is greater than or equal to the reference angle difference, the adjustment parameters of each of the spatial orientations are determined according to the distance difference between each of the position points and the simulated target.

7. The method according to claim 6, characterized in that When the offset angle difference is greater than or equal to the reference angle difference, the adjustment parameters of each spatial orientation are determined by the following steps, including: Retrieving an increase coefficient table, and determining an offset coefficient of a spatial orientation corresponding to a smaller spacing according to a correspondence between each preset spacing difference and an offset coefficient in the increase coefficient table; Retrieving a reduction coefficient table, and determining the offset coefficient of the spatial orientation corresponding to the larger spacing according to the correspondence between each preset spacing difference and the offset coefficient in the reduction coefficient table; The adjustment parameter includes an offset coefficient, the offset coefficient in the increase coefficient table is greater than a reference constant, and the offset coefficient in the decrease coefficient table is less than the reference constant.

8. The method according to claim 1, characterized in that When the simulated target is greater than the reference constant, dynamically adjusting the analog quantity ratio according to the real-time distance includes: Calculating the distance ratio of the reciprocal of the real-time distance of each simulated target to the sum of the total reciprocal; Obtain the reference ratio corresponding to each of the simulated targets at the real-time distance. For each of the simulated targets, multiply its reference ratio by the sum of the reference constant and the distance ratio difference between the simulated target and the remaining simulated targets to obtain the analog quantity ratio corresponding to the simulated target.

9. The method according to claim 8, characterized in that The process of calculating the distance ratio of the real-time distance of each simulated target to the total real-time distance also includes: The simulated targets whose priority is less than the reference priority and whose real-time distance is greater than the scaled distance are removed.

10. A digital twin-driven virtual-real scene synchronization system, characterized in that: include: A positioning module is used to obtain the real-time location of the user and dynamically calculate the simulated value at the real-time location based on the real-time distance between the real-time location and each simulated target; An offset module is used to determine the spatial position parameters of the receiving units and the simulated target corresponding to each spatial orientation according to the user's head posture information, and offset the simulated values ​​based on the adjustment parameters to obtain the offset value of each receiving unit; The adjustment module is used to dynamically adjust the analog quantity ratio according to the real-time distance when the simulation target is greater than a reference constant.

Images