A method for eliminating visual discrepancies in the dynamic real-time perspective fusion of virtual characters and real-world scenes.

Abstract

This invention relates to the field of virtual scene development technology, and provides a method for eliminating visual discrepancies in the dynamic real-time perspective fusion of virtual characters and real-world scenes. The method includes the steps of image recognition of real-world road video footage, conversion into Unity world spatial coordinates, rotating the virtual camera in Unity by adjusting the horizontal angle of ∠AOB towards the target direction, and perspective fusion of the 3D virtual character with the real-world road video footage. This invention converts screen coordinates to Unity's world spatial coordinates, calculates the angle the virtual camera should rotate, and adjusts the rendering position of the 3D virtual character to the corresponding position in the real-world video footage by adjusting the virtual camera's angle value. This achieves the correlation and fitting between the originally unrelated real-world scene and the 3D virtual character, creating a seamless perspective fusion scene between the virtual character and the real-world road, eliminating application experience obstacles that angle discrepancies may cause.

Description

Technical Field

[0001] This invention relates to the field of virtual scene development technology, specifically to a method for eliminating visual discrepancies in the dynamic real-time perspective fusion of virtual characters and real-world scenes. Background Technology

[0002] The technology that integrates virtual character images with real-world road videos in terms of visual perspective involves shooting real-world road videos and collecting road data. The captured real-world road videos are then fitted with a virtual road route model constructed using the collected road data (latitude, longitude, elevation, and slope), providing users with an immersive interactive experience.

[0003] In actual development, a discrepancy frequently arises between the 3D virtual character and the actual video footage of the road. This leads to a visual perspective inconsistency between the 3D virtual character's position in the cycling software and the road video footage. This is especially noticeable in complex road scenarios such as curves, where the displayed position of the 3D virtual character deviates significantly from its expected position in the actual video footage, causing the 3D virtual character to move outside the captured road scene, or even "run out of the frame." This inconsistency disrupts the user's immersion and may reduce the playability and realism of the interactive experience. Summary of the Invention

[0004] To address the problems existing in the background technology, the present invention provides a method for eliminating visual deviations through dynamic real-time perspective fusion of virtual characters and real-world scenes, comprising the following steps:

[0005] S1: The real-shot road video footage is processed frame by frame using image recognition technology; the identified targets are the lane lines and road edges in the road footage, and the screen coordinates of a certain point on the lane line are obtained, denoted as PosScreenTarget coordinates;

[0006] S2: Render a 3D virtual character using the Unity engine's virtual camera, and convert the screen coordinates into spatial coordinates in the Unity world, which is the target position of the 3D virtual character;

[0007] S3: In the Unity world, the target position is denoted as A, the original position of the 3D virtual character is denoted as B, and the Unity camera position is denoted as O. The three positions form a triangle. Connecting OA and OB forms an included angle, denoted as ∠AOB. By modifying the horizontal angle of the virtual camera in Unity, the angle value of ∠AOB is rotated towards the target direction.

[0008] S4: Adjust the rendering position of the 3D virtual character to the PosScreenTarget coordinates to achieve perspective fusion between the 3D virtual character and the actual road scene, ensuring that the movement path of the 3D virtual character is perfectly matched with the actual shooting scene under any circumstances, eliminating visual deviation.

[0009] In the preferred embodiment, step S1 includes the following specific processes:

[0010] S11: Based on machine vision technology, lane lines are identified.

[0011] S12: Segment the identified lane lines and implement a function interface that can obtain the corresponding horizontal coordinate of the screen based on the given vertical coordinate of the screen. The interface is defined as: float GetXByY(float Y).

[0012] In the preferred embodiment, step S2 includes the following specific processes:

[0013] S21: Using the bottom of the 3D virtual character as its own coordinate point, obtain the Y-axis coordinate value of this coordinate point on the screen. 人S

[0014] S22: Y 人S Substituting into the `float GetXByY(float Y)` interface, we get X. 人S At this point, the screen coordinates (X) will be obtained. 人S Y 人S This refers to the target position on the screen where the 3D virtual character should be displayed;

[0015] S23: Transform the screen target position point into Unity space coordinates; the distance between the 3D virtual character and the virtual camera is denoted as L. Based on the value of L, adjust the screen coordinates (X... 人S Y 人S Perform a coordinate transformation to convert the coordinates into Unity3D world space coordinates, denoted as point A(X). 人U1 Y 人U1 Z 人U1 );

[0016] A(X 人U1 Y 人U1 Z 人U1 )=Camera.main.ScreenToWorldPoint(X 人S Y 人S L).

[0017] In the preferred embodiment, step S3 specifically includes:

[0018] S31: Let the world space coordinates of the 3D virtual character in Unity3D be (X...). 人U2 Y 人U2 Z 人U2 Let the world space coordinates of the virtual camera in Unity3D be (X). 相机 Y 相机 Z 相机 The coordinates of the two points mentioned above are directly obtained from the Unity3D engine; therefore, the point (X...) 人U1 Y 人U1 Z 人U1 ), (X 人U2 Y 人U2 Z 人U2 ), (X 相机 Y 相机 Z 相机 They form a triangle ▲AOB;

[0019] S32: Let θ be the size of ∠AOB of the above triangle. Then, θ is the angle by which the virtual camera in Unity needs to be rotated. The calculation process is as follows:

[0020] Calculate vectors OA and OB

[0021] Vector OA = [X 人U1 -X 相机 ,Y 人U1 -Y 相机 Z 人U1 -Z 相机 ];

[0022] Vector OB = [X 人U2 -X 相机 ,Y 人U2 -Y 相机 Z 人U2 -Z 相机 ];

[0023] Calculate the dot product and magnitude of these two vectors:

[0024] Dot product: OA·OB=(X 人U1 -X 相机 (X) 人U2 -X 相机 )+(Y 人U1 -Y 相机 (Y) 人U2 -Y 相机 )+(Z 人U1 -Z 相机 (Z) 人U2 -Z 相机 )

[0025] The magnitude of vector OA:

[0026] The magnitude of vector OB:

[0027]

[0028] In the preferred embodiment, the specific process of S4 includes:

[0029] S41: Rotate the virtual camera in Unity by an angle of θ, which will adjust the rendering position of the 3D virtual character from its previous position to the ideal target position.

[0030] S42: Fit all the GPS-recorded points with the images taken at those points to obtain a series of camera angle data. Substitute all the angle data into the camera at each point to eliminate the visual deviation caused by the angle deviation.

[0031] The beneficial effects achieved by this invention are as follows:

[0032] The captured road footage, produced by a camera, while giving the road a perspective and three-dimensional appearance, is technically still a video stream and lacks physical attributes like coordinates and depth found in 3D space. The virtual character, rendered by the Unity engine using a virtual camera, possesses physical attributes related to the world space coordinates, rotation angle, and scaling values ​​of a 3D game world. There are no direct correlations between the real-world footage and the 3D virtual character; that is, it's impossible to correlate lane lines and other elements in the real-world footage with the rendered position of the 3D virtual character.

[0033] This invention performs image recognition on lane lines in a real-world scene, calculating the screen coordinates of a point on the lane line. These screen coordinates are then converted to Unity's world space coordinates to calculate the required rotation angle of the virtual camera. By adjusting the virtual camera's angle, the rendering position of the 3D virtual character is aligned with the corresponding position in the real-world video frame. This achieves a seamless integration and fitting between the previously unrelated real-world scene and the 3D virtual character, creating a seamless perspective fusion between the virtual character and the real-world road, eliminating potential application experience obstacles caused by angle discrepancies. By fitting the relative position of the virtual character as it moves through the constructed virtual road information to the actual captured image, users can experience the virtual world more naturally in cycling apps, enhancing the realism, immersion, and entertainment value of the software experience. This innovative solution achieves the fusion of virtual and real-world elements, potentially bringing technological advancements to the development of cycling apps and VR games, thereby meeting users' demands for a higher level of virtual experience. Attached Figure Description

[0034] Figure 1 It is a comparison diagram showing the virtual character's correct position in the actual road video footage before and after the deviation was eliminated;

[0035] Figure 2 This is a diagram showing the position of a virtual character in a real-life road video before eliminating discrepancies.

[0036] Figure 3 It is the image before lane line recognition in step S1;

[0037] Figure 4 It is the image after lane line recognition in step S1;

[0038] Figure 5 It is a schematic image of the lane lines marked by orange circles in step S1;

[0039] Figure 6 The Y-axis coordinate value of the 3D virtual character on the screen in step S2 is Y. 人S Schematic diagram;

[0040] Figure 7 The target position (X) where the 3D virtual character should be displayed on the screen in step S2. 人S Y 人S ) Schematic diagram;

[0041] Figure 8 It is in step S3 (X) 人U1 Y 人U1 Z 人U1 A diagram showing the location of the object in Unity3D;

[0042] Figure 9 This is a schematic diagram of triangle AOB formed by the three points in step S3;

[0043] Figure 10 This is a schematic diagram of the image before the virtual camera rotates by an angle θ in step S4;

[0044] Figure 11 This is a schematic diagram of the image formed after the virtual camera rotates by an angle θ in step S4. Detailed Implementation

[0045] To facilitate understanding of the present invention by those skilled in the art, specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0046] By comparing the displayed position of the 3D virtual character on the screen with the actual captured road footage, the reasons for the discrepancy were analyzed, and the following conclusions were drawn:

[0047] Because the cameras used for shooting real-world road scenes have image stabilization, cameras without image stabilization cannot be used to capture shaky footage while the camera is in motion. The camera uses two gyroscope sensors to detect camera shake: one for translational shake detection and the other to compensate for angular tilt changes during telephoto shots. The gyroscopes can calculate the linear and angular velocities of image changes when shooting handheld. Due to image stabilization, there is a lag between the video footage and the actual real-time image from the user's location.

[0048] Because the GPS and vehicle attitude acquisition devices that collect road data are in motion when real-scene road video is being filmed, and the orientation of the attitude acquisition devices is constantly changing, attitude recognition is mainly achieved through gyroscopes. In the process of calculating the attitude azimuth angle, relevant inertial algorithms are also used for calculation and correction.

[0049] Because the camera's image stabilization algorithm and the attitude acquisition device's inertial algorithm exist, and these two algorithms belong to different manufacturers and functional categories, it can be determined that these two algorithms cannot be completely identical. Therefore, when at the same location, the angle of the road image captured by the camera will inevitably deviate from the azimuth angle value obtained by the attitude acquisition device. This deviation is the fundamental reason for the discrepancy between the 3D virtual character image and the actual video footage of the real road.

[0050] Based on the above analysis, this invention proposes a method for eliminating visual deviations in the dynamic real-time perspective fusion of virtual characters and real-world scenes, which includes the following steps:

[0051] S1: The actual road video footage is processed frame by frame using image recognition technology; the identified targets are the lane lines and road edges in the road image, and the screen coordinates of a certain point on the lane line are obtained, denoted as PosScreenTarget coordinates; the specific process of step S1 includes:

[0052] S11: Based on machine vision technology, lane lines are identified.

[0053] S12: Segment the identified lane lines and implement a function interface that can obtain the corresponding horizontal coordinate of the screen based on the given vertical coordinate of the screen. The interface is defined as: float GetXByY(float Y).

[0054] S2: Render a 3D virtual character using the Unity engine's virtual camera, and convert the screen coordinates into spatial coordinates in the Unity world, which is the target position of the 3D virtual character; the specific process of step S2 includes:

[0055] S21: Using the bottom of the 3D virtual character as its own coordinate point, obtain the Y-axis coordinate value of this coordinate point on the screen.人S

[0056] S22: Y 人S Substituting into the `float GetXByY(float Y)` interface, we get X. 人S At this point, the screen coordinates (X) will be obtained. 人S Y 人S This refers to the target position on the screen where the 3D virtual character should be displayed;

[0057] S23: Transform the screen target position point into Unity space coordinates; the distance between the 3D virtual character and the virtual camera is denoted as L. Based on the value of L, adjust the screen coordinates (X... 人S Y 人S Perform a coordinate transformation to convert the coordinates into Unity3D world space coordinates, denoted as point A(X). 人U1 Y 人U1 Z 人U1 );

[0058] A(X 人U1 Y 人U1 Z 人U1 )=Camera.main.ScreenToWorldPoint(X 人S Y 人S L).

[0059] S3: In the Unity world, the target position is denoted as A, the original position of the 3D virtual character is denoted as B, and the Unity camera position is denoted as O. The three positions form a triangle. Connecting OA and OB forms an included angle, denoted as ∠AOB. By modifying the horizontal angle of the virtual camera in Unity, the angle value of ∠AOB is rotated towards the target direction. The specific process of step S3 includes:

[0060] S31: Let the world space coordinates of the 3D virtual character in Unity3D be (X...). 人U2 Y 人U2 Z 人U2 Let the world space coordinates of the virtual camera in Unity3D be (X). 相机 Y 相机 Z 相机 The coordinates of the two points mentioned above are directly obtained from the Unity3D engine; therefore, the point (X...) 人U1 Y 人U1 Z 人U1 ), (X 人U2 Y 人U2 Z 人U2 ), (X 相机 Y 相机 Z 相机They form a triangle ▲AOB;

[0061] S32: Let θ be the size of ∠AOB of the above triangle. Then, θ is the angle by which the virtual camera in Unity needs to be rotated. The calculation process is as follows:

[0062] Calculate vectors OA and OB

[0063] Vector OA = [X 人U1 -X 相机 ,Y 人U1 -Y 相机 Z 人U1 -Z 相机 ];

[0064] Vector OB = [X 人U2 -X 相机 ,Y 人U2 -Y 相机 Z 人U2 -Z 相机 ];

[0065] Calculate the dot product and magnitude of these two vectors:

[0066] Dot product: OA·OB=(X 人U1 -X 相机 (X) 人U2 -X 相机 )+(Y 人U1 -Y 相机 (Y) 人U2 -Y 相机 )+(Z 人U1 -Z 相机 (Z) 人U2 -Z 相机 )

[0067] The magnitude of vector OA:

[0068] The magnitude of vector OB:

[0069]

[0070] S4: Adjust the rendering position of the 3D virtual character to the PosScreenTarget coordinates to achieve perspective fusion between the 3D virtual character and the actual road scene, ensuring that the movement path of the 3D virtual character is perfectly matched with the actual shooting scene under any circumstances, eliminating visual bias. Specifically, S41: Rotate the virtual camera in Unity by an angle of θ, which will adjust the rendering position of the 3D virtual character from its previous position to the ideal target position.

[0071] S42: Fit all the GPS-recorded points with the images taken at those points to obtain a series of camera angle data. Substitute all the angle data into the camera at each point to eliminate the visual deviation caused by the angle deviation.

[0072] The above embodiments of the present invention do not constitute a limitation on the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A visual deviation elimination method for dynamically and real-time perspective fusion of a virtual character and a real scene, characterized in that, It includes the following steps: S1: using image recognition technology to identify the real shot road video frame by frame; the target to be identified is the lane line and road edge of the road picture, and the screen coordinates of a certain position point of the lane line are obtained, which are recorded as PosScreenTarget coordinates; S2: render the 3D virtual character through the virtual camera of the unity engine, and convert the screen coordinates into the space coordinates of the unity world, i.e. the target position of the 3D virtual character; S3: in the unity world, the target position is recorded as A, the original position of the 3D virtual character is recorded as B, and the position of the unity camera is recorded as O. The three position points form a triangle, and the connecting lines OA and OB form an angle, which is recorded as ∠AOB. By modifying the horizontal angle of the virtual camera in unity, the angle value of ∠AOB is rotated in the target direction; S4: adjust the rendering position of the 3D virtual character to the PosScreenTarget coordinates, realize the perspective fusion of the 3D virtual character and the real shot road picture, and ensure that the motion path of the 3D virtual character completely fits the actual shot picture in any case, eliminating the visual deviation; The specific process of step S1 includes: S11: identify the lane line based on machine vision technology; S12: segment the identified lane line to realize that according to the given screen vertical coordinates, the function interface of the corresponding segmented screen horizontal coordinates can be obtained, and the interface is defined as: float GetXByY(float Y); The specific process of step S2 includes: S21: taking the bottom of the 3D virtual character as its own coordinate point, obtaining the Y-axis coordinate value Y of the coordinate point on the screen 人S S22: Y 人S is substituted into the float GetXByY(float Y) interface to obtain X 人S At this time, the screen coordinate point (X 人S , Y 人S ) is obtained, which is the target position at which the 3D virtual character should be presented on the screen; S23: Coordinate conversion is performed on the screen target position point to convert it into unity space coordinates; the distance between the 3D virtual character and the virtual camera is denoted as L, and based on the value of L, coordinate transformation is performed on the screen coordinate point (X 人S , Y 人S ) to convert it into the world space coordinates of Unity3D, denoted as point A (X 人U1 , Y 人U1 , Z 人U1 ); A (X 人U1 , Y 人U1 , Z 人U1 ) = Camera.main.ScreenToWorldPoint(X 人S , Y 人S , L).

2. The visual deviation elimination method for dynamically and real-timely perspective fusion of a virtual character and a real scene according to claim 1, characterized in that: The specific process of step S3 includes: S31: record the world space coordinates of the 3D virtual character in Unity3D as (X 人U2 , Y 人U2 , Z 人U2 ), and record the world space coordinates of the virtual camera in Unity3D as (X 相机 , Y 相机 , Z 相机 ), wherein the two coordinate values are directly obtained by the Unity3D engine; and points (X 人U1 , Y 人U1 , Z 人U1 ), (X 人U2 , Y 人U2 , Z 人U2 ) and (X 相机 , Y 相机 , Z 相机 ) form a triangle ▲AOB; S32: the size of ∠AOB of the above triangle is , then this is the angle that needs to be rotated for the virtual camera in unity, and the calculation process is as follows: Calculate vector OA and vector OB ； ； Calculate the dot product and the length of the two vectors: Dot product: The magnitude of the vector OA is: The magnitude of the vector OB is: 。 3. The visual deviation elimination method for dynamically and real-timely perspective fusion of a virtual character and a real scene according to claim 1, characterized in that: The specific process of S4 includes: S41: rotate the virtual camera in the unity, the rotation angle is The rendering position of the 3D virtual character will be adjusted from the previous position to the ideal target position. S42: fit all the GPS recorded points with the pictures taken at the points to obtain a series of camera rotation angle data, and substitute all the rotation angle data into the camera at each point to eliminate the visual deviation caused by the angle deviation.

Images