Virtual imaging method and system applied to virtual shooting, computing equipment and storage medium

Abstract

The invention provides a virtual imaging method and system applied to virtual shooting, computing equipment and a storage medium, and relates to the technical field of virtual imaging. The method comprises the following steps: synchronously acquiring object surface reflection characteristic data by using a multi-wavelength light source, and generating various imaging parameters through an image feature extraction network; the dynamic attitude data and binocular vision depth information are fused, and virtual and real scene dynamic space coordinate mapping is constructed; calculating the surface reflection intensity distribution of the virtual object in real time through a light propagation simulation module in combination with the specular reflectance and deformation correction parameters, and dynamically adjusting the relative position of the virtual object and the real light source; generating a continuous and smooth shadow boundary transition effect based on a material attenuation model; and performing space-time calibration on the shadow effect by using the dynamic attitude sequence to realize frame-level synchronous rendering of virtual and real light source positions and shadow changes. According to the technical scheme provided by the invention, through optical parameter modeling and dynamic path optimization, the problem of light and shadow distortion of the virtual object in a real scene is solved.

Description

Technical Field

[0001] The present application relates to the field of virtual imaging technology, and in particular to a virtual imaging method, system, computing device and storage medium applied to virtual photography. Background Art

[0002] In virtual shooting scenes such as film and television special effects and virtual live broadcasts, the realism of virtual-real fusion is highly dependent on the consistency of light and shadow between virtual objects and real scenes. With the increasing demand for dynamic shooting, existing technologies need to meet three core requirements: real-time capture of the characteristics of real scene light sources and dynamic changes in object surface reflections; high-precision matching of the deformation and attenuation of virtual object shadows and the spatial relationship with real environment light sources; and smooth transition of shadow boundaries under complex materials such as metal and transparent materials to avoid the sense of separation between virtual and real light and shadow.

[0003] The current mainstream solution is the static illumination mapping technology based on a monocular RGB camera. It uses a single white light source to illuminate the object, uses a monocular RGB camera to capture surface color information, and combines pre-trained illumination estimation models, such as the SH illumination coefficient prediction network, to infer the ambient light distribution. Then, an offline modeling tool is used to generate a static shadow map of the virtual object, and the virtual object is implanted into the scene based on the Euler angle coordinate system transformation. In dynamic scenes, the optical flow method is used to track the movement trajectory of the object, and the shadow position is adjusted through linear interpolation.

[0004] The solution is limited by the combination of a single white light source and a monocular RGB camera, and is unable to analyze multi-wavelength reflection characteristics, resulting in a mismatch between the highlight area of ​​the virtual object and the spectral characteristics of the real scene, and color distortion is prone to occur in metal or fabric materials. It relies on static shadow maps and optical flow methods to track dynamic postures. When the object moves quickly, the shadow offset error can reach 3-5 pixels due to model update delays, which requires manual correction frame by frame. In addition, the pre-trained lighting model only supports diffuse reflection approximation of basic materials, lacks modeling of subsurface scattering effects of translucent materials, and relies on Gaussian blur processing for shadow boundaries, resulting in a harsh split in the virtual-real transition area, greatly increasing the cost of subsequent repair. Summary of the invention

[0005] The present application provides a virtual imaging method, system, computing device and storage medium for virtual shooting, so as to solve the problems of delayed real-time matching of virtual and real light sources and abrupt transition of shadows of complex materials in the prior art.

[0006] In a first aspect, the present application provides a virtual imaging method for virtual shooting, comprising: Synchronously collecting and photographing reflection characteristic data of the surface of an object through a multi-wavelength light source, inputting the reflection characteristic data into a pre-trained image feature extraction network, and generating shadow feature parameters, wherein the shadow feature parameters include the specular reflectivity and diffuse reflectance coefficient of the surface of the object; Collect the dynamic pose data of the captured object, and fuse the dynamic pose data with the depth information obtained by the binocular vision positioning system to generate the spatial coordinate mapping relationship of the virtual-real scene; Input the specular reflectivity and the deformation correction parameters in the spatial coordinate mapping relationship into the light propagation path simulation module to calculate the reflection intensity distribution of each area on the surface of the virtual object, and at the same time, adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic pose data; Input the reflection intensity distribution and the adjusted relative position relationship into the light propagation model including the material attenuation characteristics to generate the shadow boundary transition effect with continuous deformation matching; Based on the dynamic pose change sequence in the spatial coordinate mapping relationship, align the shadow boundary transition effect with the light source position in the real scene in space-time to complete the light and shadow synchronous rendering of the virtual-real scene.

[0007] Optionally, input the specular reflectivity and the deformation correction parameters in the spatial coordinate mapping relationship into the light propagation path simulation module to calculate the reflection intensity distribution of each area on the surface of the virtual object, including: Split the specular reflectivity into multiple local reflectivity sets according to the spatial positions of the vertices on the surface of the virtual object, and correct the light propagation direction offset amount for each local reflectivity set according to the spatial deformation parameters corresponding to the vertices in the spatial coordinate mapping relationship; Based on the corrected light propagation direction offset amount, calculate the initial value of the specular reflection intensity of each vertex at the preset light incident angle, and perform direction continuity interpolation on the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship to generate the reflection intensity distribution of the continuous area on the surface of the virtual object.

[0008] Optionally, adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic pose data, including: Convert the displacement change amount in the dynamic pose data into the position change amount of the virtual object coordinate system relative to the real scene light source coordinate system; Taking the initial position of the light source in the real scene as the reference point, reversely superimpose the position change amount along the moving direction vector to generate the compensation position offset amount of the light source relative to the virtual object; According to the compensation position offset amount and the dynamic reference position of the vertices on the surface of the virtual object, calculate the real-time projection distance of the light source in the moving direction of the virtual object, and dynamically correct the irradiation angle of the light source in the virtual scene based on the projection distance; Superimpose the corrected irradiation angle after tilt correction and the compensation position offset to obtain the relative position update parameter between the light source and the virtual object, and perform incremental iteration on the relative position update parameter based on the update frequency of the dynamic attitude data to complete the adjustment of the relative position relationship between the light source and the virtual object.

[0009] Optionally, fuse the dynamic attitude data with the depth information obtained by the binocular vision positioning system to generate a mapping relationship between the virtual and real scene space coordinates, including: Establish a three-dimensional coordinate system with the centroid of the photographed object as the origin based on the baseline parameter of the binocular vision positioning system; convert the joint rotation angle and displacement in the dynamic attitude data into three-dimensional vectors in the three-dimensional coordinate system; Compound and superimpose the three-dimensional vectors with the three-dimensional coordinates of the joint feature points in the depth information obtained by the binocular vision positioning system to generate compound space data containing dynamic deformation parameters; Calculate the spatial offset of each vertex on the surface of the virtual object based on the deformation parameters in the compound space data; generate the movement trajectory of the vertices on the surface of the virtual object changing with the dynamic attitude based on the spatial offset and combined with the object contour boundary in the depth information obtained by the binocular vision positioning system; Establish a dynamic mapping relationship between the vertex coordinates of the virtual object and the real scene coordinates based on the movement trajectory.

[0010] Optionally, input the reflection intensity distribution and the adjusted relative position relationship into a light propagation model including material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching, including: Based on the initial specular reflection intensity value of each vertex on the surface of the virtual object in the reflection intensity distribution, combined with the real-time projection distance of the light source in the adjusted relative position relationship, calculate the direct light contribution value of each vertex under the illumination of the light source; According to the preset material absorption coefficient in the material attenuation characteristics, layer-by-layer attenuate the direct light contribution value along the light propagation path from the light source to the surface of the virtual object to obtain the attenuated light intensity; Based on the deformation compensation coefficient in the spatial coordinate mapping relationship, perform dynamic deformation matching correction on the light intensity so that the change in light intensity between adjacent vertices conforms to the continuous deformation trend of the surface of the virtual object; Superimpose the corrected light intensity with the diffuse reflection coefficient of the surface of the virtual object to generate the final reflection intensity of each vertex under the illumination of the real scene light source; Based on the spatial distribution of the final reflection intensity, calculate the gradient of the change in reflection intensity between adjacent vertices, and perform smooth transition interpolation on the boundary of the shadow area based on the change gradient to generate a shadow boundary transition effect with continuous deformation matching.

[0011] Optionally, based on the dynamic pose change sequence in the spatial coordinate mapping relationship, perform spatio-temporal alignment of the shadow boundary transition effect with the light source position in the real scene to complete the synchronous rendering of light and shadow in the virtual and real scenes, including: According to the amount of pose difference at consecutive time nodes in the dynamic pose change sequence in the spatial coordinate mapping relationship, calculate the movement direction and displacement speed of each vertex on the surface of the virtual object in the three-dimensional space coordinate system; Extract the movement trajectory of the light source in the real scene, and establish a direction difference parameter between the movement direction in the light source movement trajectory and the movement direction of the virtual object vertices; Match the light and dark change rate of each pixel area in the shadow boundary transition effect with the direction difference parameter to obtain the deformation amount of the shadow boundary caused by the movement of the light source; According to the time interval of the dynamic pose change sequence, perform sequential synchronous interpolation calculation on the displacement speed of the virtual object vertices and the light source movement trajectory to generate the correlation ratio of the shadow deformation amount with respect to the light source movement speed; Based on the correlation ratio, perform dynamic scaling adjustment on the light and dark distribution area of the shadow boundary transition effect, so that the bending degree of the virtual shadow boundary changes synchronously with the light source movement trajectory; superimpose and fuse the adjusted shadow boundary transition effect with the contour of the illumination area of the real scene light source, so that the deformation process of the virtual shadow boundary forms a dynamic following relationship with the movement process of the real light source.

[0012] Optionally, perform direction continuity interpolation on the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship to generate the reflection intensity distribution of the continuous area on the surface of the virtual object, including: According to the difference in spatial deformation parameters between adjacent vertices in the spatial coordinate mapping relationship, calculate the change amount of the deformation gradient in the direction of the vertex connection line; Based on the change amount of the deformation gradient, set multiple intermediate interpolation points on the connection line between adjacent vertices, and calculate the interpolation result of the spatial deformation parameter corresponding to each intermediate interpolation point according to the distance ratio between each intermediate interpolation point and the adjacent vertices; Perform weighted fusion of the interpolation result of the spatial deformation parameter and the initial value of the specular reflection intensity of the corresponding vertex to obtain the reflection intensity value of the intermediate interpolation point; According to the reflection intensity values of all intermediate interpolation points, construct the reflection intensity distribution of the continuous area on the surface of the virtual object.

[0013] In a second aspect, the present application provides a virtual imaging system for virtual shooting, including: An acquisition module, configured to synchronously acquire reflection characteristic data of the surface of a shooting object through a multi-wavelength light source, input the reflection characteristic data into a pre-trained image feature extraction network, and generate imaging feature parameters including the specular reflectivity and diffuse reflection coefficient of the object surface; A generation module, configured to construct a spatial attitude monitoring array based on multiple motion sensors, acquire dynamic attitude data of the shooting object in real time, and fuse the dynamic attitude data with depth information obtained through a binocular vision positioning system to generate a virtual-real scene spatial coordinate mapping relationship; A calculation module, configured to input the specular reflectivity and deformation correction parameters in the spatial coordinate mapping relationship into a light propagation path simulation module to calculate the reflection intensity distribution of each region on the surface of the virtual object, and at the same time, adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic attitude data; The generation module is further configured to input the reflection intensity distribution and the relative position relationship into a light propagation model including material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching, wherein the deformation correction parameters are synchronously updated through real-time attitude change data of the spatial attitude monitoring array and the shadow boundary transition effect; An alignment module, configured to perform spatio-temporal alignment of the shadow boundary transition effect and the light source position in the real scene based on the dynamic attitude change sequence in the spatial coordinate mapping relationship to complete the light and shadow synchronous rendering of the virtual-real scene.

[0014] In a third aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the virtual imaging method for virtual shooting as described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, and when the computer program is executed by a computer, it implements the virtual imaging method for virtual shooting as described in the first aspect.

[0016] In the embodiments of the present application, the reflection characteristic data of the object surface is synchronously collected by a multi-wavelength light source, and the specular reflectivity and diffuse reflection coefficient are accurately separated by combining an image feature extraction network, effectively solving the problem of spectral distortion of complex materials; the dynamic attitude data and binocular vision depth information are fused to construct a high-precision virtual-real space coordinate mapping relationship, significantly reducing the shadow offset error in dynamic scenes; based on the light propagation path simulation module, the reflection intensity distribution on the surface of the virtual object is calculated in real time, and a continuous and smooth shadow boundary transition effect is generated by combining the material attenuation model, eliminating the sense of fragmentation at the virtual-real fusion boundary; finally, through the spatio-temporal alignment algorithm of the dynamic attitude sequence, the real-time synchronization rendering of the virtual shadow and the real light source is realized, achieving a virtual-real light and shadow fusion effect at the film and television level.

[0017] Further, the specular reflectivity is split into a local reflectivity set according to the vertices on the surface of the virtual object, the light propagation direction offset is corrected by combining the spatial deformation parameters, and a continuous reflection intensity distribution is generated by interpolation calculation of the deformation parameters of adjacent vertices. Through vertex-level reflectivity splitting and deformation correction, the calculation accuracy of the light direction in the highlight area on the surface of the virtual object is improved, avoiding abnormal light spots caused by dynamic deformation of the curved surface; by combining interpolation of the deformation parameters of adjacent vertices, the continuity of the reflection intensity distribution of complex curved surfaces is ensured, eliminating the problem of blocky light spots under the traditional global model, and realizing high-fidelity light and shadow simulation of dynamic surfaces.

[0018] These aspects or other aspects of the present application will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 Shows a flowchart of a virtual imaging method applied to virtual shooting provided by the present application; Figure 2 Shows a schematic structural diagram of a virtual imaging system applied to virtual shooting provided by the present application; Figure 3 Shows a schematic structural diagram of a computing device provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] In order to enable those skilled in the art to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application.

[0022] In some of the processes described in the specification, claims, and the above-mentioned drawings of this application, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0023] Researchers have found that existing virtual shooting technologies have core problems such as inaccurate capture of multi-wavelength reflection characteristics, lag in real-time matching of virtual and real light sources, and abrupt transition of complex material shadows in dynamic light and shadow fusion. Based on this, this application provides a virtual imaging method applied to virtual shooting. This method realizes high-fidelity synchronization of the light and shadow deformation and attenuation effects of virtual objects in real scenes through multi-spectral reflection characteristic separation, dynamic space mapping, and material attenuation modeling. This solution is applicable to scenarios such as film and television special effects production, AR / VR real-time interactive live broadcast, etc.

[0024] The entire R & D process reflects the accurate capture of the surface reflection characteristics of objects through multi-wavelength light sources, the construction of a high-precision virtual and real space mapping by combining dynamic postures and binocular vision data, the real-time calculation of the reflection intensity distribution of virtual objects and the synchronization of light source positions, and the generation of natural shadow transitions based on the material attenuation model to achieve dynamic matching of virtual and real light and shadow.

[0025] Next, the technical solutions in the embodiments of this application will be clearly and completely described with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of this application.

[0026] Figure 1 The following is a flowchart of a virtual imaging method applied to virtual shooting provided for the embodiments of this application, as Figure 1 shown, this method includes: 101. Synchronously collect the reflection characteristic data of the surface of the shooting object through a multi-wavelength light source, input the reflection characteristic data into a pre-trained image feature extraction network to generate shadow formation feature parameters, and the shadow formation feature parameters include the specular reflectivity and diffuse reflection coefficient of the object surface; Multi-wavelength light source synchronous acquisition refers to the process of simultaneously irradiating the surface of an object with a light source array containing multiple color spectra, and recording the object's reflected light intensity data by switching different wavelength light sources in a time-sharing manner. The imaging feature parameters refer to the quantitative indicators of the object's surface optical properties extracted from the reflection data through algorithms, including specular reflectivity and diffuse reflection coefficient. Among them, the specular reflectivity describes the direct reflection ability of the object's surface to the light source, and the diffuse reflection coefficient describes the scattered reflection ability of the object's surface to the light.

[0027] In the embodiments of the present application, first, a programmable light source controller is used to drive a multi-wavelength LED light source array to cyclically switch the red, green, and blue light sources to irradiate the surface of the object according to a preset time sequence, and a high-frame-rate industrial camera is synchronously triggered to capture the reflection images of the object's surface at each wavelength, forming a multi-spectral reflection characteristic data set. Subsequently, the multi-spectral characteristic data is input into a pre-trained convolutional neural network CNN. The convolutional neural network extracts the reflection characteristics of different wavelengths through convolutional layers, and calculates the reflection characteristics through a fully connected layer to output imaging feature parameters including specular reflectivity and diffuse reflection coefficient.

[0028] In a virtual shooting scenario, when shooting a robot character made of a composite material of metal and plastic, a multi-wavelength light source array is used to emit light of different bands to irradiate the surface of the robot. The industrial camera synchronously captures the strong specular reflection data of the metal parts of the robot under blue light and the uniform diffuse reflection data of the plastic parts under red light, forming a multi-spectral image. The pre-trained convolutional neural network analyzes the multi-spectral image and outputs imaging feature parameters including high specular reflectivity of the metal area and low specular reflectivity parameters of the plastic area, providing basic data for subsequent light and shadow simulation.

[0029] 102. Collect the dynamic attitude data of the photographed object, and fuse the dynamic attitude data with the depth information obtained through the binocular vision positioning system to generate a spatial coordinate mapping relationship between the virtual and real scenes; The dynamic attitude data refers to the object's motion state information obtained in real time through sensors, including position, rotation angle, and motion trajectory. The spatial coordinate mapping relationship refers to a mathematical conversion model that precisely aligns the three-dimensional coordinate system of the virtual object with the real scene coordinate system, and is used to ensure the spatial position consistency of the virtual object in the real scene.

[0030] In the embodiments of the present application, inertial sensors are deployed on the surface of an object to collect the motion attitude data of the object in real time. At the same time, a binocular camera is used to capture a scene, and a stereo matching algorithm is used to fuse the dynamic attitude data with the depth information obtained by a binocular vision positioning system to generate a depth map, and the three-dimensional point cloud data of the depth map is extracted. The dynamic attitude data and the three-dimensional point cloud data are input into an extended Kalman filter, and after aligning the timestamps, they are fused into six-degree-of-freedom pose parameters. Finally, a transformation matrix between the virtual coordinate system and the real scene coordinate system is constructed to obtain the spatial coordinate mapping relationship between the virtual and real scenes.

[0031] Continuing with the above example, when the robot performs a rapid rotation action, the inertial sensors installed at the joints collect the attitude change data in real time, including the position offset and rotation angle of the robot. The binocular vision system synchronously captures the scene and generates a depth map. The dynamic attitude data is aligned with the three-dimensional point cloud data of the depth map through a fusion algorithm to construct an accurate spatial mapping relationship between the virtual model of the robot and the ground of the real scene, ensuring the spatial consistency between the virtual object and the real environment during the rotation process.

[0032] 103. Input the specular reflectivity and the deformation correction parameter in the spatial coordinate mapping relationship into the light propagation path simulation module to calculate the reflection intensity distribution of each region on the surface of the virtual object, and at the same time, adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic attitude data; The deformation correction parameter refers to the surface vertex displacement compensation value calculated according to the dynamic deformation of the object, which is used to correct the light propagation path. The reflection intensity distribution refers to the numerical distribution map of the reflected light intensity of each region on the surface of the virtual object at different light incident angles.

[0033] In the embodiments of the present application, the specular reflectivity is divided into multiple local sets according to the grid vertices on the surface of the virtual object, and the direction of the light propagation path of each vertex in the multiple local sets is offset and corrected in combination with the deformation correction parameter in the spatial coordinate mapping relationship. Based on the corrected direction of the light propagation path, the initial value of the specular reflection intensity of each vertex at a preset light source angle is calculated by using Monte Carlo integration, and a continuous surface reflection intensity distribution is generated through a bilinear interpolation algorithm. At the same time, according to the dynamic attitude data, the relative position relationship between the virtual object and the light source in the real scene is updated in real time by using a quaternion rotation matrix.

[0034] Continuing with the above example, when the robotic arm undergoes a bending deformation, the system calculates the displacement of the surface vertices based on the deformation correction parameters, and adjusts the propagation path direction of the light in the metal elbow area of the robot according to the displacement. The Monte Carlo integration algorithm calculates the specular reflection intensity of each surface vertex based on the corrected light propagation path direction, and generates a continuous surface reflection distribution map through an interpolation algorithm. At the same time, according to the real-time rotation data of the robot's head, the illumination angle of the virtual spotlight is dynamically adjusted to keep the light source direction synchronized with the robot's movement.

[0035] 104. Input the reflection intensity distribution and the adjusted relative position relationship into the light propagation model including the material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching. The material attenuation characteristic refers to the difference in the attenuation degree of light propagation by different materials. For example, the high reflection attenuation of metal and the low reflection attenuation of cloth. Continuous deformation matching refers to the characteristic that the shadow boundary smoothly transitions with the dynamic deformation of the object surface.

[0036] In the embodiment of the present application, the Fresnel equation and the Lambert cosine law are called based on the difference in material types to calculate the light attenuation coefficient. The reflection intensity distribution and the adjusted relative position relationship are input into the light propagation model with material attenuation characteristics, and the level set algorithm is used to generate the initial shadow boundary, and the sawtooth is eliminated through a Laplacian smoothing filter, and finally a shadow boundary transition effect with continuous deformation matching is output.

[0037] Continuing with the above example, for the material difference between the metal torso and the plastic arm of the robot, combined with the shadow attenuation effect calculated based on the light incident angle, the reflection intensity distribution and the adjusted relative position relationship are input into the light propagation model that calls the high attenuation coefficient of metal and the low attenuation coefficient of plastic. Based on the light propagation model after inputting the relative position relationship, after generating the initial shadow boundary using the level set algorithm, the smoothing filter processes to make the transition region of the initial shadow boundary gradually change naturally, and finally a soft shadow boundary transition effect is formed at the junction of metal and plastic, avoiding the visual discontinuity of traditional hard-edge shadows.

[0038] 105. Based on the dynamic pose change sequence in the spatial coordinate mapping relationship, align the shadow boundary transition effect with the light source position in the real scene in space and time to complete the synchronous rendering of light and shadow in the virtual and real scenes.

[0039] The dynamic pose change sequence refers to the continuous motion trajectory data of an object arranged in chronological order. Space-time alignment refers to the process of synchronously matching the virtual shadow effect and the real scene light source in the time and space dimensions.

[0040] In the embodiments of the present application, the dynamic attitude change sequence in the spatial coordinate mapping relationship is segmented into discrete attitude sequences according to timestamps, and based on the discrete attitude sequences, the shadow boundary transition effect is mapped frame by frame to the real scene coordinate system according to the corresponding timestamps through a preset rigid body transformation matrix. Combining the physical coordinates of the real scene light source position, the least squares method is used to optimize the position offset of the virtual shadow, and finally the pixel-level alignment of the virtual and real light and shadow is achieved.

[0041] Continuing with the above example, during the rapid running of the robot, the system parses the dynamic attitude sequence of the robot frame by frame, and maps the virtual shadow effect to the ground coordinate system of the real scene according to the timestamp. The coordinates of the mapped virtual shadow effect are matched with the position coordinates of the real light source on site through an optimization algorithm, and the position offset of the virtual shadow is corrected in real time to ensure that when the robot moves at high speed, the projected shadow stretches or compresses naturally with the movement and is completely synchronized with the real light source, completing the light and shadow synchronous rendering of the virtual and real scenes.

[0042] This solution accurately separates the surface reflection characteristics of objects through multi-wavelength light sources, constructs a high-precision spatial mapping by combining dynamic attitudes and depth data, calculates the surface reflection distribution of virtual objects in real time and synchronizes the light source position, generates a natural shadow transition based on the material attenuation model, and finally achieves frame-level matching of virtual and real light and shadow through spatio-temporal alignment. This method solves the problems of specular highlight distortion, shadow offset, and virtual-real boundary splitting in traditional solutions, and is applicable to high-precision dynamic light and shadow fusion scenarios such as movie special effects and AR live broadcasts.

[0043] In some embodiments, the specular reflectivity and the deformation correction parameters in the spatial coordinate mapping relationship are input into the light propagation path simulation module to calculate the reflection intensity distribution of each region on the surface of the virtual object, including: 201. The specular reflectivity is split into multiple local reflectivity sets according to the spatial positions of the vertices on the surface of the virtual object, and the light propagation direction offset of each local reflectivity set is corrected according to the spatial deformation parameters corresponding to the vertices in the spatial coordinate mapping relationship. The local reflectivity set refers to multiple sub-regions obtained by dividing the surface of the virtual object according to grid vertices, and each sub-region contains a set of vertices and their corresponding specular reflectivity data. The spatial deformation parameter refers to the vertex displacement calculated according to the dynamic attitude change of the object, which is used to describe the degree of local deformation of the surface. The correction of the light propagation direction offset means adjusting the incident direction of light on the object surface according to the deformation parameter to compensate for the optical path deviation caused by the deformation.

[0044] In the embodiments of the present application, first, according to the vertex density on the surface of the virtual object, the specular reflectivity is split into multiple local reflectivity sets, and each set corresponds to a sub-region. For example, the mesh vertices in the elbow area of the robotic arm are divided into independent sets. Subsequently, based on the deformation parameters corresponding to the vertices in the spatial coordinate mapping relationship, the displacement vector of each vertex is extracted. The light propagation direction offset of each local reflectivity set is corrected by the normal vector correction method in the ray tracing algorithm. The displacement vector is projected onto the original normal vector direction of the vertex, and the deformed light propagation direction offset is calculated. The light incident direction is adjusted according to the light propagation direction offset to generate the corrected light propagation direction offset. For example, if the vertex displacement causes the normal vector to deflect by 5 degrees, the light incident direction is synchronously deflected by 5 degrees to match the deformed surface geometry.

[0045] 202. Based on the corrected light propagation direction offset, calculate the initial value of the specular reflection intensity of each vertex at a preset light incident angle, and perform direction continuity interpolation on the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship to generate the reflection intensity distribution of the continuous region on the surface of the virtual object.

[0046] The initial value of the specular reflection intensity refers to the initial value of the vertex reflection light intensity calculated according to the corrected light direction. Direction continuity interpolation refers to using the deformation parameters of adjacent vertices to generate a smooth transition reflection intensity distribution through an interpolation algorithm.

[0047] In the embodiments of the present application, based on the corrected light propagation direction offset, the Monte Carlo integration algorithm is used to perform multiple ray path samplings on each vertex, and the average value of the reflected light intensity is statistically calculated as the initial value of the specular reflection intensity at a preset light incident angle. For example, 1024 samplings are performed on the robotic elbow vertex to calculate the average reflection intensity at a 60-degree incident angle of the spotlight. Subsequently, the displacement gradient and the normal vector change rate in the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship are extracted. The displacement gradient and the normal vector change rate are input into the bilinear interpolation algorithm, and the initial value of the specular reflection intensity is weighted and fused along the mesh edge. According to the corresponding displacement gradient weight distribution ratio, a smooth transition value in the intermediate region is generated, and finally, a continuous and non-abrupt reflection intensity distribution of the continuous region on the surface of the virtual object is formed.

[0048] The following is a specific example: When virtual shooting the bending scene of a robotic arm, first split the vertexes of the surface mesh of the robot's elbow into multiple local reflectivity sets. According to the vertex displacement data generated by the bending action of the robot, calculate the normal vector offset of each vertex and correct the light incident direction to ensure that the highlight reflection direction of the virtual spotlight on the metal surface precisely matches the deformed geometric shape. Subsequently, based on the corrected light direction, calculate the initial value of the specular reflection intensity of each vertex through Monte Carlo integration, and use the displacement gradient parameters of adjacent vertexes for bilinear interpolation to generate a continuous highlight gradient distribution from the elbow to the forearm, eliminating the blocky light spots caused by traditional discrete sampling. Finally, make the metal surface present a natural and smooth reflection effect during dynamic bending.

[0049] Through the local reflectivity splitting and the correction of the light direction driven by deformation parameters, this solution dynamically adapts to the influence of the deformation of the object surface on the light propagation path, solves the problem of highlight direction deviation of the traditional global reflection model under complex deformations such as dynamic bending and stretching, and ensures that the specular reflection direction of the virtual object precisely matches the surface geometry after deformation. Combining the Monte Carlo integration and the bilinear interpolation algorithm, generate a continuous and smooth reflection intensity distribution, eliminate the blocky light spots and reflection mutations caused by traditional discrete sampling, and make the reflection intensity of the complex curved surface present a natural gradient effect.

[0050] In some embodiments, the relative position relationship between the virtual object and the light source in the real scene is adjusted in real time according to the dynamic pose data, including: 301. Convert the displacement change amount in the dynamic pose data into the position change amount of the virtual object coordinate system relative to the real scene light source coordinate system; The displacement change amount refers to the position offset data of the virtual object in the real scene coordinate system, usually described in the form of a three-dimensional vector. The position change amount of the virtual object coordinate system relative to the real scene light source coordinate system refers to converting the displacement of the virtual object into the relative displacement of the coordinate system with the light source as the origin.

[0051] In the embodiment of the present application, first extract the displacement change amount of the virtual object from the dynamic pose data, and the displacement change amount includes the translation component of the virtual object in the three-dimensional space. Through the homogeneous coordinate transformation matrix, convert the vector of the displacement change amount of the virtual object from the object local coordinate system to the global coordinate system with the real scene light source as the origin to obtain the corresponding position change amount. For example, when the object moves in a certain direction, the displacement vector needs to be recalculated according to the initial position of the light source to ensure the accurate mapping of the displacement change amount in the light source coordinate system as the position change amount.

[0052] 302. Taking the initial position of the light source in the real scene as the reference point, reversely superimpose the position change amount along the moving direction vector to generate the compensation position offset amount of the light source relative to the virtual object; The compensation position offset refers to the position correction amount generated in the opposite direction of the object movement direction to offset the relative position deviation of the light source caused by the displacement of the virtual object.

[0053] In the embodiments of the present application, taking the initial position of the light source in the real scene as the reference point, the vector of the position change amount obtained in step 301 is reversed, that is, the direction of the displacement change amount vector is reversed, and the compensation position offset of the light source relative to the virtual object is generated by multiplying by a proportionality coefficient according to the dynamic requirements of the scene. For example, if the object moves to the right, the compensation offset is generated to the left to ensure that the virtual object still maintains a logically consistent spatial relationship with the light source after moving.

[0054] 303. Calculate the real-time projection distance of the light source in the moving direction of the virtual object according to the compensation position offset and the dynamic reference position of the vertex on the surface of the virtual object, and perform dynamic tilt correction on the irradiation angle of the light source in the virtual scene based on the projection distance; The real-time projection distance refers to the vertical projection length of the light source in the moving direction of the virtual object, which is used to calculate the tilt correction amount of the light source irradiation angle. The dynamic tilt correction refers to adjusting the pitch angle or yaw angle of the light source according to the projection distance so that the light direction matches the object movement trend.

[0055] In the embodiments of the present application, according to the compensation position offset and the dynamic reference position of the vertex on the surface of the virtual object, first project the light source coordinates onto the moving direction vector of the virtual object by using the vector projection formula to obtain the real-time projection distance corresponding to the light source. Based on the real-time projection distance and the irradiation angle of the light source in the virtual scene, calculate the pitch angle correction amount of the light source through the tangent function to perform dynamic tilt correction on the pitch angle of the light source. For example, when the projection distance increases, the pitch angle of the light source increases synchronously, making the light closer to vertical irradiation.

[0056] 304. Superimpose the tilt-corrected irradiation angle and the compensation position offset to obtain the relative position update parameter between the light source and the virtual object, and perform incremental iteration on the relative position update parameter based on the update frequency of the dynamic attitude data to complete the adjustment of the relative position relationship between the light source and the virtual object.

[0057] Incremental iteration refers to performing step-by-step cumulative correction on the relative position parameter according to the update frequency of the dynamic attitude data to avoid screen jitter caused by sudden changes.

[0058] In the embodiments of the present application, the compensation position offset in step 302 and the tilt angle correction value in step 303 are superimposed to generate a relative position update parameter of the light source and the virtual object. According to the update frequency of the dynamic attitude data, an incremental Kalman filter is used to iteratively optimize the position update parameter to complete the adjustment of the relative position relationship between the light source and the virtual object. For example, only a part of the position correction amount of the current frame is applied in each frame, and the remaining part is accumulated to subsequent frames to ensure smooth adjustment of the light source position without jumps.

[0059] The following is a specific example: In the scenario of the virtual shooting robot's lateral movement, the system first converts the displacement change amount of the robot moving to the right into a rightward shift vector in the light source coordinate system; then, based on the initial position of the light source, a leftward compensation offset is generated to adjust the light source to the left to offset the logical deviation caused by the robot's displacement; next, the projection distance of the light source in the robot's moving direction is calculated, and the light is slightly tilted downward through pitch angle correction to match the surface geometric change when the robot is tilted; finally, the compensation offset and the angle correction value are superimposed, and the light source position is smoothly adjusted step by step according to the frame rate, so that the highlight area of the robot's metal shell gradually shifts with the light source, and at the same time, the shadow deformation is consistent with the real physical law, presenting a film-level light and shadow effect without jumps and smear.

[0060] This solution generates a dynamic correction of the relative position relationship between the virtual object and the real light source through displacement conversion and compensation offset, solves the problem of the logical misalignment of the light source caused by the movement of the object; combines the tilt correction driven by the projection distance and the incremental iterative optimization to achieve smooth adjustment of the light source angle and position, avoiding frame jumps. In a dynamic scene, the specular reflection and shadow deformation of the virtual object can match the change of the light source position in real time, significantly improving the realism and smoothness of the virtual-real interaction, and is applicable to virtual shooting scenes that require high-frequency light source interaction.

[0061] In some embodiments, the dynamic attitude data is fused with the depth information obtained by the binocular vision positioning system to generate a virtual-real scene space coordinate mapping relationship, including: 401. Establish a three-dimensional coordinate system with the centroid of the photographed object as the origin based on the baseline parameters of the binocular vision positioning system; convert the joint rotation angle and displacement amount in the dynamic attitude data into three-dimensional vectors in the three-dimensional coordinate system; The baseline parameter refers to the physical distance and angle between two cameras in the binocular vision positioning system, which is used to construct the basic geometric model for three-dimensional space measurement. The three-dimensional coordinate system refers to a right-handed coordinate system with the centroid of the photographed object as the origin and the binocular vision baseline direction as the reference axis.

[0062] In the embodiments of the present application, first, a three-dimensional coordinate system with the centroid of the photographed object as the origin is established according to the baseline parameters of the binocular vision positioning system. For example, if the distance between the binocular cameras is a specific distance and the included angle is a specific angle, the X-axis of the coordinate system extends along the baseline direction, the Y-axis is perpendicular upward, and the Z-axis points in the depth direction. Then, the joint rotation angles in the dynamic pose data are converted into rotation matrices, and combined with the displacement amounts to generate three-dimensional vectors in the three-dimensional coordinate system. For example, the pitch angle data of the robot arm is converted into three-dimensional motion vectors in the coordinate system to describe its motion direction and amplitude in space.

[0063] 402. Compound and superimpose the three-dimensional vectors with the three-dimensional coordinates of the joint feature points in the depth information obtained by the binocular vision positioning system to generate compound spatial data including dynamic deformation parameters; Compound spatial data refers to a data set that fuses the three-dimensional vectors of the dynamic pose and the coordinates of the surface feature points of the object in the binocular vision depth information, and includes the geometric deformation and motion trajectory information of the object.

[0064] In the embodiments of the present application, the three-dimensional vectors generated in step 401 are fused with the three-dimensional coordinates of the joint feature points extracted from the depth information of the binocular vision positioning system. The dynamic pose data and the visual data are time-synchronized and aligned through a Kalman filter, and weights are assigned according to the data confidence. For example, the weight of the dynamic pose data is relatively high, and the weight of the visual data is relatively low. After weighted fusion of the two, compound spatial data including dynamic deformation parameters is generated to describe the real-time changes in the surface geometry of the object.

[0065] 403. Deduce the spatial offsets of each vertex on the surface of the virtual object according to the deformation parameters in the compound spatial data; based on the spatial offsets and combined with the object contour boundary in the depth information obtained by the binocular vision positioning system, generate the motion trajectory of the vertices on the surface of the virtual object changing with the dynamic pose; The vertex spatial offset refers to the three-dimensional position displacement of the grid vertices on the surface of the virtual object due to the change in the dynamic pose. The motion trajectory refers to the continuous position change path formed by the vertices as the object moves.

[0066] In the embodiments of the present application, the deformation parameters are extracted from the compound spatial data, and the displacement vectors of each vertex on the surface of the virtual object are deduced through a finite element analysis algorithm. For example, when the robot elbow bends, the tangential displacement of the forearm vertices is calculated according to the curvature change rate. Subsequently, based on the spatial offsets and combined with the point cloud data of the object contour boundary in the depth information obtained by the binocular vision positioning system, a B-spline curve fitting algorithm is used to generate a smooth motion trajectory of the vertices on the surface of the virtual object changing with the dynamic pose, ensuring that the trajectory conforms to the physical motion constraints.

[0067] 404. Establish a dynamic mapping relationship between the vertex coordinates of the virtual object and the real - world scene coordinates based on the motion trajectory.

[0068] The dynamic mapping relationship refers to a real - time conversion model between the vertex coordinates of the virtual object and the real - world scene coordinates, which is used to describe the corresponding relationship of the vertex position in the virtual and real scenes as the object moves.

[0069] In the embodiment of the present application, based on the vertex motion trajectory generated in step 403, a mapping function from vertex coordinates to real - world scene coordinates is established through a non - rigid deformation model, which is used to describe the dynamic mapping relationship between the vertex coordinates of the virtual object and the real - world scene coordinates. For example, the thin - plate spline interpolation algorithm is used to convert the motion trajectory of the robot elbow vertex into a projection matrix in the real - world ground coordinate system, and the mapping parameters are updated frame by frame according to the time stamp to achieve dynamic sub - millimeter - level alignment of virtual and real coordinates.

[0070] The following is a specific example: In the virtual shooting of the robot arm bending scene, the system establishes a three - dimensional coordinate system with the robot shoulder as the origin based on the binocular vision baseline parameters, and converts the arm rotation angle and displacement into three - dimensional motion vectors in the coordinate system; then, the dynamic attitude data and the elbow feature point coordinates extracted by binocular vision are fused to generate composite spatial data containing the curvature change rate; based on the curvature change rate, the displacement of the forearm vertex is calculated, and combined with the arm outer contour point cloud to generate a smooth motion trajectory; finally, a dynamic mapping relationship between the forearm vertex and the real - world desktop is established through the deformation interpolation algorithm, so that during the bending process of the virtual arm, the surface vertex position of it matches the real desktop projection in real time, and the highlight reflection and shadow deformation of the metal joint transition naturally with the movement, presenting a virtual - real fusion effect without offset and split.

[0071] This solution deeply fuses binocular vision and dynamic attitude data to construct a high - precision composite spatial description, solves the problem of spatial mapping error of traditional single - data sources; generates a smooth vertex motion trajectory based on deformation parameters and contour constraints to ensure that the deformation of the virtual object conforms to physical laws; realizes real - time sub - millimeter - level alignment of virtual and real coordinates through a dynamic mapping function, significantly improving the spatial consistency of virtual - real fusion in dynamic scenes.

[0072] In some embodiments, the reflection intensity distribution and the adjusted relative position relationship are input into a light propagation model including material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching, including: 501. Based on the initial specular reflection intensity values of each vertex on the surface of the virtual object in the reflection intensity distribution, combined with the real - time projection distance of the light source in the adjusted relative position relationship, calculate the direct light contribution value of each vertex under the illumination of the light source; The direct light contribution value refers to the initial light intensity value generated by the vertices on the surface of a virtual object under direct illumination from a light source. The real-time projection distance is the perpendicular projection length from the light source to the surface of the virtual object, which is used to quantify the attenuation degree of the illumination intensity of the light source on the vertices.

[0073] In the embodiments of the present application, based on the initial specular reflection intensity of each vertex on the surface of the virtual object in the reflection intensity distribution, and in combination with the real-time projection distance of the light source in the adjusted relative position relationship, the Phong illumination model is used to calculate the direct light contribution value of the vertex under the illumination of the light source. Specifically, the ray tracing algorithm is used to detect whether the path from the light source to the vertex is blocked. If not blocked, the cosine value of the incident angle of the ray and the vertex normal vector is calculated, and it is multiplied by the specular reflection intensity to obtain the initial light contribution value. For example, the smaller the angle between the vertex normal vector and the ray direction, the larger the cosine value, and the higher the direct light contribution value.

[0074] 502. According to the preset material absorption coefficient in the material attenuation characteristics, along the light propagation path from the light source to the surface of the virtual object, the direct light contribution value is attenuated layer by layer to obtain the attenuated light intensity; The material absorption coefficient refers to the attenuation ratio of the light intensity on the light propagation path by different materials. For example, the high absorption rate of metal materials and the low absorption rate of cloth materials. Layer-by-layer attenuation refers to the process of segmentally weakening the light intensity along the light propagation path.

[0075] In the embodiments of the present application, according to the material type, the preset material absorption coefficient in the attenuation characteristics is called, and segmented integral calculation is performed along the light path from the light source to the vertices on the surface of the virtual object. The light path is divided into several equal-length sub-segments, and each sub-segment attenuates the direct light contribution value layer by layer according to the exponential function of the product of the absorption coefficient and the distance. For example, the absorption coefficient of metal materials is relatively high, and when the light propagates the same distance, the attenuation amplitude is significantly greater than that of plastic materials, and finally the attenuated light intensity is obtained.

[0076] 503. Based on the deformation compensation coefficient in the spatial coordinate mapping relationship, the light intensity is dynamically deformed and matched and corrected so that the change in light intensity between adjacent vertices conforms to the continuous deformation trend of the surface of the virtual object; The deformation compensation coefficient refers to the light intensity correction parameter generated according to the dynamic deformation degree of the surface of the virtual object, which is used to make the light change consistent with the deformation trend.

[0077] In the embodiments of the present application, based on the deformation compensation coefficient in the spatial coordinate mapping relationship, non-linear deformation matching correction is performed on the attenuated light intensity. For example, if the surface curvature of a vertex increases due to deformation, the correction weight of the light intensity is increased according to the curvature change rate, so that the difference in light intensity between adjacent vertices is positively correlated with the degree of deformation. Specifically, a Gaussian function is used to assign weights to the deformation parameters to generate a smoothly corrected light intensity distribution, so that the change in light intensity between adjacent vertices conforms to the continuous deformation trend of the virtual object surface.

[0078] 504. Superimpose the corrected light intensity and the diffuse reflection coefficient of the virtual object surface to generate the final reflection intensity of each vertex under the illumination of the real scene light source; The final reflection intensity refers to the total reflected light intensity value of a vertex after comprehensive direct light contribution, attenuation correction, and deformation matching, and includes the superposition result of specular reflection and diffuse reflection.

[0079] In the embodiments of the present application, the corrected light intensity is linearly superimposed with the diffuse reflection coefficient of the vertices on the virtual object surface. For example, after the high specular reflection intensity and low diffuse reflection coefficient of a metal vertex are superimposed, a reflection intensity distribution with a bright highlight area and a soft dark part is generated, while after the low specular reflection and high diffuse reflection coefficient of a cloth vertex are superimposed, a uniform scattering effect is presented, generating the final reflection intensity of each vertex under the illumination of the real scene light source.

[0080] 505. According to the spatial distribution of the final reflection intensity, calculate the change gradient of the reflection intensity between adjacent vertices, and based on the change gradient, perform smooth transition interpolation on the boundary of the shadow area to generate a shadow boundary transition effect with continuous deformation matching.

[0081] The change gradient of the reflection intensity refers to the difference rate of the reflection intensity between adjacent vertices, which is used to quantify the sharpness of the shadow boundary. Smooth transition interpolation refers to blurring the shadow boundary according to the gradient value to eliminate jaggedness.

[0082] In the embodiments of the present application, according to the spatial distribution of the final reflection intensity, calculate the change gradient of the reflection intensity between adjacent vertices. For example, when the gradient value is greater than the threshold, it is determined as the shadow boundary. Subsequently, based on the change gradient, perform smooth transition interpolation on the boundary of the shadow area, and use a bicubic interpolation algorithm to smooth the reflection intensity in the boundary area to generate a shadow boundary transition effect with continuous deformation matching. For example, in the high-gradient area at the joint of the robot arm, the light intensity value of the transition pixel is generated by interpolation, so that the shadow boundary transitions from a sharp edge to a natural gradient effect.

[0083] The following is a specific example: In the scenario of the virtual shooting robot arm bending, the system first calculates the direct light contribution value of the metal surface based on the initial value of the specular reflection intensity of the elbow vertex and the real-time projection distance of the light source; then, according to the high absorption coefficient of the metal material, it performs layer-by-layer attenuation along the light propagation path to weaken the light intensity to the natural dark part level; then, in combination with the curvature change rate caused by the elbow bending deformation, it dynamically corrects the attenuated light intensity to ensure that the light intensity of the inner vertex of the bend is lower than that of the outer side; the corrected light intensity is superimposed with the diffuse reflection coefficient to generate a reflection intensity distribution with a smooth transition between the metal highlights and dark parts; finally, based on the reflection intensity gradient of adjacent vertices, the shadow boundary between the elbow and the forearm is smoothed through the bicubic interpolation algorithm, making the sharp edge transition into a natural gradient, completely eliminating the pixel splitting at the virtual-real fusion boundary, and presenting a film-level light and shadow effect.

[0084] This solution calculates the direct light contribution value for each vertex, combines the attenuation simulation along the light path according to the material absorption characteristics, and accurately restores the light and dark change rules of complex materials such as metal and plastic under dynamic light sources; dynamically corrects the light intensity based on the deformation compensation coefficient, making the light and shadow distribution strictly synchronized with the deformation trends such as bending and stretching of the object surface, and eliminating the highlight misalignment problem of traditional static models in dynamic scenes; generates a natural transition between highlights and dark parts of the reflection intensity by superimposing the diffuse reflection coefficient, and performs smooth optimization on the shadow boundary based on the gradient-driven interpolation algorithm, completely solving the sawtooth and pixel-level splitting at the virtual-real fusion boundary.

[0085] In some embodiments, based on the dynamic pose change sequence in the spatial coordinate mapping relationship, the shadow boundary transition effect is spatially and temporally aligned with the light source position in the real scene to complete the light and shadow synchronous rendering of the virtual-real scene, including: 601. Calculate the moving direction and displacement speed of each vertex on the surface of the virtual object in the three-dimensional space coordinate system according to the pose difference amount of consecutive time nodes in the dynamic pose change sequence in the spatial coordinate mapping relationship; The dynamic pose change sequence refers to the virtual object motion trajectory data recorded in chronological order, including the position and rotation information of each time node. The pose difference amount refers to the change amplitude of the object pose between adjacent time nodes, and is used to deduce the vertex moving direction and displacement speed.

[0086] In the embodiments of the present application, the pose difference amount of consecutive time nodes is extracted from the dynamic pose change sequence in the spatial coordinate mapping relationship, and the displacement vector and rotation angle of adjacent two frames are calculated by the difference method. Based on the three-dimensional space coordinate system, the velocity interpolation algorithm is used to generate the moving direction vector of the vertex in the three-dimensional axis direction, and the displacement speed of the moving direction vector is calculated. For example, when the vertex of the robot arm moves along a certain axis within a specific time period, the displacement speed is obtained by dividing the displacement amount by the time interval.

[0087] 602. Extract the light source movement trajectory in the real scene, and establish a direction difference parameter between the movement direction in the light source movement trajectory and the movement direction of the virtual object vertex; The light source movement trajectory refers to the movement path of the light source in the three-dimensional space in the real scene. The direction difference parameter refers to the angular deviation value between the light source movement direction and the movement direction of the virtual object vertex.

[0088] In the embodiment of the present application, the light source movement trajectory in the real scene is tracked by an optical flow method or a sensor, and the direction vector of the light source movement trajectory is extracted. The light source direction vector and the vertex movement direction vector obtained in step 601 are input into the vector included angle formula to calculate the direction difference parameter between the light source direction vector and the vertex movement direction vector. For example, when there is an included angle between the light source movement direction and the vertex movement direction, the direction difference parameter is the degree of this included angle.

[0089] 603. Match the light and dark change rate of each pixel area in the shadow boundary transition effect with the direction difference parameter to obtain the shadow boundary deformation amount caused by the light source movement; The light and dark change rate refers to the speed at which the pixel brightness in the shadow boundary area changes with time. The shadow boundary deformation amount refers to the position offset amount of the shadow boundary caused by the light source movement.

[0090] In the embodiment of the present application, the light and dark change rate of each pixel area in the shadow boundary transition effect is extracted, and the light and dark change rate is matched with the direction difference parameter in step 602. The Kalman filter is used to allocate weights to the light and dark change rate and the direction difference parameter to generate the shadow boundary deformation amount caused by the light source movement. For example, the larger the direction difference parameter, the higher the weight of the shadow boundary deformation amount correction.

[0091] 604. According to the time interval of the dynamic attitude change sequence, perform a time series synchronous interpolation calculation on the displacement speed of the virtual object vertex and the light source movement trajectory to generate a correlation ratio of the shadow deformation amount with respect to the light source movement speed; The time series synchronous interpolation refers to the process of time-aligning the displacement speed of the virtual object vertex and the light source movement trajectory according to the time interval of the dynamic attitude change sequence. The correlation ratio refers to the dynamic relationship coefficient between the shadow deformation amount and the light source movement speed.

[0092] In the embodiment of the present application, according to the time interval of the dynamic attitude change sequence, the vertex displacement speed calculated in step 601 and the light source movement trajectory extracted in step 602 are input into the bilinear interpolation algorithm to align the movement data of the displacement speed and the light source movement trajectory according to the time stamp. The proportional integral controller is used to generate the correlation ratio of the shadow deformation amount changing with the light source movement speed. For example, when the light source movement speed doubles, the correlation ratio of the shadow deformation amount is synchronously increased.

[0093] 605. Dynamically scale and adjust the light and dark distribution area of the shadow boundary transition effect based on the correlation ratio, so that the bending degree of the virtual shadow boundary changes synchronously with the moving trajectory of the light source; superimpose and fuse the adjusted shadow boundary transition effect with the contour of the illumination area of the real scene light source, so that the deformation process of the virtual shadow boundary forms a dynamic following relationship with the moving process of the real light source.

[0094] The dynamic scaling adjustment refers to scaling and deforming the shadow boundary area according to the correlation ratio, so that its deformation degree is synchronized with the movement of the light source. The dynamic following relationship refers to the real-time matching between the deformation process of the virtual shadow boundary and the movement of the real light source.

[0095] In the embodiment of the present application, based on the correlation ratio in step 604, the light and dark distribution area of the shadow boundary transition effect is dynamically scaled. The thin plate spline interpolation algorithm is used to adjust the bending degree of the virtual shadow boundary, so that the bending degree changes synchronously with the moving trajectory of the light source. Finally, the adjusted shadow boundary transition effect is superimposed on the contour of the real light source illumination area, and the smooth fusion of the virtual and real boundaries is achieved through the Alpha blending algorithm, so that the deformation process of the virtual shadow boundary forms a dynamic following relationship with the moving process of the real light source.

[0096] The following is a specific example: In a fast-moving scenario of a virtual shooting robot, the system first calculates the moving direction and speed of the elbow vertex of the robot in three-dimensional space according to the dynamic pose sequence; then tracks the moving trajectory of the real spotlight and calculates the angle difference between its direction and the moving direction of the elbow vertex; matches the light and dark change rate and direction difference parameters of the elbow shadow boundary to generate the shadow deformation amount caused by the movement of the light source; aligns the vertex speed and the light source trajectory through time series synchronous interpolation to establish a dynamic correlation ratio between the shadow deformation amount and the light source speed; finally, dynamically scale the shadow boundary according to the ratio and fuse it with the real light source contour through the blending algorithm, so that the shadow of the robot elbow is stretched and deformed in real time as the spotlight moves, and the bending degree of the shadow boundary is strictly synchronized with the light source trajectory, presenting a virtual-real light and shadow fusion effect without delay and pixel misalignment.

[0097] Through the spatio-temporal alignment of the dynamic pose sequence and the light source movement trajectory, this solution accurately calculates the spatial difference parameters between the movement directions of the virtual object vertices and the real light source, solving the problems of shadow deformation and the delay misalignment of the light source movement in dynamic scenarios; based on the matching relationship between the light and dark change rate and the direction difference, it quantifies the influence of the light source movement on the deformation amount of the shadow boundary, ensuring that the shadow deformation direction is strictly synchronized with the light source movement trend; through the time series interpolation calculation, it establishes a dynamic correlation ratio between the shadow deformation amount and the light source speed, realizing the scaling adjustment of the shadow boundary and the frame-level matching with the real light source movement; finally, through the virtual-real boundary fusion algorithm, the bending degree and stretching deformation of the virtual shadow boundary are linked in real time with the real light source movement trajectory, eliminating the problems of shadow lag or deformation distortion in the traditional solution.

[0098] In some embodiments, the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship are interpolated in terms of direction continuity to generate the reflection intensity distribution of the continuous region on the virtual object surface, including: 701. Calculate the deformation gradient change amount in the direction of the vertex connection line according to the difference in the spatial deformation parameters between adjacent vertices in the spatial coordinate mapping relationship; The difference in spatial deformation parameters refers to the displacement or rotation difference amount generated by adjacent vertices during the dynamic deformation process. The deformation gradient change amount refers to the change rate of the deformation parameters along the vertex connection line, which is used to quantify the degree of deformation continuity.

[0099] In the embodiments of this application, first, the spatial deformation parameters of adjacent vertices are extracted from the spatial coordinate mapping relationship, such as displacement vectors or rotation matrices. The difference in the spatial deformation parameters between adjacent vertices is calculated by the vector difference method, such as the displacement vector difference between vertex A and vertex B. Based on the geometric length of the vertex connection line and the difference in the deformation parameters, the deformation gradient change amount in the direction of the vertex connection line is calculated. For example, if the vertex displacement difference is a specific value and the connection line length is a specific distance, the gradient change amount is the ratio of the displacement difference to the connection line length.

[0100] 702. Based on the deformation gradient change amount, set multiple intermediate interpolation points on the connection line between adjacent vertices, and calculate the interpolation result of the spatial deformation parameter corresponding to each intermediate interpolation point according to the distance ratio between each intermediate interpolation point and the adjacent vertices; The intermediate interpolation point refers to a virtual point inserted along the direction of the connection line between adjacent vertices, which is used to fill the blank area of the deformation parameters between the vertices. The distance ratio refers to the proportion of the connection line length between the intermediate interpolation point and the adjacent vertices.

[0101] In the embodiments of the present application, based on the change amount of the deformation gradient calculated in step 701, a plurality of intermediate interpolation points are evenly set on the line connecting adjacent vertices. According to the distance ratios of each interpolation point to adjacent vertices A and B, a bilinear interpolation algorithm is used to calculate the interpolation result of the spatial deformation parameter corresponding to each intermediate interpolation point. For example, if the distance of the interpolation point from vertex A is 30% of the line length, its deformation parameter is the weighted fusion result of 70% of the parameter of vertex A and 30% of the parameter of vertex B.

[0102] 703. Weightedly fuse the interpolation result of the spatial deformation parameter with the initial value of the specular reflection intensity of the corresponding vertex to obtain the reflection intensity value of the intermediate interpolation point; Specular reflection intensity weighted fusion means superimposing the interpolation result of the deformation parameter of the intermediate interpolation point and the initial value of the specular reflection intensity of its corresponding vertex according to weights to generate the reflection intensity value of the interpolation point.

[0103] In the embodiments of the present application, the interpolation result of the spatial deformation parameter obtained in step 702 and the initial value of the specular reflection intensity of the corresponding adjacent vertex are input into the weight assignment model. For example, if the interpolation result of the spatial deformation parameter is close to vertex A, the reflection intensity weight of vertex A is higher. The reflection intensity value of the intermediate interpolation point is calculated through a linear weighting formula to ensure smooth transition of the reflection intensity between the intermediate interpolation point and the adjacent vertices.

[0104] 704. Construct the reflection intensity distribution of the continuous region on the surface of the virtual object according to the reflection intensity values of all intermediate interpolation points.

[0105] Continuous reflection intensity distribution means generating a reflection intensity map covering all regions on the surface of the virtual object through the reflection intensity values of the interpolation points to eliminate the sudden change of the reflection intensity between vertices.

[0106] In the embodiments of the present application, based on the reflection intensity values of all intermediate interpolation points, a surface fitting algorithm, such as non-uniform rational B-spline, is used to construct the reflection intensity distribution of the continuous region on the surface of the virtual object. For example, taking the reflection intensity of the interpolation point as the control point, a smooth reflection intensity surface is generated to ensure that the reflection intensity value at any position on the surface dynamically matches its deformation parameter.

[0107] The following is a specific example: In the scenario of the virtual shooting robot arm bending, the system first calculates the change amount of the deformation gradient along the connection direction according to the displacement difference of adjacent vertices at the elbow, quantifying the deformation continuity of the metal surface during the bending process. Subsequently, a plurality of intermediate interpolation points are evenly inserted on the vertex connection line, and the deformation parameters are distributed according to the distance ratio. For example, the closer the interpolation point is to the vertex, the closer its deformation parameter is to the vertex data. Then, the deformation parameter of the interpolation point and the initial value of the vertex specular reflection intensity are fused according to the weight to generate the reflection intensity value of the interpolation point, ensuring a smooth transition of the light intensity in the bending area. Finally, all the reflection intensity values of the interpolation points are integrated through a surface fitting algorithm to construct a continuous reflection intensity distribution covering the elbow surface, making the metal highlight gradually change naturally from the vertex to the interpolation point, completely eliminating the reflection intensity mutation caused by the traditional discrete vertex model and presenting a film-level dynamic light and shadow effect.

[0108] This solution accurately captures the dynamic deformation trend of the virtual object surface by quantifying the change amount of the deformation gradient of adjacent vertices; based on the calculation of the deformation parameters of the intermediate interpolation points and the weighted fusion of the reflection intensity, it fills the blank area of the reflection intensity between the vertices and eliminates the reflection intensity mutation problem caused by the traditional discrete vertex model; combined with the surface fitting algorithm, it constructs a continuous reflection intensity distribution covering the object surface to ensure that the reflection intensity strictly matches the surface geometry in any deformation state.

[0109] Figure 2 The following is a schematic structural diagram of a virtual imaging system applied to virtual shooting according to an embodiment of the present application. As Figure 2 shown, the system includes: An acquisition module 21, configured to synchronously acquire the reflection characteristic data of the surface of the shooting object through a multi-wavelength light source, input the reflection characteristic data into a pre-trained image feature extraction network, and generate imaging feature parameters including the specular reflectivity and diffuse reflection coefficient of the object surface; A generation module 22, configured to construct a spatial attitude monitoring array based on a plurality of motion sensors, acquire the dynamic attitude data of the shooting object in real time, and fuse the dynamic attitude data with the depth information obtained through a binocular vision positioning system to generate a spatial coordinate mapping relationship between the virtual and real scenes; A calculation module 23, configured to input the specular reflectivity and the deformation correction parameter in the spatial coordinate mapping relationship into a light propagation path simulation module to calculate the reflection intensity distribution of each area on the surface of the virtual object, and at the same time, adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic attitude data; The generation module 22 is further configured to input the reflection intensity distribution and the relative position relationship into a light propagation model including material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching, where the deformation correction parameter is synchronously updated through the real-time attitude change data of the spatial attitude monitoring array and the shadow boundary transition effect; An alignment module 24, configured to perform spatio-temporal alignment of the shadow boundary transition effect and the light source position in the real scene based on the dynamic pose change sequence in the spatial coordinate mapping relationship, so as to complete the synchronous rendering of light and shadow in the virtual and real scenes.

[0110] Figure 2 The virtual imaging system applied to virtual shooting described above can execute Figure 1 For the virtual imaging method applied to virtual shooting in the foregoing embodiments, its implementation principle and technical effects will not be elaborated herein. For the virtual imaging system applied to virtual shooting in the above embodiments, the specific manners in which each module and unit perform operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0111] In a possible design, Figure 2 The virtual imaging system applied to virtual shooting in the foregoing embodiments can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, and among them, the one or more computer instructions are called and executed by the processing component 32.

[0112] The processing component 32 is used for the above Figure 1 The virtual imaging method applied to virtual shooting in the foregoing embodiments.

[0113] Among them, the processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above method.

[0114] The storage component 31 is configured to store various types of data to support the operation of the terminal. The storage component may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0115] Of course, the computing device may necessarily further include other components, such as input / output interfaces, display components, communication components, etc.

[0116] The input / output interface provides an interface between the processing component and the peripheral interface module, and the peripheral interface module may be an output device, an input device, etc.

[0117] The communication component is configured to facilitate communication between the computing device and other devices in a wired or wireless manner, etc.

[0118] Among them, the computing device may be a physical device or an elastic computing host provided by a cloud computing platform, etc. At this time, the computing device may refer to a cloud server, and the above-mentioned processing component, storage component, etc. may be basic server resources rented or purchased from a cloud computing platform.

[0119] The embodiments of the present application also provide a computer storage medium storing a computer program, and when the computer program is executed by a computer, the above-mentioned Figure 1 virtual imaging method applied to virtual shooting shown in the embodiments can be implemented.

[0120] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0122] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A virtual imaging method applied to virtual shooting, characterized in that Including: Synchronously collect the reflection characteristic data of the object surface by a multi-wavelength light source, input the reflection characteristic data into a pre-trained image feature extraction network to generate shadow feature parameters, where the shadow feature parameters include the specular reflectivity and diffuse reflection coefficient of the object surface; Collect the dynamic pose data of the object, and fuse the dynamic pose data with the depth information obtained by a binocular vision positioning system to generate the spatial coordinate mapping relationship between the virtual and real scenes; Input the specular reflectivity and the deformation correction parameters in the spatial coordinate mapping relationship into a light propagation path simulation module to calculate the reflection intensity distribution of each area on the virtual object surface, and at the same time, adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic pose data; Input the reflection intensity distribution and the adjusted relative position relationship into a light propagation model including material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching; Based on the dynamic pose change sequence in the spatial coordinate mapping relationship, perform spatio-temporal alignment on the shadow boundary transition effect and the light source position in the real scene to complete the light and shadow synchronous rendering of the virtual and real scenes.

2. The method according to claim 1, wherein Input the specular reflectivity and the deformation correction parameters in the spatial coordinate mapping relationship into a light propagation path simulation module to calculate the reflection intensity distribution of each area on the virtual object surface, including: Split the specular reflectivity into multiple local reflectivity sets according to the spatial positions of the vertices on the virtual object surface, and correct the light propagation direction offset amount for each local reflectivity set according to the spatial deformation parameters corresponding to the vertices in the spatial coordinate mapping relationship; Based on the corrected light propagation direction offset amount, calculate the initial value of the specular reflection intensity of each vertex at a preset light incidence angle, and perform direction continuity interpolation on the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship to generate the reflection intensity distribution of the continuous area on the virtual object surface.

3. The method according to claim 1, wherein Adjust the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic pose data, including: Convert the displacement change amount in the dynamic pose data into the position change amount of the virtual object coordinate system relative to the real scene light source coordinate system; Taking the initial position of the light source in the real scene as the reference point, reversely superimpose the position change amount along the moving direction vector to generate the compensation position offset amount of the light source relative to the virtual object; According to the compensation position offset amount and the dynamic reference position of the vertices on the virtual object surface, calculate the real-time projection distance of the light source in the moving direction of the virtual object, and perform dynamic tilt correction on the irradiation angle of the light source in the virtual scene based on the projection distance; Superimpose the tilted corrected irradiation angle and the compensation position offset amount to obtain the relative position update parameter between the light source and the virtual object, and perform incremental iteration on the relative position update parameter based on the update frequency of the dynamic pose data to complete the adjustment of the relative position relationship between the light source and the virtual object.

4. The method according to claim 1, wherein Fuse the dynamic attitude data with the depth information obtained by the binocular vision positioning system to generate a mapping relationship between the virtual and real scene space coordinates, including: Establish a three-dimensional coordinate system with the centroid of the photographed object as the origin based on the baseline parameters of the binocular vision positioning system; convert the joint rotation angles and displacement amounts in the dynamic attitude data into three-dimensional vectors in the three-dimensional coordinate system; Compound and superimpose the three-dimensional vectors with the three-dimensional coordinates of the joint feature points in the depth information obtained by the binocular vision positioning system to generate composite spatial data containing dynamic deformation parameters; Deduce the spatial offsets of each vertex on the surface of the virtual object according to the deformation parameters in the composite spatial data; based on the spatial offsets and combined with the object contour boundary in the depth information obtained by the binocular vision positioning system, generate the movement trajectories of the vertices on the surface of the virtual object changing with the dynamic attitude; Establish a dynamic mapping relationship between the vertex coordinates of the virtual object and the real scene coordinates based on the movement trajectories.

5. The method according to claim 1, characterized in that, Input the reflection intensity distribution and the adjusted relative position relationship into the light propagation model including the material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching, including: Based on the initial specular reflection intensity values of each vertex on the surface of the virtual object in the reflection intensity distribution, and combined with the real-time projection distance of the light source in the adjusted relative position relationship, calculate the direct light contribution value of each vertex under the illumination of the light source; According to the preset material absorption coefficient in the material attenuation characteristics, layer by layer attenuate the direct light contribution value along the light propagation path from the light source to the surface of the virtual object to obtain the attenuated light intensity; Based on the deformation compensation coefficient in the spatial coordinate mapping relationship, perform dynamic deformation matching correction on the light intensity so that the change in light intensity between adjacent vertices conforms to the continuous deformation trend of the surface of the virtual object; Superimpose the corrected light intensity with the diffuse reflection coefficient of the surface of the virtual object to generate the final reflection intensity of each vertex under the illumination of the real scene light source; According to the spatial distribution of the final reflection intensity, calculate the change gradient of the reflection intensity between adjacent vertices, and perform smooth transition interpolation on the boundary of the shadow area based on the change gradient to generate a shadow boundary transition effect with continuous deformation matching.

6. The method according to claim 1, wherein Based on the dynamic attitude change sequence in the spatial coordinate mapping relationship, perform spatio-temporal alignment of the shadow boundary transition effect with the light source position in the real scene to complete the light and shadow synchronous rendering of the virtual and real scenes, including: According to the attitude difference amount of consecutive time nodes in the dynamic attitude change sequence in the spatial coordinate mapping relationship, calculate the movement direction and displacement speed of each vertex on the surface of the virtual object in the three-dimensional space coordinate system; Extract the movement trajectory of the light source in the real scene and establish a direction difference parameter between the movement direction in the light source movement trajectory and the movement direction of the virtual object vertex; Match the light and dark change rate of each pixel area in the shadow boundary transition effect with the direction difference parameter to obtain the shadow boundary deformation amount caused by the movement of the light source; According to the time interval of the dynamic posture change sequence, perform temporal synchronous interpolation calculation on the displacement speed of the virtual object vertices and the light source movement trajectory to generate the correlation ratio of the shadow deformation amount with respect to the light source movement speed; Based on the correlation ratio, dynamically scale and adjust the light and dark distribution area of the shadow boundary transition effect, so that the curvature of the virtual shadow boundary changes synchronously with the light source movement trajectory; superimpose and fuse the adjusted shadow boundary transition effect with the illumination area contour of the real scene light source, so that the deformation process of the virtual shadow boundary forms a dynamic following relationship with the movement process of the real light source.

7. The method according to claim 3, wherein Perform direction continuity interpolation on the spatial deformation parameters of adjacent vertices in the spatial coordinate mapping relationship to generate the reflection intensity distribution of the continuous area on the virtual object surface, including: According to the difference in spatial deformation parameters between adjacent vertices in the spatial coordinate mapping relationship, calculate the change amount of the deformation gradient in the direction of the vertex connection line; Based on the deformation gradient change amount, set multiple intermediate interpolation points on the connection line between adjacent vertices, and calculate the interpolation result of the spatial deformation parameter corresponding to each intermediate interpolation point according to the distance ratio between each intermediate interpolation point and the adjacent vertices; Perform weighted fusion on the interpolation result of the spatial deformation parameter and the initial value of the specular reflection intensity of the corresponding vertex to obtain the reflection intensity value of the intermediate interpolation point; According to the reflection intensity values of all intermediate interpolation points, construct the reflection intensity distribution of the continuous area on the virtual object surface.

8. A virtual imaging system applied to virtual shooting, characterized in that Including: An acquisition module for synchronously acquiring the reflection characteristic data of the surface of the photographed object through a multi-wavelength light source, and inputting the reflection characteristic data into a pre-trained image feature extraction network to generate imaging feature parameters including the specular reflectivity and diffuse reflection coefficient of the object surface; A generation module for constructing a spatial attitude monitoring array based on multiple motion sensors, acquiring the dynamic attitude data of the photographed object in real time, and fusing the dynamic attitude data with the depth information obtained through a binocular vision positioning system to generate a spatial coordinate mapping relationship between the virtual and real scenes; A calculation module for inputting the specular reflectivity and the deformation correction parameter in the spatial coordinate mapping relationship into a light propagation path simulation module to calculate the reflection intensity distribution of each area on the virtual object surface, and at the same time, adjusting the relative position relationship between the virtual object and the light source in the real scene in real time according to the dynamic attitude data; The generation module is further configured to input the reflection intensity distribution and the relative position relationship into an illumination propagation model including material attenuation characteristics to generate a shadow boundary transition effect with continuous deformation matching, wherein the deformation correction parameter is synchronously updated through the real-time attitude change data of the spatial attitude monitoring array and the shadow boundary transition effect; An alignment module for spatio-temporally aligning the shadow boundary transition effect with the position of the light source in the real scene based on the dynamic posture change sequence in the spatial coordinate mapping relationship to complete the light and shadow synchronous rendering of the virtual and real scenes.

9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the virtual imaging method for virtual shooting according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, A computer program is stored, and when the computer program is executed by a computer, it implements the virtual imaging method for virtual shooting according to any one of claims 1 to 7.

Images