A URP 3D video fusion method based on Unity3D

Abstract

The present invention belongs to the field of virtual reality technology, and proposes a URP three-dimensional video fusion method based on Unity3D. The present invention includes three-dimensional video fusion, changing virtual camera properties, changing projection video properties, multi-projector splicing, controlling projection penetration, shielding video black edges, and parameter adjustment panel functions. The present invention is based on the Unity3D engine, and by simulating the projection of a surveillance camera, the depth information is used to seamlessly integrate the surveillance screen with the 3D model, achieving the effect of combining virtuality and reality, and writing scripts to complete various specific functions of the video fusion system. The present invention has low cost, high controllability, and can be developed across platforms. Data does not need to pass through a third-party platform, and privacy is more secure. At the same time, compared with traditional solutions, development is simpler and parameter adjustment is more convenient, providing a reliable solution for video fusion developers.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a URP three-dimensional video fusion method based on Unity3D. Background Art

[0002] Nowadays, surveillance can be seen everywhere, which greatly guarantees the safety of people's lives, but it is not easy to extract effective information from the vast surveillance images. Because 3D data has advantages such as intuitiveness and comprehensiveness compared to 2D data, 3D data is gaining more and more attention and is being used more and more widely. At the same time, with the development of technologies such as the metaverse and virtual reality, three-dimensional video fusion will become necessary, that is, seamlessly integrating 2D surveillance images with virtual three-dimensional scenes. If this technology is further combined with currently popular technologies such as digital twins or artificial intelligence, it will create more value and have broad prospects.

[0003] Virtual reality technology (VR for short), also known as virtual environment, spiritual environment or artificial environment, refers to the technology of using computers to generate a virtual world that can directly exert visual, auditory and tactile sensations on participants and allow them to observe and operate interactively.

[0004] Video fusion technology is a branch of virtual reality technology, and it can also be said to be a development stage of virtual reality. Video fusion technology refers to the fusion of one or more image sequence videos of a scene or model captured by a video acquisition device with a related virtual scene to generate a new virtual scene or model of this scene. The addition of video to this model results in an information combination that neither the virtual scene nor the video itself can complete alone. The purpose of video fusion is to increase the interactivity between virtual scenes and reality, reduce the uncertainty of information in computer models, increase the information carrying capacity of virtual models, build a bridge between reality and virtuality, and expand the application field of virtual reality technology.

[0005] For these application scenarios, some companies have developed corresponding software or website systems (a WebGL-based image projection fusion method, patent publication number: CN113283543A) for developers to use, but the video fusion systems developed in this way are mostly run on the Web side, which cannot give full play to the full performance of the client, and need to be developed with the help of a third-party platform. Not only does it require a huge fee, but the various data in the development system will also pass through the third-party platform, which has certain privacy and security risks and is greatly restricted. At the same time, video fusion technology often belongs to a subsystem under the digital twin system. When developing video fusion technology, it is often necessary to develop the basic functions of the digital twin system first, which invisibly increases the development cost. Since the existing video fusion methods (such as video fusion methods based on three-dimensional scenes) are closed source, and the methods are complex, involving a lot of computer graphics knowledge, it will be challenging to develop video fusion methods from scratch. In order to solve these problems, the present invention will be developed with the help of the Unity3D open source engine to break the deadlock.

[0006] Unity3D engine is an open source visual development platform that can create 3D games and projects in real time. The platform provides a complete set of software solutions, supporting platforms including mobile phones, tablets, PCs, game consoles, augmented reality and virtual reality devices. The platform has a mature visual interface, physical engine system and built-in integrated library. Developers can often complete the construction of scenes and the implementation of specific logic and functions through a small number of scripts and by dragging or adding state machines. Developers can also write or modify related plug-ins and other advanced operations to achieve specific functions required. When the project is running, it can easily and quickly modify and debug the parameters of the project in real time, quickly modify the level of refinement of model loading, and connect to the database and backend. Unity3D not only has a complete, mature and fast development mode, but also supports compatibility with all platforms. Projects running on the PC side often only need to modify a small part of the code to run on Android, Mac, Web, and other platforms. Summary of the invention

[0007] The purpose of the present invention is to provide a URP three-dimensional video fusion method based on Unity3D, based on the Unity3D engine, by simulating the projection of a surveillance camera, using depth information to seamlessly merge the surveillance image with the 3D model, achieving the effect of combining virtuality and reality, and writing scripts to complete various specific functions of the video fusion system. The development platform used in the present invention is Unity3D, which has low cost, can be developed across platforms, supports third-party plug-ins, has high controllability, and data does not need to pass through a third-party platform, and privacy is more secure.

[0008] The technical solution of the present invention is: a URP three-dimensional video fusion method based on Unity3D, including three-dimensional video fusion, changing virtual camera properties, changing projection video properties, multi-projector splicing, controlling projection penetration, shielding video black edges and parameter adjustment panel; the three-dimensional video fusion function uses depth information to seamlessly integrate the monitoring screen with the scene 3D model by simulating the projection of a monitoring camera; changing the virtual camera properties includes changing the position, rotation angle or focal length of the simulated monitoring camera; changing the projection video properties is to change the projection aspect ratio and projection transparency, and to cut the projection; multi-projector splicing is multi-projector splicing projection, changing the projection layer order, and multi-projector combined movement; controlling projection penetration is to prevent the blocked part from being penetrated by the projection, and hierarchically controlling the model layer receiving the projection; shielding the video black edges is to shield the irregular black edges according to the projection transparency Mask; a humanized parameter adjustment panel, the parameters of which can be saved in a json file in real time.

[0009] The three-dimensional video fusion is specifically as follows:

[0010] Step 1.1, create an empty game object in the scene as a virtual camera;

[0011] Step 1.2, create a custom depth shader file for drawing the depth texture of each virtual camera; write a state control statement in the shader to control the rendering state of the virtual camera output, specifically set to backface culling, turn off depth writing, and always pass the depth test; create a custom fragment function to convert the depth texture passed by the virtual camera into a linear depth value, expressed as depthValue, and its calculation formula is:

[0012] depthValue=Linear01Depth(SAMPLE_TEXTURE2D_X(_CameraDepthTexture,

[0013] sampler_CameraDepthTexture,screenPos).r,_ZBufferParams)

[0014] Among them, Linear01Depth() is the 0-1 linear depth conversion function of the Unity editor, SAMPLE_TEXTURE2D_X() is the texture sampling function of the Unity editor, _CameraDepthTexture is the depth texture of the virtual camera, sampler_CameraDepthTexture is the camera depth texture sampler, screenPos is the viewport coordinate of the screen, r is the red channel, and _ZBufferParams is the linearized depth buffer area parameter;

[0015] Step 1.3, create a custom depth post-processing script, follow the URP post-processing rules; in the depth post-processing script, define whether the depth drawing function is enabled;

[0016] Step 1.4, custom create a depth rendering pass script and a depth rendering pass function script to add the depth drawing function to the URP; in the depth rendering pass script, obtain the post-processing stack through the Volume framework to obtain whether the virtual camera has enabled depth drawing; use the Blit() function of the Unity editor, use the material of the depth shader file created in step 1.2, and perform special rendering on the output of the virtual camera so that the output is a depth map; in the depth rendering pass function script, initialize the depth rendering pass in the Create() function of the Unity editor, and add the depth rendering pass to the URP rendering pipeline in the AddRenderPasses() function of the Unity editor;

[0017] Step 1.5, add the depth rendering channel function script in step 1.4 to the rendering function of the forward rendering asset of the URP; enable the depth texture function in the rendering pipeline asset of the URP; the URP rendering pipeline has the function of depth rendering in the post-processing stage;

[0018] Step 1.6, enable the post-processing option of the virtual camera Camera component, create a cube sub-object for each virtual camera to limit the scope of local post-processing; uncheck the grid rendering so that each cube sub-object is not rendered; bind the Volume script provided by the post-processing module in URP to the virtual camera, and add the previously customized depth drawing post-processing effect to the Volume script; the virtual camera has the function of depth drawing;

[0019] Step 1.7, add the Video Player component of the Unity editor to the virtual camera, set the corresponding video source information; set the material attribute of the Video Player component to the main texture; the Unity editor automatically transfers the main texture attribute of the current object material to the picture information of the video;

[0020] Step 1.8, create a custom projection control script and add the projection control script to the virtual camera; define the camera rendering texture; initialize the camera rendering texture in the Start() function of the Unity editor; the length and width of the camera rendering texture are initialized to 1024x1024, and the image format is initialized to the rendering texture of R32_SFloat; the initialized camera rendering texture is assigned to the target texture attribute of the virtual camera Camera component as the output texture of the virtual camera; pass the output texture to the projected material through the SetTexture() function of the Unity editor;

[0021] Step 1.9, in order to realize the interaction between the virtual camera and the three-dimensional scene, the virtual camera is made to have a mesh entity; in the projection control script of step 1.8, the custom virtual camera update function is used to initialize and update the data of the virtual camera; in the camera update function, the FOV, aspect ratio, near and far plane information of the virtual camera Camera component are obtained, and a mesh entity of the virtual camera is generated. The mesh entity is used as a projection medium to limit the projection range; at the same time, the observation transformation matrix and projection matrix provided by the Unity editor are obtained through the virtual camera Camera component, and the two are multiplied to obtain the observation projection matrix VP, and the observation projection matrix VP is passed to the projection material;

[0022] Step 1.10, create a custom projection file to bind the material of the mesh entity in step 1.9; in the vertex function of the Unity editor, transform the vertex to the observation space of the main camera and output it as the input of the next fragment function, expressed as o.ray, and its formula is as follows:

[0023] o.ray=mul(UNITY_MATRIX_MV,v.vertex).xyz*float3(1,1,-1);

[0024] Among them, mul is matrix multiplication, UNITY_MATRIX_MV is the model perspective transformation matrix of the Unity editor, v.vertex is the input vertex, and the float3(1,1,-1) vector is used to negate the result coordinate z;

[0025] Step 1.11, in the fragment function of the projection shader file of step 1.10, first project all vertices onto the original plane of the main view camera, then obtain the depth map of the main view camera and sample it; parse the main view camera depth map through the Linear01Depth() function of the Unity editor to obtain a linear depth value of 0-1; multiply all vertices by the obtained linear depth value, and then project the vertices to the surface of each object that the mesh entity of step 1.8 touches in the scene; use the clip function of the Unity editor to remove the projection area that does not meet the conditions, and restore the intersection of the mesh entity and the scene in the world space, that is, the projection area, for subsequent video screen output. The calculation formula is as follows:

[0026] clip(float2(clipX,clipY)-abs(opos.xy));

[0027] Among them, clip() is the culling function of the Unity editor. When any component of the parameter vector or number passed in is less than 0, the fragment is culled. float2 is a two-dimensional vector. clipX and clipY are the positions of the projection point in the mesh entity model space. abs() is the absolute value function of the Unity editor. opos.xy is the position of the projection point in the mesh entity model space. When the position of the projection point exceeds the model range, it will be culled.

[0028] Step 1.12, in the fragment function of the projection shader file of step 1.10, manually simulate the process of projecting the projection point to the virtual camera according to the observation projection matrix VP passed in step 1.9 and the output position information of the projection point obtained in step 1.11, and obtain the viewport coordinates of the projection point in the virtual camera screen space, expressed as projUV, and the calculation formula is as follows:

[0029] float2 projUV=clipPos.xy*0.5+0.5;

[0030] Among them, clipPos.xy is the normalized device coordinate system position of the projection point calculated based on the virtual camera;

[0031] Use the viewport coordinates as UV and sample the camera rendering texture passed in step 1.8 to get the depth value, expressed as depth2, which is calculated as follows:

[0032] float depth2=tex2D(_DepthTexture,projUV).x;

[0033] Among them, tex2D is the sampling function of the Unity editor, and _DepthTexture is the rendering texture carrying depth information of the virtual camera;

[0034] After obtaining the depth information relative to the virtual camera, use the clip function to remove the fragments greater than this depth to achieve the occlusion effect. The calculation formula is as follows:

[0035] clip(depth2-opos.z / FarDistance+_DepthThreshold);

[0036] Among them, clip is the culling function. When its parameter is less than 0, the fragment is culled. opos.z is the depth of the projection point in the virtual camera model space. FarDistance is the far plane value of the virtual camera. _DepthThreshold is the culling threshold defined by the user to correct the overall culling effect.

[0037] The adjusted viewport coordinates are used as the video texture passed in the UV sampling step 1.7 and output to complete the projection of the basic video image to the three-dimensional scene; the calculation formula is as follows:

[0038] return float4(tex2D(_MainTex,mainUV),_Alpha*maskAlpha);

[0039] Among them, float4 is a four-dimensional vector, tex2D is a sampling function, _MainTex is a video texture, mainUV is the viewport coordinate UV modified by the user, _Alpha is the projection transparency passed in by the user, and maskAlpha is the result of sampling the mask texture.

[0040] The specific modification of the virtual camera attributes is as follows:

[0041] Step 2.1, move the surveillance camera position; modify the position by adjusting the Position property of the virtual camera transformation component;

[0042] Step 2.2, change the rotation angle of the surveillance camera; modify the rotation angle by adjusting the Rotation property of the virtual camera Transform component;

[0043] Step 2.3, change the focal length of the surveillance camera; add a Camera component to the virtual camera, and modify the focal length by adjusting the FOV value, that is, the virtual camera performs proportional scaling projection.

[0044] The specific method of changing the projection video attribute is as follows:

[0045] Step 3.1, call the projection control script in step 1.8, define the projection aspect ratio, projection transparency, and cropping variables;

[0046] Step 3.2, change the projection aspect ratio; in the camera update function of the projection control script, use the Mathf.Max function of the Unity editor to get the larger value of the projection aspect ratio variable and 0.01 to ensure that the projection aspect ratio is always positive; modify the built-in aspect property of the Camera component to directly modify the projection aspect ratio;

[0047] Step 3.3, change the projection transparency; in the projection shader file of step 1.10, turn off depth writing, turn on the default transparency blending, and set the blending factor; in the camera update function of the projection control script of step 1.8, use the SetFloat() function of the Unity editor to pass the projection transparency to the projection material; and in the final output stage of the fragment function of the projection shader file, multiply the projection transparency passed in by the projection control script to modify the final projection transparency;

[0048] Step 3.4, clip the projection; in the camera update function of the projection control script in step 1.8, use the SetFloa() function of the Unity editor to pass the U direction clipping value and the V direction clipping value to the projection material; in the fragment function of the projection shader file in step 1.10, use the clip function to perform a certain regular operation on the incoming UV direction clipping value, remove any projection points that do not meet the conditions, and then clip the projection. The calculation formula is as follows:

[0049] clip(mainUV.x-_uMin);

[0050] clip(_uMax-mainUV.x);

[0051] clip(mainUV.y-_vMin);

[0052] clip(_vMax-mainUV.y);

[0053] Among them, the clip function is the culling function, mainUV is the coordinate of the projection point in the viewport space, _uMin is the minimum value in the U direction, and the points whose x coordinate in the viewport space is less than this value are eliminated, _uMax is the maximum value in the U direction, and the points whose x coordinate in the viewport space is greater than this value are eliminated. The judgment principle of the V direction is the same as that of the U direction.

[0054] The multi-projector splicing is specifically as follows:

[0055] Step 4.1, stitching projections of multiple projectors; drag the virtual camera object from the hierarchy panel of the Unity editor to the Project panel to create a prefab; then drag the prefab into the scene panel to create one or more virtual cameras; stitch the projection images together by adjusting the parameters;

[0056] Step 4.2, change the projection layer order; in the projection control script of step 1.8, define the rendering priority priority; in the camera update function, set the rendering queue of the mesh entity. The specific calculation formula is as follows:

[0057] projMat.renderQueue=2001+priority;

[0058] Among them, projMat is the material corresponding to the mesh entity, renderQueue is the rendering queue. In the Unity editor, the larger the rendering queue, the later the object will be rendered; priority is the priority, and there is a rendering order between different projections, so as to achieve the effect of changing the layer order;

[0059] Step 4.3, combined movement of multiple projectors; create an empty game object as a parent object, and bind the same parent object to multiple projectors; when the parent object moves, all child objects will move together to achieve combined movement of multiple projectors.

[0060] The control projection penetration is specifically as follows:

[0061] Step 5.1, the blocked part is not penetrated by the projection; create a model for blocking the projection, assign it any destination material, the destination material cannot be a transparent material, because the transparent material has the depth writing function turned off by default; set the rendering priority of the destination material to a large value; the larger the better;

[0062] Step 5.2, hierarchically control the model layer that receives the projection; in the Layer panel in the Unity editor, create an Ignore layer, which prevents the 3D model from being blocked; select the Camera component of the virtual camera, and add the Ignore layer in its Culling Mask property; increase the rendering priority of the 3D model to a level that is not affected by the projection effect.

[0063] The specific method of shielding the black edge of the video is as follows:

[0064] Step 6.1, create a Mask map with a size of 1:1 with the projected video, where 1 represents completely opaque, 0 represents completely transparent, the pixel positions of the black edges of the video are set to 0, and the rest of the positions are set to 1;

[0065] Step 6.2, define the mask map in the projection control script of step 1.8; in the camera update function, use the SetTexture() function of the Unity editor to pass the Mask map to the projection material;

[0066] Step 6.3, in the fragment function of the projection shader file in step 1.10, use the viewport coordinate UV in step 1.12 to sample the r channel of the Mask map, and modify the projection transparency by multiplying it by the r channel value in the final fragment output stage to achieve the shielding effect.

[0067] The parameter adjustment panel is specifically:

[0068] Step 7.1, create a custom projection editor extension script to extend the editor panel to make the parameter panel more readable for users;

[0069] Step 7.2, define several serialized attributes that need to be stored in the local hard disk in the projection control script of step 1.8;

[0070] Step 7.3, in the projection editor extension script, bind and initialize each serialized attribute in step 7.2 through the OnEnabl() function of the Unity editor, rewrite the OnInspectorGUI() function of the Unity editor to define the rendering rules of the parameter panel, and implement Chinese localization and mouse hover prompts for all exposed parameters in this function;

[0071] Step 7.4, customize the file creation tool script to implement text file reading and writing operations;

[0072] Step 7.5, create a projection Json configuration script, add the serializable feature to it, declare the script as serializable storage, and redefine the variables in the projection control script in step 1.8 to be serialized to the local disk in the configuration script;

[0073] Step 7.6, create a read function in the projection control script of step 1.8; use the file tool script of step 7.4 to read the text file; then use the JsonUtility() function of the Unity editor to convert the text file content into a Json configuration script object; take the value in the Json configuration script object and assign it to the corresponding variable in the projection control script to complete the data reading operation; create a write function, first assign the data of the projection control script to the Json configuration script object, and then the file tool script is responsible for writing the Json configuration script object to the local disk.

[0074] Beneficial effects of the present invention: The present invention provides a URP three-dimensional video fusion method based on Unity3D. Based on the Unity3D engine, by simulating the projection of a surveillance camera, the depth information is used to seamlessly merge the surveillance image with the 3D model, and scripts are written to complete various specific functions of the video fusion system. The Unity3D used in this solution is an open source engine with low cost; the platform is powerful, has excellent visual effects, and is convenient for cross-platform development (PC, Web, Android); the controllability is high, and the data does not need to pass through a third-party platform, and the privacy is more secure; the present invention is based on the URP rendering pipeline, and implements highly customized programming of the rendering pipeline to achieve video fusion effects; the focal length, aspect ratio, and transparency of the projection can be changed; the projection can be cropped; the irregular black edges can be shielded according to the transparency Mask of the projection; it has a humanized parameter adjustment panel, and the parameters can be saved in real time in a json file; it can realize multi-projection splicing and multi-projector combination functions; it can control whether the object is penetrated by the projection, and the model layer receiving the projection can be hierarchically controlled to enable video fusion. DETAILED DESCRIPTION

[0075] The present invention is further described below.

[0076] Embodiment 1: 3D video fusion.

[0077] The 3D video fusion function of the present invention uses depth information to seamlessly fuse the monitoring screen with the 3D model by simulating the projection of the monitoring camera, thereby achieving a virtual-real combination effect. The user can change the viewing angle to observe the video fusion area in more detail.

[0078] In the present invention, the part blocked by the 3D model will not be penetrated by the projection, and the model layer that accepts the projection can be controlled in layers, and small objects such as trees and furniture can be set not to accept the projection. The cube does not accept the projection, that is, whether to accept the projection is controlled in layers.

[0079] In the present invention, the black edge of the projected video can be shielded. The transparency of the projected video is changed by a mask with a size of 1:1 with the projected video, thereby shielding the black edge of the projected video. In the Mask map, 1 represents completely opaque, 0 represents completely transparent, and the default is all 1. In the Mask map, the pixel position of the black edge of the video should be set to 0.

[0080] Example 2: Change virtual camera and projected video properties.

[0081] The present invention can change the properties of the virtual camera. The position can be modified by adjusting the Position property of the virtual camera Transform component; the rotation angle can be modified by adjusting the Rotation property of the virtual camera Transform component; and the focal length can be modified by adjusting the FOV value of the Camera component, that is, a geometric zoom effect.

[0082] The present invention can change the properties of the projection video, use the UV of the material to change the aspect ratio of the projection video or cut the projection, and use the alpha variable to change the transparency of the projection video.

[0083] Embodiment 3: splicing and combining multiple projectors.

[0084] The present invention can realize the function of splicing and combining multiple projectors. Multiple virtual cameras are created and appropriate parameters are adjusted to splice the projection images together, so as to realize the function of splicing multiple projectors. The layer order of the projected video can be changed by adjusting the rendering priority. By binding the same parent object to multiple projectors, the function of combining multiple projectors can be realized.

Claims

1. A URP three-dimensional video fusion method based on Unity3D, characterized in that: The URP 3D video fusion method based on Unity3D includes 3D video fusion, changing virtual camera properties, changing projection video properties, multi-projector splicing, controlling projection penetration, shielding video black edges and adjusting parameters panel; the 3D video fusion function uses depth information to seamlessly merge the monitoring screen with the scene 3D model by simulating the projection of the monitoring camera; Changing virtual camera properties includes changing the simulated surveillance camera position, rotation angle or focal length; Change the projection video properties to change the projection aspect ratio, projection transparency, and crop the projection; Multi-projector stitching includes multi-projector stitching projection, changing the order of projection layers, and moving multiple projectors together; controlling projection penetration includes preventing the blocked part from being penetrated by the projection, and controlling the model layer that receives the projection in layers; shielding the black edges of the video includes shielding the irregular black edges according to the projection transparency Mask; The three-dimensional video fusion is specifically as follows: Step 1.1, create an empty game object in the scene as a virtual camera; Step 1.2, create a custom depth shader file for drawing the depth texture of each virtual camera; Write state control statements in the shader to control the rendering state of the virtual camera output. Specifically, set it to backface culling, turn off depth writing, and always pass the depth test; customize the creation of fragment functions to convert the depth texture passed by the virtual camera into a linear depth value, expressed as depthValue, whose calculation formula is: depthValue=Linear01Depth(SAMPLE_TEXTURE2D_X(_CameraDepthTexture, sampler_CameraDepthTexture,screenPos).r,_ZBufferParams) Among them, Linear01Depth() is the 0-1 linear depth conversion function of the Unity editor, SAMPLE_TEXTURE2D_X() is the texture sampling function of the Unity editor, _CameraDepthTexture is the depth texture of the virtual camera, sampler_CameraDepthTexture is the camera depth texture sampler, screenPos is the viewport coordinate of the screen, r is the red channel, and _ZBufferParams is the linearized depth buffer area parameter; Step 1.3, create a custom depth post-processing script, follow the URP post-processing rules; in the depth post-processing script, define whether the depth drawing function is enabled; Step 1.4, custom create a depth rendering pass script and a depth rendering pass function script to add the depth drawing function to the URP; in the depth rendering pass script, obtain the post-processing stack through the Volume framework to obtain whether the virtual camera has enabled depth drawing; use the Blit() function of the Unity editor, use the material of the depth shader file created in step 1.2, and perform special rendering on the output of the virtual camera so that the output is a depth map; in the depth rendering pass function script, initialize the depth rendering pass in the Create() function of the Unity editor, and add the depth rendering pass to the URP rendering pipeline in the AddRenderPasses() function of the Unity editor; Step 1.5, add the depth rendering channel function script in step 1.4 to the rendering function of the forward rendering asset of the URP; enable the depth texture function in the rendering pipeline asset of the URP; the URP rendering pipeline has the function of depth rendering in the post-processing stage; Step 1.6, enable the post-processing option of the virtual camera Camera component, create a cube sub-object for each virtual camera to limit the scope of local post-processing; uncheck the grid rendering so that each cube sub-object is not rendered; bind the Volume script provided by the post-processing module in URP to the virtual camera, and add the previously customized depth drawing post-processing effect to the Volume script; the virtual camera has the function of depth drawing; Step 1.7, add the Video Player component of the Unity editor to the virtual camera, set the corresponding video source information; set the material attribute of the Video Player component to the main texture; the Unity editor automatically transfers the main texture attribute of the current object material to the picture information of the video; Step 1.8, create a custom projection control script and add the projection control script to the virtual camera; define the camera rendering texture; initialize the camera rendering texture in the Start() function of the Unity editor; the length and width of the camera rendering texture are initialized to 1024x1024, and the image format is initialized to the rendering texture of R32_SFloat; the initialized camera rendering texture is assigned to the target texture attribute of the virtual camera Camera component as the output texture of the virtual camera; pass the output texture to the projected material through the SetTexture() function of the Unity editor; Step 1.9, in order to realize the interaction between the virtual camera and the three-dimensional scene, the virtual camera is made to have a mesh entity; in the projection control script of step 1.8, the custom virtual camera update function is used to initialize and update the data of the virtual camera; in the camera update function, the FOV, aspect ratio, near and far plane information of the virtual camera Camera component are obtained, and a mesh entity of the virtual camera is generated. The mesh entity is used as a projection medium to limit the projection range; at the same time, the observation transformation matrix and projection matrix provided by the Unity editor are obtained through the virtual camera Camera component, and the two are multiplied to obtain the observation projection matrix VP, and the observation projection matrix VP is passed to the projection material; Step 1.10, create a custom projection file to bind the material of the mesh entity in step 1.9; in the vertex function of the Unity editor, transform the vertex to the observation space of the main camera and output it as the input of the next fragment function, expressed as o.ray, and its formula is as follows: o.ray=mul(UNITY_MATRIX_MV,v.vertex).xyz*float3(1,1,-1) Among them, mul is matrix multiplication, UNITY_MATRIX_MV is the model perspective transformation matrix of the Unity editor, v.vertex is the input vertex, and the float3(1,1,-1) vector is used to negate the result coordinate z; Step 1.11, in the fragment function of the projection shader file of step 1.10, first project all vertices onto the original plane of the main view camera, then obtain the depth map of the main view camera and sample it; parse the main view camera depth map through the Linear01Depth() function of the Unity editor to obtain a linear depth value of 0-1; multiply all vertices by the obtained linear depth value, and then project the vertices to the surface of each object that the mesh entity of step 1.8 touches in the scene; use the clip function of the Unity editor to remove the projection area that does not meet the conditions, and restore the intersection of the mesh entity and the scene in the world space, that is, the projection area, for subsequent video screen output. The calculation formula is as follows: clip(float2(clipX,clipY)-abs(opos.xy)) Among them, clip is the culling function of the Unity editor. When any component of the parameter vector or number passed in is less than 0, the fragment is culled. float2 is a two-dimensional vector. clipX and clipY are the positions of the projection point in the mesh entity model space. abs is the absolute value function of the Unity editor. opos is the position of the projection point in the mesh entity model space. When the position of the projection point exceeds the model range, it will be culled. Step 1.12, in the fragment function of the projection shader file of step 1.10, manually simulate the process of projecting the projection point to the virtual camera according to the observation projection matrix VP passed in step 1.9 and the output position information of the projection point obtained in step 1.11, and obtain the viewport coordinates of the projection point in the virtual camera screen space, expressed as projUV, and the calculation formula is as follows: float2 projUV=clipPos.xy*0.5+0.5 Among them, clipPos.xy is the normalized device coordinate system position of the projection point calculated based on the virtual camera; Use the viewport coordinates as UV and sample the camera rendering texture passed in step 1.8 to get the depth value, expressed as depth2, which is calculated as follows: float depth2=tex2D(_DepthTexture,projUV).x Among them, tex2D is the sampling function of the Unity editor, and _DepthTexture is the rendering texture carrying depth information of the virtual camera; After obtaining the depth information relative to the virtual camera, use the clip function to remove the fragments greater than this depth to achieve the occlusion effect. The calculation formula is as follows: clip(depth2-opos.z / FarDistance+_DepthThreshold) Among them, clip is the culling function. When its parameter is less than 0, the fragment is culled. opos.z is the depth of the projection point in the virtual camera model space. FarDistance is the far plane value of the virtual camera. _DepthThreshold is the culling threshold defined by the user to correct the overall culling effect. The adjusted viewport coordinates are used as the video texture passed in the UV sampling step 1.7 and output to complete the projection of the basic video image to the three-dimensional scene; the calculation formula is as follows: return float4(tex2D(_MainTex,mainUV),_Alpha*maskAlpha) Among them, float4 is a four-dimensional vector, tex2D is a sampling function, _MainTex is a video texture, mainUV is the viewport coordinate UV modified by the user, _Alpha is the projection transparency passed in by the user, and maskAlpha is the result of sampling the mask texture.

2. The URP three-dimensional video fusion method based on Unity3D according to claim 1, characterized in that: The specific modification of the virtual camera attributes is as follows: Step 2.1, move the surveillance camera position; modify the position by adjusting the Position property of the virtual camera transformation component; Step 2.2, change the rotation angle of the surveillance camera; modify the rotation angle by adjusting the Rotation property of the virtual camera Transform component; Step 2.3, change the focal length of the surveillance camera; add a Camera component to the virtual camera, and modify the focal length by adjusting the FOV value, that is, the virtual camera performs proportional scaling projection.

3. The URP three-dimensional video fusion method based on Unity3D according to claim 1, characterized in that: The specific method of changing the projection video attribute is as follows: Step 3.1, call the projection control script in step 1.8, define the projection aspect ratio, projection transparency, and cropping variables; Step 3.2, change the projection aspect ratio; in the camera update function of the projection control script, use the Mathf.Max function of the Unity editor to get the larger value of the projection aspect ratio variable and 0.01 to ensure that the projection aspect ratio is always positive; modify the built-in aspect property of the Camera component to directly modify the projection aspect ratio; Step 3.3, change the projection transparency; in the projection shader file of step 1.10, turn off depth writing and turn on the default transparency blending; in the camera update function of the projection control script of step 1.8, use the SetFloat() function of the Unity editor to pass the projection transparency to the projection material; and in the final output stage of the fragment function of the projection shader file, multiply the projection transparency passed in by the projection control script to modify the final projection transparency; Step 3.4, clip the projection; in the camera update function of the projection control script in step 1.8, use the SetFloa() function of the Unity editor to pass the U direction clipping value and the V direction clipping value to the projection material; in the fragment function of the projection shader file in step 1.10, use the clip function to perform a certain regular operation on the incoming UV direction clipping value, remove any projection points that do not meet the conditions, and then clip the projection. The calculation formula is as follows: clip(mainUV.x-_uMin) clip(_uMax-mainUV.x) clip(mainUV.y-_vMin) clip(_vMax-mainUV.y) Among them, the clip function is the culling function, mainUV is the coordinate of the projection point in the viewport space, _uMin is the minimum value in the U direction, and the points whose x coordinate in the viewport space is less than this value are eliminated, _uMax is the maximum value in the U direction, and the points whose x coordinate in the viewport space is greater than this value are eliminated. The judgment principle of the V direction is the same as that of the U direction.

4. The URP three-dimensional video fusion method based on Unity3D according to claim 3, characterized in that: The multi-projector splicing is specifically as follows: Step 4.1, stitching projections of multiple projectors; drag the virtual camera object from the hierarchy panel of the Unity editor to the Project panel to create a prefab; then drag the prefab into the scene panel to create one or more virtual cameras; stitch the projection images together by adjusting the parameters; Step 4.2, change the projection layer order; in the projection control script of step 1.8, define the rendering priority priority; in the camera update function, set the rendering queue of the mesh entity. The specific calculation formula is as follows: projMat.renderQueue=2001+priority Among them, projMat is the material corresponding to the mesh entity, renderQueue is the rendering queue. In the Unity editor, the larger the rendering queue, the later the object will be rendered; priority is the priority, and there is a rendering order between different projections, so as to achieve the effect of changing the layer order; Step 4.3, combined movement of multiple projectors; create an empty game object as a parent object, and bind the same parent object to multiple projectors; when the parent object moves, all child objects will move together to achieve combined movement of multiple projectors.

5. The URP three-dimensional video fusion method based on Unity3D according to claim 4, characterized in that: The control projection penetration is specifically as follows: Step 5.1: The blocked part is not penetrated by the projection. Create a model for blocking the projection and assign it any destination material. The destination material cannot be a transparent material. Set the rendering priority of the destination material to a large value. Step 5.2, hierarchically control the model layer that receives projection; in the Layer panel in the Unity editor, create an Ignore layer, which prevents the 3D model from being blocked; select the Camera component of the virtual camera, and add the Ignore layer in its CullingMask property; increase the rendering priority of the 3D model to a level that is not affected by the projection effect.

6. The URP three-dimensional video fusion method based on Unity3D according to claim 5, characterized in that: The specific method of shielding the black edge of the video is as follows: Step 6.1, create a Mask map with a size of 1:1 with the projected video, where 1 represents completely opaque, 0 represents completely transparent, the pixel positions of the black edges of the video are set to 0, and the remaining positions are set to 1; Step 6.2, define the mask map in the projection control script of step 1.8; in the camera update function, use the SetTexture() function of the Unity editor to pass the Mask map to the projection material; Step 6.3, in the fragment function of the projection shader file in step 1.10, use the viewport coordinate UV in step 1.12 to sample the r channel of the Mask map, and modify the projection transparency by multiplying it by the r channel value in the final fragment output stage to achieve the shielding effect.

7. The URP three-dimensional video fusion method based on Unity3D according to claim 6, characterized in that: The parameter adjustment panel is specifically: Step 7.1, create a custom projection editor extension script to extend the editor panel to make the parameter panel more readable for users; Step 7.2, define several serialized attributes that need to be stored in the local hard disk in the projection control script of step 1.8; Step 7.3, in the projection editor extension script, bind and initialize each serialized attribute in step 7.2 through the OnEnabl() function of the Unity editor, rewrite the OnInspectorGUI() function of the Unity editor to define the rendering rules of the parameter panel, and implement Chinese localization and mouse hover prompts for all exposed parameters in this function; Step 7.4, customize the file creation tool script to implement text file reading and writing operations; Step 7.5, create a projection Json configuration script, add the serializable feature to it, declare the script as serializable storage, and redefine the variables in the projection control script in step 1.8 to be serialized to the local disk in the configuration script; Step 7.6, create a read function in the projection control script of step 1.8; use the file tool script of step 7.4 to read the text file; then use the JsonUtility() function of the Unity editor to convert the text file content into a Json configuration script object; take the value in the Json configuration script object and assign it to the corresponding variable in the projection control script to complete the data reading operation; create a write function, first assign the data of the projection control script to the Json configuration script object, and then the file tool script is responsible for writing the Json configuration script object to the local disk.