A method and device for combining virtual and real images

Abstract

The present invention provides a method and system for combining virtual and real images, wherein the method includes the following steps: obtaining a real-scene image and a virtual image in a studio, and obtaining a texture map according to the image; setting a three-dimensional screen model in a virtual scene; determining the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model; synthesizing the texture map and the three-dimensional model according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target image. The present invention applies the shot image as a texture to the three-dimensional model in the virtual scene, uses the three-dimensional engine to automatically perform depth sorting for virtual and real synthesis, and conveniently adjusts the positional relationship between the real-scene image and the virtual image, so that the TV director can get rid of the limitations of space and prop production, and freely create in a wide range of imaginary space, thereby improving the TV station's ability to create and produce programs, reducing the cost of program production, and improving the production quality of the virtual studio system.

Description

Technical Field

[0001] The present invention relates to the technical field of television program production, and in particular to a method and device for combining virtual and real images. Background Art

[0002] The virtual studio system is a new TV program production system that has emerged in recent years with the rapid development of computer technology and the continuous improvement of color key technology. In traditional virtual studio products, the host shoots in a blue box, and the picture is cut out and synthesized with the three-dimensional background rendered by the graphics workstation. Since the host lacks performance immersion in the blue box, the cutout synthesis method cannot provide the host with lighting effects. With the application of large LED screens in the studio, there is currently a virtual studio technical solution that uses a large screen to replace the blue box, and synthesizes the shooting picture (real scene) and the foreground virtual implanted picture (virtual scene) through pixel aliasing. However, the pixel aliasing synthesis method uses the shooting picture as the bottom layer and the implanted picture as the overlay layer, which will result in no depth relationship between the virtual and real layers of the picture, and cannot change the front and back mixed relationship between the host and the foreground object. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a method for combining virtual and real images, and a corresponding system for combining virtual and real images, which overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, on the one hand, an embodiment of the present invention discloses a method for combining virtual and real images, including:

[0005] Acquire a real-scene picture of a studio shot by a real-scene camera, and obtain a first texture map according to the real-scene picture;

[0006] Acquire a studio virtual picture shot by a virtual camera, and obtain a second texture map according to the virtual picture;

[0007] A screen three-dimensional model is set in a virtual scene, and the virtual camera is set in the virtual scene;

[0008] Determining texture map coordinates corresponding to mesh nodes in the three-dimensional model of the screen;

[0009] The first texture map, the second texture map and the three-dimensional screen model are synthesized according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target picture.

[0010] Further, the determining of the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen includes:

[0011] Acquire a first camera coordinate system corresponding to the picture captured by the real-view camera;

[0012] Calculating a first plane projection corresponding to the screen three-dimensional model according to a first viewing matrix and a first projection matrix corresponding to the first camera coordinate system;

[0013] Setting a virtual camera in the virtual scene, and acquiring a second camera coordinate system corresponding to a picture shot by the virtual camera;

[0014] Calculating a second plane projection corresponding to the screen three-dimensional model according to a second viewing matrix and a second projection matrix corresponding to the second camera coordinate system;

[0015] Taking the upper left corner of the screen three-dimensional model as the origin, the coordinates of each pixel in the first plane projection and the second plane projection are normalized, and the normalized coordinates are used as the texture mapping coordinates of the three-dimensional model mesh node projected to the pixel.

[0016] Further, the determining of the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen also includes:

[0017] According to a first mesh node without defined texture mapping coordinates, searching for a second mesh node adjacent to the first mesh node and with defined texture mapping coordinates;

[0018] The first texture mapping coordinates corresponding to the first mesh node are obtained by calculating through coordinate interpolation according to the second texture mapping coordinates corresponding to the second mesh node.

[0019] Furthermore, obtaining a texture map according to the real scene includes:

[0020] Performing texture source sampling on the real scene according to the texture mapping coordinates;

[0021] Performing color sampling on the real scene according to the texture map coordinates;

[0022] The texture source and color are filled into the mesh nodes corresponding to the texture map coordinates.

[0023] Furthermore, setting the screen three-dimensional model in the virtual scene includes:

[0024] According to the size of the physical screen in the studio, a three-dimensional screen model with matching size is set in the virtual scene.

[0025] Furthermore, setting a virtual camera in the virtual scene includes:

[0026] According to the resolution of the real-scene camera, a virtual camera with a matching resolution is set in the virtual scene.

[0027] On the other hand, the present invention also provides a virtual and real picture combination system, which is applied to a virtual studio, comprising:

[0028] A first texture map acquisition module is used to acquire a real-scene picture of the studio shot by a real-scene camera, and obtain a first texture map according to the real-scene picture;

[0029] A second texture map acquisition module is used to acquire a virtual studio image shot by a virtual camera, and obtain a second texture map according to the virtual image;

[0030] A screen three-dimensional model creation module, used for setting the screen three-dimensional model in a virtual scene, wherein the virtual camera is set in the virtual scene;

[0031] A texture mapping coordinate acquisition module, used to determine the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen;

[0032] The target picture synthesis module is used to synthesize the first texture map, the second texture map and the three-dimensional screen model according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target picture.

[0033] Furthermore, the texture map coordinate acquisition module includes:

[0034] A first camera coordinate system acquisition module, used to acquire a first camera coordinate system corresponding to the picture shot by the real-view camera;

[0035] A first plane projection calculation module, used to calculate a first plane projection corresponding to the screen three-dimensional model according to a first viewing matrix and a first projection matrix corresponding to the first camera coordinate system;

[0036] A second camera coordinate system acquisition module, used to set a virtual camera in the virtual scene and acquire a second camera coordinate system corresponding to the picture shot by the virtual camera;

[0037] A second plane projection calculation module, used to calculate a second plane projection corresponding to the screen three-dimensional model according to a second viewing matrix and a second projection matrix corresponding to the second camera coordinate system;

[0038] The first texture mapping coordinate acquisition submodule is used to normalize the coordinates of each pixel in the first plane projection and the second plane projection with the upper left corner of the screen three-dimensional model as the origin, and use the normalized coordinates as the texture mapping coordinates of the three-dimensional model grid node projected to the pixel.

[0039] Furthermore, the texture map coordinate acquisition module includes:

[0040] A second texture mapping coordinate acquisition submodule, configured to search, according to a first grid node without defined texture mapping coordinates, a second grid node adjacent to the first grid node with defined texture mapping coordinates;

[0041] The third texture mapping coordinate acquisition submodule is used to obtain the first texture mapping coordinates corresponding to the first mesh node by calculating through coordinate interpolation according to the second texture mapping coordinates corresponding to the second mesh node.

[0042] Furthermore, the first texture map acquisition module includes:

[0043] A first texture source sampling module, used for sampling the texture source of the real scene according to the texture mapping coordinates;

[0044] A first color sampling module, used for performing color sampling on the real scene according to the texture map coordinates;

[0045] The first filling module is used to fill the texture source and color into the grid nodes corresponding to the texture mapping coordinates.

[0046] Furthermore, the second texture map acquisition module includes:

[0047] A second texture source sampling module, used for sampling the texture source of the virtual picture according to the texture mapping coordinates;

[0048] A second color sampling module, used for performing color sampling on the virtual picture according to the texture map coordinates;

[0049] The second filling module is used to fill the texture source and color into the grid nodes corresponding to the texture mapping coordinates.

[0050] Furthermore, the size of the three-dimensional screen model is consistent with the size of the studio screen.

[0051] Furthermore, a virtual camera is provided in the virtual scene, and the resolution of the virtual camera is consistent with that of the real-scene camera.

[0052] On the other hand, an embodiment of the present invention further provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the virtual-real image combination method when executed by the processor.

[0053] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the virtual-real image combination method are implemented.

[0054] The technical solution of the present invention obtains the real scene picture of the studio, and obtains the first texture map according to the real scene picture; obtains the virtual scene picture of the studio, and obtains the second texture map according to the virtual scene; sets the screen three-dimensional model in the virtual scene; determines the texture map coordinates corresponding to the grid nodes in the screen three-dimensional model; synthesizes the first texture map, the second texture map and the three-dimensional model according to the texture map coordinates corresponding to the grid nodes in the screen three-dimensional model to obtain the target picture. The present invention uses the real scene shooting picture and the virtual picture as textures to apply to the three-dimensional model in the virtual scene, uses the three-dimensional engine to automatically perform depth sorting for virtual-real synthesis, and conveniently adjusts the positional relationship between the real scene picture and the virtual picture, so that the TV director can get rid of the limitations of space and prop production, and freely create in a wide range of imagination, thereby improving the TV station's ability to create and produce programs and reducing the cost of program production. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a method for combining virtual and real images provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of a connection solution for an XR virtual studio in an embodiment of the present invention;

[0057] Figure 3 A structural block diagram of a virtual-real image combining device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] Figure 1 A flowchart of a method for combining virtual and real images provided in an embodiment of the present invention is applied to a virtual studio. The technical solution of this method integrates and expands the technologies of VR (virtual reality) and AR (augmented reality), so it is called XR technology (Extended Reality). The virtual studio product built with this technology is called XR virtual studio. XR virtual studio is a new TV program production system that has emerged with the rapid development of computer technology and the continuous improvement of color key technology. In a virtual studio system, Figure 2 As shown, the working status information of the camera is transmitted to the graphics workstation, and the computer obtains the distance and relative position between the foreground object and the camera accordingly, and can calculate the relative position in the virtual scene. The method includes the following steps:

[0060] Step 101: obtaining a real-scene picture of a studio shot by a real-scene camera, and obtaining a first texture map according to the real-scene picture;

[0061] In this embodiment, the real-scene camera is used to shoot the real-scene picture of the host on the large screen in the studio. The image collected by the real-scene camera is converted into a digital image in the form of a standard television signal and input into the computer as a texture source. Each digital image is an M*N array in the computer. The value of each element (called a pixel) in the image of M rows and N columns is the color value of the image point. A rectangular coordinate system m, n is defined on the image, and the coordinates (m, n) of each pixel are respectively the image coordinate system coordinates with the pixel as the unit. Since (m, n) only indicates the number of columns and rows in the array where the pixel is located, the position of the pixel in the image is not expressed in physical units. Therefore, it is necessary to establish an image coordinate system expressed in physical units (mm). The image coordinate system takes a certain point in the image (the intersection of the camera optical axis and the image plane) as the origin. The X-axis and the Y-axis are parallel to the m and n axes respectively. The physical dimensions of each pixel in the X-axis and Y-axis directions are dX and dY, and the relationship between the two coordinate systems can be established.

[0062] The origin of the camera coordinate system is the optical center of the camera. The x-axis and y-axis are parallel to the X and Y axes of the image. The z-axis is the optical axis of the camera, which is perpendicular to the image plane. The intersection of the optical axis and the image plane is the origin of the image coordinate system. The rectangular coordinate system is called the first camera coordinate system. After the first camera coordinate system is established, texture source sampling and color sampling are performed on the real scene to obtain the texture map corresponding to the real scene.

[0063] Step 102: obtaining a virtual studio image captured by a virtual camera, and obtaining a second texture map according to the virtual image;

[0064] The method for obtaining the second texture map according to the virtual picture is similar to the above-mentioned step 101, and the applicant briefly explains it here, that is, the image captured by the virtual camera is converted into a digital image in the form of a standard television signal and input into the computer as a texture source. After establishing a second camera coordinate system corresponding to the virtual camera, texture source sampling and color sampling are performed on the virtual picture to obtain a second texture map corresponding to the virtual picture.

[0065] It should be noted that in order to facilitate the application of XR virtual studios and improve production efficiency, in this embodiment, the second texture map corresponding to the virtual screen can be called from a database, which is produced according to the production requirements of the studio and archived for backup.

[0066] Step 103: setting a screen three-dimensional model in a virtual scene, wherein the virtual camera is set in the virtual scene;

[0067] According to the size of the physical screen in the studio, a 3D screen model with matching size is set in the virtual scene. In this embodiment, in order to facilitate the synthesis of virtual and real images, the size of the 3D screen model is set to be consistent with the size of the physical screen. At the same time, a virtual camera is set in the virtual scene. In order to facilitate the synthesis of virtual and real images, the resolution of the virtual camera is set according to the resolution of the real camera. Preferably, the resolution of the virtual camera is the same as the resolution of the real camera.

[0068] Step 104, determining the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen;

[0069] According to the first view matrix and the first projection matrix corresponding to the first camera coordinate system, the first plane projection corresponding to the screen three-dimensional model is calculated; the second camera coordinate system corresponding to the virtual camera shooting picture is obtained; according to the second view matrix and the second projection matrix corresponding to the second camera coordinate system, the second plane projection corresponding to the screen three-dimensional model is calculated; with the upper left corner of the screen three-dimensional model as the origin, the coordinates of each pixel in the first plane projection and the second plane projection are normalized, and the normalized coordinates are used as the texture mapping coordinates of the three-dimensional model grid node projected to the pixel. Specifically, determining the texture mapping coordinates corresponding to the grid node in the screen three-dimensional model includes the following steps: a. Determine the world space coordinate system of the screen three-dimensional model; b. Perform a perspective transformation to camera space according to the view matrix; c. Perform a perspective projection transformation to projection space according to the projection matrix; d. After clipping and viewport transformation, obtain the coordinate value of the plane projection pixel.

[0070] It should be noted that the above coordinate normalization process: scale the pixel coordinates to between 0 and 1, retain the distribution of the original data, that is, take the upper left corner of the screen 3D model as the origin (0,0), divide the pixel horizontal coordinate by the resolution width as the normalized horizontal coordinate, and divide the pixel vertical coordinate by the resolution height as the normalized vertical coordinate. For example, if the virtual scene resolution is (1920, 1080) and a pixel position is (960, 540), the normalized coordinate is (0.5, 0.5).

[0071] In an optional embodiment, determining the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen further includes:

[0072] According to a first mesh node without defined texture mapping coordinates, searching for a second mesh node adjacent to the first mesh node and with defined texture mapping coordinates;

[0073] The first texture mapping coordinates corresponding to the first mesh node are obtained by calculating through coordinate interpolation according to the second texture mapping coordinates corresponding to the second mesh node.

[0074] For example: when it is necessary to define the first texture mapping coordinates corresponding to the first grid node, the second texture mapping coordinates corresponding to the second adjacent grid node are (0.61, 0.45), and the coordinate difference is (0, 0.09). Accordingly, the first texture mapping coordinates corresponding to the first grid node are (0.61, 0.36).

[0075] The process of obtaining the first texture map according to the real scene includes: sampling the texture source of the real scene according to the texture map coordinates; sampling the color of the real scene according to the texture map coordinates; and filling the texture source and color into the grid nodes corresponding to the texture map coordinates.

[0076] The process of obtaining a second texture map according to the virtual picture includes: sampling the texture source of the virtual picture according to the texture map coordinates; sampling the color of the virtual picture according to the texture map coordinates; and filling the texture source and color into the grid nodes corresponding to the texture map coordinates.

[0077] Step 105: synthesize the first texture map, the second texture map and the three-dimensional screen model according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target image.

[0078] In this application, the synthesis of real scene images and virtual images is performed using a 3D engine. Commonly used 3D engines include D3D and OpenGL 3D engines. During the synthesis process, the 3D engine automatically calculates the front and back occlusion relationship of multiple objects according to the distance. In this embodiment, the object in front of the 3D model on the screen is used as the foreground primitive, and the object behind the 3D model on the screen is used as the background primitive. The 3D engine automatically performs the correct occlusion mixing operation based on the depth sorting to complete the virtual-real synthesis.

[0079] The technical solution of the present invention obtains the real scene picture of the studio, and obtains the first texture map according to the real scene picture; obtains the virtual scene picture of the studio, and obtains the second texture map according to the virtual scene; sets the screen three-dimensional model in the virtual scene; determines the texture map coordinates corresponding to the grid nodes in the screen three-dimensional model; synthesizes the first texture map, the second texture map and the three-dimensional model according to the texture map coordinates corresponding to the grid nodes in the screen three-dimensional model to obtain the target picture. The present invention uses the real scene shooting picture and the virtual picture as textures to apply to the three-dimensional model in the virtual scene, uses the three-dimensional engine to automatically perform depth sorting for virtual-real synthesis, and conveniently adjusts the positional relationship between the real scene picture and the virtual picture, so that the TV director can get rid of the limitations of space and prop production, and freely create in a wide range of imagination, thereby improving the TV station's ability to create and produce programs and reducing the cost of program production.

[0080] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0081] In order to implement the above virtual and real picture combination method, the embodiment of the present invention further provides a virtual and real picture combination device, including:

[0082] The first texture map acquisition module 301 is used to acquire a real-scene picture of the studio shot by a real-scene camera, and obtain a texture map according to the real-scene picture;

[0083] The second texture map acquisition module 302 is used to acquire a virtual studio image shot by a virtual camera and obtain a texture map according to the virtual image;

[0084] A screen 3D model creation module 303, used to set the screen 3D model in a virtual scene, wherein the virtual camera is set in the virtual scene;

[0085] A texture mapping coordinate acquisition module 304 is used to determine the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen;

[0086] The target picture synthesis module 305 is used to synthesize the first texture map, the second texture map and the three-dimensional screen model according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target picture.

[0087] In an optional embodiment, the texture map coordinate acquisition module 304 may include:

[0088] A first camera coordinate system acquisition module, used to acquire a first camera coordinate system corresponding to the picture shot by the real-view camera;

[0089] A first plane projection calculation module, used to calculate a first plane projection corresponding to the screen three-dimensional model according to a first viewing matrix and a first projection matrix corresponding to the first camera coordinate system;

[0090] A second camera coordinate system acquisition module, used to set a virtual camera in the virtual scene and acquire a second camera coordinate system corresponding to the picture shot by the virtual camera;

[0091] A second plane projection calculation module, used to calculate a second plane projection corresponding to the screen three-dimensional model according to a second viewing matrix and a second projection matrix corresponding to the second camera coordinate system;

[0092] The first texture mapping coordinate acquisition submodule is used to normalize the coordinates of each pixel in the first plane projection and the second plane projection with the upper left corner of the screen three-dimensional model as the origin, and use the normalized coordinates as the texture mapping coordinates of the three-dimensional model grid node projected to the pixel.

[0093] In an optional embodiment, the texture map coordinate acquisition module 304 may also include:

[0094] A second texture mapping coordinate acquisition submodule, configured to search, according to a first grid node without defined texture mapping coordinates, a second grid node adjacent to the first grid node with defined texture mapping coordinates;

[0095] The third texture mapping coordinate acquisition submodule is used to obtain the first texture mapping coordinates corresponding to the first mesh node by calculating through coordinate interpolation according to the second texture mapping coordinates corresponding to the second mesh node.

[0096] Furthermore, the first texture map acquisition module 301 may include:

[0097] A first texture source sampling module, used for sampling the texture source of the real scene according to the texture mapping coordinates;

[0098] A first color sampling module, used for performing color sampling on the real scene according to the texture map coordinates;

[0099] The first filling module is used to fill the texture source and color into the grid nodes corresponding to the texture mapping coordinates.

[0100] Furthermore, the second texture map acquisition module 302 may include:

[0101] A second texture source sampling module, used for sampling the texture source of the virtual picture according to the texture mapping coordinates;

[0102] A second color sampling module, used for performing color sampling on the virtual picture according to the texture map coordinates;

[0103] The second filling module is used to fill the texture source and color into the grid nodes corresponding to the texture mapping coordinates.

[0104] In an optional embodiment, the size of the three-dimensional screen model is consistent with the size of the studio screen.

[0105] In an optional embodiment, the resolution of the virtual camera is consistent with that of the real-view camera.

[0106] Based on the above description, the technical solution of the present invention obtains the real scene picture of the studio, and obtains the first texture map according to the real scene picture; obtains the virtual picture of the studio, and obtains the second texture map according to the virtual picture; sets the screen three-dimensional model in the virtual scene; determines the texture map coordinates corresponding to the grid nodes in the screen three-dimensional model; synthesizes the first texture map, the second texture map and the three-dimensional model according to the texture map coordinates corresponding to the grid nodes in the screen three-dimensional model to obtain the target picture. The present invention applies the real scene shooting picture and the virtual picture as texture to the three-dimensional model in the virtual scene, uses the three-dimensional engine to automatically sort the depth to synthesize the virtual and the real, and conveniently adjusts the positional relationship between the real scene picture and the virtual picture, so that the TV director can get rid of the limitations of space and prop production, and freely create in a wide range of imagination space, improve the TV station's ability to create and produce programs, and reduce the cost of program production.

[0107] An embodiment of the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned virtual-real image synthesis method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0108] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned virtual-real image synthesis method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0110] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0111] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0115] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0116] The above is a detailed introduction to a virtual-real image synthesis method and a virtual-real image synthesis device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for combining virtual and real images, applied to a virtual studio, characterized in that: include: Acquire a real-scene picture of a studio shot by a real-scene camera, and obtain a first texture map according to the real-scene picture; Acquire a studio virtual picture shot by a virtual camera, and obtain a second texture map according to the virtual picture; A screen three-dimensional model is set in a virtual scene, and the virtual camera is set in the virtual scene; Determining texture mapping coordinates corresponding to mesh nodes in the three-dimensional model of the screen, including: Acquire a first camera coordinate system corresponding to the picture captured by the real-view camera; Calculating a first plane projection corresponding to the screen three-dimensional model according to a first viewing matrix and a first projection matrix corresponding to the first camera coordinate system; Acquire a second camera coordinate system corresponding to the image captured by the virtual camera; Calculating a second plane projection corresponding to the screen three-dimensional model according to a second viewing matrix and a second projection matrix corresponding to the second camera coordinate system; Taking the upper left corner of the screen three-dimensional model as the origin, normalizing the coordinates of each pixel in the first plane projection and the second plane projection, and using the normalized coordinates as the texture mapping coordinates of the three-dimensional model mesh node projected to the pixel; The first texture map, the second texture map and the three-dimensional screen model are synthesized according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target picture.

2. The method according to claim 1, characterized in that: The determining of the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen also includes: According to a first mesh node without defined texture mapping coordinates, searching for a second mesh node adjacent to the first mesh node and with defined texture mapping coordinates; The first texture mapping coordinates corresponding to the first mesh node are obtained by calculating through coordinate interpolation according to the second texture mapping coordinates corresponding to the second mesh node.

3. The method according to claim 1, characterized in that The obtaining of a first texture map according to the real scene comprises: Performing texture source sampling on the real scene according to the first texture mapping coordinates; Performing color sampling on the real scene according to the first texture mapping coordinates; The texture source and color are filled into the mesh nodes corresponding to the first texture mapping coordinates.

4. The method according to claim 1, characterized in that The obtaining of a second texture map according to the virtual picture comprises: Performing texture source sampling on the virtual picture according to the second texture mapping coordinates; Performing color sampling on the virtual picture according to the second texture map coordinates; The texture source and color are filled into the mesh nodes corresponding to the second texture mapping coordinates.

5. The method according to claim 1, characterized in that The step of setting the screen three-dimensional model in the virtual scene comprises: According to the size of the physical screen in the studio, setting a three-dimensional screen model with matching size in the virtual scene; The virtual camera is arranged in the virtual scene, and includes: According to the resolution of the real-scene camera, a virtual camera with a matching resolution is set in the virtual scene.

6. A virtual and real picture combination system, applied to a virtual studio, characterized in that: include: A first texture map acquisition module is used to acquire a real-scene picture of the studio shot by a real-scene camera, and obtain a first texture map according to the real-scene picture; A second texture map acquisition module is used to acquire a virtual studio image shot by a virtual camera, and obtain a second texture map according to the virtual image; A screen three-dimensional model creation module, used for setting the screen three-dimensional model in a virtual scene, wherein the virtual camera is set in the virtual scene; A texture mapping coordinate acquisition module, used to determine the texture mapping coordinates corresponding to the grid nodes in the three-dimensional model of the screen; The texture map coordinate acquisition module includes: A first camera coordinate system acquisition module, used to acquire a first camera coordinate system corresponding to the picture shot by the real-view camera; A first plane projection calculation module, used to calculate a first plane projection corresponding to the screen three-dimensional model according to a first viewing matrix and a first projection matrix corresponding to the first camera coordinate system; A second camera coordinate system acquisition module, used to set a virtual camera in the virtual scene and acquire a second camera coordinate system corresponding to the picture shot by the virtual camera; A second plane projection calculation module, used to calculate a second plane projection corresponding to the screen three-dimensional model according to a second viewing matrix and a second projection matrix corresponding to the second camera coordinate system; A first texture mapping coordinate acquisition submodule, used to normalize the coordinates of each pixel in the first plane projection and the second plane projection with the upper left corner of the screen three-dimensional model as the origin, and use the normalized coordinates as the texture mapping coordinates of the three-dimensional model grid node projected to the pixel; The target picture synthesis module is used to synthesize the first texture map, the second texture map and the three-dimensional screen model according to the texture map coordinates corresponding to the grid nodes in the three-dimensional screen model to obtain a target picture.

7. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images