Method and system for fusing three-dimensional model and video
By analyzing the three-dimensional data model and decoding the video picture, and fusing it and rendering it through the OpenGL interface, the problem of difficulty in simulating real scenes in the existing technology is solved, and efficient fusion and rendering effects of three-dimensional models and videos are achieved.
Patent Information
- Application Number
- CN202510050478.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-13
- Publication Date
- 2025-04-29
AI Technical Summary
Existing video decoding methods are difficult to effectively integrate three-dimensional models and videos, resulting in the inability to simulate real scenes.
By analyzing the three-dimensional data model and decoding the video screen, it is fused, and rendered through the OpenGL interface, the corresponding texture is loaded to achieve the effect of simulating the real scene.
It realizes efficient integration of three-dimensional models and videos, and is suitable for applications in various scenarios, with high degree of automation and good simulation scene effects.
Smart Images
Figure CN119906816A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of video decoding, and in particular discloses a method and a system for fusing a three-dimensional model and a video. Background Art
[0002] In the prior art, the video decoding process is as follows:
[0003] 1. The pure video decoding process is:
[0004] Compressed coded data -> pixel data. For example, decoding H.264 is "H.264 code stream -> YUV".
[0005] 2. The general video decoding process is:
[0006] The video bitstream is generally stored in a certain packaging format (such as MP4, AVI, etc.), which usually also includes audio bitstream and other contents.
[0007] For videos in encapsulated formats, you need to extract the video bitstream from the encapsulated format first and then decode it. For example, to decode a video file in MKV format, the process is "MKV->H.264 bitstream->YUV".
[0008] Assimp library is a model loading library commonly used in OpenGL, and its full name is Open Asset Import Library. It supports model files in multiple formats, such as obj, 3ds, c4e, etc. Models are generally made through tool software such as Blender, 3DS Max or Maya, and then model files can be exported. When using OpenGL, it is necessary to parse the data content in these files, which mainly include vertex data, normals, texture coordinates, etc., as well as material, lighting and other information. Only after parsing these data can subsequent rendering work be done.
[0009] Therefore, using OpenGL to simulate real scenes in video images is a technical problem that needs to be solved urgently. Summary of the invention
[0010] The present invention provides a method and system for fusing a three-dimensional model and a video, aiming to solve at least one defect existing in the existing video decoding method.
[0011] One aspect of the present invention relates to a method for fusing a three-dimensional model and a video, comprising the following steps:
[0012] Parse 3D data models and decode video images;
[0013] The parsed 3D data model is integrated with the decoded video image, and then rendered through the OpenGL interface. After loading the corresponding texture, the effect of simulating the real scene is achieved.
[0014] Furthermore, the steps of parsing the three-dimensional data model and decoding the video picture include:
[0015] For videos in a packaged format, first extract the video bitstream from the packaged format;
[0016] Decode the extracted video code stream to obtain each frame of the video.
[0017] Furthermore, the parsed three-dimensional data model is merged with the decoded video image, and then rendered through the OpenGL interface. After the corresponding texture is loaded to achieve the effect of simulating the real scene, the three-dimensional data model is imported. When importing the three-dimensional data model, Assimp is used to load the scene data of the three-dimensional data model into a Scene type object, and a data structure with a corresponding relationship is generated for the scene nodes and model nodes of the three-dimensional data model.
[0018] Furthermore, the parsed 3D data model is merged with the decoded video image, and then rendered through the OpenGL interface. After the corresponding texture is loaded to achieve the effect of simulating the real scene, a file name parameter of the Mesh class is used as a parameter of the LoadMesh() function, and the model is loaded with the help of Assimp. The scene data of the loaded 3D data model is parsed, and the scene data of the 3D data model is filled into the vertex cache, index cache and texture object.
[0019] Furthermore, the steps of fusing the parsed 3D data model with the decoded video image, rendering through the OpenGL interface, and loading the corresponding texture to achieve the effect of simulating the real scene include:
[0020] Using aiProcess_Triangulate operation to identify 3D data models that are not composed of triangles and convert the 3D data models that are not composed of triangles into a triangle-based mesh model;
[0021] Use the aiProcess_GenSmoothNormals operation to identify 3D data models without vertex normals and generate vertex normals in the 3D data models without vertex normals;
[0022] Use aiProcess_FlipUVsv operation to flip the texture coordinates along the Y axis of the 3D data model.
[0023] Another aspect of the present invention relates to a system for fusing a three-dimensional model and a video, comprising:
[0024] Parsing and decoding module, used for parsing 3D data model and decoding video images;
[0025] The fusion rendering module is used to fuse the parsed 3D data model with the decoded video image, and then render it through the OpenGL interface and load the corresponding texture to achieve the effect of simulating the real scene.
[0026] Furthermore, the parsing and decoding module includes:
[0027] The extraction unit, for a video in a package format, first extracts a video bit stream from the package format;
[0028] The acquisition unit decodes the extracted video code stream to obtain each frame of the video.
[0029] Furthermore, in the fusion rendering module, a three-dimensional data model is imported. When the three-dimensional data model is imported, Assimp is used to load the scene data of the three-dimensional data model into a Scene type object, and a data structure with a corresponding relationship is generated for the scene nodes and model nodes of the three-dimensional data model.
[0030] Furthermore, in the fusion rendering module, a file name parameter of the Mesh class is used as a parameter of the LoadMesh() function, and the model is loaded with the help of Assimp, the scene data of the loaded 3D data model is parsed, and the scene data of the 3D data model is filled into the vertex cache, index cache and texture object.
[0031] Furthermore, the fusion rendering module includes:
[0032] A conversion unit, for converting a three-dimensional data model that is not composed of triangles into a mesh model based on triangles by using an aiProcess_Triangulate operation;
[0033] A generation unit, used to generate vertex normals from a three-dimensional data model without vertex normals by using aiProcess_GenSmoothNormals operation;
[0034] The flip unit is used to flip the texture coordinates along the Y axis of the three-dimensional data model using the aiProcess_FlipUVsv operation.
[0035] The beneficial effects achieved by the present invention are:
[0036] The present invention provides a method and system for fusing a three-dimensional model and a video, which parses a three-dimensional data model and decodes a video screen; fuses the parsed three-dimensional data model with the decoded video screen, and then renders it through an OpenGL interface, and after loading the corresponding texture, achieves the effect of simulating a real scene. The present invention parses the three-dimensional data model, fuses it with the decoded video screen, and then renders it through an OpenGL interface to achieve the effect of simulating a real scene. The method is suitable for applications in various scenes, has a high degree of automation, and has a good effect of simulating scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for fusing a three-dimensional model and a video according to the present invention;
[0038] Figure 2 A schematic diagram of the operation effect of a method for fusing a three-dimensional model and a video according to the present invention;
[0039] Figure 3 It is a structural diagram of Assimp in a method for fusing a three-dimensional model and a video of the present invention;
[0040] Figure 4 A schematic diagram of Mesh in a method for fusing a three-dimensional model and a video according to the present invention;
[0041] Figure 5 A schematic diagram of a three-dimensional model in a method for fusing a three-dimensional model and a video according to the present invention;
[0042] Figure 6 A schematic diagram of a model without texture in a method for fusing a three-dimensional model and a video according to the present invention;
[0043] Figure 7 A schematic diagram of a model after texture loading in a method for fusing a three-dimensional model and a video according to the present invention;
[0044] Figure 8 A schematic diagram of a video decoding process in a method for fusing a three-dimensional model and a video according to the present invention;
[0045] Fig. 9 A schematic diagram of the data structure of FFMPEG decoding in a method for fusing a three-dimensional model and a video according to the present invention;
[0046] Fig.10 A schematic diagram of pixel conversion in a method for fusing a three-dimensional model and a video according to the present invention. DETAILED DESCRIPTION
[0047] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0048] like Figure 1 and Figure 2 As shown, the first embodiment of the present invention proposes a method for fusing a three-dimensional model and a video, comprising the following steps:
[0049] Step S100: parsing the three-dimensional data model and decoding the video picture.
[0050] After the three-dimensional data model is parsed, it is fused with the decoded video image.
[0051] Step S200: Fusing the parsed 3D data model with the decoded video image, rendering through the OpenGL interface, and loading the corresponding texture to achieve the effect of simulating the real scene.
[0052] The 3D data model is integrated with the video image and then rendered through the OpenGL interface to achieve the effect of simulating the real scene. It is suitable for applications in various scenarios.
[0053] Further, see Figures 1 to 10 In the method for fusing a 3D model and a video provided in this embodiment, step S100 includes:
[0054] Step S110: For the video in the encapsulation format, first extract the video code stream from the encapsulation format.
[0055] For videos in encapsulated formats, you need to extract the video bitstream from the encapsulated format first and then decode it.
[0056] Step S120: decode the extracted video code stream to obtain each frame of video.
[0057] The video file is decoded through a decoder, and then each frame is obtained and pasted onto a certain surface of the 3D model in the form of a texture map.
[0058] Preferably, see Figures 1 to 10 In the method for fusing a three-dimensional model and a video provided in this embodiment, in step S200, a three-dimensional data model is imported. When the three-dimensional data model is imported, Assimp is used to load the scene data of the three-dimensional data model into a Scene type object, and a data structure with a corresponding relationship is generated for the scene nodes and model nodes of the three-dimensional data model.
[0059] Import 3D data models. When importing a model file, Assimp loads all model and scene data into a Scene type object, and generates a data structure with corresponding relationships for scene nodes and model nodes. All model and scene data are contained in the scene object, such as Material and Mesh. Similarly, the root node reference of the scene is also contained in this scene object.
[0060] Preferably, see Figures 1 to 10 In the method for fusing a 3D model and a video provided in this embodiment, in step S200, a file name parameter of the Mesh class is used as a parameter of the LoadMesh() function, the model is loaded with the help of Assimp, the scene data of the loaded 3D data model is parsed, and the scene data of the 3D data model is filled into the vertex cache, index cache and texture object.
[0061] The root node of the scene may also contain many child nodes and an index collection pointing to mMeshes[] that stores the model point cloud data. The mMeshes[] on the root node stores the actual Mesh object, and the mMesshes[] on each child node is just a reference to the mMeshes[] in the root node. A Mesh object itself contains all the relevant data required for rendering, such as vertex positions, normal vectors, texture coordinates, patches, and the material of the object.
[0062] The video file is decoded through a decoder, and each frame is obtained and pasted onto a certain surface of the model in the form of a texture map. For example, the scene in the model is a house with a TV in the house. The decoded video image is pasted onto the surface of the TV and then rendered. The resulting effect is that the TV in the 3D model will play real images, integrating real people into the 3D model.
[0063] Preferably, see Figures 1 to 10 In the method for fusing a 3D model and a video provided in this embodiment, step S200 includes:
[0064] Step S210: using the aiProcess_Triangulate operation to identify a three-dimensional data model that is not composed of triangles, and converting the three-dimensional data model that is not composed of triangles into a triangle-based mesh model.
[0065] In this embodiment, three operations are provided, the first operation is aiProcess_Triangulate operation, which converts a model that is not composed of triangles into a triangle-based mesh model. For example, a quadrilateral mesh can be transformed into a triangle mesh by generating two triangles from each quadrilateral.
[0066] Step S220: using the aiProcess_GenSmoothNormals operation to identify a three-dimensional data model that does not contain vertex normals, and generate vertex normals in the three-dimensional data model that does not contain vertex normals.
[0067] The second operation is the aiProcess_GenSmoothNormals operation, which generates vertex normals for models that originally did not have them. Remember that these processing methods are non-overlapping bitmasks, so you can use the "or" operation to combine these operations.
[0068] Step S230: Use the aiProcess_FlipUVsv operation to flip the texture coordinates along the Y axis of the three-dimensional data model.
[0069] The third operation is the aiProcess_FlipUVsv operation, which flips the texture coordinates along the y axis. You need to choose the appropriate operation based on the imported model data. If the mesh is successfully loaded, you will get a pointer to the aiScene object. This object contains the contents of the entire model, and the different structures of the model are kept in an aiMesh structure. Next, call the InitFromScene() function to initialize the Mesh object.
[0070] The present invention relates to a system for fusing three-dimensional models and videos, comprising a parsing and decoding module and a fusion rendering module, wherein the parsing and decoding module is used to parse a three-dimensional data model and decode a video picture; the fusion rendering module is used to fuse the parsed three-dimensional data model with the decoded video picture, and then render through an OpenGL interface, and achieve the effect of simulating a real scene after loading corresponding textures.
[0071] After the parsing and decoding module parses the three-dimensional data model, it is integrated with the decoded video image.
[0072] The fusion rendering module fuses the 3D data model with the video image, and then renders it through the OpenGL interface to achieve the effect of simulating the real scene. It is suitable for applications in various scenarios.
[0073] Further, see Figures 1 to 10 In the system for fusing three-dimensional models and videos provided in this embodiment, the parsing and decoding module includes an extraction unit and an acquisition unit, wherein the extraction unit first extracts a video code stream from the encapsulation format for the video in the encapsulation format; the acquisition unit decodes the extracted video code stream to obtain each frame of the video.
[0074] For the video in the encapsulation format, the extraction unit needs to first extract the video code stream from the encapsulation format and then decode it.
[0075] The acquisition unit decodes the video file through a decoder, and then acquires each frame of the picture, and sticks it on a certain surface of the three-dimensional model in the form of texture mapping.
[0076] Preferably, see Figures 1 to 10In the system for fusing three-dimensional models and videos provided in this embodiment, a three-dimensional data model is imported into the fusion rendering module. When the three-dimensional data model is imported, Assimp is used to load the scene data of the three-dimensional data model into a Scene type object, and a data structure with a corresponding relationship is generated for the scene nodes and model nodes of the three-dimensional data model.
[0077] The Fusion Rendering module imports 3D data models. When a model file is imported, Assimp loads all model and scene data into a Scene type object, and generates a data structure with corresponding relationships for scene nodes and model nodes. All model and scene data are contained in the scene object, such as Material and Mesh. Similarly, the scene's root node reference is also contained in this scene object.
[0078] Further, see Figures 1 to 10 In the system for fusing 3D models and videos provided in this embodiment, in the fusion rendering module, a file name parameter of the Mesh class is used as a parameter of the LoadMesh() function, and the model is loaded with the help of Assimp, and the scene data of the loaded 3D data model is parsed, and the scene data of the 3D data model is filled into the vertex cache, index cache and texture object.
[0079] The root node of the Fusion Rendering Module scene may also contain many child nodes and an index set pointing to mMeshes[] that stores the model point cloud data. The mMeshes[] on the root node stores the actual Mesh object, and the mMesshes[] on each child node is just a reference to the mMeshes[] in the root node. A Mesh object itself contains all the relevant data required for rendering, such as vertex positions, normal vectors, texture coordinates, patches, and the material of the object.
[0080] The video file is decoded through a decoder, and each frame is obtained and pasted onto a certain surface of the model in the form of a texture map. For example, the scene in the model is a house with a TV in the house. The decoded video image is pasted onto the surface of the TV and then rendered. The resulting effect is that the TV in the 3D model will play real images, integrating real people into the 3D model.
[0081] Further, see Figures 1 to 10In the system for fusing three-dimensional models and videos provided in this embodiment, the fusion rendering module includes a conversion unit, a generation unit and a flip unit, wherein the conversion unit is used to convert a three-dimensional data model that is not composed of triangles into a triangle-based mesh model by using the aiProcess_Triangulate operation; the generation unit is used to generate vertex normals from a three-dimensional data model that does not contain vertex normals by using the aiProcess_GenSmoothNormals operation; and the flip unit is used to flip the texture coordinates along the Y-axis of the three-dimensional data model by using the aiProcess_FlipUVsv operation.
[0082] In this embodiment, three operations are provided, the first operation is the aiProcess_Triangulate operation of the conversion unit, which converts a model that is not composed of triangles into a triangle-based mesh model. For example, a quadrilateral mesh can be transformed into a triangle mesh by generating two triangles from each quadrilateral.
[0083] The second operation is the aiProcess_GenSmoothNormals operation of the generation unit, which generates vertex normals for models that originally did not have them. Remember that these processing methods are non-overlapping bit masks, so the operations can be combined using the "or" operation.
[0084] The third operation is the aiProcess_FlipUVsv operation of the flip unit, which flips the texture coordinates along the y axis. You need to choose the appropriate operation based on the imported model data. If the mesh is successfully loaded, you will get a pointer to the aiScene object. This object contains the content of the entire model, and the different structures of the model are kept in an aiMesh structure. Next, call the InitFromScene() function to initialize the Mesh object.
[0085] The method and system for fusing a 3D model and a video provided in this embodiment are implemented in code as follows:
[0086] The Mesh class is the interface between Assimp and the OpenGL program. The Mesh class uses a file name parameter as a parameter of the LoadMesh() function to load the model with the help of Assimp, then parses the loaded model data and fills the model data into the vertex buffer, index buffer and texture object. In order to render the mesh, the Render() function is defined. The internal structure of the Mesh class matches the method of Assimp loading models. Assimp uses an aiScene object to represent the loaded mesh, which encapsulates the mesh structure containing the various parts of the model. The aiScene object must contain at least one mesh structure, and complex models can contain multiple mesh structures. The member m_Entries of the Mesh class is a vector of MeshEntry structures, each of which corresponds to a mesh structure in the aiScene object. This structure contains the index of the vertex buffer, index buffer and material. Currently, a material is only one texture, and because mesh entities can share materials, a separate vector is set for each material (m_Textures). MeshEntry::MaterialIndex points to a texture in m_Textures.
[0087] class Mesh
[0088] {
[0089] public:
[0090] Mesh();
[0091] ~Mesh();
[0092] bool LoadMesh(const std::string&Filename);
[0093] void Render();
[0094] private:
[0095] bool InitFromScene(const aiScene*pScene,const std::string&Filename);
[0096] void InitMesh(unsigned int Index,const aiMesh*paiMesh);
[0097] bool InitMaterials(const aiScene*pScene,const std::string&Filename);
[0098] void Clear();
[0099] #define INVALID_MATERIAL 0xFFFFFFFF
[0100] struct MeshEntry{
[0101] MeshEntry();
[0102] ~MeshEntry();
[0103] bool Init(const std::vector&Vertices,
[0104] const std::vector&Indices);
[0105] GLuint VB;
[0106] GLuint IB;
[0107] unsigned int NumIndices;
[0108] unsigned int MaterialIndex;
[0109] };
[0110] std::vector <meshentry>m_Entries;
[0111] std::vector<Texture*> m_Textures;
[0112] };
[0113] Loading a mesh starts with this function. An instance of the Assimp::Importer class is created on the stack and its ReadFile method is called. This function requires two parameters: the full path to the model file and some processing options. Assimp can perform many useful optimizations on the loaded model. For example, it can generate normals for models without normals, optimize the structure of the model to improve performance, etc. We can choose the appropriate operation according to our needs. Here we use three operations it provides: the first is aiProcess_Triangulate, which converts a model that is not composed of triangles into a triangle-based mesh model. For example, a quad mesh can be transformed into a triangle mesh by generating two triangles from each quad in it; the second operation is aiProcess_GenSmoothNormals, which generates vertex normals for models that originally do not have vertex normals. Remember that these processing methods are non-overlapping bit masks, so you can use the "or" operation to combine these operations; the third operation is aiProcess_FlipUVsv, which flips the texture coordinates along the y axis. You need to choose the appropriate operation based on the imported model data. If the mesh is loaded successfully, we get a pointer to an aiScene object. This object contains the content of the entire model, and the different structures of the model are kept in an aiMesh structure. Next, call the InitFromScene() function to initialize the Mesh object.
[0114] bool Mesh::LoadMesh(const std::string&Filename)
[0115] {
[0116] / / Release the previously loaded mesh(if it exists)
[0117] Clear();
[0118] bool Ret = false;
[0119] Assimp::Importer Importer;
[0120] const aiScene*pScene=Importer.ReadFile(Filename.c_str(),aiProcess_Triangulate|aiProcess_GenSmoothNormals|aiProcess_FlipUVs);
[0121] if(pScene){
[0122] Ret=InitFromScene(pScene,Filename);
[0123] }
[0124] else{
[0125] printf("Error parsing'%s':'%s'\n",Filename.c_str(),Importer.GetErrorString());
[0126] }
[0127] return Ret;
[0128] }
[0129] First, we allocate storage space for m_Entries and m_Textures according to the number of m_Entries and m_Textures needed. The number can be obtained through the members mNumMeshes and mNumMaterials in the aiScene object. Next, we traverse the mMeshes array in the aiScene object and initialize the m_Entries instances one by one. Finally, initialize the material.
[0130] bool Mesh::InitFromScene(const aiScene*pScene,const std::string&Filename)
[0131] {
[0132] m_Entries.resize(pScene->mNumMeshes);
[0133] m_Textures.resize(pScene->mNumMaterials);
[0134] / / Initialize the meshes in the scene one by one
[0135] for(unsigned int i=0; i <m_Entries.size();i++){
[0136] const aiMesh*paiMesh=pScene->mMeshes[i];
[0137] InitMesh(i,paiMesh);
[0138] }
[0139] return InitMaterials(pScene,Filename);
[0140] }
[0141] To use the corresponding library of FFmpeg, you need to register it. Only after registering it can you use the encoder and decoder normally. Open the file and get the corresponding FFmpeg global context according to the file name information.
[0142] / / / register
[0143] av_register_all();
[0144] / / / Open the file
[0145] AVFormatContext*pFormatCtx; / / File context, which describes the composition and basic information of a media file or media stream
[0146] pFormatCtx = avformat_alloc_context(); / / allocate pointer
[0147] if(avformat_open_input(&pFormatCtx,file_path,NULL,NULL)!=0){ / / Open the file, store the information in the file context, and then
[0148] printf("Unable to open file");
[0149] return -1;
[0150] }
[0151] It is necessary to detect the stream information and obtain the encoding format of the stream encoding. If the stream information is not detected, the encoding type obtained by the stream encoder may be empty, and the original format cannot be known during subsequent data conversion, resulting in errors. Then the original data is decoded.
[0152] / / Check if there is information flow in the file
[0153] if(avformat_find_stream_info(pFormatCtx,NULL)<0){
[0154] printf("No information stream was found in the file");
[0155] return -1;
[0156] }
[0157] / / Check if the video stream is stored in the file
[0158] int videoStream = -1;
[0159] for(i=0;i <pformatctx->nb_streams; i++){
[0160] if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO){
[0161] videoStream = i;
[0162] }
[0163] }
[0164] / / If videoStream is -1, it means no video stream is found
[0165] if (videoStream == -1) {
[0166] printf("No video stream found in the file");
[0167] return -1;
[0168] }
[0169] / / Check if there is an audio stream in the file
[0170] int audioStream=-1
[0171] for(i=0;i <pformatctx->nb_streams; i++){
[0172] if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO){
[0173] audioStream = i;
[0174] }
[0175] }
[0176] / / If audioStream is -1, it means no audio stream is found
[0177] if (audioStream == -1) {
[0178] printf("No audio stream found in the file");
[0179] return -1;
[0180] }
[0181] / / Get the original data
[0182] int y_size=pCodecCtx->width*pCodecCtx->height;
[0183] packet=(AVPacket*)malloc(sizeof(AVPacket)); / / allocate a packetav_new_packet(packet,y_size); / / allocate packet data
[0184] int ret,got_picture;
[0185] while(1){
[0186] if(av_read_frame(pFormatCtx,packet)<0){ / / Read a packet
[0187] break; / / Here we think the video has been read
[0188] }
[0189] if(packet->stream_index==videoStream){
[0190] ret = avcodec_decode_video2 (pCodecCtx, pFrame, &got_picture, packet); / / decode the packet and store the original data in pFrame
[0191] if(ret<0){
[0192] printf("decode error.");
[0193] return -1;
[0194] }
[0195] }
[0196] }
[0197] 1. Introduction to Assimp
[0198] Assimp is a model loading library commonly used in Opengl, and its full name is Open Asset Import Library. It supports model files in multiple formats, such as obj, 3ds, c4e, etc. Models are generally made through tool software such as Blender, 3DS Max or Maya, and then the model files can be exported. When using Opengl, it is necessary to parse the data content in these files, which mainly include vertex data, normals, texture coordinates, etc., as well as material, lighting and other information. Only after parsing these data can we do subsequent rendering work.
[0199] When importing a model file, Assimp loads all model and scene data into a Scene type object, and generates a data structure with corresponding relationships for scene nodes and model nodes. The data structure diagram is as follows:
[0200] All models and scene data are contained in the scene object, such as Material and Mesh. Similarly, the scene's root node reference is also contained in this scene object.
[0201] The root node of the scene may also contain many child nodes and an index collection pointing to mMeshes[] that stores the model point cloud data. The mMeshes[] on the root node stores the actual Mesh object, and the mMesshes[] on each child node is just a reference to the mMeshes[] in the root node.
[0202] A Mesh object itself contains all the relevant data needed for rendering, such as vertex positions, normal vectors, texture coordinates, patches, and the material of the object.
[0203] A Mesh contains multiple faces. A Face represents the most basic shape unit in rendering, namely a primitive (basic primitives include points, lines, triangles, and rectangles). A face records the vertex index of a primitive. Through this index, the corresponding vertex position data can be found in mMeshes[].
[0204] A Mesh also contains a Material object to specify some material properties of the object, such as color, texture map (diffuse map, specular map, etc.).
[0205] Generally, a Model is usually composed of multiple meshes. For example, a human body model can be composed of limbs, head, upper body and other parts, and each part is a Mesh.
[0206] So how do you understand Mesh? If you have practiced drawing a triangle, you will know that a triangle is a Mesh, which contains vertex positions, normals, and texture coordinates. In fact, this triangle is also a Face. In actual complex models, Mesh generally contains many Faces. Face is the basic primitive for drawing, including triangles, points, lines, polygons, etc. The most commonly used is triangles.
[0207] A Mesh is a combination of vertices, edges, and faces, and is also the smallest unit for drawing 3D objects. The more Meshes a 3D model has, the closer the model is to reality, but it also brings a burden on hardware device rendering, especially for mobile devices. For example, when using Opengl ES rendering on a mobile phone, it is necessary to measure the GPU performance and the setting of the number of Meshes.
[0208] However, many formats of 3D models only have one model file, and the texture is embedded in the model. If we use the Assimp library to load it, we can load the model, but the loaded effect is all black and the texture cannot be loaded.
[0209] The loading effect is shown in the figure below, all black.
[0210] Separate the texture, save it to a local file, and load the local texture file.
[0211] First, read the local file through ReadFile and return aiScene.
[0212] Get the number of textures: scene->mNumTextures.
[0213] Get the current texture: aiTexture*texture = scene->mTextures[i]
[0214] Then take a look at the introduction of the aiTexture document.
[0215] / **Width of the texture, in pixels
[0216] *
[0217] *If mHeight is zero the texture is compressed in a format
[0218] *like JPEG. In this case mWidth specifies the size of the
[0219] *memory area pcData is pointing to, in bytes.
[0220] * /
[0221] unsigned int mWidth;
[0222] / **Height of the texture, in pixels
[0223] *
[0224] *If this value is zero, pcData points to an compressed texture
[0225] *in any format (e.g. JPEG).
[0226] * /
[0227] unsigned int mHeight;
[0228] / **Data of the texture.
[0229] *
[0230] *Points to an array of mWidth*mHeight aiTexel's.
[0231] *The format of the texture data is always ARGB8888 to
[0232] *make the implementation for user of the library as easy
[0233] *as possible.If mHeight=0this is a pointer to a memory
[0234] *buffer of size mWidth containing the compressed texture
[0235] *data.Good luck, have fun!
[0236] * /
[0237] C_STRUCT aiTexel*pcData;
[0238] char achFormatHint[HINTMAXTEXTURELEN]
[0239] mWidth: The pixel width of the texture. If the height is 0, mWidth specifies the memory area pointed to by pcData, in bytes.
[0240] mHeigth: The pixel height of the texture, like jpeg type, the value is 0.
[0241] pcData: Texture data, the RGBA value of the data is saved in this data.
[0242] achFormatHint: the format of the image, png or jpg or other.
[0243] Let's take a look at the aiTexel structure. The structure is as follows: it contains r, g, b, and a values, representing the r, g, b, and a values of each pixel.
[0244] struct aiTexel
[0245] {
[0246] unsigned char b,g,r,a;
[0247] #ifdef__cplusplus
[0248] / / ! Comparison operator
[0249] bool operator==(const aiTexel&other)const
[0250] {
[0251] return b==other.b&&r==other.r&&
[0252] g==other.g&&a==other.a;
[0253] }
[0254] / / ! Inverse comparison operator
[0255] bool operator! =(const aiTexel&other)const
[0256] {
[0257] return b! =other.b||r! =other.r||
[0258] g! =other.g||a! =other.a;
[0259] }
[0260] / / ! Conversion to a floating-point 4d color
[0261] operator aiColor4D()const
[0262] {
[0263] return aiColor4D(r / 255.f,g / 255.f,b / 255.f,a / 255.f);
[0264] }
[0265] #endif / / __cplusplus
[0266] }PACK_STRUCT;
[0267] According to the above conditions, save the image locally.
[0268] const aiScene*scene=import.ReadFile(path,aiProcess_Triangulate|aiProcess_FlipUVs);
[0269] if(!scene||scene->mFlags&AI_SCENE_FLAGS_INCOMPLETE||!scene->mRootNode){
[0270] qDebug()<<"ERROR::ASSIMP::"<<import.GetErrorString();
[0271] return;
[0272] }
[0273] directory=path.substr(0,path.find_last_of(' / '));
[0274] for(int i=0;i <scene->mNumTextures;i++)
[0275] {
[0276] aiTexture*texture=scene->mTextures[i];
[0277] char fileName
[100] ;
[0278] sprintf(fileName,"%s / %s.%s",modelDirectory.c_str(),texture->mFilename.C_Str(),texture->achFormatHint);
[0279] QFile file(fileName);
[0280] if(file.exists())
[0281] {
[0282] m_mapPath.insert(i,QString(fileName));
[0283] continue;
[0284] }
[0285] if(!file.open(QIODevice::WriteOnly))
[0286] break;
[0287] unsigned char*buffer=new unsigned char[texture->mWidth*4];
[0288] memset(buffer,0,texture->mWidth*4);
[0289] for(int x=0;x <texture->mWidth; ++x)
[0290] {
[0291] / / Copy RGBA data to the buffer
[0292] int index = x * 4;
[0293] buffer[index]=texture->pcData[x].b; / / Blue
[0294] buffer[index+1]=texture->pcData[x].g; / / Green
[0295] buffer[index+2]=texture->pcData[x].r; / / Red
[0296] buffer[index+3]=texture->pcData[x].a; / / Alpha
[0297] }
[0298] file.write((char*)buffer,texture->mWidth*4);
[0299] file.close();
[0300] m_mapPath.insert(i,QString(fileName));
[0301] delete[]buffer;
[0302] }
[0303] Finally, here is the effect after loading the corresponding texture.
[0304] 2. FFMPEG related introduction
[0305] 1. Video decoding process
[0306] 2. FFMPEG decoding function
[0307] av_register_all(): Register all components
[0308] avformat_open_input(): Open the input video file
[0309] avformat_find_stream_info(): Get video file information
[0310] avcodec_find_decoder(): Find decoder
[0311] avcodec_open2(): Open the decoder
[0312] av_read_frame(): Reads a frame of compressed data from the input file
[0313] avcodec_decode_video2(): Decodes a frame of compressed data
[0314] avcodec_close(): Close the decoder
[0315] avformat_close_input(): Close the input video file
[0316] 3. Data structure of FFMPEG decoding
[0317] 4. FFMPEG data structure analysis
[0318] AVFormatContext
[0319] iformat: AVInputFormat of input video
[0320] nb_streams: the number of AVStreams of the input video streams: the AVStream[] array of the input video
[0321] duration: the duration of the input video (in microseconds) bit_rate: the bit rate of the input video AVInputFormat
[0322] name: Encapsulation format name
[0323] long_name: the long name of the encapsulation format extensions: the extension name of the encapsulation format
[0324] id: Encapsulation format ID
[0325] Some interface functions for encapsulation format processing
[0326] AVStream
[0327] id: serial number
[0328] codec: AVCodecContext corresponding to the stream
[0329] time_base: The time base of the stream
[0330] r_frame_rate: The frame rate of the stream
[0331] AVCodecContext
[0332] codec: AVCodec of codec
[0333] width,height: width and height of the image (only for video)
[0334] pix_fmt: pixel format (only for video)
[0335] sample_rate: sampling rate (audio only)
[0336] channels: Number of channels (audio only)
[0337] sample_fmt: sampling format (audio only)
[0338] AVCodec
[0339] name: codec name
[0340] long_name: codec long name
[0341] type: codec type
[0342] id: codec ID
[0343] Some encoding and decoding interface functions
[0344] AVPacket
[0345] pts: Display timestamp
[0346] dts: decoding timestamp
[0347] data: compressed encoded data
[0348] size: compressed encoded data size
[0349] stream_index: the AVStream to which it belongs
[0350] AVFrame
[0351] data: decoded image pixel data (audio sampling data) linesize: for video, it is the size of a row of pixels in the image; for audio, it is the size of the frame.
[0352] width, height: width and height of the image (only for video) key_frame: whether it is a key frame (only for video) pict_type: frame type (only for video). For example, 1, P, B.
[0353] 5. Pixel data conversion
[0354] The decoded YUV format video pixel data is saved in data[0], data[1], and data[2] of AVFrame.
[0355] However, these pixel values are not stored continuously, and some invalid pixels are stored after each row of valid pixels.
[0356] Taking the brightness Y data as an example, data[0] contains a total of linesize[0]*height data.
[0357] However, for optimization and other considerations, linesize[0] is not actually equal to width, but a value larger than the width.
[0358] Therefore, sws_scale() needs to be used for conversion. After the conversion, invalid data is removed and width and linesize[0] are equal.
[0359] This embodiment provides a method and system for fusing a 3D model and a video. Compared with the prior art, the method parses the 3D data model and decodes the video screen; the parsed 3D data model is fused with the decoded video screen, and then rendered through the OpenGL interface, and the corresponding texture is loaded to achieve the effect of simulating a real scene. This embodiment parses the 3D data model, fuses it with the decoded video screen, and then renders it through the OpenGL interface to achieve the effect of simulating a real scene. It is suitable for applications in various scenarios, has a high degree of automation, and has a good effect of simulating scenes.
[0360] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations. < / meshentry>
Claims
1. A method for fusing a three-dimensional model and a video, characterized in that: The following steps are involved: Parse 3D data models and decode video images; The parsed three-dimensional data model is fused with the decoded video image, and then rendered through the OpenGL interface, and the corresponding texture is loaded to achieve the effect of simulating the real scene.
2. The method for fusing a three-dimensional model and a video according to claim 1, characterized in that: The steps of parsing the three-dimensional data model and decoding the video picture include: For videos in a packaged format, first extract the video bitstream from the packaged format; The extracted video code stream is decoded to obtain each frame of the video.
3. The method for fusing a three-dimensional model and a video according to claim 1, wherein: In the step of fusing the parsed three-dimensional data model with the decoded video picture, rendering through the OpenGL interface, and loading the corresponding texture to achieve the effect of simulating the real scene, the three-dimensional data model is imported. When importing the three-dimensional data model, Assimp is used to load the scene data of the three-dimensional data model into a Scene type object, and a data structure with a corresponding relationship is generated for the scene nodes and model nodes of the three-dimensional data model.
4. The method for fusing a three-dimensional model and a video according to claim 1, wherein: In the step of fusing the parsed three-dimensional data model with the decoded video picture, rendering through the OpenGL interface, and loading the corresponding texture to achieve the effect of simulating the real scene, a file name parameter of the Mesh class is used as a parameter of the LoadMesh() function, and the model is loaded with the help of Assimp, the scene data of the loaded three-dimensional data model is parsed, and the scene data of the three-dimensional data model is filled into the vertex cache, index cache and texture object.
5. The method for fusing a three-dimensional model and a video according to claim 1, wherein: The step of fusing the parsed three-dimensional data model with the decoded video image, rendering through the OpenGL interface, and loading the corresponding texture to achieve the effect of simulating the real scene includes: Using aiProcess_Triangulate operation to identify 3D data models that are not composed of triangles and convert the 3D data models that are not composed of triangles into a triangle-based mesh model; Use the aiProcess_GenSmoothNormals operation to identify 3D data models without vertex normals and generate vertex normals in the 3D data models without vertex normals; The aiProcess_FlipUVsv operation is used to flip the texture coordinates along the Y axis of the three-dimensional data model.
6. A system for fusing a three-dimensional model and a video, characterized in that: include: Parsing and decoding module, used for parsing 3D data model and decoding video images; The fusion rendering module is used to fuse the parsed three-dimensional data model with the decoded video image, and then render it through the OpenGL interface, and load the corresponding texture to achieve the effect of simulating the real scene.
7. The system for fusing a three-dimensional model and a video according to claim 6, characterized in that: The parsing and decoding module comprises: The extraction unit, for a video in a package format, first extracts a video bit stream from the package format; The acquisition unit decodes the extracted video code stream to acquire each frame of the video.
8. The system for fusing a three-dimensional model and a video according to claim 6, characterized in that: In the fusion rendering module, the three-dimensional data model is imported. When the three-dimensional data model is imported, Assimp is used to load the scene data of the three-dimensional data model into a Scene type object, and a data structure with a corresponding relationship is generated for the scene nodes and model nodes of the three-dimensional data model.
9. The system for fusing a three-dimensional model and a video according to claim 6, characterized in that: In the fusion rendering module, a file name parameter of the Mesh class is used as a parameter of the LoadMesh() function, and the model is loaded with the help of Assimp, the scene data of the loaded three-dimensional data model is parsed, and the scene data of the three-dimensional data model is filled into the vertex cache, index cache and texture object.
10. The system for fusing a three-dimensional model and a video according to claim 6, characterized in that: The fusion rendering module includes: A conversion unit, for converting a three-dimensional data model that is not composed of triangles into a mesh model based on triangles by using an aiProcess_Triangulate operation; A generation unit, used to generate vertex normals from a three-dimensional data model without vertex normals by using aiProcess_GenSmoothNormals operation; The flipping unit is used to flip the texture coordinates along the Y axis of the three-dimensional data model by using the aiProcess_FlipUVsv operation.