3D model rendering reality method and device, and medium

Abstract

The invention provides a 3D model rendering reality method. The method comprises the following steps: acquiring a multi-view image of a target object in a real scene and camera position attitude information corresponding to the multi-view image; rendering the 3D model based on the pose information to obtain a 2D rendered image; constructing an illumination change data set, and training an illumination change network; inputting a 3D model to be rendered and a background image into the trained illumination change network, and outputting an illumination adaptive rendering image; and synthesizing the rendered image and a background image to obtain a reality-enhanced synthetic image. The method has the advantages of being high in authenticity, high in generalization ability, efficient in data construction, wide in application and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer graphics and image processing, in particular to a 3D model rendering realization method, a computer device and a computer readable storage medium. BACKGROUND

[0002] In the task of aerial target detection, a virtual engine is often used to generate a 3D model of an aircraft, and a camera observation posture is set to render the 3D model into a 2D image, which is then synthesized with a real background to generate training or test data. However, in the traditional method, the surface lighting of the 3D model is usually pre-set and cannot be adjusted adaptively according to the background environment, resulting in inconsistent lighting between the model and the background in the synthesized image, unrealistic visual effects, and affecting the training effect of the subsequent detection model.

[0003] To solve this problem, existing research has attempted to use deep learning technology to train a lighting change network, so that the model can adaptively adjust the surface lighting according to the background environment while maintaining the main structure, thereby improving the realism of the synthesized image. However, such a network needs to rely on a large amount of paired training data, i.e., image pairs of the same model under different lighting conditions, which has a complex construction process and high cost, and existing methods still have deficiencies in lighting transfer accuracy and generalization ability. SUMMARY

[0004] The present application provides a 3D model rendering realization method for solving the above problems of unrealistic lighting of synthesized images, complex training data construction, etc.

[0005] The present application provides a 3D model rendering realization method, comprising the following steps: S1: obtaining multi-view images of a target object in a real scene and corresponding camera posture information thereof; S2: rendering a 3D model based on the posture information to obtain a 2D rendered image; S3: constructing a lighting change dataset, including paired data of the target object in the 2D rendered image and a real image; S4: training a lighting change network using the lighting change dataset, the network being used to adjust the surface lighting of the model according to the background environment; S5: inputting a 3D model to be rendered and a background image into the trained lighting change network to output a lighting-adaptive rendered image; S6: synthesizing the rendered image and the background image to obtain a real-sense-enhanced synthesized image.

[0006] Specifically, in the S1, the following steps are further included: S11: providing a model equipped with a first posture sensor in a real world as the observed target object; S12: providing a camera matched with a second pose sensor as an observer; S13: collecting observation images of the target object in various background environments in the real world by using the camera, and simultaneously collecting relative pose information between the camera and the target object by using the first pose sensor and the second pose sensor.

[0007] Specifically, in the S2, the following steps are further included: S21: making a 3D virtual model corresponding to the 3D model by using a virtual engine; S22: performing 2D view rendering of the 3D virtual model by using the collected relative pose information between the camera and the target object, to form the 2D rendering image.

[0008] Specifically, in the S3, the matching data of the target object in the 2D rendering image and the real image includes: obtaining the 2D rendering image formed in the S22 and the observation image formed in the S13; performing frame-by-frame matching on the 2D rendering image and the observation image, to form a light change data set.

[0009] Specifically, in the S4, the light change network training includes: S41: inputting a first foreground image in the 2D rendering image and the background image into the light change network, to generate a converted second foreground image; S42: comparing the second foreground image and the observation image, predicting an error, and feeding back the error to the light change network; S43: the light change network outputs a corrected second foreground image through error feedback, and continues to compare with the observation image, and then feeds back the predicted error to the light change network, so as to train the light change network by using the light change data set.

[0010] Specifically, the light change network includes a feature extraction network and a parameter generation network, and the S5 further includes the following steps: S51: inputting the foreground image and the background image into the feature extraction network to extract hidden layer image features; S52: inputting the hidden layer image features into the parameter generation network to obtain light conversion parameters; S53: performing light conversion on the foreground image based on the light conversion parameters.

[0011] Specifically, in the S6, the background image and the rendering image are processed by using a synthesis algorithm, wherein the synthesis algorithm uses an MDB algorithm.

[0012] Specifically, the background image is a part of the real-world image that does not contain the observation target, and the background image can provide a changing background environment for assisting in forming the illumination change dataset and training the illumination change network.

[0013] The application provides a computer device, which comprises a memory and a processor coupled with the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the 3D model rendering realization method.

[0014] The application provides a computer readable storage medium for storing a computer program, wherein the computer program is used to realize the 3D model rendering realization method when executed by a computer.

[0015] Compared with the prior art, the application realizes the adaptive fusion of the 3D model rendering image and the background illumination by constructing a real illumination change dataset and training an illumination change network with illumination migration capability, and significantly improves the realism of the synthesized image. The method has the following advantages: High realism: the model illumination is consistent with the background environment, and the visual effect is natural; Strong generalization ability: suitable for various backgrounds and model types; Efficient data construction: real acquisition data is used to construct the dataset, and manual annotation is avoided; Wide application: can be used in many fields such as aerial detection, AR / VR, game development, etc. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. Figure 1 is the flow chart of the 3D model rendering realization method provided by the application; Figure 2 is the structural block diagram of the 3D model rendering realization method provided by the application; Figure 3 is Figure 1 the flow chart of the S1 shown in the figure; Figure 4 is a 2D view rendered by a 3D virtual model; Figure 5 is a real 2D image captured by a camera; Figure 6 is Figure 1 a flow chart of the S2 shown in FIG. 2; Figure 7 is Figure 1 a flow chart of the S4 shown in FIG. 4; Figure 8 is a structure diagram of the illumination change network in an embodiment; Figure 9 is Figure 1 a flow chart of the S5 shown in FIG. 5; Figure 10 is a structure diagram of the illumination change network in an embodiment; Figure 11 is a schematic diagram of an apparatus in an embodiment of the present application; Figure 12 is a structure diagram of a computer readable storage medium in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0018] It should be noted that all directional indications, such as up, down, left, right, front, back, etc., in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0019] In addition, the description such as "first", "second" and the like in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0020] In addition, the technical solutions of each embodiment of the present application can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the scope of protection required by the present application.

[0021] Referring to Figure 1 and Figure 2 , Figure 1 is a process block diagram of the 3D model rendering realization method provided by the application, Figure 2 is a structural block diagram of the 3D model rendering realization method. The 3D model rendering realization method comprises the following steps: S1: acquiring multi-view images of a target object in a real scene and corresponding camera pose information thereof; S2: providing a 3D virtual model, rendering the 3D virtual model based on the relative pose information, and obtaining a 2D rendered image; S3: constructing a light change dataset, comprising paired data of the target object in the 2D rendered image and a real image; S4: training a light change network using the light change dataset, the network being used to adjust the light on the model surface according to the background environment; S5: inputting a 3D model to be rendered and a background image into the trained light change network, and outputting a light-adaptive rendered image; S6: synthesizing the rendered image and the background image to obtain a real-sense-enhanced synthesized image.

[0022] The application realizes adaptive fusion of 3D model rendering images and background light by constructing a real light change dataset and training a light change network with light migration capability, which significantly improves the real sense of the synthesized image. Figure 2 As shown in the prior art, the main focus is on how to directly convert and render a 3D model into a 2D image, and then synthesize the 2D image with a real background. However, the surface light of the 3D model needs to be determined in advance, and cannot change adaptively with the background, resulting in insufficient realism. The light change network trained by the deep learning technology can change the surface light of the model adaptively according to the environment while keeping the model subject unchanged, generate a 2D image with adaptive background light, and thus make the synthesized image more real.

[0023] Please continue to refer to Figure 3 , the 3D model rendering realization method is further described. Among them, Figure 3 is Figure 1 the process block diagram of S1. It should be noted that in S1, the following steps are further included: S11: providing a model equipped with a first pose sensor in a real world as the target object to be observed; S12: providing a camera equipped with a second pose sensor as an observer; S13: collecting observation images of the target object in various background environments in the real world by using the camera, and simultaneously collecting relative attitude information between the camera and the target object by using the first attitude sensor and the second attitude sensor.

[0024] In the embodiment, by arranging the attitude sensors in the camera and the target object respectively, the collection of the relative attitude information is realized, so as to facilitate the subsequent production of the 3D virtual model by the virtual engine and the collection of the rendering image by cooperating with the camera view angle.

[0025] Please continue to refer to Figure 6 After the collection of the attitude information is realized, the S2 further includes the following steps: S21: producing a 3D virtual model corresponding to the 3D model by using a virtual engine; S22: performing 2D view rendering of the 3D virtual model by using the collected relative attitude information between the camera and the target object, to form the 2D rendering image.

[0026] For the process of converting the 3D virtual model into the 2D rendering image, the following is briefly explained: Suppose the coordinates of an object in a world coordinate system are V_world, the camera is at the origin of the world coordinate system, and the direction is the coordinate system direction, then the 3D object rendering follows the world coordinates (V_world) to be converted into camera projection coordinates (V_camera), normalized device coordinates (V_ndc), and finally screen pixel coordinates (V_screen).

[0027] First, the conversion of the world coordinates (V_world) to the camera projection coordinates (V_camera) mainly depends on the following algorithm: V_camera= M_perspective * V_world Wherein, V_world is the coordinates of the vertex in the world space; M_perspective is the perspective projection matrix; V_camera is the coordinates of the vertex in the homogeneous clipping space.

[0028] Secondly, the conversion of the camera projection coordinates (V_camera) to the normalized device coordinates (V_ndc) mainly depends on the following algorithm: V_ndc = V_camera.xyz / V_camera.w Finally, the normalized device coordinates (V_ndc) in the range of [-1, 1] are mapped to the screen pixel coordinates (V_screen).

[0029] V_screen.x = (V_ndc.x + 1) * (screenWidth / 2) + viewportX V_screen.y = (1 - V_ndc.y) * (screenHeight / 2) + viewportY Based on this, complete conversion of the 3D virtual model to the 2D rendering image is realized.

[0030] Specifically, in the S3, the paired data of the 2D rendering image and the target object in the real image includes: Obtaining the 2D rendering image formed in the S22 and the observation image formed in the S13; Please refer to Figure 4 and Figure 5 , the 2D rendering image and the observation image are frame-by-frame paired to form a light change data set. Figure 4 and Figure 5 A set of training samples in the light change data set is shown, wherein Figure 4 is obtained by 2D view rendering of the camera using the 3D virtual model, Figure 5 is a real image taken. By comparing the two images, it can be seen that Figure 4 has the shape structure of the real image, but lacks color and details under different light background changes. The light change network trained in the present application is expected to restore the light effect of the real target under such background by inputting Figure 4 the target shape and structure and the background.

[0031] Please continue to refer to Figure 7 and Figure 8 , in the S4, the light change network training includes: S41: inputting the first foreground image in the 2D rendering image and the background image into the light change network to generate a converted second foreground image; S42: comparing the second foreground image and the observation image, predicting the error, and feeding the error back to the light change network; S43: the light change network outputs the second foreground image after correction through error feedback, and continues to compare with the observation image, and then feeds back the predicted error to the light change network, so as to realize training of the light change network using the light change data set.

[0032] Please continue to refer to Figure 9 and Figure 10 , the light change network includes a feature extraction network and a parameter generation network, and the S5 further includes the following steps: S51: input the foreground image and the background image into the feature extraction network to extract hidden layer image features; S52: input the hidden layer image features into the parameter generation network to obtain illumination conversion parameters; S53: perform illumination conversion on the foreground image based on the illumination conversion parameters.

[0033] In the S6, the background image and the rendering image are processed by using a synthesis algorithm, wherein the synthesis algorithm uses an MDB algorithm.

[0034] The background image is a part of a real world image that does not contain the observation target, and the background image can provide a changing background environment for assisting in forming the illumination change data set and training the illumination change network.

[0035] Compared with the prior art, the 3D model rendering realization method provided by the present application has the following technical effects: High authenticity: the model illumination is consistent with the background environment, and the visual effect is natural; Strong generalization ability: suitable for various backgrounds and model types; Efficient data construction: real acquisition data is used to construct the data set, avoiding manual annotation; Wide application: can be used in many fields such as aerial detection, AR / VR, game development, etc.

[0036] Please continue to read Figure 11 In one embodiment, a computer device 100 is provided, which includes a memory 101 and a processor 103 coupled with the memory 101. The memory 101 is configured to store program data, and the processor 103 is configured to execute the program data. The program data includes the following steps: S1: obtaining multi-view images of a target object in a real scene and corresponding camera pose information thereof; S2: providing a 3D virtual model, rendering the 3D virtual model based on the relative pose information to obtain a 2D rendering image; S3: constructing an illumination change data set, including paired data of the 2D rendering image and the target object in a real image; S4: training an illumination change network using the illumination change data set, the network being used to adjust the model surface illumination according to the background environment; S5: inputting a 3D model to be rendered and a background image into the trained illumination change network to output a rendering image with adaptive illumination; S6: synthesizing the rendering image and the background image to obtain a synthesized image with enhanced realism.

[0037] Please continue to refer to Figure 12 In one embodiment, a computer readable storage medium 300 is provided, which stores a computer program 301, the computer program 301 is executed to implement the following steps: S1: obtaining multi-view images of a target object in a real scene and corresponding camera pose information thereof; S2: providing a 3D virtual model, rendering the 3D virtual model based on the relative pose information to obtain a 2D rendered image; S3: constructing a light change dataset, including paired data of the 2D rendered image and the target object in a real image; S4: training a light change network using the light change dataset, the network being used to adjust the light on the model surface according to the background environment; S5: inputting a 3D model to be rendered and a background image into the trained light change network, and outputting a light-adaptive rendered image; S6: synthesizing the rendered image and the background image to obtain a real-sense enhanced synthesized image.

[0038] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0039] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0040] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A 3D model rendering realism method, characterized by, The method comprises the following steps: S1: obtaining multi-view images of a target object in a real scene and corresponding camera pose relative attitude information thereof; S2: providing a 3D virtual model, rendering the 3D virtual model based on the relative attitude information, and obtaining a 2D rendered image; S3: constructing a light change dataset, including paired data of the 2D rendered image and the target object in a real image; S4: training a light change network using the light change dataset, the network being used to adjust the light on the surface of a model according to a background environment; S5: inputting a 3D model to be rendered and a background image into the trained light change network, and outputting a light-adaptive rendered image; S6: synthesizing the rendered image and the background image to obtain a real-sense-enhanced synthesized image.

2. The 3D model rendering realism method of claim 1, wherein, In the S1, the following steps are further included: S11: providing a model equipped with a first attitude sensor in a real world as the target object to be observed; S12: providing a camera equipped with a second attitude sensor as an observer; S13: using the camera to collect observation images of the target object in various background environments in the real world, and using the first attitude sensor and the second attitude sensor to simultaneously collect relative attitude information between the camera and the target object.

3. The 3D model rendering realism method of claim 2, wherein, In the S2, the following steps are further included: S21: using a virtual engine to make a 3D virtual model corresponding to the 3D model; S22: using the collected relative attitude information between the camera and the target object to perform 2D view rendering of the 3D virtual model by the camera, to form the 2D rendered image.

4. The 3D model rendering realism method of claim 3, wherein, In the S3, the paired data of the 2D rendered image and the target object in a real image comprises: obtaining the 2D rendered image formed in the S22 and the observation image formed in the S13; pairing the 2D rendered image and the observation image frame by frame, to form a light change dataset.

5. The 3D model rendering realism method of claim 4, wherein, In the S4, the training of the light change network comprises: S41: inputting a first foreground image in the 2D rendered image and the background image into the light change network, to generate a converted second foreground image; S42: comparing the second foreground image and the observation image, predicting an error, and feeding back the error to the light change network; S43: the light change network outputs a corrected second foreground image through error feedback, and continues to compare with the observation image, and then feeds back the predicted error to the light change network, to realize training of the light change network using the light change dataset.

6. The 3D model rendering realism method of claim 5, wherein, The light change network comprises a feature extraction network and a parameter generation network, and the S5 further comprises the following steps: S51: inputting a foreground image and a background image into the feature extraction network, to extract hidden layer image features; S52: inputting the hidden layer image features into the parameter generation network, to obtain light conversion parameters; S53: performing light conversion on the foreground image based on the light conversion parameters.

7. The 3D model rendering realism method of claim 6, wherein, In the S6, the background image and the rendering image are processed by using a synthesis algorithm, wherein the synthesis algorithm uses an MDB algorithm.

8. The 3D model rendering realism method of claim 7, wherein, The background image is a part of the real-world image that does not contain the observation target, and the background image can provide a changing background environment for assisting in forming the illumination change dataset and training the illumination change network.

9. A computer device, comprising: The computer device comprises a memory and a processor coupled to the memory; The memory is configured to store program data, and the processor is configured to execute the program data to implement the 3D model rendering realization method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store a computer program, and the computer program is configured to implement the 3D model rendering realization method according to any one of claims 1-8 when executed by a computer.

Images