Digital plotting method based on AIGC

Abstract

The invention discloses a digital plotting method based on AIGC, and relates to the technical field of digital plotting, the method adopts a digital plotting system to work, the system comprises an information collection module, a model evaluation module and an AI plotting module, the information collection module is used for collecting various information in a to-be-processed model, and the model evaluation module is used for evaluating the information in the to-be-processed model; and the model evaluation module is used for carrying out model block division on the model to be processed by utilizing boundary features, and calculating the line complexity and the screen area ratio of each model block. The AI drawing module is used for drawing a display picture of a to-be-processed model in a replacement scene by adopting a feature point tracking algorithm and a grid tracking algorithm, the information collection module comprises a scene loading module, a visual angle movement recording module, a model information extraction module and a boundary feature analysis module, and the system has the characteristic that the advantages of the scene loading module and the visual angle movement recording module are combined.

Description

Technical Field

[0001] The present invention relates to the technical field of digital matte painting, and specifically to a digital matte painting method based on AIGC. Background Art

[0002] The combination of AIGC and digital matte painting technology can provide high-quality picture output in creating the picture atmosphere, which is more realistic than pure AI-generated shots and can better adjust and optimize the picture quality than pure live-action shots. The AI automatically matches the lens distortion and perspective, so that the model to be replaced in the new picture maintains the distortion and extension amplitude according to the original feature points, completing the replacement of the picture elements.

[0003] Two commonly used AI algorithms are the mesh tracking algorithm and the feature point tracking algorithm. Feature point tracking is to find points with unique features in the video frame and track the movement of these points in subsequent frames. Mesh tracking is to divide the image into multiple grid cells and perform tracking based on the preset grid. The effects presented by the two algorithms are generally the same, but the applicable scenarios are different.

[0004] In the prior art, often only a single algorithm is used for a certain model to be replaced. If the first algorithm is adopted, the advantage is that high-precision generation can be achieved, but the disadvantage is that in scenes with large dynamic changes (such as fast movement and large rotation), the feature points may cause mutations and need to be manually adjusted, reducing the production efficiency. If the second algorithm is adopted, although the changes of the model in the scene can be smoothly processed, the display effect is not good when facing the model to be processed with complex lines. Therefore, it is necessary to design a digital matte painting method based on AIGC that combines the advantages of both. Summary of the Invention

[0005] The purpose of the present invention is to provide a digital matte painting method based on AIGC to solve the problems raised in the above background art.

[0006] To solve the above technical problems, the present invention provides the following technical solution: A digital matte painting method based on AIGC, which uses a digital matte painting system to work. The system includes an information collection module, a model evaluation module, and an AI matte painting module. The information collection module is used to collect various information in the model to be processed and record the way of lens movement and rotation when displayed after replacing the scene. The model evaluation module is used to divide the model blocks of the model to be processed using boundary features and calculate the line complexity and screen area ratio of each model block. The AI matte painting module is used to draw the display picture of the model to be processed in the replacement scene using the feature point tracking algorithm and the mesh tracking algorithm.

[0007] According to the above technical solution, the information collection module includes a scene loading module, a perspective movement recording module, a model information extraction module, and a boundary feature analysis module. The scene loading module is electrically connected to the model information extraction module, and the model information extraction module is electrically connected to the boundary feature analysis module. The scene loading module is used to load the scene that needs to be digitally painted, select the area to be replaced, and load the model to be replaced. The perspective movement recording module is used to record the perspective route experienced by the scene video lens after reconstruction. The model information extraction module is used to read the line features, boundary features, and model size information in the model to be replaced. The boundary feature analysis module is used to analyze the boundary features of each part of the model; The model evaluation module includes a block division module, a complexity calculation module, a screen occupation ratio calculation module, a timing module, and a lens speed calculation module. The block division module is electrically connected to the boundary feature analysis module, and the complexity calculation module is electrically connected to the model information extraction module. The block division module is used to distinguish the model to be processed using the boundary features and divide it into multiple model blocks. The complexity calculation module is used to calculate the line complexity according to the line features in each model block. The screen occupation ratio calculation module is used to calculate the screen area occupation ratio of each model block in the scene video. The timing module is used to calculate the time experienced by the scene video. The lens speed calculation module is used to calculate the lens speed according to the perspective route and time experienced by the scene video lens; The AI painting module includes a feature point tracking algorithm module, a grid tracking algorithm module, a dynamic adjustment module, and a scene generation module. The feature point tracking algorithm module and the grid tracking algorithm module are both electrically connected to the dynamic adjustment module. The dynamic adjustment module is electrically connected to the scene generation module and the scene loading module. The feature point tracking algorithm module accurately tracks and draws the model to be replaced in the new scene according to the pre-identified feature points. The grid tracking algorithm module is used to divide the model to be replaced into grids and perform real-time rendering through grid-based deformation operations. The dynamic adjustment module is used to adjust the application methods of the feature point tracking algorithm and the grid tracking algorithm. The scene generation module is used to generate the reconstructed scene according to the set algorithm application method.

[0008] According to the above technical solution, it includes the following steps: S1. Load the scene that needs to be digitally painted, read the line features, boundary features, and model size information from the model to be replaced, record the perspective route experienced by the scene video lens after reconstruction, and analyze the boundary features of each part of the model; S2. Use boundary features to divide the model to be processed, generate multiple model blocks, calculate the line complexity based on the line features in each model block, and calculate the screen area ratio of each model block in the scene video in combination with the viewing route. Adjust the AI generation algorithm adopted in the new scene according to the line complexity and screen area ratio of different model blocks; S3. Calculate the dynamic change degree of the model to be replaced according to the speed of camera movement and rotation, adjust the application ratio of the feature point tracking algorithm and the grid tracking algorithm in real time in combination with the preset algorithm type, and generate the reconstructed scene according to the set algorithm application method to complete the display of the model to be processed in the new scene; S4. Record the situation where feature point mutations occur in the model blocks using the feature point tracking algorithm. If the number of feature point mutations exceeds the threshold, optimize the application ratio of the feature point tracking algorithm.

[0009] According to the above technical solution, in S2, the specific method for generating multiple model blocks is as follows: Use the edge detection algorithm to identify the contour boundary of the model to be replaced. According to the shape, color, and contour of each part in the model to be replaced, determine which parts constitute the boundary features. Obtain the point cloud data of the area near the contour boundary through the sampling grid, and cluster the point cloud in the model to be replaced according to the similarity of the boundary features to generate different model blocks.

[0010] According to the above technical solution, in S2, the specific method for adjusting the AI generation algorithm adopted in the new scene is as follows: S2-1. Extract all the edges and contour lines of each model block, count the number of edges of each model block, calculate the number of independent vertices contained in the edges and contour lines of each model block, calculate the local curvature of each edge and take the average. Use the number of edges , the number of vertices , and the average edge curvature to perform weighted calculation to obtain the line complexity of the model block , and count the line complexity of multiple models to obtain , where is the number of model blocks in the model to be replaced, is the weight coefficient of the number of edges, is the weight coefficient of the number of vertices, is the weight coefficient of the average edge curvature; S2-2. Read the perspective route experienced by the scene when generating video shots after reconstruction, establish a 3D lens transfer matrix based on the movement trajectory of the lens, analyze the distortion effect of the lens to be replaced in the model, project the points of the 3D lens transfer matrix onto the 2D plane through the current perspective, analyze the planar image formed by the lens to be replaced according to the perspective route, and determine the size of each model block for statistics, and calculate the average screen area ratio of each model block in the video based on the size of the entire scene video frame to obtain ; S2-3. For model blocks with high line complexity and large screen area ratio, adopt the feature point tracking algorithm. The specific calculation formula is the algorithm reference value , where is the model block serial number, is the line complexity weight coefficient, is the model block size weight coefficient. When , preferentially adopt the feature point tracking algorithm. When , preferentially adopt the grid tracking algorithm

[0011] According to the above technical solution, in S3, the specific method for calculating the dynamic change degree of the model to be replaced is as follows S3-1. According to the 3D lens transfer matrix established in S2-2, calculate the deformation degree of the model block caused by the movement and rotation of the lens. Take a point in each model block as a marker point . Since the position of the marker point is constantly changing according to the movement and flipping of the lens, record the position change value of the marker point within a certain specified time period and sum them up to obtain the total position change value . The larger the , the greater the dynamic change degree of the model to be replaced within the current time period , that is

[0012] According to the above technical solution, in S3, the specific method for real-time adjusting the application ratio of the feature point tracking algorithm and the grid tracking algorithm is as follows: S3-2. Record the dynamic change degrees within multiple time periods , where is the number of divided time periods. The position change value of the marker point in the latter time period is calculated based on the position of the same marker point in the previous time period. Adjust according to different dynamic change degrees . The larger the , the greater the The larger it is, the fewer model blocks adopt the feature point tracking algorithm, and the degree of dynamic change within different time periods The smaller it is, the more model blocks adopt the feature point tracking algorithm.

[0013] According to the above technical solution, in the step S4, the specific method for optimizing the application ratio of the feature point tracking algorithm is as follows: count the number of feature point mutations that occur in the model blocks that adopt the feature point tracking algorithm in the model to be replaced after the scene video picture is generated where is the number of model blocks that adopt the feature point tracking algorithm, and calculate the average number of mutations When , according to increase the size of in direct proportion to the size of When it is

[0014] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: In the present invention, the model to be processed is divided into multiple model blocks, and the boundary features are used to distinguish between different blocks. For the model blocks with a high degree of line complexity and a large screen area ratio, the feature point tracking algorithm is adopted, and for the model blocks with a low degree of line complexity and a small screen area ratio, the grid tracking algorithm is adopted, so as to improve the production efficiency while maintaining the display effect of important parts; at the same time, calculate the speed of the camera movement and rotation when the model is displayed after replacing the scene, and adjust the proportion of the model blocks that need to use the grid tracking algorithm based on the time axis, reducing the probability of feature point mutations and improving the model accuracy as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the accompanying drawings: Figure 1 is a schematic diagram of the overall module structure of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0017] Please refer to Figure 1, the present invention provides a technical solution: a digital painting method based on AIGC. This method works using a digital painting system, which includes an information collection module, a model evaluation module, and an AI painting module. The information collection module is used to collect various information in the model to be processed, and record the way of camera movement and rotation when displaying after replacing the scene. The model evaluation module is used to divide the model blocks of the model to be processed using boundary features, and calculate the line complexity and screen area ratio of each model block. The AI painting module is used to draw the display screen of the model to be processed in the replaced scene using the feature point tracking algorithm and the grid tracking algorithm; The information collection module includes a scene loading module, a perspective movement recording module, a model information extraction module, and a boundary feature analysis module. The scene loading module is electrically connected to the model information extraction module, and the model information extraction module is electrically connected to the boundary feature analysis module. The scene loading module is used to load the scene that needs to be digitally painted, select the area to be replaced and load the model to be replaced. The perspective movement recording module is used to record the perspective route experienced by the scene video camera after the scene is reconstructed. The model information extraction module is used to read the line features, boundary features, and model size information in the model to be replaced. The boundary feature analysis module is used to analyze the boundary features of each part of the model; The model evaluation module includes a block division module, a complexity calculation module, a screen ratio calculation module, a timing module, and a camera speed calculation module. The block division module is electrically connected to the boundary feature analysis module, and the complexity calculation module is electrically connected to the model information extraction module. The block division module is used to distinguish the model to be processed using boundary features and divide it into multiple model blocks. The complexity calculation module is used to calculate the line complexity according to the line features in each model block. The screen ratio calculation module is used to calculate the screen area ratio of each model block in the scene video. The timing module is used to calculate the time experienced by the scene video. The camera speed calculation module is used to calculate the camera speed according to the perspective route and time experienced by the scene video camera; The AI painting module includes a feature point tracking algorithm module, a grid tracking algorithm module, a dynamic adjustment module, and a scene generation module. Both the feature point tracking algorithm module and the grid tracking algorithm module are electrically connected to the dynamic adjustment module. The dynamic adjustment module is electrically connected to the scene generation module and the scene loading module. The feature point tracking algorithm module accurately tracks and draws the model to be replaced in the new scene according to the pre-identified feature points. The grid tracking algorithm module is used to divide the model to be replaced into grids and perform real-time rendering through grid-based deformation operations. The dynamic adjustment module is used to adjust the application methods of the feature point tracking algorithm and the grid tracking algorithm. The scene generation module is used to generate the reconstructed scene according to the set algorithm application method; It includes the following steps: S1. Load the scene to be digitally painted, read the line features, boundary features, and model size information from the model to be replaced, record the perspective route experienced by the reconstructed scene video shot, and analyze the boundary features of each part of the model; S2. Use the boundary features to divide the model to be processed, generate multiple model blocks, calculate the line complexity based on the line features in each model block, and calculate the proportion of the screen area of each model block in the scene video in combination with the perspective route. Adjust the AI generation algorithm adopted in the new scene according to the line complexity and the proportion of the screen area of different model blocks; S3. Calculate the dynamic change degree of the model to be replaced according to the speed of the camera movement and rotation, adjust the application ratio of the feature point tracking algorithm and the mesh tracking algorithm in real time in combination with the preset algorithm type, and generate the reconstructed scene according to the set algorithm application method to complete the display of the model to be processed in the new scene; S4. Record the situation where feature point mutations occur in the model blocks using the feature point tracking algorithm. If the number of feature point mutations exceeds the threshold, optimize the application ratio of the feature point tracking algorithm; In S2, the specific method for generating multiple model blocks is as follows: Use the edge detection algorithm to identify the contour boundary of the model to be replaced. According to the shape, color, and contour of each part of the model to be replaced, determine which parts constitute the boundary features. Obtain the point cloud data of the area near the contour boundary through sampling the grid, and cluster the point cloud in the model to be replaced according to the similarity of the boundary features to generate different model blocks; In S2, the specific method for adjusting the AI generation algorithm adopted in the new scene is as follows: S2-1. Extract all the edges and contour lines of each model block, count the number of edges of each model block, calculate the number of independent vertices contained in the edges and contour lines of each model block, calculate the local curvature of each edge and take the average. Use the number of edges , the number of vertices , and the average edge curvature to perform weighted calculation to obtain the line complexity of the model block , and count the line complexity of multiple models to obtain , where is the number of model blocks in the model to be replaced, is the weight coefficient of the number of edges, is the weight coefficient of the number of vertices, is the weight coefficient of the average edge curvature; S2-2. Read the perspective route experienced by the scene when generating video shots after reconstruction, establish a 3D camera transfer matrix based on the movement trajectory of the shots, analyze the distortion effect of the model shots to be replaced, project the points of the 3D camera transfer matrix onto the 2D plane through the current perspective, analyze the planar images formed by the model to be replaced according to the perspective route, and measure the size of each model block and count them. Based on the size of the entire scene video frame , calculate the average screen area ratio of each model block in the video to obtain ; S2-3. For model blocks with high line complexity and large screen area ratio, adopt the feature point tracking algorithm. The specific calculation formula is the algorithm reference value , where is the model block serial number, is the line complexity weight coefficient, is the model block size weight coefficient. When , preferentially adopt the feature point tracking algorithm. When , preferentially adopt the grid tracking algorithm; In S3, the specific method for calculating the dynamic change degree of the model to be replaced is as follows: S3-1. According to the 3D camera transfer matrix established in S2-2, calculate the deformation degree of the model block caused by the movement and rotation of the camera. Take a point in each model block as a marker point . Since the position of the marker point is constantly changing according to the movement and flipping of the camera, record the position change value of the marker point within a specified time period and sum them up to obtain the total position change value . The larger the , the greater the dynamic change degree of the model to be replaced within the current time period, that is ; In S3, the specific method for real-time adjusting the application ratio of the feature point tracking algorithm and the grid tracking algorithm is as follows: S3-2. Record the dynamic change degrees within multiple time periods , where is the number of divided time periods. The position change value of the marker point in the latter time period is calculated based on the position of the same marker point in the previous time period. Adjust according to different dynamic change degrees . The larger the , the greater the , that is, the greater the dynamic change degree within different time periods, the fewer model blocks adopt the feature point tracking algorithm. The smaller the dynamic change degree The smaller it is, the more the feature point tracking algorithm is adopted for each model block; In S4, the specific method for optimizing the application ratio of the feature point tracking algorithm is: count the number of feature point mutations that occur in the model blocks that adopt the feature point tracking algorithm in the model to be replaced after the scene video frame is generated , where is the number of model blocks that adopt the feature point tracking algorithm, and calculate the average mutation number . When , according to , increase the size of in direct proportion to its size, If not, there is no need to optimize.

[0018] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0019] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A digital painting method based on AIGC, characterized by: The method adopts a digital painting system to work, which includes an information collection module, a model evaluation module, and an AI painting module. The information collection module is used to collect various information in the model to be processed, and record the lens movement and rotation mode when the scene is replaced. The model evaluation module is used to divide the model to be processed into model blocks using boundary features, and calculate the line complexity of each model block and the screen area ratio. The AI ​​painting module is used to use a feature point tracking algorithm and a grid tracking algorithm to draw the display screen of the model to be processed in the replacement scene.

2. The AIGC-based digital painting method according to claim 1, characterized in that: The information collection module includes a scene loading module, a view movement recording module, a model information extraction module, and a boundary feature analysis module. The scene loading module is electrically connected to the model information extraction module, and the model information extraction module is electrically connected to the boundary feature analysis module. The scene loading module is used to load the scene that needs to be digitally painted, select the area that needs to be replaced and load the model to be replaced. The view movement recording module is used to reconstruct the view route experienced by the scene video lens after the scene is reconstructed. The model information extraction module is used to read the line features, boundary features, and model size information in the model to be replaced. The boundary feature analysis module is used to analyze the boundary features of each part in the model. The model evaluation module includes a block division module, a complexity calculation module, a screen ratio calculation module, a timing module, and a lens speed calculation module. The block division module is electrically connected to the boundary feature analysis module, and the complexity calculation module is electrically connected to the model information extraction module. The block division module is used to distinguish the model to be processed by using boundary features and divide it into multiple model blocks. The complexity calculation module is used to calculate the line complexity according to the line features in each model block. The screen ratio calculation module is used to calculate the screen area ratio of each model block in the scene video. The timing module is used to calculate the time experienced by the scene video. The lens speed calculation module is used to calculate the lens speed according to the viewing angle route and time experienced by the scene video lens; The AI ​​painting module includes a feature point tracking algorithm module, a grid tracking algorithm module, a dynamic adjustment module, and a scene generation module. The feature point tracking algorithm module and the grid tracking algorithm module are both electrically connected to the dynamic adjustment module. The dynamic adjustment module is electrically connected to the scene generation module and the scene loading module. The feature point tracking algorithm module accurately tracks and draws the model to be replaced in the new scene based on the feature points identified in advance. The grid tracking algorithm module is used to divide the model to be replaced into grids and perform real-time rendering through grid-based deformation operations. The dynamic adjustment module is used to adjust the application methods of the feature point tracking algorithm and the grid tracking algorithm. The scene generation module is used to generate a reconstructed scene according to the set algorithm application method.

3. The AIGC-based digital painting method according to claim 2, characterized in that: The following steps are involved: S1. Load the scene to be digitally painted, read the line features, boundary features and model size information from the model to be replaced, record the view route experienced by the reconstructed scene video lens, and analyze the boundary features of each part of the model; S2. Use boundary features to divide the model to be processed to generate multiple model blocks, calculate the line complexity according to the line features in each model block, and calculate the screen area ratio of each model block in the scene video in combination with the view route, and adjust the AI ​​generation algorithm used in the new scene according to the line complexity and screen area ratio of different model blocks; S3, calculating the dynamic change degree of the model to be replaced according to the speed of the lens movement and rotation, adjusting the application ratio of the feature point tracking algorithm and the grid tracking algorithm in real time in combination with the pre-set algorithm type, generating the reconstructed scene according to the set algorithm application method, and completing the display of the model to be processed in the new scene; S4. Record the situation where the feature point mutation occurs in the model block using the feature point tracking algorithm. If the number of feature point mutations exceeds a threshold, optimize the application ratio of the feature point tracking algorithm.

4. The AIGC-based digital painting method according to claim 3, characterized in that: In S2, the specific method for generating multiple model blocks is: using an edge detection algorithm to identify the contour boundary of the model to be replaced, determining which parts constitute boundary features based on the shape, color and contour of each part in the model to be replaced, obtaining point cloud data of the area near the contour boundary through a sampling grid, clustering the point cloud in the model to be replaced according to the similarity of the boundary features, and generating different model blocks.

5. The AIGC-based digital painting method according to claim 4, characterized in that: In S2, the specific method for adjusting the AI ​​generation algorithm used in the new scenario is: S2-1. Extract all edges and contours of each model block, count the number of edges of each model block, calculate the number of independent vertices contained in the edges and contours of each model block, calculate the local curvature of each edge and average it, and use the edge count , number of vertices , edge mean curvature Perform a weighted calculation to get the line complexity of the model block , and statistically analyze the line complexity of multiple models, and obtain ,in is the number of model blocks in the model to be replaced, is the weight coefficient of the edge number, is the weight coefficient of the number of vertices, is the weight coefficient of the mean curvature of the edge; S2-2, read the perspective route that the scene goes through when generating the video lens after reconstruction, establish a 3D lens transfer matrix according to the motion trajectory of the lens, analyze the distortion effect of the lens of the model to be replaced, project the points of the 3D lens transfer matrix onto the 2D plane through the current perspective, analyze the plane image formed by the perspective route of the model to be replaced, and calculate the size of each model block Perform statistics based on the size of the entire scene video screen , calculate the average screen area ratio of each model block in the video, and get ; S2-3. For model blocks with high line complexity and large screen area, the feature point tracking algorithm is used. The specific calculation formula is the algorithm reference value. ,in is the model block number, is the line complexity weight coefficient, is the model block size weight coefficient, when When , the feature point tracking algorithm is preferred. When , the grid tracking algorithm is preferred.

6. The AIGC-based digital painting method according to claim 5, characterized in that: In S3, the specific method for calculating the dynamic change degree of the model to be replaced is: S3-1, according to the 3D lens transfer matrix established in S2-2, calculate the deformation degree of the model block caused by the lens movement and rotation, and select a point in each model block as a marking point Since the position of the marker point is constantly changing according to the movement and flipping of the lens, the position change value of the marker point within a specified time period is recorded. And sum them up to get the total position change value , The larger the value, the more dynamic the model to be replaced is in the current time period. The larger the .

7. The AIGC-based digital painting method according to claim 6, characterized in that: In S3, the specific method for adjusting the application ratio of the feature point tracking algorithm and the grid tracking algorithm in real time is: S3-2, recording the degree of dynamic changes in multiple time periods ,in The position change value of the mark point in the next time period is calculated based on the position of the same mark point in the previous time period. right Make adjustments, The bigger the The bigger.

8. The AIGC-based digital painting method according to claim 7, characterized in that: In S4, the specific method for optimizing the application ratio of the feature point tracking algorithm is: counting the number of feature point mutations that occur in the model block using the feature point tracking algorithm in the model to be replaced after the scene video screen is generated ,in The average number of mutations is calculated for the number of model blocks using the feature point tracking algorithm. ,when When, according to The size increases in direct proportion to The size of No optimization is needed.

Images