Building video generation method based on natural language and computer program product

Abstract

The invention provides a natural language-based building video generation method and a computer program product, and relates to the technical field of artificial intelligence generation, and the method comprises the steps: collecting building images of a plurality of visual angles to form a first image set; performing size unification processing on the first image set to obtain a preprocessed image set; building feature recognition is carried out on the preprocessed image set through a pre-training language model, and a feature description text is generated to form a training data set; training the LoRA model by adopting the training data set; inputting the feature description text of the target building into the trained LoRA model, and generating a plurality of building images of different visual angles to form a second image set; inputting the second image set into a video generation model to generate a building video; performing post-processing optimization; and converting the video into a target output format. According to the method, the accurate matching degree of video generation is improved, the consistency and coherence of videos are enhanced, and meanwhile, the system resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence generation technology, and in particular to a natural language-based architectural video generation method and a computer program product. Background Art

[0002] In recent years, generative AI technology has shown great potential in the fields of architectural design and visual display. The method of generating architectural videos through natural language can transform text descriptions into intuitive visual effects, greatly improving the visualization efficiency of architectural design and enabling designers and customers to better understand and communicate design intentions. At present, conventional technologies based on natural language generation of videos mainly include natural language processing (NLP), image generation model (such as LoRA model) training and video synthesis.

[0003] In the prior art, the system first understands and annotates the input image through the NLP model to extract feature information such as architectural style, structure, material and color. NLP technology usually relies on pre-trained language models (such as BERT, GPT, Llama) to convert images into architectural feature annotations through methods such as keyword extraction and dependency analysis. Then, based on these annotation information, the LoRA model is used to generate architectural images that meet the characteristics. During the training phase, the LoRA model learns the correspondence between architectural features and images through a large number of samples, thereby generating architectural images with specific structures and styles. Finally, the generated architectural images are combined with text descriptions, and the camera perspective and transition effects are set to generate a coherent architectural dynamic video.

[0004] However, the existing technology has deficiencies in many aspects. First, the NLP model has insufficient understanding ability when dealing with complex architectural descriptions, and it is difficult to accurately capture all architectural features, resulting in low annotation accuracy and affecting the accuracy of the generated images. Secondly, the LoRA model is highly dependent on training samples when generating pictures, resulting in poor consistency in the generated images, especially when showing multi-angle or complex structure buildings, it is difficult to ensure the integrity and coherence of the details. In terms of video synthesis, the perspective and path of traditional methods are fixed, and the generated frames lack fluidity and spatial sense, making it difficult to show a coherent architectural dynamic effect. In addition, since the generation process includes multiple steps, large amount of calculation, reliance on high-performance resources, and high generation cost, it is difficult to meet the needs of large-scale applications. Summary of the invention

[0005] The purpose of the present invention is to provide a natural language-based architectural video generation method and a computer program product, which can improve generation accuracy, optimize visual effects, and reduce resource consumption compared to the prior art.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A method for generating architectural videos based on natural language comprises the following steps:

[0008] S1: collect building images to form a first image set;

[0009] S2: performing image preprocessing on the first image set to form a preprocessed image set;

[0010] S3: Processing and describing the preprocessed image set using a natural language processing model to form a training data set;

[0011] S4: Based on the diffusion model, the LoRA model is trained using the training data set;

[0012] S5: Generate building images with specific features using the LoRA model to form a second image set;

[0013] S6: Generate a building video based on a diffusion model by combining the text description and the second image set;

[0014] S7: performing post-processing optimization on the building video;

[0015] S8: Output the optimized building video.

[0016] Furthermore: the image preprocessing in S2 includes: unifying the images in the first image set into the same length and width using an image processing algorithm.

[0017] Further: S3 includes: using a natural language processing model to generate a corresponding architectural feature description for each image in the preprocessed image set, and pairing the image with the text description to form the training data set.

[0018] Further: S4 includes: optimizing and training the LoRA model through a loss function so that it can generate building images based on text features.

[0019] Further: the S6 includes: inputting a text description and the second image set, interpolating the image sequence to generate video frames, and outputting the building video.

[0020] Furthermore: the diffusion process of the diffusion model satisfies:

[0021] x t′ =a t x0+σ t ∈

[0022] Among them, x0 is the original image, x t′ is the image at time t′, ∈ is the standard Gaussian noise, a t and σ tis a parameter used to control the noise level.

[0023] Further: the S7 includes: performing color enhancement, motion blur and inter-frame interpolation processing on the building video.

[0024] Further: S8 includes: converting the optimized building video into a target resolution, frame rate and format.

[0025] The present invention also provides a computer program product, comprising a computer program stored on a computer-readable storage medium, wherein the computer program implements the steps of the above method when executed by a processor; wherein the computer program is configured with functional nodes corresponding to each of the steps in the node system of the ComfyUI framework.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. This method improves the accuracy of matching between natural language description and video generation through the systematic design of architectural image data preprocessing, feature recognition, LoRA model training and video generation. The present invention uses a pre-trained language model to perform feature recognition on architectural images, extract multi-dimensional features such as architectural style, structure, material and color, and generate detailed feature description text, so that the generated architectural video can more accurately reflect the design intent of the text description, effectively solving the problem of insufficient understanding ability of NLP models in the prior art.

[0028] Second, the consistency and coherence of the generated video are enhanced. Through targeted training of the LoRA model and video generation based on the preset camera motion path, the architectural images generated from different perspectives maintain a unified style and complete details. At the same time, the diffusion model is used for inter-frame transition generation, combined with the optical flow algorithm and dynamic blur optimization post-processing solution, which significantly improves the smoothness and spatial expression of the video, overcoming the lack of coherence between video frames in traditional methods.

[0029] 3. Reduced system resource consumption. The present invention reduces redundant calculations and improves generation efficiency through a reasonably designed data preprocessing process and model training strategy. At the same time, the use of targeted video compression and format conversion solutions enables the system to adapt to the playback requirements of different platforms, while ensuring video quality and reducing storage and transmission costs, making large-scale applications possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of a flowchart of a method for generating architectural videos based on natural language in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0033] The natural language-based architectural video generation method provided by the present invention has a complete implementation process comprising the following steps:

[0034] First, collect architectural images from multiple perspectives to form a first image set. Specifically, collect multi-angle images containing the target architectural style and structural features. These images should be representative and fully display various aspects of the architectural features.

[0035] Next, each image in the first image set is resized to obtain a preprocessed image set. The present invention uses the resize function in the Pillow library of Python to achieve image resizing. The function uses a bicubic interpolation algorithm to resize the image. Specifically, when the image is resized from the original size (H oriy ,W orig ) is scaled to the target size (H target ,W target ), the scaling factors are:

[0036]

[0037] Among them, scale H and scale w Represents the scaling ratio in the vertical and horizontal directions respectively. Bicubic interpolation uses four convolutions to calculate the weights of the surrounding 16 pixels. For the target position (x, y), it is calculated by the weighted sum of the surrounding pixels f(i, j):

[0038]

[0039] Among them, the weight w(i,j) is calculated by the bicubic convolution kernel R(x):

[0040]

[0041] Here, the parameter a is usually taken as -0.5, which is called the Catmull-Rom kernel, which ensures a smooth and natural scaling effect.

[0042] Then, the pre-trained language model is used to identify architectural features of the pre-processed image set. The present invention uses the BERT pre-trained language model to identify the structural type, style, building material, color characteristics and spatial layout characteristics of the building through keyword extraction and dependency analysis. Generate corresponding feature description text for each image, and pair the image with the text to form a training data set X2:

[0043] X2={(x i ,t i )|x i ∈X 1-1 ,t i =g(x i )}

[0044] Among them, t i is the image x i , and g is the NLP annotation model.

[0045] After obtaining the training data set, the LoRA model is trained. The training process is achieved through loss function optimization:

[0046]

[0047] Among them, f LoRA Represents the LoRA model, θ is the model parameter, and L is the loss function, which is used to measure the difference between the generated image and the target image. The specific loss function thresholds used in model training are: weight update threshold: 0.05 (weight updates below this value will be ignored to reduce the amount of calculation); early stopping threshold: 0.001 (early stopping is triggered when the validation set loss is less than this value); learning rate: initial value 1e-4, using learning rate decay.

[0048] After the model training is completed, input the feature description text of the target building to generate multiple building images from different perspectives:

[0049]

[0050] Among them, t is the input description, The generated image.

[0051] Next, the generated building image set is input into the video generation model. The present invention adopts a diffusion model to realize video generation, and the diffusion process satisfies:

[0052] x t′=a t x0+σ T ∈

[0053] Among them, x0 is the building image generated by LoRA, x T′ is the image at time t′, is standard Gaussian noise, a T and σ T is a parameter that controls the noise level.

[0054] The generated video is post-processed and optimized, including: using the optical flow algorithm to supplement frames and improve the frame rate; performing HDR color mapping to enhance the video color performance; and optimizing dynamic blur based on motion vectors to improve video smoothness. Finally, according to the requirements of the target platform, the video is converted to the corresponding encoding format, resolution, and frame rate, and appropriately compressed to adapt to different bandwidth conditions.

[0055] The method of the present invention can be implemented in the ComfyUI framework, and the entire processing flow is completed by configuring the corresponding function nodes. Practice shows that the method provided by the present invention can effectively improve the accuracy and visual effect of architectural video generation, while reducing system resource consumption, and has good application value.

[0056] The above is a brief introduction to the present invention. Weight update threshold: 0.05 (weight updates below this value will be ignored to reduce the amount of calculation)

[0057] Early stopping threshold: 0.001 (early stopping is triggered when the validation set loss is less than this value)

[0058] Learning rate: initial value 1e-4, the present invention is described in detail using the learning rate decay example.

[0059] A method for generating architectural videos based on natural language comprises the following steps:

[0060] S1: collect building images to form a first image set;

[0061] Among them, images are collected according to the age, style and type of buildings. For example, the image collection of Liao Dynasty brick and stone towers, and so on, specifically collect image collections of specific historical building ages, styles and specific building types.

[0062] Image sets have the following specific requirements:

[0063] ① Resolution above 6MP;

[0064] ② Do not allow watermarks to appear in photos;

[0065] ③ Do not include people in the photo;

[0066] ④It is best if the pictures are medium to long-range photos;

[0067] ⑤ Make the composition as straight as possible, and avoid camera rotation and distortion;

[0068] ⑥ Do not use a perspective view where the camera's horizontal angle to the building exceeds 75°;

[0069] ⑦ Do not include model photos;

[0070] ⑧It can be a picture from a human perspective or a picture of the middle of the building taken from a drone perspective. Do not use a high-angle bird's-eye view.

[0071] S2: performing image preprocessing on the first image set to form a preprocessed image set;

[0072] In some embodiments, the image preprocessing in S2 includes: unifying the images in the first image set into the same length and width using an image processing algorithm.

[0073] S3: Processing and describing the preprocessed image set using a natural language processing model to form a training data set;

[0074] In some other embodiments, S3 includes: using a natural language processing model to generate a corresponding architectural feature description for each image in the preprocessed image set, and pairing the image with the text description to form the training data set.

[0075] S4: Based on the diffusion model, the LoRA model is trained using the training data set;

[0076] In other embodiments, the LoRA model is trained by optimizing a loss function so that it can generate building images based on text features.

[0077] S5: Generate building images with specific features using the LoRA model to form a second image set;

[0078] S6: Generate a building video based on a diffusion model by combining the text description and the second image set;

[0079] In some other embodiments, the S6 includes: inputting a text description and the second image set, interpolating the image sequence to generate video frames, and outputting the building video.

[0080] The diffusion process of the diffusion model satisfies:

[0081] x T′ =a T x0+σ t ∈

[0082] Among them, x0 is the original image, x T′ is the image at time t′, ∈ is the standard Gaussian noise, at and σ t is the parameter used to control the noise level;

[0083] The concept of architectural video refers to the dynamic image of a building, which aims to show the space, structure or architectural history of the building through visual means. The specific viewing trajectory is to shoot with a drone, with the flight altitude located in the middle of the building height, and to perform 360° center-building positioning shooting around the target building.

[0084] S7: performing post-processing optimization on the building video;

[0085] In some other embodiments, the S7 includes: performing color enhancement, motion blur and inter-frame interpolation processing on the building video.

[0086] Among them, color enhancement: adjust the saturation, contrast, brightness and other parameters for the actual effect;

[0087] Motion Blur: Remove the motion blur enhancement, because I need a clear video of the target building details and surrounding environment without motion blur;

[0088] Inter-frame interpolation: Increase the interpolation rate (FPS) of the video and add frames to the video to make the video clearer and smoother.

[0089] S8: Output the optimized building video.

[0090] In some other embodiments, S8 includes: converting the optimized architectural video into a target resolution, frame rate and format.

[0091] In addition, in another embodiment, the present invention also provides a computer program product, including a computer program stored on a computer-readable storage medium, and the computer program implements the steps of the above method when executed by a processor; wherein the computer program is configured with functional nodes corresponding to each of the steps in the node system of the ComfyUI framework.

[0092] The above embodiments are only preferred embodiments of the present invention. Those skilled in the art may make various changes or modifications to the present invention without departing from the principle and essence of the present invention, all of which fall within the protection scope of the present invention.

[0093] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating architectural videos based on natural language, characterized in that: The following steps are involved: S1: collect building images to form a first image set; S2: performing image preprocessing on the first image set to form a preprocessed image set; S3: Processing and describing the preprocessed image set using a natural language processing model to form a training data set; S4: Based on the diffusion model, the LoRA model is trained using the training data set; S5: Generate building images with specific features using the LoRA model to form a second image set; S6: Generate a building video based on a diffusion model by combining the text description and the second image set; S7: performing post-processing optimization on the building video; S8: Output the optimized building video.

2. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The image preprocessing in S2 includes: unifying the images in the first image set into the same length and width using an image processing algorithm.

3. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The S3 includes: using a natural language processing model to generate a corresponding architectural feature description for each image in the preprocessed image set, and pairing the image with the text description to form the training data set.

4. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The S4 includes: optimizing and training the LoRA model through a loss function so that the model can generate building images based on text features.

5. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The S6 includes: inputting a text description and the second image set, interpolating an image sequence to generate a video frame, and outputting the building video.

6. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The diffusion process of the diffusion model satisfies: x t′ =a t x0+σ t ∈ Among them, x0 is the original image, x t′ is the image at time t′, ∈ is the standard Gaussian noise, a t and σ t is a parameter used to control the noise level.

7. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The S7 includes: performing color enhancement, motion blur and inter-frame interpolation processing on the building video.

8. The method for generating architectural videos based on natural language according to claim 1, characterized in that: The S8 includes: converting the optimized building video into a target resolution, frame rate and format.

9. A computer program product, characterized in that It includes a computer program stored on a computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 8; wherein, the computer program is configured with a functional node corresponding to each of the steps in the node system of the ComfyUI framework.

Images