A structured light image synthesis method based on illumination decoupling and large language model agent

Abstract

The application discloses a structured light image synthesis method based on illumination decoupling and a large language model agent, comprising: obtaining an original image and multi-dimensional configuration parameters input by a user, and generating a structured prompt word by using a large language model agent; extracting a light bar mapping image and generating a constraint mask for limiting a reconstruction range according to a spatial geometric relationship of the original image; inputting the original image, the structured prompt word and the constraint mask into a first-stage basic generation network, decoupling illumination features in network forward propagation, and generating a basic base map; performing feature extraction on the basic base map and inputting the basic base map into a second-stage detail refinement network to obtain texture gating weights; and performing image fusion by using a physical rendering method based on material reflectivity according to the basic base map, the light bar mapping image and the texture gating weights, and outputting a synthesized image. The application realizes high-quality generation and natural fusion of a target object, and improves the structural integrity and visual authenticity of a generated image.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of computer vision, natural language processing and generative artificial intelligence, and particularly relates to a structured light and shadow image synthesis method based on illumination decoupling and large language model proxy. Background Technology

[0002] In digital interior design and whole-house customization scenarios, naturally integrating structured objects with specific physical forms and complex lighting properties (such as curtains, blinds, and other interior furnishing components with regular light-blocking, light-transmitting, or folding characteristics) into the user's original image of the real interior scene is a core image editing and generation technology. Existing technologies are mainly based on diffusion models or generative adversarial networks, performing local image editing by inputting user commands and masked regions; however, these methods have serious shortcomings in practical applications.

[0003] First, existing models typically rely solely on simple feature words input by the user as guiding conditions, lacking a deep semantic understanding of the physical form of objects and a mandatory constraint mechanism. This makes it difficult for the models to accurately respond to the diverse needs of users for the specific physical form of target objects, resulting in generated object forms that are seriously inconsistent with actual physical requirements.

[0004] Secondly, when processing structured lighting, existing image generation networks typically inject the lighting map globally into each layer of the network. This global injection method leads to excessive interference of deep features with shallow features, which smooths out local high-frequency details of the image while transmitting the lighting distribution, resulting in blurring and seriously affecting the quality and realism of the generated image.

[0005] Furthermore, existing technologies only impose pixel-level constraints on the masked area when generating local objects, lacking global constraints on the overall spatial geometry and perspective relationships of the original image. This makes it easy for the model to introduce structural changes that are inconsistent with the original scene, producing an "illusion" phenomenon and disrupting the spatial layout and geometric consistency of the user's original scene.

[0006] In addition, existing methods often use simple global addition or multiplication operations when overlaying structured lighting, ignoring the differences in lighting response of different object surface materials (such as specular reflective materials and diffuse reflective materials) in the real world. This results in the generated results lacking physical realism in light and shadow transitions, specular distribution and shadow representation, and the visual effect is stiff and unnatural.

[0007] Therefore, there is an urgent need for a structured light and shadow image synthesis method that can achieve precise control of physical form, maintain scene spatial structure, preserve local high-frequency details, and possess a physical realism of material differences. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a structured light and shadow image synthesis method based on illumination decoupling and a large language model proxy, comprising the following steps: Obtain the original image and multi-dimensional configuration parameters input by the user, and use a large language model proxy to generate structured prompt words; Based on the spatial geometric relationships of the original image, extract the light stripe map and generate a constraint mask to limit the reconstruction range; The original image, structured prompts, and constraint masks are input into the first-stage basic generation network, and the illumination features in the forward propagation of the network are decoupled to generate a base map that retains the structure of non-target regions. Feature extraction is performed on the base map and input into the second-stage detail refinement network to obtain texture gating weights; Based on the base map, light stripe map, and texture gating weights, a physical rendering method based on material reflectivity is used for image fusion to output a composite image.

[0009] Optionally, generating structured prompt words using a large language model proxy includes: Identify the parameter types in the multidimensional configuration parameters; When a preset physical form parameter is detected, the condition enhancement logic is triggered to add a mandatory interpretation instruction for that physical form and an anti-hallucination instruction to prevent environmental changes to the parameter description. The description text following the appended instructions is translated by a large language model agent, and the output is unambiguous structured prompt words.

[0010] Optionally, based on the spatial geometric relationships of the original image, the extraction of the light stripe mapping map includes: Using Fast Fourier Transform, the lighting texture that matches the window perspective relationship in the original image is extracted, and a light stripe map with pixel value normalization is generated.

[0011] Optionally, generating a constraint mask to limit the scope of the reconstruction includes: Combining the semantic segmentation results of the original image with scene protection instructions, the region where the target object needs to be generated is delineated as the editable area using a binarization matrix, while the remaining background area is designated as the frozen area, in order to establish the constraint mask.

[0012] Optionally, the original image, structured cue words, and constraint mask are input into the first-stage basic generation network, and the illumination features in the network's forward propagation are decoupled to generate a base map that preserves the structure of non-target regions, including: The first-stage basic generation network includes a downsampling module, an intermediate module, and an upsampling module; During network forward propagation, the light stripe map is injected as a feature of the illumination map and concatenated only into the downsampling module of the generated network; And disconnect the residual connections of the illumination map features in the intermediate module and the upsampling module; The final output is a base map that preserves the structure of the non-target regions.

[0013] Optionally, feature extraction is performed on the base map and input into the second-stage detail refinement network to obtain texture gating weights, including: Extract the Laplacian high-frequency feature map, foreground grayscale map, and edge extraction map from the base map; The Laplacian high-frequency feature map, foreground grayscale map, and edge extraction map are fused and stitched with the base map in the channel dimension to construct a multi-channel feature tensor. The multi-channel feature tensor is input into the second-stage detail refinement network to obtain texture gating weights.

[0014] Optionally, based on the base map, light stripe map, and texture gating weights, a physically based rendering method using material reflectivity is employed for image fusion, and the output synthesized image includes: Using the texture gating weight as an adjustment factor, and utilizing the global illumination intensity coefficient, the light stripe map is multiplied by the texture gating weight, and then fused with the base map in a multiply-add manner to output the synthesized image.

[0015] On the other hand, the present invention also provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0016] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0017] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0018] Compared with the prior art, the present invention has the following advantages and technical effects: This invention utilizes a large language model agent to generate structured prompts based on the user-input original image and multi-dimensional configuration parameters. This transforms the user's fuzzy natural language requirements into precise guidance conditions that include physical form constraints and scene preservation instructions, thereby achieving precise control over the spatial physical form of the target object and effectively solving the problem that existing models struggle to understand complex physical states.

[0019] This invention extracts a light stripe map that matches the target area based on the spatial geometric relationship of the original image and generates a constraint mask to limit the reconstruction range. This enables precise delineation of editable areas and freezing of background areas during subsequent generation, thereby forming a global constraint on the spatial structure and perspective relationship of the original image. This effectively avoids the generation of content from altering non-target areas, eliminates background illusions, and maintains the consistency and credibility of the spatial layout of the user's original scene.

[0020] This invention generates a base map that preserves the structure of non-target regions by inputting the original image, structured cue words, and constraint masks into the generative network and decoupling the lighting features during the forward propagation of the network. This limits the transmission of lighting information to the shallow layers of the network, preventing excessive interference of deep features with shallow high-frequency textures. As a result, while preserving the global lighting distribution, the local edges and surface details of the object are fully preserved, overcoming the defects of texture blurring and detail loss in traditional methods.

[0021] This invention extracts features from a base map and inputs them into a thinning network to obtain texture gating weights. This allows for the accurate capture and differentiation of the reflective properties of different material surfaces, providing prior material information for subsequent light and shadow fusion. This solves the problem of harsh light and shadow caused by existing methods' inability to distinguish between specular and diffuse materials.

[0022] This invention uses a physical rendering method based on material reflectivity to fuse and output a composite image based on a base map, light stripe map, and texture gating weights. This enables light intensity to produce differentiated responses on different material surfaces that conform to real physical laws, achieving a natural fusion of structured light and shadow with the object surface. This significantly improves the physical realism and visual naturalness of the generated image in terms of light and shadow transitions, highlight distribution, and shadow representation.

[0023] In summary, through the synergistic effect of the above-mentioned technical means, the present invention has achieved excellent results in physical shape control, scene structure preservation, high-frequency detail retention, and material-differentiated lighting fusion. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0027] Example 1 like Figure 1 As shown, this embodiment provides a structured light and shadow image synthesis method based on illumination decoupling and large language model proxy, including the following steps: Obtain the original image and multi-dimensional configuration parameters input by the user, and use a large language model proxy to generate structured prompt words; Based on the spatial geometric relationships of the original image, extract the light stripe map and generate a constraint mask to limit the reconstruction range; The original image, structured prompts, and constraint masks are input into the first-stage basic generation network, and the illumination features in the forward propagation of the network are decoupled to generate a base map that retains the structure of non-target regions. Feature extraction is performed on the base map and input into the second-stage detail refinement network to obtain texture gating weights; Based on the base map, light stripe map, and texture gating weights, a physical rendering method based on material reflectivity is used for image fusion to output a composite image.

[0028] As a specific implementation method, this embodiment uses a large language model agent to perform intent recognition and semantic reconstruction based on the user-input original image and multi-dimensional configuration parameters, generating structured positive prompts with physical state interpretation and anti-hallucination instructions; based on the spatial geometry and perspective relationship of the original image, feature alignment is performed to extract a basic light stripe mapping map matching the target region, and combined with the structured positive prompts, a scene-preserving constraint mask is established to limit the reconstruction range; the original image, the structured positive prompts, and the scene-preserving constraint mask are input into the first-stage basic generation network, and then... During the forward propagation of the network, a lighting feature decoupling strategy is executed to generate a base map that retains the perspective structure of non-target areas in the original scene and includes basic lighting and target object structure. Multi-channel high-frequency edge features are extracted from the base map, and the extracted features are input into the second-stage detail refinement network for residual learning. The network outputs a detail residual map for correcting local edges and texture gating weights for distinguishing material reflectivity. Based on the corrected base map, the base light stripe map, and the texture gating weights, parameterized calculations are performed using a preset physically realistic material reflectivity rendering formula to obtain the final synthesized image.

[0029] The feasible method for establishing the structured positive prompt words specifically includes: identifying parameter types based on multi-dimensional configuration parameters input by the user; if a parameter containing a specific physical form is identified, conditional enhancement logic is triggered to add a mandatory interpretation instruction for the specific physical form and an anti-hallucination instruction to prevent environmental changes to the parameter description; based on the description text after the added instructions, an unambiguous English structured positive prompt word is translated and output through a large language model proxy.

[0030] The feasible method of extracting a light stripe map based on the spatial geometric relationship of the original image includes: using a fast Fourier transform to extract the lighting texture that matches the window perspective relationship in the original image, and generating a light stripe map with pixel value normalization.

[0031] Furthermore, generating a constraint mask to limit the reconstruction range includes: combining the semantic segmentation results of the original image with scene protection instructions, using a binarization matrix to delineate the area where the target object needs to be generated as the editable area, and the remaining background area as the frozen area, in order to establish the constraint mask.

[0032] The method for obtaining the base map by executing the illumination feature decoupling strategy in the network forward propagation is feasible and specifically includes: differentially allocating the injection of illumination map features based on the downsampling module, intermediate module and upsampling module in the first-stage basic generation network; injecting and splicing the illumination map features only into the downsampling module, and disconnecting the residual connections of the illumination map features in the intermediate module and the upsampling module to isolate the interference of global illumination on shallow high-frequency textures, thereby generating the base map.

[0033] The feasible method involves extracting features from the base map and inputting them into a second-stage detail refinement network to obtain texture gating weights. This includes: extracting a Laplacian high-frequency feature map, a foreground grayscale map, and an edge extraction map from the base map; fusing and stitching the Laplacian high-frequency feature map, the foreground grayscale map, and the edge extraction map with the base map in the channel dimension to construct a multi-channel feature tensor; and inputting the multi-channel feature tensor into the second-stage detail refinement network to obtain texture gating weights.

[0034] The feasible method for obtaining the final synthesized image specifically includes: using the texture gating weight as an adjustment factor to distinguish the light response intensity of different materials; using the global illumination intensity coefficient, multiplying the base light stripe map with the texture gating weight and then adding it to the base map to complete the fusion of light and shadow features and output the final synthesized image.

[0035] This embodiment provides a structured lighting and shadow image generation method that combines a large language model agent with multimodal feature fusion. It can dynamically construct strongly constrained cue words based on user parameters using an LLM agent, precisely controlling the physical spatial form of objects. It can decouple lighting features from the image generation network, effectively isolating excessive global illumination interference with high-frequency textures. By constructing a scene constraint mechanism, it can limit the image reconstruction area, ensuring the generation effect of the target area while maintaining the structural stability and spatial consistency of non-target areas in the original scene. Lighting and shadow details can be verified and corrected through high-frequency multi-channel feature input and physical rendering equations. This application achieves high-quality generation and natural integration of target objects through dynamic reconstruction of large model cue words and low-level lighting decoupling and physical-level fusion, thereby improving the structural integrity and visual realism of the generated images.

[0036] Example 2 This embodiment provides a structured light and shadow image generation method that combines large language model proxy and multimodal feature fusion, and is particularly suitable for smart curtain generation. The method specifically includes the following steps: Step S1: Based on the original image and multi-dimensional configuration parameters input by the user, a structured positive prompt word is established using a large language model agent.

[0037] The system first obtains the original indoor scene image and configuration parameters uploaded by the user through the front-end interactive interface, such as opening and closing method, curtain style, and shading rate. The large language model agent performs intent recognition and parameter type classification on the received parameters. When specific parameters related to physical form are recognized, such as "single-open" and "tied up," the system triggers conditional enhancement logic. The agent automatically adds anti-illusion and physical state interpretation constraint instructions to the basic description. The constraint instructions generated internally by the system are: "Mandatory constraints: single-opening curtains must be opened from one side; curtains must be in a tucked-up state; all original elements in the target room except for the windows must be preserved, and the room layout must not be changed." Subsequently, the LLM Agent translates and reconstructs the integrated instructions into English structured positive prompts that conform to the generative network input specifications, serving as global text guidance conditions for the subsequent generation process.

[0038] As an additional implementation method, the implementation of the large language model agent in step 1 can be replaced by a lightweight language model that has been fine-tuned by a specific prompt word project and deployed locally or on a private server, in addition to calling the general large language model through the cloud API. This can also complete the semantic reconstruction of the conditional text and the translation of prompt words.

[0039] Step S2: Based on spatial feature alignment, extract the basic light stripe map and establish the scene-preserving constraint mask.

[0040] To achieve accurate lighting and shadow mapping and preserve the original scene, the system analyzes the spatial geometry and perspective of the input indoor scene image. Using Fast Fourier Transform, it extracts the lighting texture matching the window perspective of the original image, generating a basic light stripe mapping map with pixel values ​​normalized to [0,1]. Simultaneously, combining the semantic segmentation results of the original image with the scene preservation instructions in step S1, the system establishes a scene-preserving constraint mask to limit the image reconstruction range. This mask, in the form of a binary matrix, precisely delineates the window area where curtains need to be generated as the editable area, while the remaining indoor background, such as walls and furniture, is frozen, thus eliminating the possibility of the generated model producing a "background illusion" from the ground up.

[0041] Step S3: Execute the lighting decoupling strategy through the first-stage basic generation network to generate a base map containing the basic structure and lighting.

[0042] The original image features, the structured positive prompts, and the scene-preserving constraint mask are simultaneously input into a diffusion model based on the U-Net architecture. To address the technical pain point of traditional models where global illumination maps easily lead to the loss of local high-frequency details, this embodiment innovatively introduces an illumination feature decoupling strategy in the network's forward propagation. In the specific execution of the U-Net architecture, the system injects and concatenates the illumination map features only into the encoder's downsampling module; while in the intermediate module and the decoder's upsampling module, the connection of the illumination map features is forcibly disconnected. Through this decoupling strategy, the network effectively learns the global illumination distribution at a deep level, while isolating illumination interference at a shallow level to preserve high-frequency textures, ultimately outputting a base map with accurate structure and harmonious basic illumination. .

[0043] Step S4: Extract multi-channel high-frequency features based on the base map, and obtain the detail residual map and texture gating weights through the second-stage detail refinement network.

[0044] Since the base map may not be refined enough in terms of edge physical contours and material micro-properties, this embodiment introduces a second-stage detail refinement network for residual learning. The system first processes the base map generated in step S3. (3-channel RGB features) The Laplacian high-frequency feature map, foreground grayscale image, and edge extraction map are extracted. These features are fused and concatenated along the channel dimension to construct a 6-channel feature tensor containing rich edge and texture information, which is then input into the detail refinement network. After calculation, the detail refinement network outputs a detail residual map used to correct local edge artifacts. And a normalized single-channel matrix, texture gating weights The numerical distribution of this weight matrix accurately maps the local material reflectivity differences, which is used to distinguish between specular and diffuse regions.

[0045] Step S5: Use the physical rendering formula to fuse the base image, light stripe map and texture gating weights, calculate and output the final composite image.

[0046] In the final lighting and shadow blending stage, the system abandons traditional global addition or direct multiplication, and instead performs parametric physically based lighting rendering based on the actual material reflectivity distribution. This forms the base map after detail correction. The basic light stripe mapping map extracted in step S2 and the texture gating weights output in step S4 Substitute the following physical rendering formulas for calculation: ); In the formula, Represents the final synthesized image; A parameterizable global illumination intensity coefficient. Based on the light stripe map, This is a texture gating weight used to distinguish between different material responses such as specular and diffuse reflection. Through this multiplicative formula mechanism, the light stripe map can dynamically overlay the light spots of blinds or the light stripes of curtain folds onto the base map; simultaneously, it is subject to the texture gating weight. Due to limitations, the light gain exhibits reasonable physical differences on different material surfaces, such as being brighter in the high-reflection area and attenuating in the dark area, thus giving the final generated curtain image a good sense of physical realism and spatial integration.

[0047] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A structured light and shadow image synthesis method based on illumination decoupling and large language model proxy, characterized in that, Includes the following steps: Obtain the original image and multi-dimensional configuration parameters input by the user, and use a large language model proxy to generate structured prompt words; Based on the spatial geometric relationships of the original image, extract the light stripe map and generate a constraint mask to limit the reconstruction range; The original image, structured prompts, and constraint masks are input into the first-stage basic generation network, and the illumination features in the forward propagation of the network are decoupled to generate a base map that retains the structure of non-target regions. Feature extraction is performed on the base map and input into the second-stage detail refinement network to obtain texture gating weights; Based on the base map, light stripe map, and texture gating weights, a physical rendering method based on material reflectivity is used for image fusion to output a composite image.

2. The method according to claim 1, characterized in that, Generating structured prompt words using a large language model proxy includes: Identify the parameter types in the multidimensional configuration parameters; When a preset physical form parameter is detected, the condition enhancement logic is triggered to add a mandatory interpretation instruction for that physical form and an anti-hallucination instruction to prevent environmental changes to the parameter description. The description text following the appended instructions is translated by a large language model agent, and the output is unambiguous structured prompt words.

3. The method according to claim 1, characterized in that, Based on the spatial geometric relationships of the original image, the extraction of the light stripe mapping map includes: Using Fast Fourier Transform, the lighting texture that matches the window perspective relationship in the original image is extracted, and a light stripe map with pixel value normalization is generated.

4. The method according to claim 1, characterized in that, Generating a constraint mask to limit the scope of the reconstruction includes: Combining the semantic segmentation results of the original image with scene protection instructions, the region where the target object needs to be generated is delineated as the editable area using a binarization matrix, while the remaining background area is designated as the frozen area, in order to establish the constraint mask.

5. The method according to claim 1, characterized in that, The original image, structured cue words, and constraint masks are input into the first-stage basic generation network, and the illumination features in the network's forward propagation are decoupled to generate a base map that preserves the structure of non-target regions, including: The first-stage basic generation network includes a downsampling module, an intermediate module, and an upsampling module; During network forward propagation, the light stripe map is injected as a feature of the illumination map and concatenated only into the downsampling module of the generated network; And disconnect the residual connections of the illumination map features in the intermediate module and the upsampling module; The final output is a base map that preserves the structure of the non-target regions.

6. The method according to claim 1, characterized in that, Feature extraction is performed on the base map and input into the second-stage detail refinement network to obtain texture gating weights, including: Extract the Laplacian high-frequency feature map, foreground grayscale map, and edge extraction map from the base map; The Laplacian high-frequency feature map, foreground grayscale map, and edge extraction map are fused and stitched with the base map in the channel dimension to construct a multi-channel feature tensor. The multi-channel feature tensor is input into the second-stage detail refinement network to obtain texture gating weights.

7. The method according to claim 1, characterized in that, Based on the base map, light stripe map, and texture gating weights, an image fusion method based on material reflectivity is used to perform image fusion, and the output synthesized image includes: Using the texture gating weight as an adjustment factor, and utilizing the global illumination intensity coefficient, the light stripe map is multiplied by the texture gating weight, and then fused with the base map in a multiply-add manner to output the synthesized image.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-7.

Images