A scene rendering method and device for fine-grained semantic manipulation

Abstract

The application relates to the technical field of scene rendering, and discloses a scene rendering method for fine-grained semantic control, which can describe fine-grained semantics of a demand text by constructing a dependency syntax tree in a semantic understanding unit, can identify fine-grained information in the demand text, can accurately locate a picture area, and can realize high matching between a rendering result and the demand text. In addition, the demand text is converted into the dependency syntax tree, and a reasoning module is assembled on the tree, so that the association between scene graph features and text features is strengthened, and positioning is more accurate. A regularizer is designed to constrain a scene generation model, can accurately modify visual information of a target area, and will not affect other irrelevant areas. The application does not need manual delineation of an editing area, can directly understand the demand of the text, can perform fine-grained local editing on an original scene graph, and can realize convenient and controllable rendering by allowing a user to propose editing demands in a text form, add, delete and modify the scene graph.

Description

Technical Field

[0001] The present invention relates to the technical field of scene rendering, in particular to a fine-grained semantic manipulation-based scene rendering method and apparatus. Background Art

[0002] With the rapid development of artificial intelligence and the continuous improvement of people's living standards, the demand for spatial design of indoor and outdoor scenes and the like is constantly increasing. Visualized scene design has become an important application scenario, and the corresponding rendering technology has gradually gained extensive attention from researchers. Whether in the art field, scientific research field or engineering field, research on applying computer technology to scene design is increasingly popular. The traditional scene design process generally first carries out user demand analysis, then a designer assembles scene components according to the demands, and finally presents a visualized result to the user through rendering. Rendering involves a large number of calculation steps, requiring the use of high-performance machines and consuming a large amount of running time, and some complex scenes even take several days. Moreover, in the field of spatial design, users' designs are usually still vague. Some users cannot even determine specific design requirements. They tend to view effect drawings and make modifications to confirm the design scheme. For example, in indoor scene design, a user is dissatisfied with a certain wall and requests to modify it into a window to increase light. For such a demand, the designer needs to manually delineate the area to be modified, adjust the relevant scene graph components, for example, find a small window patch to replace the wall area, and then re-render the global scene for output. In response to the variable demands of users, global rendering will undoubtedly multiply the design cost and restrict the development of scene design towards simplification and popularization. One solution is to provide interactive scene design and fast preview functions to make indoor design intelligent and scene rendering fast. Users only need to directly submit a modification demand description in the form of text, and the machine can understand the user's demand through natural language processing technology, automatically locate the area to be modified, perform editing operation on the area and then render the result. This manner of local rendering rather than global rendering can effectively reduce the cost. However, this simple manner involves two types of challenges, including how to understand the user's text description to accurately locate the operation area, and the problems of incompatibility and inconsistency between the local rendering result and the surrounding area.

[0003] Traditional rendering techniques generally employ two classic methods: rasterization and ray tracing. Rasterization is primarily used for real-time rendering. It first maps the vertices of a 3D scene onto a 2D screen, then rasterizes these vertices into fragments of the screen, and then colors each fragment based on lighting calculations. Finally, the colored fragments, after being sorted and discarded, are sent to a color buffer and processed through a swap chain in the graphics interface to become the colors displayed on the screen. Compared to rasterization, ray tracing is more realistic, but due to the exponential increase in the number of rays, computation time is significantly increased. Ray tracing tracks the light rays projected from the camera into the scene, attempting to find possible paths between objects and light sources to simulate the realistic behavior of light in a medium to output the rendered image. This rendering method typically updates the entire scene graph, making it difficult to modify local areas. When dealing with frequently changing user modification requests, this global update approach is very costly.

[0004] To address this problem, a simple approach is to use locally editable rendering techniques. Currently, the mainstream methods for this task can be categorized into two types. The first type requires the user to manually define the area to be updated and then edit that area. Editing can be done by using a search engine to retrieve relevant content for replacement. Alternatively, neural networks can be used, mapping to a high-dimensional latent space to learn the relationships between visual information in different regions, thereby identifying relevant content for editing. However, replacement-based retrieval methods struggle to ensure compatibility between the edited area and neighboring areas, easily leading to inconsistencies and abrupt transitions. Neural network-based methods often rely on posterior knowledge to capture the correspondence between latent encodings and scene graph regions. Since the process of identifying latent semantics is heavily influenced by the analyzed samples, different samples will yield different recognition structures, leading to model instability. Moreover, discovering semantically rich subspaces within a given high-dimensional latent space is a very challenging task. Existing manipulation models are typically linear, limiting the diversity of editing results. Besides this method based on manually defined regions, the second type of editable rendering technique allows the machine to automatically locate the editable area simply by providing a textual description of the requirements. This simplifies the process, lowers the technical barrier to entry, and thus improves the user experience. To achieve this functionality, text understanding technology and image object detection technology are required. Currently, mainstream object detection technologies can be divided into two categories: single-level detection and two-level detection. Single-level detection models use dense sampling to directly classify objects in a single process and use predefined bounding boxes or keypoints of various scales and aspect ratios to locate objects. Two-level detection models attempt to find any number of object proposals in the image in the first stage, and then classify and locate them in the second stage.

[0005] Currently, semantically manipulable rendering technologies are weak. Research indicates that only a few related technologies exist, such as image editing, which can modify images based on text. To achieve this, CN114612290A matches the required text with a pre-defined text template to determine the instruction feature encoding. This instruction feature encoding is then combined with the feature encoding of the image to be edited to complete the modification. CN114612290A designs corresponding image editing networks for different text templates, thereby accelerating the model training process. However, because the required text template needs to be predetermined, these methods support limited modification operations and lack scalability. Therefore, CN115035213A directly fuses the feature encoding of the image to be edited with the feature encoding of the required text through residual blocks to obtain the hidden layer image feature encoding. Then, it takes a weighted sum of this hidden layer image feature encoding and the original image feature encoding to modify the image information. However, these methods generally treat text and images as data features input into the model, directly generating results. Because it lacks analysis and understanding of the details, this method struggles to support fine-grained editing and is prone to problems such as poor editing results and mismatches between the resulting images and the required text descriptions. For example, without understanding the directional terms "right" and "above," it's difficult to determine whether the user wants to modify the "sofa" or the "wall." Furthermore, generating results requires learning the mapping relationship between text, images, and the modified results, which necessitates training with a large amount of labeled data, which is costly to obtain. Summary of the Invention

[0006] The purpose of this invention is to provide a scene rendering method and apparatus that can understand fine-grained requirements in user descriptions, automatically identify things that need to be changed in the scene graph, generate appropriate new content, and render the results with fine-grained semantic manipulation.

[0007] To achieve the above objectives, the present invention provides a scene rendering method with fine-grained semantic manipulation, comprising the following steps:

[0008] S1: Input scene graph I into the object detection unit. The object detection unit locates each entity object in scene graph I and encodes the visual information of the region where the entity is located, outputting a region feature encoding set V = {v1, ..., v1}. M}, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features;

[0009] S2: Transform the requirement text Q = {q1,…,q} m The input is a semantic understanding unit, which uses a syntax parser to process the requirement text Q = {q1, ...,q}. m} is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree node is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated;

[0010] S3: Encode the region features from the target detection unit into a set V = {v1, ..., v...} M} and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit Input to the localization inference unit, the localization inference unit calculates the relationship between each image region i and the localization inference unit. Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area.

[0011]

[0012] Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l ;

[0013] S4: Encode the features v from the text identified by the semantic understanding unit (edit operation op) and the region to be edited (the region l determined by the localization reasoning unit). l The input content rendering unit encodes the features of the region to be edited (l) based on the editing operation (op) identified from the text. l The modified features are then input into the generator for rendering. The content rendering unit uses a GAN network as the generator, and a regularizer is designed to train the generator, that is, to add the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

[0014]

[0015] As a preferred embodiment, in step S1, the target detection unit includes a backbone feature extraction network, a higher-order feature modeling network, and a target detector.

[0016] The backbone feature extraction network is composed of a CSPNet network and a DarkNet53 fully convolutional neural network. CSPNet divides the original input feature V0 into two parts, V′0 and V″0, and inputs them into a cross-stage hierarchical structure with dual pathways for merging. The DarkNet53 fully convolutional neural network contains 53 convolutional layers, each followed by a batch regularization layer and an activation layer. The DarkNet53 fully convolutional neural network does not have pooling layers; instead, it uses convolutional layers with a stride of 2 to perform the feature map downsampling process. The DarkNet53 fully convolutional neural network also contains 5 CSPNet modules.

[0017] The high-order feature modeling network consists of two parts: the SPP network and the PANet network. The SPP network performs multi-scale max pooling on the feature map to increase the network's receptive field and uncover important contextual information in the scene map. The PANet network adds a bottom-up shortcut to the feature pyramid network FPN, allowing fine-grained local information to be directly used in the top layer.

[0018] The target detector uses an anchor box detection algorithm to predict target entities. It consists of three YOLO heads and uses the K-means algorithm to cluster sample targets to obtain the prior box size. Then, it uses relative offsets to calculate the size and position of the predicted box containing the target object. It also uses CIOU error. Let the true bounding box containing the target object be a, and the calculated predicted box be b. The CIOU error is calculated as follows:

[0019]

[0020]

[0021]

[0022] Among them, IOU (a,b) The intersection-union ratio (IU) of the ground truth bounding boxes and the predicted bounding boxes, ρ 2 (a,b) is the Euclidean distance between the center points of the ground truth box and the predicted box, d is the diagonal distance of the smallest box containing both the ground truth box and the predicted box, and (gw,gh) and (pw,ph) are the width and height of the ground truth box and the predicted box, respectively.

[0023] As a preferred option, a depthwise separable convolutional network is used to replace the standard 3×3 convolutional layer with a stride of 1 in the CSPNet module. The depthwise separable convolution consists of depthwise convolution and pointwise convolution. The depthwise convolution uses a convolutional kernel of size k×k to process the input feature map, and the number of convolutional kernels is consistent with the number of channels c1 of the feature map. The pointwise convolution uses a convolutional kernel of size c2 of size 1×1 to integrate the results after the depthwise convolution and change the number of channels of the final output feature map.

[0024] As a preferred approach, in the SPP network, the input feature map with c SPP channels is divided into two parts and input into the optimized network. One part of the features is concatenated with multi-scale features through skip connections. In the PANet network, two consecutive residual blocks are surrounded by skip connections to replace the five consecutive convolutional layers in the original network.

[0025] As a preferred embodiment, in step S2, the semantic understanding unit uses an open-source dependency parser to obtain the syntax tree of the required text, and then uses a bidirectional tree-based LSTM neural network to represent the features of the nodes in the tree. The bidirectional tree-based LSTM network can capture the relational dependencies between nodes in the book and encode the contextual semantic information of parent and child nodes. Specifically, for node t in the tree, this unit assigns its corresponding word q t The representation vector W emb e t The input is fed into a bidirectional tree-structured LSTM network to ultimately obtain the feature encoding h of that node. t = Among them W emb For trainable word embedding matrix, e t One-hot encoding for corresponding words, The outputs of the tree-structured LSTM networks in the bottom-up and top-down directions are respectively. Finally, the average value of the feature codes of all nodes in the tree is taken as the global feature code q of the requirement text, and it is mapped to the corresponding control instruction code. in These are the trainable weights and biases, respectively.

[0026] As a preferred embodiment, in step S3, the localization reasoning unit includes a scoring module, an integration module, and a relationship reasoning module. Each module updates the temporary localization score for each region of the scene graph at the corresponding node t. in,

[0027] The scoring module evaluates the similarity between the region feature encoding and the target node feature encoding in each scene graph, and only assembles to the leaf and root nodes; it first calculates the feature encoding v for each region. i With the node feature encoding h tSimilarity S vh (v i ,h t = fc(L2norm(fc(v)) i )⊙h t In this context, fc is a fully connected layer. If this module is located at a leaf node, the calculated similarity score is directly used as the temporary localization score. It is then sent to the module assembled by the parent node; if it is at the root node, the similarity is added to the temporary scores of other child nodes for that region. As the final positioning score, N t Let be the set of nodes under node t;

[0028] Integration Module: As shown in the following formula, this module integrates the temporary location scores sent from child nodes to node t and transmits the results to the parent node.

[0029]

[0030] Relational Reasoning Module: This module performs compound machine reasoning based on the relational terms between two grammatical components in the requirement text, as shown below:

[0031]

[0032] This module first aggregates region feature codes based on the similarity between each region feature code and the feature code of that node to obtain the context region feature code. This feature is used to characterize the region information of the scene graph based on textual relationships; this contextual region feature encoding is then used to calculate a provisional localization score for a single region, as shown in the following formula:

[0033]

[0034] Where v i h represents the feature encoding of each visual region. t This represents the feature encoding of a node in the syntax tree; L2norm represents the L2 norm, which is the square root of the sum of squares of the elements of the pointer. Making the L2 norm regularity term ||W||2 as small as possible ensures that each element of W is small, close to zero but not equal to 0; this results in a model with strong robustness against interference. The Softmax function maps the outputs of multiple neurons to the (0,1) interval, thus supporting multi-class prediction.

[0035] The scoring module, integration module, and relational reasoning module are assembled into each node of the dependency syntax tree according to the rules. First, the scoring module is assembled into all leaf nodes and the root node. For other nodes, the assembly module or the relational reasoning module is determined based on the node feature encoding, as shown in the following formula. Finally, this unit selects the region l with the highest positioning score at the root node module as the region to be edited.

[0036] EditArea←argmaxsoftmax(fc(h t )).

[0037] As a preferred option, in step S4, the content rendering unit uses a GAN network as a generator. To train the GAN network, a regularizer is designed to constrain the model to modify only the visual information of the target region without affecting other unrelated regions. Moreover, through adversarial learning, the compatibility between the edited region and the neighboring regions can be maximized.

[0038] As a preferred approach, for the scene region feature encoding set V = {v1,…,v} M}, sample from the standard Gaussian distribution N(0,I) To interfere with the feature encoding of the region to be edited v l ,sampling This interferes with the feature encoding of other regions; correspondingly, the GAN network outputs two interfering scene images, namely... and On the one hand, the interference in the feature encoding of the region to be edited should not affect the information of other regions, as shown in the following formula.

[0039]

[0040] M1 is a binary mask used to cover the area to be edited, thus allowing attention to changes in other areas. On the other hand, interfering with the feature encoding of other areas should not affect the information of the area to be edited, as shown in the following formula.

[0041]

[0042] M2 is a binary mask used to cover other areas, thus focusing on changes in the region to be edited; ultimately, as shown below, the training objective of the GAN network is the interference generation loss. and generation loss The sum, where λ1 and λ2 are preset parameters,

[0043]

[0044] The present invention also provides a scene rendering apparatus for fine-grained semantic manipulation, comprising:

[0045] The object detection unit is used to locate each entity object in scene graph I, encode the visual information of the region where the entity is located, and output the region feature encoding set V = {v1, ..., v1}. M}, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features;

[0046] The semantic understanding unit is used to process the requirement text Q = {q1, ...,q} using a syntax parser. m} is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree node is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated;

[0047] The localization inference unit is used to encode the set V = {v1, ..., v1} based on the regional features from the target detection unit. M} and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit Locate the region to be edited and calculate the relationship between each image region i and the image region i. Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area.

[0048]

[0049] Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l ;

[0050] The content rendering unit is used to encode the features of the region to be edited (l) based on the editing operation (op) identified from the text. lThe modified features are then input into the generator for rendering. The content rendering unit uses a GAN network as the generator, and a regularizer is designed to train the generator, that is, to add the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

[0051]

[0052] As a preferred embodiment, the object detection unit includes a backbone feature extraction network, a higher-order feature modeling network, and an object detector. The backbone feature extraction network is composed of a CSPNet network and a DarkNet53 fully convolutional neural network. The DarkNet53 fully convolutional neural network contains 53 convolutional layers, each followed by a batch regularization layer and an activation layer, and has no pooling layers. It uses convolutional layers with a stride of 2 to replace pooling layers for feature map downsampling. The DarkNet53 fully convolutional neural network also includes 5 CSPNet modules and introduces depthwise separable convolutions to optimize the residual modules in the CSPNet structure. The higher-order feature modeling network consists of two parts: an SPP network and a PANet network. The object detector consists of three YOLO heads.

[0053] The semantic understanding unit uses an open-source dependency parser to obtain the syntax tree of the required text. Then, a bidirectional tree-structured LSTM neural network is used to represent the features of the nodes in the tree.

[0054] The localization reasoning unit includes a scoring module, an integration module, and a relational reasoning module. The scoring module is used to evaluate the similarity between the regional feature encoding of each scene graph and the feature encoding of the target node in the tree, and is only assembled to the leaf node and the root node. The integration module is used to integrate the temporary localization scores sent to node t by the child node and transmit the results to the parent node. The relational reasoning module is used to perform compound machine reasoning based on the relation words between two grammatical components in the requirement text.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] This invention constructs a dependency syntax tree using a semantic understanding unit, enabling the description of fine-grained semantics in the request text. This allows for the identification of fine-grained information within the request text, accurately locating image regions and achieving a high match between the rendering result and the request text. Therefore, this invention provides a deeper understanding of the text and possesses semantic reasoning capabilities. Furthermore, converting the request text into a dependency syntax tree and assembling a reasoning module on the tree strengthens the correlation between scene graph features and text features, resulting in more accurate localization. This invention also designs a regularizer to constrain the scene generation model, accurately modifying the visual information of the target region without affecting other unrelated regions. This invention eliminates the need for manual delineation of editing areas, directly understanding the textual requirements and enabling fine-grained local editing of the original scene graph. Users can submit editing requests in text form, adding, deleting, and modifying scene graphs, achieving convenient and controllable rendering. Attached Figure Description

[0057] Figure 1 This is a flowchart of a scene rendering method for fine-grained semantic manipulation according to an embodiment of the present invention.

[0058] Figure 2 This is a structural diagram of the scene rendering method for fine-grained semantic manipulation according to an embodiment of the present invention.

[0059] Figure 3 This is a structural diagram of the backbone feature extraction network in an embodiment of the present invention.

[0060] Figure 4 This is a diagram of the depthwise separable convolutional structure of the DarkNet53 fully convolutional neural network according to an embodiment of the present invention.

[0061] Figure 5 This is a diagram of the SPP network structure of the high-order feature modeling network in an embodiment of the present invention.

[0062] Figure 6 This is a diagram of the PANet network structure, a high-order feature modeling network according to an embodiment of the present invention.

[0063] Figure 7 This is a structural diagram of the semantic understanding unit in an embodiment of the present invention. Detailed Implementation

[0064] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0065] Example 1

[0066] like Figures 1 to 7 As shown, a scene rendering method with fine-grained semantic manipulation according to a preferred embodiment of the present invention includes the following steps:

[0067] S1: Input scene graph I into the object detection unit. The object detection unit locates each entity object in scene graph I and encodes the visual information of the region where the entity is located, outputting a region feature encoding set V = {v1, ..., v1}. M}, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features;

[0068] S2: Transform the requirement text Q = {q1,…,q} m The input is a semantic understanding unit, which uses a syntax parser to process the requirement text Q = {q1, ...,q}. m} is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree node is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated;

[0069] S3: Encode the region features from the target detection unit into a set V = {v1, ..., v...} M} and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit Input to the localization inference unit, the localization inference unit calculates the relationship between each image region i and the localization inference unit. Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area.

[0070]

[0071] Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l ;

[0072] S4: Encode the features v from the text identified by the semantic understanding unit (edit operation op) and the region to be edited (the region l determined by the localization reasoning unit).l The input content rendering unit encodes the features of the region to be edited (l) based on the editing operation (op) identified from the text. l The modified features are then input into the generator for rendering. The content rendering unit uses a GAN network as the generator, and a regularizer is designed to train the generator, that is, to add the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

[0073]

[0074] This embodiment constructs a dependency syntax tree by a semantic understanding unit, enabling the description of fine-grained semantics in the request text. This allows for the identification of fine-grained information within the request text, accurately locating image regions and achieving a high match between the rendering result and the request text. Therefore, this embodiment demonstrates a deeper understanding of the text and possesses semantic reasoning capabilities. Furthermore, converting the request text into a dependency syntax tree and assembling a reasoning module on the tree strengthens the correlation between scene graph features and text features, resulting in more accurate localization. This embodiment also designs a regularizer to constrain the scene generation model, accurately modifying the visual information of the target region without affecting other unrelated regions. This embodiment eliminates the need for manual delineation of the editing area, directly understanding the textual requirements and enabling fine-grained local editing of the original scene graph. Users can submit editing requests in text form, adding, deleting, and modifying scene graphs, achieving convenient and controllable rendering.

[0075] Example 2

[0076] The difference between this embodiment and Embodiment 1 is that, based on Embodiment 1, this embodiment provides further explanation of each step of the rendering method.

[0077] The fine-grained semantic manipulation scene rendering method in this embodiment includes the following steps:

[0078] S1: Input scene graph I into the object detection unit. The object detection unit locates each entity object in scene graph I and encodes the visual information of the region where the entity is located, outputting a region feature encoding set V = {v1, ..., v1}. M}, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features.

[0079] The object detection unit is used to identify physical entities in the scene graph, such as "walls" and "tables," and encodes the visual information of the areas where these entities are located, thereby obtaining the features of the scene graph regions. Considering the complex structure of indoor scenes, the rich variety of object elements, and the influence of factors such as lighting, angle, and object occlusion, traditional object detection algorithms struggle to meet the requirements of practical applications in terms of detection accuracy and response speed. Therefore, this unit is based on the YOLOv4 framework to detect physical entities in the scene graph. When scene graph I is input, this method locates various physical objects in the graph, such as "sofa," "window," and "table."

[0080] Specifically, in step S1, the target detection unit includes a backbone feature extraction network, a higher-order feature modeling network, and a target detector.

[0081] The backbone feature extraction network is composed of a combination of CSPNet and DarkNet53 fully convolutional neural networks; such as Figure 3 As shown, CSPNet divides the original input feature V0 into two parts, V′0 and V″0, and inputs them into a cross-stage hierarchical structure with dual pathways for merging; DarkNet53 fully convolutional neural network contains 53 convolutional layers, each followed by a batch normalization layer and a leaky activation layer. (ReLU layer); The DarkNet53 fully convolutional neural network does not have pooling layers. Instead, it uses convolutional layers with a stride of 2 to replace pooling layers for feature map downsampling, which effectively prevents the loss of low-level features caused by pooling layers. The DarkNet53 fully convolutional neural network also includes 5 CSPNet modules; this structure not only reduces the number of parameters but also extracts rich and comprehensive deep features, eliminates the defect of redundant gradient information generated during backpropagation, and enhances the network's learning ability. To reduce computational cost and achieve a lightweight network, this module introduces depthwise separable convolutions to optimize the residual modules in the CSPNet structure. This module replaces the standard 3×3 convolutional layers with a stride of 1 in the CSPNet modules with depthwise separable convolutional networks, which can effectively reduce network parameters and improve response speed. Figure 4As shown, depthwise separable convolution consists of depthwise convolution and pointwise convolution. Depthwise convolution uses a k×k kernel to process the input feature map, with the number of kernels matching the number of channels c1 in the feature map. Pointwise convolution uses c2 kernels of size 1×1 to integrate the results of depthwise convolution and changes the number of channels in the final output feature map. In the original network, the number of parameters A1 and the computational cost B1 required for a standard k×k convolution are shown in Equation (1), while the number of parameters A2 and the computational cost B2 required for a depthwise separable convolution with the same kernel size are shown in Equation (2). By combining Equations (1) and (2), it can be seen that the number of parameters and the computational cost of the improved convolution operation can be reduced to 1 / c2 + 1 / k of the original convolution. 2 .

[0082]

[0083]

[0084] The high-order feature modeling network consists of two parts: the SPP network and the PANet network. The SPP network performs multi-scale max pooling on the feature map, increasing the network's receptive field and uncovering important contextual information in the scene map. The PANet network adds a bottom-up shortcut to the Feature Pyramid Network (FPN), allowing fine-grained local information to be directly used in the top layer, enhancing the network's ability to detect targets of different sizes. Considering the role of CSPNet in promoting model training, this module optimizes the structure of the SPP and PANet networks. Figure 5 The comparison before and after optimization is shown. In the SPP network, the input feature map with c SPP channels is input into the optimized network in two parts. One part of the features is concatenated with multi-scale features through skip connections; for example... Figure 6 As shown, in the PANet network, two consecutive residual blocks are surrounded by skip connections to replace the five consecutive convolutional layers in the original network.

[0085] The target detector uses the anchor box detection algorithm to predict the target entity. It consists of three YOLO heads. The K-means algorithm is used to cluster the sample targets to obtain the prior box size. Then, the size and position of the predicted box of the target object are calculated using the relative offset. This module uses CIOU error. This error calculation method uses the Euclidean distance between the boxes to evaluate the loss when the detected box and the real box do not overlap. In addition, the length and width loss of the box are also considered to ensure that the predicted box fits the real box better. Let the real box of the target object be a and the calculated predicted box be b. The calculation formula of CIOU error is shown in (3):

[0086]

[0087] Among them, IOU (a,b) The intersection-union ratio (IU) of the ground truth bounding boxes and the predicted bounding boxes, ρ 2 (a,b) is the Euclidean distance between the center points of the ground truth box and the predicted box, d is the diagonal distance of the smallest box containing both the ground truth box and the predicted box, and (gw,gh) and (pw,ph) are the width and height of the ground truth box and the predicted box, respectively.

[0088] S2: Transform the requirement text Q = {q1,…,q} m The input is a semantic understanding unit, which uses a syntax parser to process the requirement text Q = {q1, ...,q}. m} is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree nodes is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated. For example... Figure 7 As shown.

[0089] The semantic understanding unit is responsible for understanding the semantics in the user's modification request description. Traditional methods typically encode the text description into a single feature vector and then calculate the similarity between this vector and the feature encoding of each visual region to locate the image region to be edited. Because they do not model the fine-grained semantic information in the text, such as recognizing entities and relationships, these methods struggle to understand the subtle information contained within complex request texts; they also struggle to accurately locate image regions, leading to significant discrepancies between the rendering results and the request text. To address this issue, this embodiment employs the method described in step S2 above, which encodes the request text Q = {q1,…,q...}. m The text is converted into a dependency syntax tree to enable fine-grained analysis of the required text, thereby improving the ability to understand and reason about the text semantically.

[0090] Specifically, in step S2, the semantic understanding unit uses an open-source dependency parser (i.e., the Spacy tool) to obtain the syntax tree of the required text, and then uses a bidirectional tree-structured LSTM neural network to represent the features of the nodes in the tree. Unlike ordinary LSTM networks, the bidirectional tree-structured LSTM network can capture the relational dependencies between nodes in the book and encode the contextual semantic information of parent and child nodes. Specifically, for node t in the tree, this unit assigns its corresponding word q t The representation vector W emb et The input is fed into a bidirectional tree-structured LSTM network to obtain the feature encoding of the node. Among them W emb For trainable word embedding matrix, e t One-hot encoding for corresponding words, The outputs of the tree-structured LSTM networks in the bottom-up and top-down directions are respectively. Finally, the average value of the feature codes of all nodes in the tree is taken as the global feature code q of the requirement text, and it is mapped to the corresponding control instruction code. in These are the trainable weights and biases, respectively.

[0091] S3: Encode the region features from the target detection unit into a set V = {v1, ..., v...} M} and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit The input localization inference unit, as shown in equation (4), calculates the localization inference unit for each image region i with respect to... Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area.

[0092]

[0093] Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l .

[0094] The localization inference unit jointly considers the set of regional feature codes V = {v1, ..., v1} from the preceding unit. M} and the dependency syntax tree node feature encoding set of the demand text Locating the area to be edited. To achieve accurate localization, the model, in addition to understanding the given scene graph and the requirement text, also needs to calculate the fine-grained relationship between the two. To this end, this unit designs a tree-structured modular network that standardizes the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural network module that calculates a temporary localization score for the region. These temporary scores are then integrated from the bottom up to obtain the final localization score for each region. Specifically, the designed tree-structured modular network consists of three types of modules: a scoring module, an integration module, and a relational reasoning module. Each module updates the temporary localization score for each region of the scene graph at the corresponding node t. The result is then sent to the parent node. The three modules described below will be introduced:

[0095] In step S3, the localization reasoning unit includes a scoring module, an integration module, and a relational reasoning module. Each module updates the temporary localization score for each region of the scene graph at the corresponding node t. in,

[0096] The scoring module evaluates the similarity between the region feature encoding and the target node feature encoding in each scene graph, and only assembles to the leaf and root nodes; it first calculates the feature encoding v for each region. i With the node feature encoding h t Similarity S vh (v i ,h t = fc(L2norm(fc(v)) i )⊙h t In this context, fc is a fully connected layer. If this module is located at a leaf node, the calculated similarity score is directly used as the temporary localization score. It is then sent to the module assembled by the parent node; if it is at the root node, the similarity is added to the temporary scores of other child nodes for that region. As the final positioning score, N t Let be the set of nodes under node t;

[0097] Integration Module: As shown in formula (5), this module integrates the temporary positioning scores sent from the child nodes to node t, and transmits the results to the parent node.

[0098]

[0099] Relational Reasoning Module: This module performs compound machine reasoning based on the relation words between two grammatical components in the requirement text, as shown in Equation (6):

[0100]

[0101] This module first aggregates region feature codes based on the similarity between each region feature code and the feature code of that node to obtain the context region feature code. This feature is used to characterize the regional information of the scene graph based on textual relationships, such as "the wall above the sofa". This contextual region feature encoding is then used to calculate a provisional localization score for a single region, as shown in Equation (7):

[0102]

[0103] Where v i h represents the feature encoding of each visual region. tThis represents the feature encoding of a node in the syntax tree; L2norm represents the L2 norm, which is the square root of the sum of squares of the elements of the pointer. Making the L2 norm regularity term ||W||2 as small as possible ensures that each element of W is small, close to zero but not equal to 0; this results in a model with strong robustness against interference. The Softmax function maps the outputs of multiple neurons to the (0,1) interval, thus supporting multi-class prediction.

[0104] The scoring module, integration module, and relational reasoning module are assembled into each node in the dependency syntax tree according to the rules. First, the scoring module is assembled into all leaf nodes and the root node. For other nodes, the integration module or the relational reasoning module is assembled according to the node feature encoding, as shown in Equation (8). Finally, this unit selects the region l with the highest positioning score at the root node module as the region to be edited.

[0105] EditArea←argmaxsoftmax(fc(h t ))(8).

[0106] S4: Encode the features v from the text identified by the semantic understanding unit (edit operation op) and the region to be edited (the region l determined by the localization reasoning unit). l The input content rendering unit encodes the features of the region to be edited (l) based on the editing operation (op) identified from the text. l Modifications are made, and the modified features are input into the generator for rendering. In recent years, generative adversarial networks (GANs) have demonstrated their great potential in content generation. Considering that GANs can generate high-quality and high-resolution images, the content rendering unit uses a GAN network as the generator and designs a regularizer to train the generator, thereby ensuring that the modification of the feature encoding of the target region will not affect other irrelevant regions, and can also enhance the compatibility of the modified region and the adjacent regions, avoiding abrupt connections. As shown in Equation (9), this unit adds the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

[0107]

[0108] In step S4, the content rendering unit uses a GAN network as the generator. To train the GAN network, a regularizer is designed to constrain the model to modify only the visual information of the target region without affecting other unrelated regions. Furthermore, through adversarial learning, the compatibility between the edited region and its neighboring regions is maximized, avoiding hard connections. Specifically, for the scene region feature encoding set V = {v1,…,v…} M}, sample from the standard Gaussian distribution N(0,I) To interfere with the feature encoding of the region to be edited v l ,sampling This interferes with the feature encoding of other regions; correspondingly, the GAN network outputs two interfering scene images, namely... and On the one hand, the interference with the feature encoding of the region to be edited should not affect the information of other regions, as shown in equation (10).

[0109]

[0110] M1 is a binary mask used to cover the area to be edited, thereby focusing on changes in other areas; on the other hand, interfering with the feature encoding of other areas should not affect the information of the area to be edited, as shown in equation (11).

[0111]

[0112] M2 is a binary mask used to cover other areas, thus focusing on changes in the region to be edited; ultimately, as shown below, the training objective of the GAN network is the interference generation loss. and generation loss The sum of λ1 and λ2, where λ1 and λ2 are preset parameters, as shown in equation (12).

[0113]

[0114] This embodiment proposes a fine-grained semantic manipulation method for scene rendering. Unlike traditional rendering methods, this method eliminates the need for manual delineation of editing areas, directly understands textual requirements, and enables fine-grained local editing of the original scene image. Users can submit editing requests in text form, adding, deleting, and modifying elements in the scene image, achieving convenient and controllable rendering. This embodiment demonstrates a deeper understanding of text and possesses semantic reasoning capabilities. Furthermore, by converting the request text into a dependency syntax tree and assembling a reasoning module on the tree, the correlation between scene image features and text features is strengthened, resulting in more accurate localization. This embodiment designs a regularizer to constrain the scene generation model, accurately modifying the visual information of the target area without affecting other unrelated areas. Compared to existing technologies, the target detection network designed in this embodiment has fewer parameters, faster response speed, is better at discovering deep features of the scene image, and achieves higher accuracy in indoor target detection.

[0115] Example 3

[0116] This embodiment provides a scene rendering apparatus based on the fine-grained semantic manipulation scene rendering method of Embodiment 1 or Embodiment 2, comprising:

[0117] The object detection unit is used to locate each entity object in scene graph I, encode the visual information of the region where the entity is located, and output the region feature encoding set V = {v1, ..., v1}. M}, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features;

[0118] The semantic understanding unit is used to process the requirement text Q = {q1, ...,q} using a syntax parser. m} is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree node is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated;

[0119] The localization inference unit is used to encode the set V = {v1, ..., v1} based on the regional features from the target detection unit. M} and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit Locate the region to be edited and calculate the relationship between each image region i and the image region i. Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area.

[0120]

[0121] Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l ;

[0122] The content rendering unit is used to encode the features of the region to be edited (l) based on the editing operation (op) identified from the text. l The modified features are then input into the generator for rendering. The content rendering unit uses a GAN network as the generator, and a regularizer is designed to train the generator, that is, to add the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

[0123]

[0124] Furthermore, the object detection unit includes a backbone feature extraction network, a higher-order feature modeling network, and an object detector. The backbone feature extraction network is composed of a CSPNet network and a DarkNet53 fully convolutional neural network. The DarkNet53 fully convolutional neural network contains 53 convolutional layers, each followed by a batch regularization layer and an activation layer, and has no pooling layers. It uses convolutional layers with a stride of 2 instead of pooling layers for the feature map downsampling process. The DarkNet53 fully convolutional neural network also contains 5 CSPNet modules and introduces depthwise separable convolutions to optimize the residual modules in the CSPNet structure. The higher-order feature modeling network consists of two parts: an SPP network and a PANet network. The object detector consists of three YOLO heads.

[0125] The semantic understanding unit uses an open-source dependency parser to obtain the syntax tree of the required text. Then, a bidirectional tree-structured LSTM neural network is used to represent the features of the nodes in the tree.

[0126] The localization reasoning unit includes a scoring module, an integration module, and a relational reasoning module. The scoring module is used to evaluate the similarity between the regional feature encoding of each scene graph and the feature encoding of the target node in the tree, and is only assembled to the leaf node and the root node. The integration module is used to integrate the temporary localization scores sent to node t by the child node and transmit the results to the parent node. The relational reasoning module is used to perform compound machine reasoning based on the relation words between two grammatical components in the requirement text.

[0127] For other aspects of the apparatus in this embodiment, please refer to Embodiment 2.

[0128] In summary, this invention provides a fine-grained semantic manipulation method for scene rendering. This interactive method allows for local editing of scene images, and the editing operation is simple, triggered by verbal descriptions without the need for manual definition of editing locations as in traditional methods. Users can submit textual descriptions of their editing needs, adding, deleting, and modifying parts of the image for convenient and controllable rendering. The device first identifies objects in the scene image, then analyzes the user's textual description of their requirements, finely locating the scene area to be edited and the type of operation. Finally, it uses an adversarial neural network to render a high-quality scene image. This method aims to understand user needs through natural language processing technology, automatically locating the areas to be edited and modified without manual definition. The result is rendered by editing these areas. Unlike traditional methods, this incremental rather than full-update approach significantly reduces operational costs and enables interactive rendering. This manipulation method is data-driven, extracting typical semantic features from large-scale image datasets. Users can specify the scene structure and modify scene content; this method can edit according to these requirements and also reduces the manual traces in compositing operations, resulting in smoother rendering results. Specifically, this method first identifies entities and relationships in the scene graph, then analyzes the user's requirement description text to locate the area to be edited and the type of operation. Subsequently, an unconstrained adversarial neural renderer is developed to edit this area, and through adversarial learning, the rendered result is made to approximate the real image as closely as possible, solving the compatibility problem between the generated scene and the real result. Compared to traditional methods, this technology does not require users to manually define the editing area, offering greater flexibility and user-friendliness. This visually adjustable scene design can effectively improve efficiency in application areas such as architectural space design. Furthermore, this invention also provides a fine-grained semantic manipulation scene rendering device with the aforementioned beneficial effects, which will not be elaborated further here.

[0129] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A scene rendering method with fine-grained semantic manipulation, characterized in that, Includes the following steps: S1: Input scene graph I into the object detection unit. The object detection unit locates each entity object in scene graph I and encodes the visual information of the region where the entity is located, outputting a region feature encoding set V = {v1, ..., v1}. M }, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features; S2: Transform the requirement text Q = {q1,…,q} m The input is a semantic understanding unit, which uses a syntax parser to process the requirement text Q = {q1, ...,q}. m } is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree node is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated; S3: Encode the region features from the target detection unit into a set V = {v1, ..., v...} M } and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit Input to the localization inference unit, the localization inference unit calculates the relationship between each image region i and the localization inference unit. Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area. Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l ; S4: Encode the features v from the text identified by the semantic understanding unit (edit operation op) and the region to be edited (the region l determined by the localization reasoning unit). l The input content rendering unit encodes the features of the region to be edited (l) based on the editing operation (op) identified from the text. l The modified features are then input into the generator for rendering. The content rendering unit uses a GAN network as the generator, and a regularizer is designed to train the generator, that is, to add the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

2. The scene rendering method with fine-grained semantic manipulation according to claim 1, characterized in that, In step S1, the target detection unit includes a backbone feature extraction network, a higher-order feature modeling network, and a target detector. The backbone feature extraction network is composed of a CSPNet network and a DarkNet53 fully convolutional neural network; CSPNet divides the original input feature V0 into two parts V0. ′ The inputs are combined with V0″ and fed into a cross-stage hierarchical structure with dual paths; the DarkNet53 fully convolutional neural network contains 53 convolutional layers, each followed by a batch regularization layer and an activation layer; the DarkNet53 fully convolutional neural network does not have pooling layers, and uses convolutional layers with a stride of 2 instead of pooling layers for feature map downsampling; the DarkNet53 fully convolutional neural network also contains 5 CSPNet modules; The high-order feature modeling network consists of two parts: the SPP network and the PANet network. The SPP network performs multi-scale max pooling on the feature map to increase the network's receptive field and uncover important contextual information in the scene map. The PANet network adds a bottom-up shortcut to the feature pyramid network FPN, allowing fine-grained local information to be directly used in the top layer. The target detector uses an anchor box detection algorithm to predict target entities. It consists of three YOLO heads and uses the K-means algorithm to cluster sample targets to obtain the prior box size. Then, it uses relative offsets to calculate the size and position of the predicted box containing the target object. It also uses CIOU error. Let the true bounding box containing the target object be a, and the calculated predicted box be b. The CIOU error is calculated as follows: Among them, IOU (a,b) The intersection-union ratio (IU) of the ground truth bounding boxes and the predicted bounding boxes, ρ 2 (a,b) is the Euclidean distance between the center points of the ground truth box and the predicted box, d is the diagonal distance of the smallest box containing both the ground truth box and the predicted box, and (gw,gh) and (pw,ph) are the width and height of the ground truth box and the predicted box, respectively.

3. The scene rendering method with fine-grained semantic manipulation according to claim 2, characterized in that, The standard 3×3 convolutional layer with a stride of 1 in the CSPNet module is replaced by a depthwise separable convolutional network. The depthwise separable convolution consists of depthwise convolution and pointwise convolution. The depthwise convolution uses a k×k kernel to process the input feature map, and the number of kernels is consistent with the number of channels c1 of the feature map. The pointwise convolution uses c2 kernels of size 1×1 to integrate the results of the depthwise convolution and change the number of channels of the final output feature map.

4. The scene rendering method with fine-grained semantic manipulation according to claim 2, characterized in that, In the SPP network, the input feature map with c SPP channels is divided into two parts and input into the optimized network. One part of the features is concatenated with multi-scale features through skip connections. In the PANet network, skip connections are used to surround two consecutive residual blocks, replacing the original network's five consecutive convolutional layers.

5. The scene rendering method with fine-grained semantic manipulation according to claim 1, characterized in that, In step S2, the semantic understanding unit uses an open-source dependency parser to obtain the syntax tree of the required text, and then uses a bidirectional tree-based LSTM neural network to represent the features of the nodes in the tree. The bidirectional tree-based LSTM network can capture the relational dependencies between nodes in the book and encode the contextual semantic information of parent and child nodes. Specifically, for node t in the tree, this unit assigns its corresponding word q t The representation vector W emb e t The input is fed into a bidirectional tree-structured LSTM network to obtain the feature encoding of the node. Among them W emb For trainable word embedding matrix, e t One-hot encoding for corresponding words, The outputs of the tree-structured LSTM networks in the bottom-up and top-down directions are respectively. Finally, the average value of the feature codes of all nodes in the tree is taken as the global feature code q of the requirement text, and it is mapped to the corresponding control instruction code. in These are the trainable weights and biases, respectively.

6. The scene rendering method with fine-grained semantic manipulation according to claim 2, characterized in that, In step S3, the localization reasoning unit includes a scoring module, an integration module, and a relational reasoning module. Each module updates the temporary localization score for each region of the scene graph at the corresponding node t. in, The scoring module evaluates the similarity between the region feature encoding and the target node feature encoding in each scene graph, and only assembles to the leaf and root nodes; it first calculates the feature encoding v for each region. i With the node feature encoding h t Similarity S vh (v i ,h t = fc(L2norm(fc(v)) i )⊙h t In this context, fc is a fully connected layer. If this module is located at a leaf node, the calculated similarity score is directly used as the temporary localization score. It is then sent to the module assembled by the parent node; if it is at the root node, the similarity is added to the temporary scores of other child nodes for that region. As the final positioning score, N t Let be the set of nodes under node t; Integration Module: As shown in the following formula, this module integrates the temporary location scores sent from child nodes to node t and transmits the results to the parent node. Relational Reasoning Module: This module performs compound machine reasoning based on the relational terms between two grammatical components in the requirement text, as shown below: This module first aggregates region feature codes based on the similarity between each region feature code and the feature code of that node to obtain the context region feature code. This feature is used to characterize the region information of the scene graph based on textual relationships; this contextual region feature encoding is then used to calculate a provisional localization score for a single region, as shown in the following formula: Where v i h represents the feature encoding of each visual region. t The L2 norm represents the feature encoding of a node in the syntax tree; L2norm represents the L2 norm, which is the square root of the sum of squares of the elements of the pointer; making the L2 norm regular term ||W||2 as small as possible can make each element of W very small, close to zero but not equal to 0; the resulting model has strong anti-interference ability; the Softmax function can map the output of multiple neurons to the (0,1) interval, thereby supporting multi-class prediction; The scoring module, integration module, and relational reasoning module are assembled to each node in the dependency syntax tree according to the rules. First, the scoring module is assembled to all leaf nodes and the root node. For other nodes, the integration module or the relational reasoning module is assembled according to the node feature encoding, as shown in the following formula. Finally, this unit selects the region l with the highest positioning score at the root node module as the region to be edited. EditArea←argmaxsoftmax(fc(h t ))。 7. The scene rendering method with fine-grained semantic manipulation according to claim 1, characterized in that, In step S4, the content rendering unit uses a GAN network as a generator. To train the GAN network, a regularizer is designed to constrain the model to modify only the visual information of the target region without affecting other unrelated regions. Moreover, through adversarial learning, the compatibility between the edited region and the neighboring regions can be maximized.

8. The scene rendering method with fine-grained semantic manipulation according to claim 7, characterized in that, For the scene region feature encoding set V = {v1, ..., v...} M }, sample from the standard Gaussian distribution N(0,I) To interfere with the feature encoding of the region to be edited v l ,sampling This interferes with the feature encoding of other regions; correspondingly, the GAN network outputs two interfering scene images, namely... and On the one hand, the interference in the feature encoding of the region to be edited should not affect the information of other regions, as shown in the following formula. M1 is a binary mask used to cover the area to be edited, thus allowing attention to changes in other areas. On the other hand, interfering with the feature encoding of other areas should not affect the information of the area to be edited, as shown in the following formula. M2 is a binary mask used to cover other areas, thus focusing on changes in the region to be edited; ultimately, as shown below, the training objective of the GAN network is the interference generation loss. and generation loss The sum, where λ1 and λ2 are preset parameters, 9. A scene rendering device with fine-grained semantic manipulation, characterized in that, include: The object detection unit is used to locate each entity object in scene graph I, encode the visual information of the region where the entity is located, and output the region feature encoding set V = {v1, ..., v1}. M }, where M is the number of detected regions, and V contains elements It describes the visual information of the entity target in the i-th region and the entity category to which it belongs, where d v Encode the dimension of the region features; The semantic understanding unit is used to process the requirement text Q = {q1, ...,q} using a syntax parser. m } is converted into a dependency parsing tree, and the feature set of the corresponding tree node is It is used to describe the fine-grained semantics of the requirement text; based on this, the global feature encoding of the requirement text is obtained. Understanding user intent to generate corresponding control command codes Where m is the required text length, d q The required text feature encoding dimension, op dimension, and the region feature encoding dimension d output by the scene graph object detection unit are given. v Consistent, the feature set H of the dependency syntax tree node is used as reference information to assist the subsequent localization unit in accurately locating the region to be updated; The localization inference unit is used to encode the set V = {v1, ..., v1} based on the regional features from the target detection unit. M } and the dependency syntax tree node feature encoding set from the demand text of the semantic understanding unit Locate the region to be edited and calculate the relationship between each image region i and the image region i. Positioning score S(v) i The highest-scoring item (H) is used to determine the editing area. Furthermore, the localization inference unit employs a tree-structured modular network, standardizing the visual localization process within a dependency syntax tree. Each node in the tree is equipped with a neural module network that computes a temporary localization score for the region. These temporary scores are then integrated from the bottom up to ultimately obtain the localization score for each region. The region l with the highest localization score is selected as the region to be edited, and the feature encoding of the region l is v. l ; The content rendering unit is used to encode the features of the region to be edited (l) based on the editing operation (op) identified from the text. l The modified features are then input into the generator for rendering. The content rendering unit uses a GAN network as the generator, and a regularizer is designed to train the generator, that is, to add the manipulation instruction encoding op into the feature encoding v of the region to be edited. l The modified region feature encoding set is then input into the GAN network to output the rendered scene. Where α is a preset parameter.

10. A scene rendering apparatus for fine-grained semantic manipulation according to claim 9, characterized in that, The object detection unit includes a backbone feature extraction network, a higher-order feature modeling network, and an object detector. The backbone feature extraction network is composed of a CSPNet network and a DarkNet53 fully convolutional neural network. The DarkNet53 fully convolutional neural network contains 53 convolutional layers, each followed by a batch regularization layer and an activation layer, and has no pooling layers. It uses convolutional layers with a stride of 2 to replace pooling layers for feature map downsampling. The DarkNet53 fully convolutional neural network also contains 5 CSPNet modules and introduces depthwise separable convolutions to optimize the residual modules in the CSPNet structure. The high-order feature modeling network consists of two parts: the SPP network and the PANet network; the object detector consists of three YOLO heads. The semantic understanding unit uses an open-source dependency parser to obtain the syntax tree of the required text; then it uses a bidirectional tree-structured LSTM neural network to represent the features of the nodes in the tree. The localization reasoning unit includes a scoring module, an integration module, and a relational reasoning module. The scoring module is used to evaluate the similarity between the regional feature encoding of each scene graph and the feature encoding of the target node in the tree, and is only assembled to the leaf node and the root node. The integration module is used to integrate the temporary localization scores sent to node t by the child node and transmit the results to the parent node. The relational reasoning module is used to perform compound machine reasoning based on the relation words between two grammatical components in the requirement text.

Images