A method and system for expressing invisible scenes based on semantic attributes

Abstract

The present invention proposes a method and system for expressing invisible scenes based on semantic attributes, which relate to the field of computer vision technology. A semantic mapping network is added to the basic model of a scene expression network to obtain a semantic attribute embedded scene expression network; the constructed semantic attribute embedded scene expression network is trained to establish a mapping from the semantic attribute space to the implicit expression space; the semantic attributes of the invisible scene are input into the trained semantic attribute embedded scene expression network to synthesize a view of the invisible scene; the present invention uses semantic attributes to establish a connection between seen scenes and invisible scenes, expands neural scene expression from a single visual modality to a visual and textual dual modality, uses semantic information to realize the expression of invisible scenes, generalizes scene expression to invisible scenes without views, and solves the problem of view dependence in current scene expression methods.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method and system for expressing invisible scenes based on semantic attributes. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, neural scene representation has developed faster and faster, and its impact in the fields of computer vision and graphics has become increasingly obvious. Since actual application scenarios often change, cross-scene generalization is an important issue in neural scene representation.

[0004] Cross-scene generalization methods can be divided into two categories: global generalization methods and local generalization methods. Global generalization is to obtain the global expression features of each scene and use them as conditions to input them into the scene expression model for decoding to achieve new viewpoint synthesis of each scene. There are two ways to obtain global expression features. The first is amortized reasoning, such as GQN and NeRF-VAE; the second is automatic decoding, such as SRNs and LFNs.

[0005] The second type of cross-scene generalization method is local generalization. Local generalization uses the local properties of the scene to express the new scene, but it also requires partial views and poses of the scene. The more existing views, the better the expression effect.

[0006] The cross-scene generalization methods mentioned above all require image and pose information of the new scene under partial viewing angles, which belong to the generalization expression between visible scenes. However, for invisible scenes without any view and pose information, current methods are difficult to generalize.

[0007] Therefore, the problem of accurate generalization for unseen scenes is a topic worthy of further research. Summary of the Invention

[0008] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for expressing invisible scenes based on semantic attributes, which utilizes semantic attributes to establish a connection between seen scenes and invisible scenes, expands neural scene expression from a single visual modality to a dual modality of vision and text, utilizes semantic information to realize the expression of invisible scenes, and generalizes scene expression to invisible scenes without views, thus solving the problem of view dependence in current scene expression methods.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] A first aspect of the present invention provides an invisible scene expression method based on semantic attributes;

[0011] A method for expressing invisible scenes based on semantic attributes, comprising:

[0012] On the basis of the scene expression network model, a semantic mapping network is added to obtain a semantic attribute embedded scene expression network;

[0013] Train the constructed semantic attribute embedded scene expression network to establish a mapping from the semantic attribute space to the implicit expression space;

[0014] The semantic attributes of the unseen scene are input into the trained semantic attribute embedded scene representation network to synthesize the view of the unseen scene.

[0015] Furthermore, the scene expression network basic model includes a 2D-3D projection module, a scene characterization function, a hypernetwork, a differentiable ray marching, and a pixel generator. It takes the parameters of the target camera perspective and the initial depth map as input and outputs the view of the scene.

[0016] Furthermore, the semantic mapping network connects the hypernetwork in the scene expression network basic model in the following way:

[0017] Input the scene's semantic description vector into the semantic mapping network to obtain the scene's implicit encoding;

[0018] The implicit code is decoded through the hypernetwork to obtain the weight parameter value of the scene representation function.

[0019] Furthermore, the semantic mapping network is composed of a multi-layer fully connected neural network.

[0020] Furthermore, the semantic description vector includes information about the type, size, position, color, and material of objects in the scene.

[0021] Furthermore, it also includes adding a combined expression module before the pixel generator of the scene expression network basic model.

[0022] Furthermore, the combined expression module adopts a combined expression strategy that converts a single multi-object scene expression into multiple parallel single-object scene expressions. It inputs parallel single-object scene semantic description vectors and obtains scene expression vectors of parallel single-object scenes, and finally performs weighted fusion to obtain the scene expression vector of the multi-object scene.

[0023] A second aspect of the present invention provides an invisible scene expression system based on semantic attributes.

[0024] A semantic attribute-based invisible scene representation system, including a network construction module, a network training module, and a view synthesis module:

[0025] The network construction module is configured as follows: adding a semantic mapping network to the scene expression network basic model to obtain a semantic attribute embedded scene expression network;

[0026] The network training module is configured to: train the constructed semantic attribute embedded scene expression network and establish a mapping from the semantic attribute space to the implicit expression space;

[0027] The view synthesis module is configured to input the semantic attributes of the unseen scene into the trained semantic attribute embedded scene representation network to synthesize the view of the unseen scene.

[0028] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the invisible scene expression method based on semantic attributes as described in the first aspect of the present invention.

[0029] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, the steps of the invisible scene expression method based on semantic attributes as described in the first aspect of the present invention are implemented.

[0030] One or more of the above technical solutions have the following beneficial effects:

[0031] This paper proposes a method for representing invisible scenes based on semantic attributes, which for the first time begins to address the problem of accurate generalization of completely invisible scenes. During the training phase, based on the SRNs unsupervised learning framework, a semantic mapping module is learned to embed semantic attributes into the implicit scene expression corresponding to the visible scene, and an association is established between the semantic attribute space and the implicit scene expression space. During the testing phase, the association between the two is utilized to synthesize unseen views through semantic attributes, thus getting rid of the dependence on views.

[0032] To address the high complexity problem of expressing unseen scenes containing multiple objects, the present invention also proposes a combined expression strategy, which converts the expression of multi-object scenes into the expression of multiple parallel single-object scenes, reducing the number of training samples and improving training efficiency.

[0033] The present invention uses semantic attributes to establish a connection between seen scenes and invisible scenes, expands the neural scene expression from a single visual modality to a visual and textual dual modality, uses semantic information to express invisible scenes, and generalizes scene expression to invisible scenes without views, solving the problem of view dependence in current scene expression methods.

[0034] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 Representing network structure graphs for semantic attribute embedded scenarios.

[0037] Figure 2 This is a flow chart of the method of the first embodiment.

[0038] Figure 3 Schematic diagram of the semantic mapping network process.

[0039] Figure 4 Schematic diagram of the process of combining expression modules.

[0040] Figure 5 This is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention; unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0044] Example 1

[0045] This embodiment discloses a method for expressing invisible scenes based on semantic attributes, and proposes a semantic attribute embedded scene representation network (Semantic Attribute-Embedded Scene Representation Networks, SAE-SRNs). The network structure of SAE-SRNs is as follows: Figure 1As shown in the figure, a semantic mapping network and a combination expression strategy are added to the original SRNs (Scene Representation Networks) basic model.

[0046] A method for expressing invisible scenes based on semantic attributes, the specific steps are as follows Figure 2 As shown, including:

[0047] Step S1: Add a semantic mapping network to the scene expression network basic model to obtain a semantic attribute embedded scene expression network;

[0048] like Figure 1 As shown in the figure, the original scene representation network SRNs basic model includes a 2D-3D projection module, a scene representation function, a hypernetwork, a differentiable ray marching, and a pixel generator. The specific process of synthesizing scene views is as follows:

[0049] (1) Input the camera view parameters (intrinsic parameter K, external parameter [R|t]) and the initial depth map d0 into the 2D-3D projection module to generate the world coordinates (x, y, z);

[0050] (2) Input the world coordinates into the scene representation function Φ to generate the scene expression vector V i ;

[0051] (3) V i Input into the differentiable Ray Marching to generate the depth map d i ;

[0052] (4) The generated depth map d i Feedback to the 2D-3D projection module generates new world coordinates (x, y, z);

[0053] (5) Then input it into the scene representation function Φ to generate a new scene expression vector V i+1 After n iterations of optimization, the final scene expression vector V is output. n ;

[0054] (6)V n The input is sent to the pixel generator for decoding, and the image under the target perspective is finally output.

[0055] The scene representation function Φ is the key to scene expression. The weight parameters of the scene representation function Φ are determined by the hypernetwork. For the cross-scene generalized expression task, the hypernetwork takes the implicit code z of the scene as input and outputs the weight parameters of the scene representation function Φ. By inputting different implicit codes, the generalized representation of multiple scenes is achieved; the hypernetwork (HyperNetwork) is a neural network that can output the weight parameters of the fully connected network, thereby realizing the control of the lower-level fully connected network.

[0056] In order to achieve better expression of different scenarios, this embodiment adds a semantic mapping network T to the front end of the super network. The semantic mapping network consists of a multi-layer fully connected neural network.

[0057] like Figure 3 As shown in Figure 1, the semantic mapping network T takes the semantic description vector s of the scene as input and converts it into the implicit code z of the scene. The implicit code z will be input into the subsequent hypernetwork for decoding to express the scene described by s. The semantic mapping network T is shown in formula (1):

[0058]

[0059] Among them, s is the m-dimensional semantic vector, z is the n-dimensional latent code, and the dimension of the semantic attribute m is usually determined by the attributes of the dataset, n = 256 (the same as the original scene representation network SRNs basic model).

[0060] Step S2: Train the constructed semantic attribute embedded scene expression network to establish a mapping from the semantic attribute space to the implicit expression space.

[0061] Based on the SRNs unsupervised learning framework, the semantic attribute embedded scene expression network is trained to construct a mapping from the semantic attribute space to the implicit expression space. Specifically:

[0062] Step S201: Construct a training data set, which includes camera viewing angle parameters, real scene images and corresponding scene semantic description vectors. The scene semantic description vectors contain information such as the type, size, position, color and material of objects in the scene.

[0063] Step S202: Use the training data set to perform unsupervised training on the semantic attribute embedded scene expression network. By taking the distance between the target perspective image predicted by the semantic attribute embedded scene expression network and the real scene image as the loss function value, the parameters in the network are continuously updated until the optimal state is reached.

[0064] Step S3: Input the semantic attributes of the invisible scene into the trained semantic attribute embedded scene representation network to synthesize the view of the invisible scene. The specific method is:

[0065] The random camera view parameters, initial depth map and semantic attributes of the invisible scene are input into the semantic attribute embedded scene expression network. After multiple iterative optimizations, the final scene expression vector matrix is ​​generated and input into the pixel generator for decoding. Finally, the view of the invisible scene is output. The appearance and geometry of the invisible scene are expressed through semantic attributes, and the view of the invisible scene is synthesized.

[0066] Although the problem of generating visual modal expressions from textual modalities of invisible scenes has been solved by adding a semantic mapping network, this semantic attribute embedded scene expression network can only express invisible scenes containing a single object. When the network is applied to invisible scenes containing multiple objects, the accuracy of the synthesized image will decrease.

[0067] To address this problem, this embodiment also proposes an object combination expression strategy, which selects the scene expression vector of each object based on depth. Because at a certain pixel position under a certain perspective, only the closer object part will be seen, while the farther object will be blocked. In this way, the combination expression strategy simplifies the task of multi-object scene expression into parallel single-object scene expression. To this end, a combination expression module is added before the pixel generator of the semantic attribute embedded scene expression network to convert the scene expression of multiple objects in the same scene into multiple parallel single-object scene expressions. Finally, the scene expression vectors of each single-object scene are weightedly fused to obtain the scene expression vector of the multi-object scene, such as Figure 4 As shown, specifically:

[0068] (1) For a multi-object scene containing L objects The L semantic vectors of the scene Input them into the model respectively, and after passing through the scene representation function Φ, the scene expression vector matrix of L scenes is obtained At the same time, the depth maps of these scenes are obtained

[0069] (2) For each pixel (i, j), according to the estimated depth From the obtained parallel single-object scene representation vector Select the scene representation V′ closest to the camera (i,j) As the scene expression vector of the original multi-object scene at this pixel point.

[0070] The reason for choosing the scene representation closest to the camera is that there is an occlusion problem in multi-object scenes, and often the front object occludes the back object. Therefore, for the light corresponding to pixel (i, j), the feature vector of the first object passed by the light is used as the feature vector of the multi-object scene, while the feature vectors of the objects behind it are directly ignored due to occlusion. The first object passed by the light is also the object closest to the camera.

[0071] (3) Expression vector V′ of each pixel (i,j) The expression vector matrix V′ that constitutes the entire multi-object scene is input into the pixel generator to synthesize the invisible multi-object scene. view.

[0072] In order to ensure the continuity and differentiability of the neural network, the scene expression vectors of each object are weighted and fused using formula (2) to obtain the scene expression vector of the scene.

[0073]

[0074] Where L is the multi-object scene The number of objects in V (i,j),k is the scene representation vector at pixel coordinate (i, j) in the kth single-object scene, V′ (i,j) is the multi-object scene expression vector at the (i, j) pixel coordinate. k According to the depth d (i,j) The calculated weight, α, is a parameter that reflects how quickly the weight changes with depth. In order to better simulate the change in depth weight of objects at different distances from the camera, the α value should be as large as possible. However, due to the limitations of the float data storage range and experimental verification, the best effect is achieved when α = 30.

[0075] Example 2

[0076] This embodiment discloses an invisible scene expression system based on semantic attributes;

[0077] like Figure 5 As shown in Figure 1, a semantic attribute-based invisible scene representation system includes a network construction module, a network training module, and a view synthesis module:

[0078] The network construction module is configured as follows: adding a semantic mapping network to the scene expression network basic model to obtain a semantic attribute embedded scene expression network;

[0079] The network training module is configured to: train the constructed semantic attribute embedded scene expression network and establish a mapping from the semantic attribute space to the implicit expression space;

[0080] The view synthesis module is configured to input the semantic attributes of the unseen scene into the trained semantic attribute embedded scene representation network to synthesize the view of the unseen scene.

[0081] Example 3

[0082] The purpose of this embodiment is to provide a computer-readable storage medium.

[0083] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the invisible scene expression method based on semantic attributes as described in the first embodiment of the present disclosure.

[0084] Example 4

[0085] The purpose of this embodiment is to provide an electronic device.

[0086] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the invisible scene expression method based on semantic attributes as described in the first embodiment of the present disclosure are implemented.

[0087] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for expressing invisible scenes based on semantic attributes, characterized in that: include: On the basis of the scene expression network model, a semantic mapping network is added to obtain a semantic attribute embedded scene expression network; The constructed semantic attribute embedded scene expression network is trained to establish a mapping from the semantic attribute space to the implicit expression space. Specifically: Construct a training dataset, which includes camera view parameters, real scene images, and corresponding scene semantic description vectors. The scene semantic description vectors contain information about the type, size, position, color, and material of objects in the scene. The semantic attribute embedded scene representation network is unsupervisedly trained using a training dataset. The distance between the target view image predicted by the semantic attribute embedded scene representation network and the real scene image is used as the loss function value, and the parameters in the network are continuously updated until the optimal state is reached. The semantic attributes of the unseen scene are input into the trained semantic attribute embedded scene representation network to synthesize the view of the unseen scene.

2. The invisible scene expression method based on semantic attributes according to claim 1, characterized in that: The scene expression network basic model includes a 2D-3D projection module, a scene representation function, a hypernetwork, differentiable RayMarching, and a pixel generator. It takes the parameters of the target camera perspective and an initial depth map as input and outputs a view of the scene.

3. The invisible scene expression method based on semantic attributes according to claim 1, characterized in that: The semantic mapping network connects the super network in the scene expression network basic model in the following way: Input the scene's semantic description vector into the semantic mapping network to obtain the scene's implicit encoding; The implicit code is decoded through the hypernetwork to obtain the weight parameter value of the scene representation function.

4. The invisible scene expression method based on semantic attributes according to claim 1, characterized in that: The semantic mapping network is composed of a multi-layer fully connected neural network.

5. The invisible scene expression method based on semantic attributes according to claim 1, characterized in that: It also includes adding a combined expression module before the pixel generator of the scene expression network basic model.

6. The invisible scene expression method based on semantic attributes according to claim 5, characterized in that: The combined expression module adopts a combined expression strategy that converts a single multi-object scene expression into multiple parallel single-object scene expressions. It inputs parallel single-object scene semantic description vectors and obtains scene expression vectors of parallel single-object scenes, and finally performs weighted fusion to obtain the scene expression vector of the multi-object scene.

7. A semantic attribute-based invisible scene expression system, characterized by: Including network construction module, network training module and view synthesis module: The network construction module is configured to: add a semantic mapping network to the scene expression network basic model to obtain a semantic attribute embedded scene expression network; The network training module is configured to train the constructed semantic attribute embedded scene expression network and establish a mapping from the semantic attribute space to the implicit expression space. Specifically: Construct a training dataset, which includes camera view parameters, real scene images, and corresponding scene semantic description vectors. The scene semantic description vectors contain information about the type, size, position, color, and material of objects in the scene. The semantic attribute embedded scene representation network is unsupervisedly trained using a training dataset. The distance between the target view image predicted by the semantic attribute embedded scene representation network and the real scene image is used as the loss function value, and the parameters in the network are continuously updated until the optimal state is reached. The view synthesis module is configured to input the semantic attributes of the unseen scene into the trained semantic attribute embedded scene representation network to synthesize the view of the unseen scene.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the invisible scene expression method based on semantic attributes as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the invisible scene expression method based on semantic attributes as described in any one of claims 1 to 6 are implemented.

Images