Virtual scene construction method and system, electronic equipment and storage medium

Abstract

The invention discloses a virtual scene construction method and system, electronic equipment and a storage medium. The method comprises the steps of obtaining article label data and image data; performing identification and segmentation processing on the image data according to the article label data to obtain a mask file and a feature value file; performing three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic tag; and editing and rendering the three-dimensional model according to the semantic tag to generate a virtual scene. According to the embodiment of the invention, the three-dimensional article can be edited according to the semantic tag so as to generate the corresponding virtual scene, and the method can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a virtual scene construction method, system, electronic device and storage medium. Background Art

[0002] In the related technology, there is a way to manually model the virtual scene, that is, first build a white film for the environment, then model each single object separately, and then edit the model to generate a room model. Since the objects are modeled separately, the objects themselves can be labeled. However, in actual applications, it is found that this method can be automated to a certain extent when modeling, but the construction of the overall model and the labeling of objects still need to be done manually, which is time-consuming and inefficient.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a virtual scene construction method, system, electronic device and storage medium, which can improve the efficiency of constructing virtual scenes.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application provides a method for constructing a virtual scene, the method comprising:

[0006] Acquire object label data and image data; the object label data is used to mark the object to be identified; the image data includes an image corresponding to the object to be identified;

[0007] The image data is identified and segmented according to the object label data to obtain a mask file and a feature value file; the mask file is used to mask the object to be identified, and the feature value file is used to mark the object to be identified;

[0008] Performing three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with semantic labels;

[0009] The three-dimensional model is edited and rendered according to the semantic label to generate a virtual scene.

[0010] In some embodiments, the obtaining of the object tag data and the image data comprises the following steps:

[0011] Acquire user input data, and identify the user input data to obtain the item label data;

[0012] Perform image acquisition processing on the actual scene to obtain the image data.

[0013] In some embodiments, the identifying and segmenting the image data according to the object tag data to obtain a mask file and a feature value file includes the following steps:

[0014] Performing data preprocessing on the item label data to obtain label prompt words;

[0015] Inputting the image data into an image segmentation model according to the label prompt word to perform object recognition and segmentation to obtain a corresponding mask file;

[0016] Performing feature value extraction processing on the mask file to obtain the feature value file.

[0017] In some embodiments, inputting the image data into an image segmentation model to perform object recognition and segmentation processing according to the label prompt word to obtain the mask file includes the following steps:

[0018] Inputting the image data into the image segmentation model; the image segmentation model comprises an image encoder, a hint encoder and a mask decoder;

[0019] Encoding the image data by the image encoder to obtain an image representation;

[0020] The label prompt word is converted by the prompt encoder to obtain a prompt representation;

[0021] Fusing the image representation and the prompt representation to obtain a fused representation;

[0022] Inputting the fused representation into the mask decoder for decoding to obtain a segmentation mask;

[0023] The image data is segmented according to the segmentation mask to obtain the mask file.

[0024] In some embodiments, performing three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label includes the following steps:

[0025] Performing conversion processing on the image data to obtain Gaussian basis elements;

[0026] The Gaussian primitives are subjected to semantic information mapping processing according to the mask file and the eigenvalue file to obtain a three-dimensional model with semantic labels.

[0027] In some embodiments, the performing semantic information mapping processing on the Gaussian primitives according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label comprises the following steps:

[0028] Performing semantic feature calculation processing on the Gaussian primitives to obtain a first feature;

[0029] Performing region positioning processing on the Gaussian primitives according to the mask file to obtain a positioning region;

[0030] Performing feature value mapping processing on the positioning area according to the feature value file to obtain a second feature;

[0031] Calculate a semantic loss value according to the first feature and the second feature;

[0032] The Gaussian primitives are trained according to the semantic loss values ​​to generate the three-dimensional model with the semantic labels.

[0033] In some embodiments, the editing and rendering of the three-dimensional model according to the semantic tag to generate a virtual scene specifically includes the following steps:

[0034] Get interactive instructions;

[0035] Editing the semantic tag according to the interactive instruction to obtain an editing result;

[0036] Performing object editing processing on the three-dimensional model according to the editing result to obtain an edited model;

[0037] The editing model is rendered to obtain the virtual scene.

[0038] To achieve the above object, another aspect of the embodiment of the present application provides a virtual scene construction system, the system comprising:

[0039] The first module is used to obtain object label data and image data; the object label data is used to mark the object to be identified; the image data includes an image corresponding to the object to be identified;

[0040] The second module is used to identify and segment the image data according to the object label data to obtain a mask file and a feature value file;

[0041] A third module is used to perform three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label;

[0042] The fourth module is used to edit and render the three-dimensional model according to the semantic label to generate a virtual scene.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The embodiments of the present application include at least the following beneficial effects: the present application provides a virtual scene construction method, system, electronic device and storage medium, the scheme obtains object label data and image data; identifies and segments the image data according to the object label data to obtain a mask file and a feature value file, and can pre-mark the image with the object label data to facilitate subsequent object modeling, making object identification more accurate, reducing computing power requirements, and improving the object labeling speed; in addition, the scheme performs three-dimensional modeling on the image data according to the mask file and the feature value file to obtain a three-dimensional model with semantic labels; edits and renders the three-dimensional model according to the semantic labels to generate a virtual scene, and can label the three-dimensional objects while the three-dimensional model is being constructed, and can edit the three-dimensional objects according to the semantic labels to generate the corresponding virtual scene, thereby improving the construction efficiency of the virtual scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a virtual scene construction method provided in an embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0048] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0049] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0050] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0051] Figure 6 This is a system architecture diagram of a virtual scene construction provided by an embodiment of the present application;

[0052] Figure 7 This is a modeling data collection flow chart provided in an embodiment of the present application;

[0053] Figure 8 It is a flowchart of building a three-dimensional model provided in an embodiment of the present application;

[0054] Fig. 9 It is a structural schematic diagram of a virtual scene construction system provided in an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0057] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0058] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0061] 1) 3D Gaussian Splatting (3DGS) is a rendering technique in 3D computer graphics. 3DGS uses Gaussian distribution to represent points or volumes in 3D space and uses these distributions to efficiently render complex 3D scenes.

[0062] 2) Segment Anything Model (SAM) is an advanced algorithm for image segmentation in the field of computer vision. It can segment objects using a single point, bounding box or text as a prompt word.

[0063] 3) 3D modeling is the use of 3D production software to construct a model with 3D data in a virtual 3D space.

[0064] 4) Virtual Reality (VR) is a computer simulation system that can create and experience a virtual world. It uses computers to generate a simulated environment and immerse users in it. Virtual reality technology uses real-life data, electronic signals generated by computer technology, and combines them with various output devices to transform them into phenomena that people can feel. These phenomena can be real objects in reality or substances that are invisible to the naked eye, which are expressed through three-dimensional models.

[0065] In the related technology, there are two main general methods for constructing three-dimensional models and identifying objects: 1. Manual modeling method, constructing a white film for the environment, modeling individual objects separately, and generating a room model after editing the model. Since the objects are modeled separately, the objects themselves can be labeled. This method can use a certain degree of automation when modeling, but the construction of the overall model and the labeling of objects still need to be done manually, which is time-consuming and inefficient. 2. Use professional equipment to construct a three-dimensional model of the overall environment, use deep learning to perform feature recognition on the modeled point cloud, select the boundaries of the objects, and realize point cloud segmentation. This method uses three-dimensional modeling first and then object recognition and segmentation. This method requires a lot of computing power. Due to the immaturity of point cloud recognition and object selection technology, the accuracy of object recognition is not high, and a certain degree of manual intervention is still required.

[0066] In view of this, a virtual scene construction and processing method, system, electronic device and storage medium are provided in the embodiments of the present application. The scheme obtains object label data and image data; identifies and segments the image data according to the object label data to obtain a mask file and a feature value file, and can pre-mark the image with the object label data to facilitate subsequent object modeling, making object identification more accurate, reducing computing power requirements, and improving the object labeling speed; in addition, the scheme performs three-dimensional modeling on the image data according to the mask file and the feature value file to obtain a three-dimensional model with semantic labels; edits and renders the three-dimensional model according to the semantic labels to generate a virtual scene, and can label the three-dimensional objects while the three-dimensional model is being constructed. The three-dimensional objects can be edited according to the semantic labels to generate the corresponding virtual scene, thereby improving the construction efficiency of the virtual scene.

[0067] A virtual scene construction method provided in an embodiment of the present application relates to the field of computer vision technology. A virtual scene construction method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements a virtual scene construction method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0070] Figure 1 is an optional flowchart of a virtual scene construction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0071] Step S101, obtaining object label data and image data; the object label data is used to mark the object to be identified; the image data includes an image corresponding to the object to be identified;

[0072] Step S102, identifying and segmenting the image data according to the object label data to obtain a mask file and a feature value file;

[0073] Step S103, performing three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label;

[0074] Step S104: Edit and render the three-dimensional model according to the semantic label to generate a virtual scene.

[0075] Steps S101 to S104 shown in the embodiment of the present application are to obtain the corresponding item label data and image data by obtaining the user's input. Among them, the item label data is used to mark the items to be identified, and the labels can be generated by voice or text input of the items to be identified. The image data includes the image corresponding to the item to be identified, and can also include camera parameters, depth information, etc. Then the image data is identified and segmented according to the item label data, and the image can be segmented by a visual segmentation model to obtain a mask file and a feature value file, wherein the mask file is used to mask the items to be identified, and the feature value file is used to mark the items to be identified. Then the mask file, the feature value file and the image data are input into the three-dimensional production software for three-dimensional modeling, and a label with semantic information is generated based on the mask file and the feature value file, so that the three-dimensional model after modeling has a semantic label. Finally, the three-dimensional model is edited and rendered according to the semantic label, and the items are deleted, replaced or added through the semantic label, and the processed three-dimensional model is displayed in the client by loading it into the local or platform cloud rendering to obtain the final virtual scene.

[0076] See also Figure 2 In step S101 of some embodiments, obtaining the object tag data and image data includes the following steps:

[0077] Step S201, obtaining user input data, and identifying the user input data to obtain the item tag data;

[0078] Step S202: performing image acquisition processing on the actual scene to obtain the image data.

[0079] In step S201 of some embodiments, data input by a user is obtained. The user input method may include voice input or text input. Voice data may be recorded by a client, and voice recognition may be performed on the voice data to obtain item label data. The item label data includes item generated labels, which may be green plants, tables, etc. The client may include devices such as smart phones and cameras.

[0080] In step S202 of some embodiments, the embodiment of the present application performs image acquisition on the actual scene, and the actual scene is the real scene corresponding to the virtual scene to be constructed. For example, in the home scene, it is necessary to perform image acquisition on the real home scene, and shoot the items and environment in the room. In addition, the acquired image data needs to include the image of the item to be identified corresponding to the item label data. If the item label data is green plants, when performing image acquisition, the panorama is shot and the green plant image is focused on. The green plants can be photographed in detail, or there are certain requirements for the number of green plant images obtained, which can be set according to the actual situation. In a feasible embodiment, the user shoots indoor videos or related item information according to the guidance of the client. First, the whole house can be scanned and photographed in a long-range manner, and then the close-up along the edge of the wall corner is shot from top to bottom, and the information such as the wall corner and the ceiling corner is supplemented. The center of the house is scanned and photographed from the outside, and specific item details can be supplemented. It should be noted that the images obtained in the above shooting steps must meet a certain number before they can be submitted, and the specific number can be set according to the actual situation. After the captured images meet certain conditions, they can be submitted to the platform for verification. By default, a room requires at least 500 images, which can be adjusted according to actual conditions.

[0081] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application can improve the recognition accuracy of the model for the object to be identified in the image through the user inputting the object label data, and reduce the cost of subsequent labeling. The embodiment of the present application also standardizes the data collection process and introduces a certain verification mechanism to ensure the quality and quantity of data collection, so as to reduce the occurrence of model missing, inaccurate object recognition, etc.

[0082] See also Figure 3 In step S102 of some embodiments, the image data is identified and segmented according to the object tag data to obtain a mask file and a feature value file, including the following steps:

[0083] Step S301, preprocessing the item label data to obtain label prompt words;

[0084] Step S302, inputting the image data into an image segmentation model to perform object recognition and segmentation processing according to the label prompt word to obtain the mask file;

[0085] Step S303: performing feature value extraction processing on the mask file to obtain the feature value file.

[0086] In step S301 of some embodiments, data preprocessing is performed on the object tag data, and the category tag input by the client is obtained by identification to generate a predefined tag. The predefined tag is a JSON file, and the file stores tag prompt words and corresponds to specific values ​​as identifiers. For example, the file includes {"houseplant":0, "table":1, "chair":2}, which indicates an indoor green plant tag with identification number 0, a table tag with identification number 1, and a chair tag with identification number 2. The predefined tag is used as a tag prompt word, and the tag prompt word is used to input the image segmentation model for image recognition and segmentation processing. It should be noted that the embodiments of the present application can also preprocess the image data, such as removing blurred images, reading camera parameters, depth information, etc.

[0087] In step S302 of some embodiments, the image data is input into an image segmentation model according to the label prompt word. The image segmentation model may adopt a SAM visual segmentation model. The image is segmented by the image segmentation model, and a mask file is finally output. The mask file is used to mask the objects to be identified. The feature value file is used to generate a 32-bit feature value to mark different objects, and is combined with the original image file, camera parameters, depth information, etc. as input data for generating a three-dimensional model.

[0088] In step S303 of some embodiments, feature value files are obtained by performing feature value extraction processing on the original image corresponding to the mask file. These feature values ​​may include pixel-level category labels, object bounding boxes, object areas and perimeters, etc.

[0089] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application uses an image segmentation model to identify and segment a two-dimensional image to obtain a mask file and a feature value file, and can provide a data basis for the subsequent generation of a three-dimensional model based on the mask file and the feature value file.

[0090] In step S302 of some embodiments, inputting the image data into an image segmentation model to perform object recognition and segmentation processing on the object to be recognized according to the label prompt word to obtain the mask file includes the following steps:

[0091] Inputting the image data into the image segmentation model; the image segmentation model comprises an image encoder, a hint encoder and a mask decoder;

[0092] Encoding the image data by the image encoder to obtain an image representation;

[0093] The label prompt word is converted by the prompt encoder to obtain a prompt representation;

[0094] Fusing the image representation and the prompt representation to obtain a fused representation;

[0095] Inputting the fused representation into the mask decoder for decoding to obtain a segmentation mask;

[0096] The image data is segmented according to the segmentation mask to obtain the mask file.

[0097] In an embodiment of the present application, the image segmentation model can adopt the SAM model, which can realize the recognition and segmentation of any object according to text instructions or image recognition. The embodiment of the present application uses a deep learning framework such as PyTorch to load the SAM model as an image segmentation model, and the image segmentation model includes an image encoder, a prompt encoder and a mask decoder. By inputting the image into the image encoder for image coding processing, a feature representation of the image is obtained, that is, an image representation, and then by inputting a prompt information, the prompt information is a label prompt word, and the label prompt word is converted into a feature representation of the prompt, that is, a prompt representation. Then the image representation and the prompt representation are element-level summed and fused to obtain a fused representation, and the fused representation is input to the mask decoder for decoding processing to obtain a segmentation mask, so that each pixel in the image can be assigned to different categories or objects according to the segmentation mask to obtain a mask file, and the granularity of the segmentation result can also be adjusted as needed, such as the whole, part and sub-part.

[0098] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application inputs the image data into the image segmentation model according to the label prompt words to perform object recognition and segmentation processing to obtain a mask file, which can improve the accuracy of the model in identifying objects according to the label prompt words.

[0099] See also Figure 4 In step S103 of some embodiments, the three-dimensional modeling processing is performed on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label, including the following steps:

[0100] Step S401, converting the image data to obtain Gaussian basis elements;

[0101] Step S402: Perform semantic information mapping processing on the Gaussian primitives according to the mask file and the feature value file to obtain a three-dimensional model with semantic labels.

[0102] In step S401 of some embodiments, the image data may be converted by a three-dimensional modeling algorithm such as a 3DGS algorithm, and Gaussian primitives may be generated according to the input image, camera parameters, depth information, etc. in the image data. Specifically, an initial point cloud is generated from the input image, each point cloud point is converted into a 3D Gaussian sphere, and attributes such as a mean, a covariance matrix, a color, and an opacity are assigned to it, and then the parameters of the Gaussian function are optimized by using differentiable Gaussian rasterization and a random gradient descent method, multiple three-dimensional Gaussian spheres are projected onto a two-dimensional plane, and the color value of each pixel is calculated to generate Gaussian primitives.

[0103] In step S402 of some embodiments, the embodiments of the present application perform semantic information mapping processing on Gaussian primitives according to the mask file and the eigenvalue file, and use the mask file and the eigenvalue file to map the two-dimensional semantic information to a three-dimensional Gaussian sphere to obtain a three-dimensional model with a semantic label, and the semantic label is generated by the semantic information.

[0104] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application performs three-dimensional modeling processing on image data through mask files and feature value files, and can generate corresponding label data while constructing the three-dimensional model, so that objects in the three-dimensional model can be quickly annotated, thereby improving the speed of object annotation and reducing computing power requirements.

[0105] In step S402 of some embodiments, the performing semantic information mapping processing on the Gaussian primitives according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label includes the following steps:

[0106] Performing semantic feature calculation processing on the Gaussian primitives to obtain a first feature;

[0107] Performing region positioning processing on the Gaussian primitives according to the mask file to obtain a positioning region;

[0108] Performing feature value mapping processing on the positioning area according to the feature value file to obtain a second feature;

[0109] Calculate a semantic loss value according to the first feature and the second feature;

[0110] The Gaussian primitives are trained according to the semantic loss values ​​to generate the three-dimensional model with the semantic labels.

[0111] In an embodiment of the present application, a semantic feature is calculated for each Gaussian ball in the Gaussian primitive, wherein the Gaussian ball can be regarded as a spatial range defined by a spherical Gaussian function, and the Gaussian primitive is used to describe the Gaussian distribution characteristics within this spatial range. In image processing, the Gaussian ball is used as part of a feature extraction method, and this feature extraction method is associated with an object recognition task. The semantic features of the Gaussian ball can be inferred based on this semantic task to obtain a first feature, which is the semantic feature of the Gaussian primitive obtained by initial construction. Then, the object position corresponding to the Gaussian primitive is processed for regional positioning according to the mask file to obtain a positioning area, and then the feature value of each area is mapped to a point on the surface of the Gaussian sphere according to the feature value file to obtain a second feature, which is the feature mapped to the corresponding Gaussian primitive through the feature value file. According to the first feature and the second feature, a semantic loss value can be calculated, and a Gaussian-semantic joint training is performed through the semantic loss value, and a Gaussian primitive parameter (Semantic_label) is added as a semantic label to record the 32-bit feature value of the object. Among them, the training of Gaussian primitives can utilize differentiable Gaussian rasterization, which is trained through stochastic gradient descent similar to neural networks, and the parameters of the Gaussian function are optimized in combination with the semantic loss value, and finally a three-dimensional model with semantic labels is generated.

[0112] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application marks the attributes of objects in the three-dimensional model by calculating the similarity between the modeling semantics and the input semantics, and finally generates a three-dimensional model with semantic information, thereby improving the speed of object labeling and facilitating the subsequent construction of virtual scenes.

[0113] See also Figure 5 In step S104 of some embodiments, the editing and rendering of the three-dimensional model according to the semantic tag to generate a virtual scene specifically includes the following steps:

[0114] Step S501, obtaining an interaction instruction;

[0115] Step S502, editing the semantic tag according to the interactive instruction to obtain an editing result;

[0116] Step S503, performing object editing processing on the three-dimensional model according to the editing result to obtain an edited model;

[0117] Step S504: Rendering the editing model to obtain the virtual scene.

[0118] In step S501 of some embodiments, an interaction instruction input by a user on a client is obtained. The interaction instruction may be a voice input instruction or a keyboard input instruction, etc., which is not limited here.

[0119] In step S502 of some embodiments, semantic tags can be annotated, deleted, replaced, etc. in the three-dimensional model according to the interactive instruction. For example, the semantic tags are annotated. By default, the annotation status of the semantic tags is set to invisible. Therefore, by setting the visible status of the semantic tags to visible, the semantic tags can be displayed in the three-dimensional model. The display method can be text or the like, and can be displayed below the corresponding objects. The semantic tags can also be deleted according to the interactive instruction, that is, the corresponding tag data in the three-dimensional model is directly deleted, and the replacement processing replaces the corresponding tag data in the three-dimensional model with other tag data.

[0120] In step S503 of some embodiments, the three-dimensional model is subjected to object editing processing according to the editing result to obtain an edited model. For example, after deleting the semantic tag of an object, the corresponding object in the three-dimensional model is also deleted to obtain a processed edited model. The embodiment of the present application can also obtain an object model modeled by a third-party platform by connecting with a third-party platform, or can generate an object model based on 3DGS by taking a photo of the object by the client, and then merge the obtained object model with the three-dimensional model with the semantic tag to obtain an edited model. The embodiment of the present application can obtain a processed edited model by merging a single object model based on 3DGS in the platform object model library with a three-dimensional model through interaction with the client.

[0121] In step S504 of some embodiments, the editing model is rendered to obtain the virtual scene. The processed editing model can be displayed in the client by loading it locally or rendering it on the platform cloud to generate a virtual scene.

[0122] The following is a detailed description and explanation of the solution of the embodiment of the present application in conjunction with a specific application example:

[0123] The embodiments of the present application can be applied to the field of computer vision technology, and can be applicable to scenes such as home scenes and smart parks that require three-dimensional modeling. Figure 6The virtual scene construction method of the embodiment of the present application can be applied to the client, 3DGS platform and e-commerce platform. Data collection is performed through the data collection module in the client. Data collection can include image collection, and semantic definition interaction is performed on the items in advance. The interaction method adopts voice input or text input, etc. The client can also receive the three-dimensional model returned by the 3DGS platform for display to generate a virtual scene. Then, data processing, model generation, model operation and display processing are performed through the 3DGS platform, wherein the data processing includes a data processing module, a semantic label module and an image semantic labeling module, wherein the data processing module is used to pre-process the data input by the client, the semantic label module is used to convert the text or voice data input by the client into semantic labels, and the image semantic labeling module is used to label the image according to the semantic label. In the model generation process, a 3GDS environment model with semantic information is generated, and the environment model and the item model can be managed by docking with the e-commerce platform. The management process is the model operation and display process. By labeling the items in the environment model, the items in the environment model can also be added, deleted, replaced, etc., so as to obtain the final model and render it, and the rendering result is displayed in the client to construct and generate a virtual scene. In the embodiment of the present application, the client realizes the input data required for modeling and object identification through the data processing process, especially the need to add the definition of identifiable objects, and because the accuracy of three-dimensional modeling and object identification is highly dependent on the quality of data collection, the basic process of collection is particularly defined, and the client also supports the display of the model. The platform will pre-process the collected data, especially the module for semantic annotation of the input image, which needs to convert the definition of the object to be identified into the semantic annotation of the image and carry it to the generated 3D model. After the 3D model with semantic tags is generated, the objects can be filtered based on specific semantic tags for annotation, deletion, addition and replacement. The relevant operations can be pushed to the client through the rendering model for display, and the user can instantly see the effect of replacing objects in the home environment. The embodiment of the present application can lay the foundation for the subsequent development of the digital twin field, and docking with the e-commerce platform can bring users a new shopping mode of what you see is what you get.

[0124] See also Figure 7In the process of collecting modeling data, start the client, enter the type of object to be identified, and then collect images of the real scene corresponding to the virtual scene construction. When the number of indoor scanning pictures is not met, perform a long-range scanning shot from left to back at the entrance of the house, and then perform a close-up scanning and moving shot along the edge of the house corner, and then perform fixed-point shooting in the middle of the house, and take additional shots of the corners of the wall and ceiling. Finally, take pictures of specific objects, that is, objects that need to be identified, until the number of indoor scanning pictures is met, and then verify the camera parameter information and depth information. The camera parameters include internal and external parameters, and complete the data collection and submit it to the modeling platform. Please refer to Figure 8 The modeling platform constructs a three-dimensional model through a computing server, determines multi-view camera parameters through camera parameter calibration, obtains multi-label semantic information through multi-view semantic information extraction, and combines depth information and camera parameters to model a three-dimensional model. The model is modeled through supervised algorithm design, and 2D expression optimization, camera parameter joint optimization, depth loss constraint, and semantic joint optimization are performed to train the three-dimensional model to obtain a Gaussian implicit model. The corresponding virtual scene can be generated by editing the Gaussian implicit model. Since the embodiment of the present application places object segmentation and recognition before modeling during the virtual scene construction process, the recognition is more accurate, the computing power requirement is small, and the object labeling speed is faster.

[0125] The embodiment of the present application can pre-label and quickly replace household items in the virtual scene through the virtual scene construction method, and can realize the carrying of the attributes of the items while building the 3D model. Without manual intervention, the items in the three-dimensional model can be labeled and replaced quickly. The scheme allows users to define the items to be identified and modeled through voice input / text through the semantic definition module, pre-process and label the collected image data, and carry the labeled semantic information in the Gaussian sphere attributes of the 3D model, so as to support the labeling of items while achieving modeling. The embodiment of the present application pre-places the item segmentation and recognition before 3D modeling, so that the item labeling recognition is more accurate, and at the same time, the system computing power requirements are small, and the identification speed of item labels is faster. In addition, the platform of the embodiment of the present application can also be connected with the e-commerce platform to realize the one-click purchase function of items, providing users with an integrated solution from 3D modeling, item labeling, item replacement and display to purchase.

[0126] See also Fig. 9 The embodiment of the present application further provides a virtual scene construction system, which can implement the above virtual scene construction method, and the system includes:

[0127] The first module 901 is used to obtain object label data and image data; the object label data is used to mark the object to be identified; the image data includes an image corresponding to the object to be identified;

[0128] The second module 902 is used to identify and segment the image data according to the object label data to obtain a mask file and a feature value file;

[0129] The third module 903 is used to perform three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label;

[0130] The fourth module 904 is used to edit and render the three-dimensional model according to the semantic label to generate a virtual scene.

[0131] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above virtual scene construction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0133] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0135] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0136] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the virtual scene construction method of the embodiment of this application;

[0137] Input / output interface 1003, used to implement information input and output;

[0138] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0139] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0140] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0141] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned virtual scene construction method is implemented.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The embodiments of the present application provide a method, system, electronic device and storage medium for constructing a virtual scene. The scheme obtains object label data and image data; identifies and segments the image data according to the object label data to obtain a mask file and a feature value file, and can pre-mark the image with the object label data to facilitate subsequent object modeling, making object identification more accurate, reducing computing power requirements, and improving the object labeling speed; in addition, the scheme performs three-dimensional modeling on the image data according to the mask file and the feature value file to obtain a three-dimensional model with semantic labels; edits and renders the three-dimensional model according to the semantic labels to generate a virtual scene, and can label the three-dimensional objects while constructing the three-dimensional model. The three-dimensional objects can be edited according to the semantic labels to generate the corresponding virtual scene, thereby improving the construction efficiency of the virtual scene.

[0145] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0146] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0155] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A virtual scene construction method, characterized in that: The method comprises the following steps: Acquire object label data and image data; the object label data is used to mark the object to be identified; the image data includes an image corresponding to the object to be identified; The image data is identified and segmented according to the object label data to obtain a mask file and a feature value file; the mask file is used to mask the object to be identified, and the feature value file is used to mark the object to be identified; Performing three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with semantic labels; The three-dimensional model is edited and rendered according to the semantic label to generate a virtual scene.

2. The method according to claim 1, characterized in that The step of obtaining the object tag data and image data comprises the following steps: Acquire user input data, and identify the user input data to obtain the item label data; Perform image acquisition processing on the actual scene to obtain the image data.

3. The method according to claim 1, characterized in that The identifying and segmenting the image data according to the object tag data to obtain a mask file and a feature value file comprises the following steps: Performing data preprocessing on the item label data to obtain label prompt words; Inputting the image data into an image segmentation model according to the label prompt word to perform object recognition and segmentation processing to obtain the mask file; Performing feature value extraction processing on the mask file to obtain the feature value file.

4. The method according to claim 3, characterized in that The step of inputting the image data into an image segmentation model according to the label prompt word to perform object recognition and segmentation to obtain a corresponding mask file comprises the following steps: Inputting the image data into the image segmentation model; the image segmentation model comprises an image encoder, a hint encoder and a mask decoder; Encoding the image data by the image encoder to obtain an image representation; The label prompt word is converted by the prompt encoder to obtain a prompt representation; Fusing the image representation and the prompt representation to obtain a fused representation; Inputting the fused representation into the mask decoder for decoding to obtain a segmentation mask; The image data is segmented according to the segmentation mask to obtain the mask file.

5. The method according to claim 1, characterized in that The performing three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label comprises the following steps: Performing conversion processing on the image data to obtain Gaussian basis elements; The Gaussian primitives are subjected to semantic information mapping processing according to the mask file and the eigenvalue file to obtain a three-dimensional model with semantic labels.

6. The method according to claim 5, characterized in that The step of performing semantic information mapping processing on the Gaussian primitives according to the mask file and the eigenvalue file to obtain a three-dimensional model with semantic labels comprises the following steps: Performing semantic feature calculation processing on the Gaussian primitives to obtain a first feature; Performing region positioning processing on the Gaussian primitives according to the mask file to obtain a positioning region; Performing feature value mapping processing on the positioning area according to the feature value file to obtain a second feature; Calculate a semantic loss value according to the first feature and the second feature; The Gaussian primitives are trained according to the semantic loss values ​​to generate the three-dimensional model with the semantic labels.

7. The method according to any one of claims 1 to 5, characterized in that: The editing and rendering of the three-dimensional model according to the semantic label to generate a virtual scene specifically includes the following steps: Get interactive instructions; Editing the semantic tag according to the interactive instruction to obtain an editing result; Performing object editing processing on the three-dimensional model according to the editing result to obtain an edited model; The editing model is rendered to obtain the virtual scene.

8. A virtual scene construction system, characterized in that: The system comprises: The first module is used to obtain object label data and image data; the object label data is used to mark the object to be identified; the image data includes an image corresponding to the object to be identified; The second module is used to identify and segment the image data according to the object label data to obtain a mask file and a feature value file; A third module is used to perform three-dimensional modeling processing on the image data according to the mask file and the feature value file to obtain a three-dimensional model with a semantic label; The fourth module is used to edit and render the three-dimensional model according to the semantic label to generate a virtual scene.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images