Model processing method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a model processing method, device, equipment and medium, the method comprising: obtaining an initial three-dimensional model constructed for a target space, and a real scene video collected in the target space; matching a plurality of video frames included in the real scene video with the initial three-dimensional model based on a plurality of viewpoints set in the initial three-dimensional model to obtain a matching result; wherein the matching result is used to represent a matching relationship between the video frames and the viewpoints; performing texture mapping processing on the initial three-dimensional model according to the matching result to obtain a virtual reality model. In the above scheme, the video frames required for texture mapping are determined from the real scene video automatically, and the texture mapping processing is also automatic, which reduces the standardization requirement for shooting the target space, simplifies the job flow for generating the virtual reality model for the target space, and improves the success rate of generating the virtual reality model through automatic video frame extraction and texture mapping processing.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model processing method, apparatus, device, and medium. Background Technology

[0002] Creating a virtual reality model of a building requires staff to meticulously follow specific standard operating procedures to capture images in each room along designated paths. The aforementioned approach imposes strict standards on image capture paths, and failure to adhere strictly to these standards significantly increases the likelihood of virtual reality model generation failure. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a model processing method, apparatus, device and medium.

[0004] This disclosure provides a model processing method, including: Acquire an initial 3D model constructed for the target space, as well as real-world video captured within the target space; Based on multiple viewpoints set within the initial 3D model, multiple video frames included in the real-scene video are matched with the initial 3D model to obtain a matching result; wherein, the matching result is used to characterize the matching relationship between the video frame and the viewpoint; Based on the matching results, the initial 3D model is processed with textures to obtain a virtual reality model.

[0005] This disclosure also provides a model processing apparatus, including: The acquisition module is used to acquire an initial 3D model constructed for the target space, as well as real-scene video captured within the target space; A matching module is used to match multiple video frames included in the real-scene video with the initial 3D model based on multiple viewpoints set within the initial 3D model to obtain a matching result; wherein, the matching result is used to characterize the matching relationship between the video frame and the viewpoint; The texturing module is used to perform texturing processing on the initial 3D model according to the matching results to obtain a virtual reality model.

[0006] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the model processing method provided in this disclosure.

[0007] This disclosure also provides a computer-readable storage medium storing a computer program for executing the model processing method provided in this disclosure.

[0008] Compared with the prior art, the technical solution provided in this disclosure has the following advantages. The model processing scheme provided in this disclosure includes: acquiring an initial three-dimensional model constructed for a target space, and real-scene video captured within the target space; matching multiple video frames included in the real-scene video with the initial three-dimensional model based on multiple viewpoints set within the initial three-dimensional model to obtain a matching result; wherein the matching result is used to characterize the matching relationship between the video frames and the viewpoints; and performing texture processing on the initial three-dimensional model according to the matching result to obtain a virtual reality model. By adopting the above technical solution, matching video frames from the real-scene video captured in the target space with the initial three-dimensional model corresponding to the target space to obtain a matching result, and performing texture processing on the initial three-dimensional model according to the matching result to obtain a virtual reality model, this achieves automated determination of the video frames required for texture mapping from the real-scene video and automated texture processing, reducing the standardization requirements for shooting the target space, simplifying the workflow for generating virtual reality models for the target space, and improving the success rate of virtual reality model generation through automated video frame extraction and texture processing. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic flowchart illustrating a model processing method provided in an embodiment of this disclosure; Figure 2 A schematic diagram of a viewpoint provided for an embodiment of this disclosure; Figure 3 A flowchart illustrating another model processing method provided in this embodiment of the disclosure; Figure 4 A schematic diagram of a first segmented image provided in an embodiment of this disclosure; Figure 5 A schematic diagram of a video frame cluster and outlier video frames provided in an embodiment of this disclosure; Figure 6A schematic diagram of a first segmented image provided in an embodiment of this disclosure; Figure 7 A flowchart illustrating yet another model processing method provided in this disclosure embodiment; Figure 8 This is a schematic diagram of a graphic transformation process provided in an embodiment of the present disclosure; Figure 9 This is a schematic diagram of a wall intersection line provided in an embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of a model processing device provided in an embodiment of the present disclosure; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0012] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0013] In related technologies, the creation of virtual reality models of houses requires fixed-point shooting, specifically, manual collection of data from different points in the room, which is time-consuming. Alternatively, it requires walking around the room according to a specific Standard Operating Procedure (SOP). Both of these methods place high demands on the location and path of data collection, and the collection methods themselves pose challenges to model reconstruction, easily leading to the failure of virtual reality model generation.

[0014] To address the aforementioned problems, this disclosure provides a model processing method, which will be described below with reference to specific embodiments.

[0015] Figure 1 This is a flowchart illustrating a model processing method provided in an embodiment of this disclosure. This model processing method can be applied to a model processing device. The model processing device can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the model processing method includes: Step 101: Obtain the initial 3D model constructed for the target space, as well as the real-world video captured within the target space.

[0016] The target space can be the actual space where a virtual reality model is to be built. This target space can be a house, etc. The initial 3D model can be a floor plan 3D model, meaning it records the floor plan of the target space but does not depict the actual scene of the target space. This embodiment does not restrict the format of the initial 3D model; for example, it can be a vector Building Information Modeling (BIM) stored in a lightweight data exchange format (JavaScript Object Notation, JSON).

[0017] The real-scene video can be video actually filmed within the target space. This real-scene video can be an indoor panoramic video captured by workers within the target space without a specific standard operating procedure. This embodiment does not limit the capture scenario for this real-scene video. For example, the capture scenario could be a renovation inspection scenario, where workers can carry a panoramic camera on a shoulder strap within the target space to capture video and inspect the progress of the renovation on the site, or workers can use mobile devices such as smartphones to capture video. Accordingly, the real-scene video in this renovation inspection scenario is a renovation inspection video.

[0018] In this embodiment of the disclosure, during the process of renovation acceptance, staff will move around the target space for a relatively long time and comprehensively. During this movement, a panoramic camera worn by the staff can capture video and generate a real-scene video. The model processing device can acquire this real-scene video and obtain an initial 3D model of the target space constructed in modeling software.

[0019] Step 102: Based on the multiple viewpoints set within the initial 3D model, match the multiple video frames included in the real-scene video with the initial 3D model to obtain matching results; wherein, the matching results are used to characterize the matching relationship between video frames and viewpoints.

[0020] The viewpoint can be a virtual location within the initial 3D model for observing that model; it can be understood as the position of a virtual camera, also known as the observation point. The vertical height of the viewpoint can be matched to the height at which the staff wears the panoramic camera. Figure 2 A schematic diagram of a viewpoint provided for an embodiment of this disclosure, such as... Figure 2 As shown, the red dots represent the viewpoints from a top-down perspective within the initial 3D model. Figure 2 In the diagram, short black lines represent doors, short red lines represent windows, and the viewpoints are arranged in an array at preset intervals.

[0021] A video frame can be the smallest image unit that constitutes a live-action video. The matching result can be the match between a viewpoint and a video frame; in the matching result, one viewpoint can correspond to one video frame. A viewpoint and video frame with a matching relationship can be understood as the video frame being captured near the location corresponding to that viewpoint in the target space.

[0022] In this embodiment of the disclosure, the model processing device can extract multiple video frames from a real-world video and obtain multiple viewpoints pre-arranged in an initial 3D model. The multiple viewpoints and the multiple video frames are then matched to obtain a matching result.

[0023] Optionally, the model processing method further includes: if an object occludes in the video frame, inputting the video frame into the object erasure model to obtain a video frame that erases the object and completes the background portion occluded by the object. This embodiment does not limit the type of the object; for example, the object can be a human body. The object erasure model can be determined by fine-tuning an image editing model.

[0024] Figure 3 This is a flowchart illustrating another model processing method provided in an embodiment of the present disclosure. In some embodiments of the present disclosure, multiple video frames included in the real-scene video are matched with the initial 3D model based on multiple viewpoints set within the initial 3D model to obtain a matching result, including: Step 301: Extract candidate video frames from multiple video frames, and use the building component segmentation model to determine multiple first segmentation images of the multiple candidate video frames.

[0025] The candidate video frames can be selected video frames from the video frames to be matched with the viewpoint. The building component segmentation model can be a neural network model that segments building component regions in an image. This embodiment does not limit the structure of the building component segmentation model; for example, it can be a semantic segmentation model or a model based on the transformer architecture. Building components can be the basic units that constitute a building entity, including one or more of walls, roofs, floors, doors, and windows. The first segmented image can be an image obtained by segmenting a real-scene image into building components, and different building components in the first segmented image can be represented by different colors. The first segmented image and the candidate video frames can have a one-to-one correspondence.

[0026] In this embodiment, the model processing device can filter multiple video frames to obtain candidate video frames, input the candidate video frames into the building component segmentation model, and obtain the first segmentation image output by the building component segmentation model that corresponds one-to-one with the candidate video frames. Figure 4 This is a schematic diagram of a first segmented image provided in an embodiment of the present disclosure, such as... Figure 4 As shown, the top and bottom doors of the wall are represented by different mask colors.

[0027] In some embodiments of this disclosure, extracting candidate video frames from multiple video frames includes: clustering multiple video frames according to a preset cluster size to obtain outlier video frames and multiple video frame clusters; wherein the cluster size is smaller than a preset size; and extracting a preset number of video frames and outlier video frames from each video frame cluster as candidate video frames.

[0028] The cluster size can be used to limit the maximum number of video frames in a cluster. The preset size can be the maximum value of a pre-defined cluster size; setting this preset size limits the number of video frames in a cluster, reducing the likelihood of clustering video frames corresponding to different viewpoints into the same video frame cluster. Outlier video frames are video frames that are not clustered into a video frame cluster; they can be understood as outliers obtained from clustering, also known as external points. A video frame cluster is a set of video frames obtained from clustering. The preset quantity can be a pre-defined number, for example, 1.

[0029] In this embodiment, the model processing device can extract vector features from video frames. This embodiment does not limit the method of extracting these vector features. Then, the dimension of the vector features is compressed to a preset dimension. This embodiment does not limit the preset dimension; for example, it can be 2-dimensional or 3-dimensional. Further, the reduced-dimensional feature vectors are clustered. Video frame clusters are determined based on the feature vector clusters in the clustering results, and outlier video frames are determined based on the outlier feature vectors in the clustering results. A preset number of video frames are randomly selected from each feature vector cluster, and the selected video frames and outlier video frames are used as candidate video frames.

[0030] Figure 5 This is a schematic diagram of a video frame cluster and outlier video frames provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, dots represent video frames. Video frames of the same color belong to the same video frame cluster, while black dots represent outlier video frames. Viewpoint matching calculations are then performed on each outlier video frame. For each video frame cluster, one video frame is selected as a candidate video frame for subsequent viewpoint matching calculations. This allows for the extraction of key video frames containing significant information from multiple video frames.

[0031] Because videos of actual construction and acceptance work are often lengthy and lack specific standard operating procedures for acquisition, workers may linger in one place for extended periods, leading to unnecessary information redundancy. The proposed solution utilizes an unsupervised approach to compress the number of video frames and filters redundant frames within a vector space, thereby compressing information from multiple video frames to obtain candidate video frames.

[0032] Step 302: Based on the architectural construction information of the initial 3D model, perform 2D rendering processing on multiple viewpoints of the initial 3D model to obtain multiple second segmentation images, wherein the viewpoints and second segmentation images correspond one-to-one.

[0033] The building construction information can be used to record the building components corresponding to the meshes in the initial 3D model, which is semantic information. The second segmentation image can be a semantic segmentation map obtained by performing 2D processing on the initial 3D model while retaining the building component information. Different building components in the second segmentation image can be represented by different colors.

[0034] In this embodiment, the initial 3D model can be obtained by 3D modeling based on vector information, and each facet in the initial 3D model contains corresponding building component information. The model processing device can perform 2D rendering on the initial 3D model, and determine the color of the corresponding area on the second segmented image after 2D rendering based on the building component information corresponding to each facet in the initial 3D model. This color corresponds one-to-one with the building component area.

[0035] Figure 6 This is a schematic diagram of a first segmented image provided in an embodiment of the present disclosure, such as... Figure 6 As shown, the wall, top, and bottom doors are represented by different color blocks.

[0036] Step 303: Match multiple first segmented images with multiple second segmented images to obtain multiple matched images corresponding to multiple viewpoints.

[0037] The matching image can be a video frame that matches the viewpoint. There can be a one-to-one correspondence between the matching image and the viewpoint.

[0038] In this embodiment, the first segmented image determined based on candidate video frames and the second segmented image determined based on the viewpoint in the initial 3D model are matched by the intersection over union (IoU) ratio. The candidate video frame corresponding to the first segmented image that matches the second segmented image is determined as the matching image that matches the viewpoint corresponding to the second segmented image.

[0039] In the above scheme, the first segmented image corresponding to the candidate video frame and the second segmented image corresponding to the viewpoint in the initial 3D model are determined by segmenting building components and rendering based on the building component information in two dimensions. Based on the second segmented image corresponding to the viewpoint, matching between sampling points and video frames is achieved.

[0040] Figure 7 A flowchart illustrating yet another model processing method provided in this disclosure embodiment is shown below. Figure 7As shown, in some embodiments of this disclosure, multiple first segmented images are matched with multiple second segmented images to obtain multiple matched images corresponding to multiple viewpoints, including: Step 701: The multiple second segmented images are sequentially determined as images to be processed.

[0041] The image to be processed can be the second segmented image currently being matched.

[0042] In this embodiment, the model processing device can perform matching processing on the second segmented images as images to be processed, and the matching processing of the second segmented images can be performed in parallel.

[0043] Step 702: Determine the cross-union ratios (CUI) of multiple images between the image to be processed and multiple first segmented images, and extract the first segmented image with the largest CUI.

[0044] Among them, the image intersection-over-union ratio can characterize the degree of matching between corresponding image regions of building components in two images, and can be used to characterize the similarity between two images.

[0045] In this embodiment, the model processing device can calculate the image crossover ratio (CVR) between the image to be processed and each first segmented image, and determine the first segmented image corresponding to the maximum value among the multiple CVRs.

[0046] In some embodiments of this disclosure, determining multiple image intersection-union ratios between the image to be processed and multiple first segmented images includes: performing rotation transformation processing on each first segmented image according to multiple preset rotation angles to obtain multiple rotated images corresponding to each first segmented image; for each first segmented image, calculating multiple candidate intersection-union ratios between the multiple rotated images of the first segmented image and the image to be processed, and extracting the maximum intersection-union ratio among the multiple candidate intersection-union ratios as the image intersection-union ratio between the first segmented image and the image to be processed.

[0047] The preset angle can be a pre-set rotation angle. The rotated image can be the image obtained by rotating the first segmented image by a corresponding angle. The candidate intersection-union ratio can characterize the degree of overlap between the rotated image and the image to be processed. For example, the candidate intersection-union ratio can be a comprehensive metric based on the intersection-union ratio of image regions corresponding to multiple identical building components between images.

[0048] In this embodiment, to consider the panoramic characteristics of the panoramic image, the first segmented image corresponding to the panoramic image needs to be rotated according to a preset angle to obtain a rotated image corresponding to each preset angle. For each first segmented image, the intersection-union ratio (IUR) of multiple rotated images of the first segmented image with the image to be processed is calculated to obtain multiple candidate IURs. Specifically, the model processing device can determine the semantic information corresponding to each pixel in the two images, and determine the number of pixels with the same semantic meaning at the same position in the two images. The number of pixels is divided by the total number of pixels in one image to obtain the candidate IURs.

[0049] Furthermore, the maximum value among the candidate crossover ratios is taken as the maximum crossover ratio, and this maximum crossover ratio is taken as the image crossover ratio between the first segmented image and the image to be processed.

[0050] In the above scheme, based on the characteristics of panoramic images, the intersection-union ratio (IUR) of the first segmented image after rotation processing and the image to be processed is calculated, thus achieving accurate calculation of the IUR.

[0051] In some embodiments of this disclosure, the target space includes multiple building components; calculating multiple candidate cross-union ratios between multiple rotated images of the first segmented image and the image to be processed includes: extracting partial images of the same building components corresponding to each rotated image and the image to be processed, obtaining multiple component image pairs corresponding to each rotated image; for each rotated image, calculating multiple component cross-union ratios of the multiple component image pairs of the rotated image, and determining candidate cross-union ratios based on the multiple component cross-union ratios.

[0052] A component image pair may include a rotated image and a partial image of the same building component corresponding to the image to be processed. Other image portions besides the building component may be blacked out or made transparent. The component intersection-union ratio can be the ratio of the intersection area to the union area between the partial images of the same building component.

[0053] In this embodiment, for each rotated image and the image to be processed, the model processing device can extract partial images of the same building components from both images to obtain a component image pair corresponding to each building component. This component image pair may include a first image portion of the rotated image corresponding to the building component and a second image portion of the image to be processed corresponding to the building component. For each component image pair, the intersection-union ratio (IUGR) between the first and second image portions is calculated to obtain the component IUGR corresponding to each component image pair. Based on the weights corresponding to the building components, the component IUGRs are weighted and summed to obtain candidate IUGRs. Optionally, the weights corresponding to doors and windows may be greater than the weights corresponding to other building components.

[0054] In the above scheme, the candidate cross-union ratio between images is determined based on the cross-union ratio between the image parts corresponding to the same component in different images, thereby extending the matching relationship between components to between images and improving the accuracy of image matching.

[0055] Step 703: Determine the video frame corresponding to the first segmented image with the largest image intersection-union ratio as the matching image of the viewpoint corresponding to the image to be processed.

[0056] In this embodiment, the model processing device can take the first segmented image corresponding to the maximum intersection-union ratio of the images as the target segmented image, and determine the video frame corresponding to the target segmented image as the matching image of the viewpoint corresponding to the image to be processed.

[0057] In the above scheme, the first segmented image and the second segmented image are matched based on the second segmented image to determine the matching image of the viewpoint corresponding to the second segmented image. This realizes the selection of video frames collected near the position corresponding to the viewpoint from multiple video frames, making the subsequent texture mapping based on the matching image more natural.

[0058] Step 103: Apply texture mapping to the initial 3D model based on the matching results to obtain a virtual reality model.

[0059] Among them, the virtual reality model can be a three-dimensional model that simulates the real scene of the target space.

[0060] In this embodiment of the disclosure, the model processing device can perform texture processing on the initial 3D model based on the matching results between the viewpoint and the video frame to generate a virtual reality model.

[0061] In some embodiments of this disclosure, a virtual reality model is obtained by applying texture mapping to an initial 3D model based on the matching results, including: applying the matching image of each viewpoint to the model view of the viewpoint on the initial 3D model to obtain the virtual reality model.

[0062] Among them, the model view can be the view obtained by observing the initial 3D model from the position of the virtual camera at the viewpoint.

[0063] In this embodiment, the matching results record the matching relationship between viewpoints and matching images. For each viewpoint, the model processing device can obtain the matching image corresponding to that viewpoint from the matching results and apply the matching image as a texture map to the model view corresponding to that viewpoint to obtain a virtual reality model. Thus, the virtual reality model is determined by applying texture maps from real-world images to the model view.

[0064] In some embodiments of this disclosure, a matching image of each viewpoint is mapped onto the model view of the viewpoint in the initial three-dimensional model to obtain a virtual reality model, including: determining the partition model where the viewpoint is located within the initial three-dimensional model; wherein the partition model is a portion of the initial three-dimensional model corresponding to a room in the target space; determining graphic transformation data based on the partition model and the matching image, and performing graphic transformation processing on the matching image based on the graphic transformation data to obtain a transformed image; and mapping the transformed image onto the model view to obtain a virtual reality model.

[0065] The room model can be a 3D model of the room where the viewpoint is located. Graphical transformation data can be used to record the precise matching between the model view and the image. The graphical transformations required for the matching image can be 6-DOF (Degrees of Freedom, DoF) spatial pose data, specifically including scale data, rotation data, and translation data. The transformed image can be the image obtained after performing graphical transformations on the matching image.

[0066] In this embodiment, the model processing device can determine the partition model of the partition where the viewpoint is located in the initial 3D model. Based on the partition model and the matching image, 3D graphics are matched to obtain graphics transformation data. Further, graphics transformation processing is performed on the matching graphics based on the graphics transformation data to obtain a transformed image. Texture mapping is performed on the model view corresponding to the viewpoint based on the transformed image to obtain a virtual reality model. Figure 8 This is a schematic diagram of a graphic transformation process provided in an embodiment of the present disclosure, such as... Figure 8 As shown, the initially loaded matching image is represented by a set of gradient-colored dots located in the center region of the image. Through image transformation processing, the matching image is magnified, rotated, and translated, and this matching image matches the room layout model of the master bedroom. Optionally, the top-view outline of the model and the top-view outline of the walls can record the positions of the first and second doors, respectively. Therefore, the success of the image transformation can be quickly determined by checking whether the door positions are consistent after the image transformation.

[0067] In the above scheme, the graphic transformation data of the matching graphic is accurately determined based on the segmentation model, which improves the accuracy of the texture mapping.

[0068] In some embodiments of this disclosure, determining graphic transformation data based on the partition model and the matching image includes: determining the top-view outline of the partition model and the top-view outline of the wall in the matching image; performing graphic matching on the top-view outline of the model and the top-view outline of the wall to obtain graphic transformation data.

[0069] The model's top-down outline can be a top-view outline of the room-by-room model; specifically, it can be a vector outline of the room layout. The wall's top-down outline can be a top-view outline of the room's walls.

[0070] In this embodiment, the model processing device can project the segmented model along the Z-axis to obtain a projected image, extract the outer contour of the projected image, and obtain a top-view contour map of the model. Furthermore, layout estimation is performed on the matching image to obtain a top-view contour map of the wall. For both the model and wall top-view contour maps, starting from their geometric centers, they are divided into N parts, and then a pre-set matching algorithm is used for graphic matching to obtain graphic transformation data. Here, N is a positive integer. This embodiment does not limit the matching algorithm. By dividing the contour map before matching, the problem of greedy point set matching is avoided.

[0071] For example, if X represents the top view of the wall, Y represents the top view of the model, s represents scale adjustment, R represents rotation adjustment, and t represents translation adjustment, then it can be determined by... Complete the point set alignment to obtain the corresponding graphic transformation data.

[0072] In the above scheme, the efficient and accurate determination of graphic transformation data is achieved by matching the contour maps. Based on this graphic transformation data, accurate real-scene texture mapping can be achieved, thereby automatically generating virtual reality models.

[0073] In some embodiments of this disclosure, determining the top-view outline of the partitioned model includes: determining the wall intersection lines in the matching image; wherein the wall intersection lines include the wall top intersection line and / or the wall ground intersection line; converting the wall intersection lines from two-dimensional space to three-dimensional space and performing top-view projection processing to obtain the wall top-view outline.

[0074] Among them, the wall intersection line can be the intersection line of the wall with other surfaces. The wall-top intersection line can be the intersection line of the wall with the roof. The wall-floor intersection line can be the intersection line of the wall with the floor.

[0075] In this embodiment, the model processing device can identify wall intersections in the matching image, input the matching image marked with wall intersections into the trained spatial transformation model, and use the spatial transformation model to transform the wall intersections from two-dimensional space to three-dimensional space, obtaining the wall intersections in three-dimensional space. The model processing device then maps the wall intersections in three-dimensional space onto a horizontal reference plane to obtain a top-view outline of the wall. This spatial transformation model can be a neural network model used for two-dimensional to three-dimensional space transformation; this embodiment does not limit the specific type of the spatial transformation model. Figure 9 This is a schematic diagram of a wall intersection provided in an embodiment of the present disclosure, such as... Figure 9 As shown, the red lines marked at the junctions of the wall and the roof, and at the junctions of the wall and the ground, are the wall junction lines.

[0076] In the above scheme, the top view outline of the wall is determined based on the wall intersection line. Since the light and shadow changes of the wall intersection line are relatively obvious, the top view outline of the wall can be determined even in low light conditions, thereby determining the virtual reality model, reducing the lighting requirements for generating the virtual reality model.

[0077] The model processing method provided in this disclosure includes: acquiring an initial 3D model constructed for a target space, and real-scene video captured within the target space; matching multiple video frames included in the real-scene video with the initial 3D model based on multiple viewpoints set within the initial 3D model to obtain a matching result; wherein, the matching result is used to characterize the matching relationship between the video frames and the viewpoints; and performing texture processing on the initial 3D model according to the matching result to obtain a virtual reality model. By adopting the above technical solution, matching video frames from real-scene video captured in the target space with the initial 3D model corresponding to the target space to obtain a matching result, and performing texture processing on the initial 3D model according to the matching result to obtain a virtual reality model, this method automates the determination of video frames required for texture mapping from real-scene video and performs automated texture processing. This reduces the standardization requirements for shooting in the target space, simplifies the workflow for generating virtual reality models for the target space, and improves the success rate of virtual reality model generation through automated video frame extraction and texture processing.

[0078] The model processing method provided in this disclosure solves the problem of complex standard operating procedures caused by manual fixed-point acquisition during the generation of virtual reality models. It introduces an initial 3D model as a reference, and can automatically extract matching graphics from real-world videos based on algorithms, and automatically estimate the 6-DOF graphic transformation data of the matching image. After the operators perform their normal work, the virtual reality model is automatically generated, reducing the cost of human intervention and eliminating the need for specific standard acquisition procedures.

[0079] Figure 10 This is a schematic diagram of a model processing device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and can also be integrated into an electronic device. Figure 10 As shown, the model processing device includes: The acquisition module 1001 is used to acquire an initial three-dimensional model constructed for the target space, as well as real-scene video captured within the target space; The matching module 1002 is used to match multiple video frames included in the real-scene video with the initial three-dimensional model based on multiple viewpoints set within the initial three-dimensional model to obtain a matching result; wherein, the matching result is used to characterize the matching relationship between the video frame and the viewpoint; The texturing module 1002 is used to perform texturing processing on the initial 3D model according to the matching result to obtain a virtual reality model.

[0080] Optionally, the step of matching multiple video frames included in the real-scene video with the initial 3D model based on multiple viewpoints set within the initial 3D model to obtain a matching result includes: Candidate video frames are extracted from the plurality of video frames, and a building component segmentation model is used to determine a plurality of first segmentation images of the plurality of candidate video frames; Based on the architectural construction information of the initial 3D model, the multiple viewpoints of the initial 3D model are subjected to 2D rendering processing to obtain multiple second segmented images, wherein the viewpoints and the second segmented images correspond one-to-one; The plurality of first segmented images are matched with the plurality of second segmented images to obtain a plurality of matched images corresponding to the plurality of viewpoints.

[0081] Optionally, extracting candidate video frames from the plurality of video frames includes: The multiple video frames are clustered according to a pre-set cluster size to obtain outlier video frames and multiple video frame clusters; wherein the cluster size is smaller than the pre-set size. A predetermined number of video frames and outlier video frames are extracted from each video frame cluster as candidate video frames.

[0082] Optionally, matching the plurality of first segmented images with the plurality of second segmented images to obtain a plurality of matching images corresponding to the plurality of viewpoints includes: The plurality of second segmented images are sequentially determined as images to be processed; Determine multiple image intersection-union ratios (IU / U) between the image to be processed and the multiple first segmented images, and extract the first segmented image with the maximum IU / U; The video frame corresponding to the first segmented image with the maximum image intersection-union ratio is determined as the matching image of the viewpoint corresponding to the image to be processed.

[0083] Optionally, determining the multiple image intersection-union ratios between the image to be processed and the multiple first segmented images includes: Each of the first segmented images is rotated and transformed according to multiple preset rotation angles to obtain multiple rotated images corresponding to each of the first segmented images. For each of the first segmented images, calculate multiple candidate cross-union ratios between multiple rotated images of the first segmented image and the image to be processed, and extract the maximum cross-union ratio among the multiple candidate cross-union ratios as the image cross-union ratio between the first segmented image and the image to be processed.

[0084] Optionally, the target space includes multiple building components; The calculation of multiple candidate intersection-union ratios between multiple rotated images of the first segmented image and the image to be processed includes: Extract partial images of the same building components corresponding to each of the rotated images and the image to be processed, to obtain multiple component image pairs corresponding to each of the rotated images; For each of the rotated images, calculate the cross-union ratios of multiple component image pairs of the rotated image, and determine the candidate cross-union ratio based on the multiple component cross-union ratios.

[0085] Optionally, the step of performing texture mapping on the initial 3D model based on the matching result to obtain a virtual reality model includes: The matching image of each viewpoint is mapped onto the model view of the initial 3D model to obtain the virtual reality model.

[0086] Optionally, the step of mapping the matching image of each viewpoint onto the model view of the initial 3D model to obtain the virtual reality model includes: Determine the room model where the viewpoint of the initial 3D model is located; wherein, the room model is the portion of the initial 3D model corresponding to the room in the target space; Based on the segmentation model and the matching image, graphic transformation data is determined, and the matching image is processed according to the graphic transformation data to obtain a transformed image; The transformed image is applied to the model view to obtain the virtual reality model.

[0087] Optionally, determining the graphic transformation data based on the segmentation model and the matching image includes: Determine the top-view outline of the partitioned model and the top-view outline of the wall in the matching image; The top-view outline of the model and the top-view outline of the wall are matched to obtain the graphic transformation data.

[0088] It should be noted that, Figure 10 The model processing device shown can execute each step in the above-described model processing method embodiments and achieve each process and effect in the above-described model processing method embodiments, which will not be elaborated here.

[0089] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 11 As shown, the electronic device 1100 includes one or more processors 1101 and memory 1102.

[0090] The processor 1101 may be a central processing unit (CPU) or other form of processing unit with model processing capability and / or instruction execution capability, and may control other components in the electronic device 1100 to perform the desired function.

[0091] The memory 1102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1101 may execute the program instructions to implement the model processing method of the embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0092] In one example, the electronic device 1100 may also include an input device 1103 and an output device 1104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0093] In addition, the input device 1103 may also include, for example, a keyboard, a mouse, etc.

[0094] The output device 1104 can output various information to the outside, including determined distance information, direction information, etc. The output device 1104 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0095] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 1100 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1100 may include any other suitable components depending on the specific application.

[0096] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the model processing methods provided in the embodiments of this disclosure.

[0097] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0098] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the model processing method provided in embodiments of this disclosure.

[0099] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0100] Embodiments of this disclosure may also be vehicles equipped with the aforementioned model processing device.

[0101] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model processing method, characterized in that, include: Acquire an initial 3D model constructed for the target space, as well as real-world video captured within the target space; Based on multiple viewpoints set within the initial 3D model, multiple video frames included in the real-scene video are matched with the initial 3D model to obtain a matching result; wherein, the matching result is used to characterize the matching relationship between the video frame and the viewpoint; Based on the matching results, the initial 3D model is processed with textures to obtain a virtual reality model.

2. The method according to claim 1, characterized in that, The process of matching multiple video frames from the real-world video with the initial 3D model based on multiple viewpoints set within the initial 3D model to obtain matching results includes: Candidate video frames are extracted from the plurality of video frames, and a building component segmentation model is used to determine a plurality of first segmentation images of the plurality of candidate video frames; Based on the architectural construction information of the initial 3D model, the multiple viewpoints of the initial 3D model are subjected to 2D rendering processing to obtain multiple second segmented images, wherein the viewpoints and the second segmented images correspond one-to-one; The plurality of first segmented images are matched with the plurality of second segmented images to obtain a plurality of matched images corresponding to the plurality of viewpoints.

3. The method according to claim 2, characterized in that, The step of extracting candidate video frames from the plurality of video frames includes: The multiple video frames are clustered according to a pre-set cluster size to obtain outlier video frames and multiple video frame clusters; wherein the cluster size is smaller than the pre-set size. A predetermined number of video frames and outlier video frames are extracted from each video frame cluster as candidate video frames.

4. The method according to claim 2, characterized in that, The step of matching the plurality of first segmented images with the plurality of second segmented images to obtain a plurality of matched images corresponding to the plurality of viewpoints includes: The plurality of second segmented images are sequentially determined as images to be processed; Determine multiple image intersection-union ratios (IU / U) between the image to be processed and the multiple first segmented images, and extract the first segmented image with the maximum IU / U; The video frame corresponding to the first segmented image with the maximum image intersection-union ratio is determined as the matching image of the viewpoint corresponding to the image to be processed.

5. The method according to claim 4, characterized in that, Determining the multiple image intersection-union ratios between the image to be processed and the multiple first segmented images includes: Each of the first segmented images is rotated and transformed according to multiple preset rotation angles to obtain multiple rotated images corresponding to each of the first segmented images. For each of the first segmented images, calculate multiple candidate cross-union ratios between multiple rotated images of the first segmented image and the image to be processed, and extract the maximum cross-union ratio among the multiple candidate cross-union ratios as the image cross-union ratio between the first segmented image and the image to be processed.

6. The method according to claim 5, characterized in that, The target space includes multiple architectural components; The calculation of multiple candidate intersection-union ratios between multiple rotated images of the first segmented image and the image to be processed includes: Extract partial images of the same building components corresponding to each of the rotated images and the image to be processed, to obtain multiple component image pairs corresponding to each of the rotated images; For each of the rotated images, calculate the cross-union ratios of multiple component image pairs of the rotated image, and determine the candidate cross-union ratio based on the multiple component cross-union ratios.

7. The method according to claim 1, characterized in that, The step of performing texture mapping on the initial 3D model based on the matching result to obtain a virtual reality model includes: The matching image of each viewpoint is mapped onto the model view of the initial 3D model to obtain the virtual reality model.

8. The method according to claim 7, characterized in that, The step of mapping the matching image of each viewpoint onto the model view of the initial 3D model to obtain the virtual reality model includes: Determine the room model where the viewpoint of the initial 3D model is located; wherein, the room model is the portion of the initial 3D model corresponding to the room in the target space; Based on the segmentation model and the matching image, graphic transformation data is determined, and the matching image is processed according to the graphic transformation data to obtain a transformed image; The transformed image is applied to the model view to obtain the virtual reality model.

9. The method according to claim 8, characterized in that, The step of determining graphic transformation data based on the segmentation model and the matching image includes: Determine the top-view outline of the partitioned model and the top-view outline of the wall in the matching image; The top-view outline of the model and the top-view outline of the wall are matched to obtain the graphic transformation data.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the model processing method according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the model processing method described in any one of claims 1-9.

Images