Spatial modeling method and device, equipment and storage medium

Abstract

The invention discloses a spatial modeling method, apparatus and device, and a storage medium. The method comprises the steps of obtaining a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by performing image acquisition on a target space; the first image frame comprises a first image corresponding to the target object; the second image frame comprises a second image corresponding to the target object; replacing a first image in the first image frame with a first replacement image to obtain a third image frame, and replacing a second image in the second image frame with a second replacement image to obtain a fourth image frame; a corresponding relation exists between the first replacement image and the second replacement image; and modeling the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a space modeling method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of computer vision and three-dimensional reconstruction technology, the space modeling method has been widely applied in virtual reality, augmented reality, automatic driving and other fields. By collecting multi-view images and constructing a three-dimensional model, high-precision restoration of the real scene can be achieved, providing important data support for subsequent applications.

[0003] However, in actual application, if the images containing sensitive information are directly used for modeling, privacy leakage problems may occur. In related technologies, attempts are made to blur or delete the privacy part in the original image to protect privacy, but this processing method often affects the integrity of the effective information of the image, and thus reduces the quality and practicability of three-dimensional modeling. SUMMARY

[0004] Therefore, the present application provides a space modeling method, device, equipment and storage medium.

[0005] The technical scheme of the present application embodiment is as follows:

[0006] The present application embodiment provides a space modeling method, which comprises the following steps:

[0007] obtaining a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame comprises a first image corresponding to a target object; the second image frame comprises a second image corresponding to the target object;

[0008] replacing the first image in the first image frame with a first replacement image to obtain a third image frame, and replacing the second image in the second image frame with a second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship;

[0009] modeling the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

[0010] In the above space modeling method, the method further comprises: searching for a target three-dimensional material matching the target object from a preset object material library; and rendering the first replacement image and the second replacement image based on the target three-dimensional material.

[0011] In the above space modeling method, the preset object material library comprises three-dimensional materials corresponding to each object category in a plurality of object categories; and the object category is a category containing privacy information.

[0012] In the space modeling method, the preset object material library includes three-dimensional materials corresponding to a target object category; the three-dimensional materials are provided with initial orientations; the initial orientations are aligned with a camera coordinate system of a preset pose algorithm; the target object category includes at least a plurality of object categories; the three-dimensional material corresponding to each object category is created based on object data of the same category; and the object data does not contain private information.

[0013] In the space modeling method, rendering the first replacement image and the second replacement image based on the target three-dimensional material includes: performing three-dimensional pose discrimination on the target object in the first image and the second image respectively to obtain corresponding first pose information and second pose information; and rendering the first replacement image based on the first pose information and the second replacement image based on the second pose information by using the target three-dimensional material.

[0014] In the space modeling method, before modeling the target space based on the third image frame and the fourth image frame to obtain the corresponding three-dimensional model, the method further includes: identifying object feature information of the target object from the first image and the second image; the object feature information includes at least one or more of the shape, size, and color of the target object; and updating the first replacement image and the second replacement image based on the object feature information.

[0015] In the space modeling method, modeling the target space based on the third image frame and the fourth image frame to obtain the corresponding three-dimensional model includes: performing model training on an initial model of the target space by using the third image frame and the fourth image frame to obtain the corresponding three-dimensional model; and the three-dimensional model can output a two-dimensional image corresponding to a target view angle under the condition of inputting any view angle.

[0016] Embodiments of the present application provide a space modeling device, which comprises:

[0017] The acquisition module is configured to obtain a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame includes a first image corresponding to a target object; and the second image frame includes a second image corresponding to the target object.

[0018] The replacement module is configured to replace the first image in the first image frame with a first replacement image to obtain a third image frame, and replace the second image in the second image frame with a second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship.

[0019] The modeling module is configured to model the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

[0020] The embodiment of the present application provides a space modeling device, comprising: a processor, a memory and a communication bus; the communication bus is used for realizing the communication connection between the processor and the memory; the processor is used for executing the computer program stored in the memory, so as to realize the space modeling method.

[0021] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to realize the space modeling method.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the technical solutions provided by the embodiment of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings, wherein:

[0024] Figure 1 A flowchart of a space modeling method provided by the embodiment of the present application is shown in the figure;

[0025] Figure 2 A flowchart of an exemplary rendering replacement image provided by the embodiment of the present application is shown in the figure Figure 1 ;

[0026] Figure 3 A flowchart of an exemplary rendering replacement image provided by the embodiment of the present application is shown in the figure Figure 2 ;

[0027] Figure 4 A flowchart of an exemplary updating replacement image provided by the embodiment of the present application is shown in the figure;

[0028] Figure 5 A flowchart of an exemplary space modeling method provided by the embodiment of the present application is shown in the figure;

[0029] Figure 6 A structure diagram of a space modeling device provided by the embodiment of the present application is shown in the figure;

[0030] Figure 7 A structure diagram of a space modeling device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings.

[0032] As an alternative to Neural Radiance Fields (NeRF), three-dimensional Gaussian splats show impressive reconstruction quality and high training and rendering efficiency. It uses an explicit set of 3D Gaussian volumes to represent the scene, which are rendered into camera views for new view synthesis.

[0033] However, due to the high resolution and photorealism of the scene reconstructed by three-dimensional Gaussian, sensitive information in the scene may be leaked during the reconstruction process, thereby causing privacy problems. With the wide application of three-dimensional Gaussian scene reconstruction technology in various fields, such as virtual reality, augmented reality, autonomous driving, etc., the scene may contain personal identity information, business secrets, sensitive geographic information, etc., which, if not protected, will pose a threat to personal privacy and social security.

[0034] Some existing privacy protection methods have limitations. For example, the privacy information in the picture data used for scene reconstruction is removed or blurred. However, in the process of removing or blurring the privacy information, some important information related to scene reconstruction may be lost, thereby affecting the quality and integrity of the reconstructed scene and reducing the value of the data in actual application.

[0035] The embodiment of the present application provides a space modeling method, which is realized by a space modeling device, as shown in the following steps S101 to S103: Figure 1

[0036] Step S101, obtaining a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame includes a first image corresponding to a target object; the second image frame includes a second image corresponding to the target object.

[0037] In the embodiments of the present application, the space modeling device is an electronic device with space modeling function, which can be a tablet computer, a notebook computer, a palm computer, a personal digital assistant (PDA), a desktop computer, etc., and the specific space modeling device is not limited here.

[0038] ​In the embodiments of the present application, the space modeling device can form an image set by collecting images of the target space through a camera, a camera or other image collection devices. The image set can include multiple image frames arranged in chronological order, which cover different perspectives of the target space. Among them, the first image frame and the second image frame are image frames containing the same target object in the image set.

[0039] Illustratively, the image set is obtained by using a multi-angle camera system that rotates around the target space to capture image frames from multiple perspectives. After image collection is completed, the obtained image frames form an image set.

[0040] In the embodiments of the present application, the first image frame and the second image frame are image frames collected for different perspectives of the target space. Accordingly, the first image and the second image are images of the target object from different perspectives.

[0041] In the embodiments of the present application, the target object can refer to an object containing private information in the image frame. Illustratively, the target object can be a face, a license plate, a slogan, a plaque, etc. Therefore, considering the particularity of the target object, the space modeling device can identify the image corresponding to the target object from the image frame and desensitize the target object contained in the image, and then perform three-dimensional modeling based on the processed image frame. This can improve the quality of space modeling while ensuring the integrity of the image frame.

[0042] Step S102, replacing the first image in the first image frame with a first replacement image to obtain a third image frame, and replacing the second image in the second image frame with a second replacement image to obtain a fourth image frame. There is a corresponding relationship between the first replacement image and the second replacement image.

[0043] In the embodiments of the present application, the first image and the second image are corresponding images of the target object from different perspectives. The space modeling device can replace the first image in the first image frame with a first replacement image to obtain a third image frame, and replace the second image in the second image frame with a second replacement image. Since the first image and the second image are images of the target object from different perspectives, the same content can be used to replace the first image and the second image to ensure the unity of the image frames of the target object from different perspectives.

[0044] Exemplarily, the replacement implementation can include: identifying an image region where the target object is located from the first image frame to obtain a first image, rendering a first replacement image, and replacing the first image with the first replacement image to obtain a third image frame; identifying an image region where the target object is located from the second image frame to obtain a second image, rendering a second replacement image, and replacing the second image with the second replacement image to obtain a fourth image frame. Wherein, the first replacement image and the second replacement image can be rendered by using the target three-dimensional material corresponding to the target object.

[0045] In the embodiments of the present application, the target three-dimensional material is a material with privacy information removed, and therefore the first replacement image and the second replacement image rendered based on the target three-dimensional material are images with privacy information removed, and the third image frame and the fourth image frame obtained are image frames with privacy information removed at different angles of view of the target space.

[0046] In the embodiments of the present application, the first replacement image and the second replacement image have the same geometric properties and semantic consistency in the three-dimensional space. The first replacement image and the second replacement image respectively replace the forms of the target object at different angles of view. This replacement relationship can ensure that the first replacement image and the second replacement image can still maintain visual coherence and structural consistency in the subsequent modeling process.

[0047] In the embodiments of the present application, compared with the way of removing or blurring the privacy information in the related art, the first replacement image and the second replacement image rendered based on the target three-dimensional material can guarantee the integrity of the target object in the replacement image corresponding to the target object, protect privacy while retaining information closely related to the modeling of the target space to the greatest extent, and improve the integrity of the image frame used for subsequent modeling.

[0048] Step S103, modeling the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

[0049] In the embodiments of the present application, the third image frame and the fourth image frame are image frames with privacy information removed, and therefore the three-dimensional image obtained by modeling the space based on the third image frame and the fourth image frame also has no privacy information. In some implementations, the way of space modeling is NeRF or three-dimensional Gaussian splatting.

[0050] In this way, by obtaining an image frame containing a target object, updating the image frame by using a replacement image having a corresponding relationship, and then modeling the space based on the updated image frame, the desensitization processing of privacy information can be realized without damaging the integrity of the scene, thereby effectively protecting sensitive data while ensuring the quality and availability of the modeling result.

[0051] In some embodiments, the space modeling device can further perform the following steps S201 and S202:

[0052] Step S201: searching, from a preset object material library, a target three-dimensional material matching the target object.

[0053] In embodiments of the present application, the preset object material library includes three-dimensional materials corresponding to different objects, and can also include three-dimensional materials corresponding to different object categories. For example, the three-dimensional materials included in the preset object material library can be set based on actual requirements and application scenarios, which are not limited in the present application.

[0054] For example, for a target object of a face category, a standard human head model without facial features can be stored in the preset object material library; for a target object of a license plate category, a blank logo or a standardized rectangular map can be stored in the preset object material library.

[0055] In embodiments of the present application, if the space modeling device performs pixel-level semantic segmentation on the first image frame and the second image frame using a semantic segmentation algorithm to obtain the first image and the second image corresponding to the target object, the target three-dimensional material matching the target object can be searched from the preset object material library based on the target object.

[0056] For example, if the preset object material library includes three-dimensional materials corresponding to different objects, the space modeling device can directly match the corresponding target three-dimensional material based on the target object. If the preset object material library includes three-dimensional materials corresponding to different object types, the object category to which the target object belongs can be determined first, and then the corresponding target three-dimensional material can be matched based on the object category.

[0057] In embodiments of the present application, the three-dimensional material can also be applied to the fields of training data processing of an automatic driving system, virtual reality scene generation, augmented reality content rendering, etc. For example, in the training process of an automatic driving system, to protect the privacy of pedestrians, the automatic driving system can automatically identify and replace all pedestrian images in the training stage, and use a standard human model without facial features to replace them. In this way, the data privacy is protected, and the training effect of the model is not affected.

[0058] Step S202: rendering the first replacement image and the second replacement image based on the target three-dimensional material.

[0059] In the embodiments of the present application, when the target three-dimensional material is obtained, the space modeling device can render the first replacement image and the second replacement image based on the target three-dimensional material. The rendering process refers to projecting the target three-dimensional material to the camera perspective according to the selected target three-dimensional material and the position and attitude information of the target three-dimensional material in the target space, and generating a two-dimensional image matching the original image (the first image and the second image). The first replacement image and the second replacement image correspond to the desensitization images under two different perspectives, respectively.

[0060] For example, the third image frame and the fourth image frame are obtained as follows: the space modeling device can obtain a first perspective of the first image frame to the target space, and then render a first replacement image of the target three-dimensional material under the first perspective, and then replace the first image in the first image frame with the first replacement image to obtain the third image frame; the space modeling device can obtain a second perspective of the second image frame to the target space, and then render a second replacement image of the target three-dimensional material under the second perspective, and then replace the second image in the second image frame with the second replacement image to obtain the fourth image frame.

[0061] For example, in the rendering process, a graphics engine (such as OpenGL, DirectX, UnrealEngine, etc.) can be used to render a three-dimensional material model into a two-dimensional image consistent with the resolution of the original image according to the spatial coordinates and orientation parameters of the target object. The rendering process can include texture mapping, lighting calculation, shadow projection, and other links to ensure that the finally generated two-dimensional image is visually consistent with the original image. For example, when the target object is a face, a head model without facial features can be rendered, and the head model is placed in the position of the face in the original image, so that the head model is consistent with the original image in terms of lighting, angle, background fusion, etc.

[0062] In this way, before replacing the original image, the matching three-dimensional material is searched from the preset three-dimensional material library and the replacement image is generated by rendering, which can ensure that the replacement image is visually consistent with the original image, improve the realism and continuity in the modeling process, and avoid the decline of modeling quality caused by image mutation.

[0063] In some embodiments, the preset object material library includes three-dimensional materials corresponding to each object category in a plurality of object categories; the object category is a category containing privacy information.

[0064] In the embodiments of the present application, the plurality of object categories can be analyzed in the semantic category column [c1, c2, ···, c N ] of the preset two-dimensional semantic segmentation dataset, to determine the category [c 1_pri , c2_pri , ···, c N_pri ]. Wherein, the preset two-dimensional semantic segmentation dataset can be a PASCAL Visual Object Classes (VOC) dataset, or a Microsoft Common Objects in Context (COCO) dataset, of course, it can also be other two-dimensional semantic segmentation datasets, which can be set based on actual needs and application scenarios, and the present application does not limit this.

[0065] In the embodiments of the present application, for each object class c i , the corresponding three-dimensional materials are created by using three-dimensional modeling software or collecting non-private information of the same object data, and these materials are stored in the preset object material library in a specific data format.

[0066] Exemplarily, a face may reveal the identity of an individual, a license plate may expose vehicle information, and a slogan may involve company secrets or politically sensitive content. By including object categories containing private information in the material library management, the system can automatically identify these categories and replace the objects containing private information, thereby preventing the user's private information from being reconstructed into the final three-dimensional model.

[0067] In this way, by constructing a three-dimensional material library for private object categories, different types of private objects can be systematically addressed, the coverage and applicability of privacy protection can be improved, and the generality and scalability of the method can be enhanced.

[0068] In the embodiments of the present application, the preset object material library includes three-dimensional materials corresponding to target object categories; the three-dimensional materials are provided with initial orientations; the initial orientations are aligned with the camera coordinate system of the preset pose algorithm; the target object categories include at least a plurality of object categories; wherein the three-dimensional materials corresponding to each object category are created based on object data of the same category; the object data does not contain private information.

[0069] In the embodiments of the present application, the preset object material library can include other object categories containing private information in addition to the object categories containing private information in the above-mentioned preset two-dimensional semantic segmentation dataset.

[0070] In the embodiments of the present application, the three-dimensional materials in the preset object material library are provided with initial orientations, and the initial orientations are aligned with the camera coordinate system of the preset pose algorithm, so that a uniform initial orientation is set for each object, and the initial orientation of each object is aligned with the camera coordinate system of the pose estimation algorithm for retrieval and rendering.

[0071] In this way, by configuring a unified and standardized three-dimensional material for each type of privacy object and setting the initial orientation to match the camera coordinate system of the pose algorithm, the replacement image can be naturally integrated into the original image during rendering, reducing visual deviation and further ensuring the consistency and stability of the modeling effect.

[0072] In the embodiments of the present application, the content of the preset object material library can be flexibly adjusted according to different privacy protection needs, for example, more types of desensitization materials can be added, the rendering effect of the replacement image can be optimized, multi-language identification object replacement can be supported, and the like. In addition, personalized privacy protection strategies can also be implemented in combination with a user identity recognition mechanism, thereby further enhancing the flexibility and applicability of the preset object material library.

[0073] In some embodiments, when performing the above step S202, the space modeling device can include the following steps S301 and S302, as shown in the following table: Figure 3

[0074] Step S301, performing three-dimensional pose discrimination on the target object in the first image and the second image respectively to obtain corresponding first pose information and second pose information.

[0075] In the embodiments of the present application, the space modeling device performs three-dimensional pose discrimination on the target object in the first image and the second image respectively to obtain the first pose information and the second pose information. The pose information can be the position (coordinates) and attitude (rotation angle) of the target object category in the three-dimensional space determined by the computer vision algorithm, thereby obtaining the complete three-dimensional geometric state of the target object.

[0076] Illustratively, the implementation of three-dimensional pose discrimination: a deep learning model and a geometric estimation method can be combined, the input is a two-dimensional image containing a target object (first image and second image), and the output is a three-dimensional transformation matrix of the target object in the camera coordinate system of the preset pose algorithm. For example, the system identifies a face in a photo and calculates the position and orientation of the face relative to the camera coordinate system of the preset pose algorithm to generate an accurate 3D pose description.

[0077] In the embodiments of the present application, the first pose information and the second pose information respectively represent the three-dimensional pose parameters of the target object under two different viewing angles. The first pose information and the second pose information include a translation vector and a rotation quaternion or Euler angle, and the three-dimensional pose parameters in the first pose information and the second pose information are used for subsequent image synthesis and view rendering.

[0078] Step S302, rendering a first replacement image based on the first pose information and a second replacement image based on the second pose information using the target three-dimensional material.

[0079] ​In the embodiments of the present application, the space modeling device utilizes the target three-dimensional material, renders the first replacement image according to the first position information, and renders the second replacement image according to the second position information, can accurately generate the spatial position and orientation of the target object in the target space, and generate the replacement image consistent with the original image.

[0080] In the embodiments of the present application, according to the pose information of the target object (i.e. the position and direction of the target object in the three-dimensional space), the most suitable replacement model is selected from the preset object material library, and the same orientation and scale as the target object in the original image are generated by using the graphic rendering engine.

[0081] In this way, by accurately judging the three-dimensional pose of the target object under different viewing angles, and rendering the adaptive replacement image according to the pose information, the overall modeling accuracy and visual continuity are improved.

[0082] In some embodiments, the space modeling device, before performing the above step S103, as shown in the following steps S401 and S402: Figure 4

[0083] Step S401, identifying the object feature information of the target object from the first image and the second image; the object feature information at least includes one or more of the shape, size and color of the target object.

[0084] In the embodiments of the present application, the space modeling device can perform image recognition on the first image to identify the object feature information of the target object. The object feature information refers to a data set describing the appearance characteristics of the target object, including but not limited to the shape (such as circular, rectangular or irregular shape), size (such as area, volume or relative size), color (such as RGB value, HSV hue, etc.) of the target object. The object feature information is used for subsequent three-dimensional pose estimation and retrieval and rendering of matching objects in the three-dimensional material library.

[0085] In the embodiments of the present application, considering that the target three-dimensional material corresponding to the target object in the preset object material library may not meet the difference of the existence of the same object category in different target spaces, therefore, by extracting the object feature information of the target object, the first replacement image and the second replacement image updated based on the object feature information can be more consistent with the characteristics of the target object in the target space.

[0086] Step S402, updating the first replacement image and the second replacement image based on the object feature information.

[0087] In the embodiments of the present application, after the space modeling device knows the object feature information, the first replacement image and the second replacement image are updated to obtain the updated first replacement image and the second replacement image. ​

[0088] In the embodiments of the present application, the space modeling device finds and matches a non-sensitive substitute similar to the target object from a pre-constructed three-dimensional material library based on the object feature information of the target object identified in the previous step. The pre-set object material library stores a large number of desensitized three-dimensional models, each of which is provided with a uniform initial orientation, and the model coordinates are aligned with the camera coordinate system to facilitate subsequent rendering operations.

[0089] In the embodiments of the present application, the process of updating the first replacement image and the second replacement image not only guarantees the visual consistency of the replaced image, but also ensures the compatibility of the replaced image with the original scene in terms of geometric structure, so that the replaced image applied to subsequent space modeling can improve the accuracy of the obtained three-dimensional model.

[0090] In the embodiments of the present application, the process of space modeling can include: acquiring an original image by the image acquisition module, then extracting the object feature information of the target object by the target recognition module, and finally outputting a desensitized image that can be used for three-dimensional Gaussian scene reconstruction by calling the three-dimensional material library for matching and rendering by the replacement image generation module. The entire process can be integrated into an image processing pipeline, with close connection between each step, ensuring efficient operation of the system and stability of the results. In this way, the automatic identification and intelligent replacement of sensitive target objects are completed without affecting the overall quality of the image.

[0091] In this way, by extracting the key features of the target object and adjusting the replacement image accordingly, the appearance of the object in the original image can be more accurately simulated, making the replacement image more visually similar to the actual scene, thereby further improving the realism and accuracy of modeling.

[0092] In some embodiments, when performing the above step S103, the space modeling device can include the following steps: using the third image frame and the fourth image frame to perform model training on the initial model of the target space to obtain a corresponding three-dimensional model; the three-dimensional model can output a two-dimensional image corresponding to a target view under the condition of inputting any view angle.

[0093] In the embodiments of the present application, the model training of the three-dimensional model refers to the process of parameter optimization of the initial three-dimensional model by using multi-view image data. Illustratively, the model training includes inputting the third image frame and the fourth image frame as inputs, and the space modeling device inputs these desensitized image frames into a neural network or a three-dimensional reconstruction algorithm for updating and adjusting the structural parameters of the three-dimensional model to more accurately fit the real geometric shape and texture information of the target space. The purpose of model training is to improve the precision, stability and generalization ability of the three-dimensional model, so that the three-dimensional model has stronger view synthesis capability.

[0094] In the embodiments of the present application, the three-dimensional model refers to a digital representation generated by training in this context. The three-dimensional model not only contains the geometric structure of the scene (such as point cloud, mesh, etc.), but also contains surface texture, lighting properties and other information. The three-dimensional model has strong view synthesis capability, that is, no matter from which view angle the user observes the scene, a high-quality two-dimensional image can be rendered in real time, which can realize virtual roaming, interactive display and other functions.

[0095] In the embodiments of the present application, the initial model can be a three-dimensional Gaussian splatting (3D Gaussians Splatting, 3DGS). As an efficient spatial modeling method, 3DGS uses an explicit set of 3D Gaussian bodies to represent the scene and generates new view images through rendering, with high reconstruction quality and computational efficiency. This method is usually trained and modeled based on the original image set, and can generate realistic three-dimensional scene models. Of course, the initial model can also be a 2D Gaussian splatting (2DGS), a Gaussian opaque field (GOF), a multi-view stereo vision (MVS) or other models.

[0096] In the embodiments of the present application, in the case of inputting any view angle, the two-dimensional image corresponding to the target view angle is output, indicating that the three-dimensional model has high universality and adaptability, and can support image output of any angle, distance and field of view. In this way, the obtained three-dimensional model can be widely used in virtual reality, augmented reality, autonomous driving and other fields, thereby providing users with immersive visual experience and accurate spatial perception.

[0097] As shown in Figure 5 , a flowchart of an exemplary spatial modeling method is provided, including the following steps S501 to S5016:

[0098] Step S501, class list of semantic segmentation dataset.

[0099] Here, the semantic class list of the two-dimensional semantic segmentation dataset (corresponding to the preset two-dimensional semantic segmentation dataset discussed above) is obtained.

[0100] Step S502, enumerate object classes that may contain privacy information.

[0101] Here, each semantic class in the semantic class list is analyzed to analyze the object class that may contain privacy information, and the object class containing privacy information is obtained.

[0102] Step S503, whether all classes are enumerated.

[0103] Here, it is judged whether all object classes containing privacy information are enumerated, if yes, step S505 is executed, if not, step S504 is executed.

[0104] Step S504, obtaining the three-dimensional model of the object category without privacy information.

[0105] Here, for the currently enumerated object category, corresponding three-dimensional materials are created using three-dimensional modeling software or super-privacy-free information of the same object, which are stored in the three-dimensional de-sensitization object database (corresponding to the preset object material library discussed above) in a specific data format, and a uniform initial orientation is set for each object (corresponding to the three-dimensional material of each object category), so that the initial orientation of each object is aligned with the camera coordinate system of the pose estimation algorithm, so as to facilitate subsequent retrieval and rendering.

[0106] Step S505, three-dimensional de-sensitization object database.

[0107] Here, for each object category enumerated, a corresponding three-dimensional material is generated and stored in the three-dimensional de-sensitization object database.

[0108] Step S506, input scene scanning image.

[0109] Here, the target space is scanned to obtain the scene scanning image (corresponding to the image set discussed above).

[0110] Step S507, whether all frames are processed.

[0111] Here, the space modeling device determines whether all image frames in the scene scanning image are processed, if yes, step S515 is executed, if not, step S508 is executed.

[0112] Step S508, take an unprocessed frame for semantic segmentation to obtain a target that may contain privacy information.

[0113] Here, the space modeling device can obtain the image frames included in the scene scanning image frame by frame, and then apply the semantic segmentation algorithm to the unprocessed image frames [img1, img2, img N_frame ] to obtain the pixel-level semantic segmentation result i of each image img , wherein, represents the position of the object in the picture, represents the category to which it belongs, and the target that may contain privacy information (corresponding to the image (first image or second image) of the target object in the image frame (first image frame or second image frame) discussed above) is obtained.

[0114] Step S509, select all targets that need to be protected from privacy.

[0115] Here, the space modeling device selects all the targets that need to be protected from the acquired target that may contain privacy information through pre-set rules or further classification algorithms (corresponding to the target object in the image frame (first image frame or second image frame) discussed above).

[0116] Step S5010, whether all targets are processed.

[0117] Here, the space modeling device determines whether all targets in the current frame are processed, if yes, step S507 is executed, if not, step S5011 is executed.

[0118] Step S5011, taking an unprocessed target, calculating its three-dimensional pose.

[0119] Here, the space modeling device uses a pose estimation algorithm to determine the three-dimensional pose of the target that needs to be protected PNP, to obtain its pose information in the three-dimensional space coordinate system of the camera

[0120] Step S5012, finding the three-dimensional material corresponding to the semantic category of the target.

[0121] Here, the space modeling device can find the object category of the target from the three-dimensional desensitization object material library obtained in step S505, and then obtain the corresponding three-dimensional material based on the object category, i.e. for each target find the corresponding category c in the three-dimensional desensitization object database, select the three-dimensional material corresponding to c n . n

[0122] Step S5013, rendering a three-dimensional material map corresponding to the pose and size.

[0123] Here, the space modeling device uses a graphics rendering engine to render a three-dimensional material map that conforms to the orientation and size of the object in the image based on the three-dimensional material obtained in step S5012 and the three-dimensional pose of step S5011 (corresponding to the first replacement image or the second replacement image discussed above).

[0124] Step S5014, paste the map back to the corresponding position of the original image.

[0125] Here, the space modeling device accurately pastes the three-dimensional material map rendered in step S5013 back to the corresponding position of the original image through image synthesis technology (corresponding to the third image frame or the fourth image frame discussed above), and then executes step S5010.

[0126] ​Step S5015, performing three-dimensional Gaussian scene reconstruction using the processed picture.

[0127] Here, the spatial modeling device obtains the processed picture after processing all frames and all targets in the processed scene scanning image, and inputs the processed picture (corresponding to the third image frame and the fourth image frame discussed above) into a general three-dimensional Gaussian scene reconstruction algorithm (corresponding to the initial model corresponding to the target space discussed above) to perform three-dimensional Gaussian scene reconstruction.

[0128] Step S5016, three-dimensional Gaussian scene without privacy information.

[0129] Here, the spatial modeling device performs three-dimensional Gaussian scene reconstruction based on the processed picture to obtain a three-dimensional Gaussian scene without privacy information (corresponding to the three-dimensional model discussed above).

[0130] In the embodiments of the present application, before the privacy information is desensitized, a three-dimensional material library is first constructed: the categories of various objects that may contain privacy information are determined, and a three-dimensional material without privacy information is selected for each category of objects. For the input scene scanning image, frame-by-frame pixel-level target recognition is performed to accurately determine the targets in the picture that may involve privacy information, such as human faces, slogans, plaques, etc., and to select the targets that really need privacy protection. Then, the three-dimensional pose of the target that needs privacy protection is determined. Subsequently, according to the identified target and its pose, a specific orientation and size of the map that matches the target is searched and rendered in the three-dimensional material library, the rendered map is accurately pasted back to the corresponding position of the original picture, and the processed picture is input into the subsequent training process of three-dimensional Gaussian scene reconstruction, thereby obtaining a desensitized three-dimensional Gaussian scene. In this way, while protecting privacy, information closely related to scene reconstruction is maximally preserved, the quality, integrity and data availability of the reconstructed scene are guaranteed, and the three-dimensional Gaussian scene reconstruction technology can be more safely, efficiently and high-quality applied in privacy-sensitive fields.

[0131] The embodiments of the present application provide an exemplary spatial modeling method, and the implementation manner can include steps S11 to S14:

[0132] S11, obtaining a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame includes a first image corresponding to a target object; and the second image frame includes a second image corresponding to the target object.

[0133] S12, find a target three-dimensional material matching the target object from a preset object material library; render a first replacement image and a second replacement image based on the target three-dimensional material; wherein the preset object material library includes three-dimensional materials corresponding to each object category in a plurality of object categories; the object category is a category containing private information.

[0134] S13, replace the first image in the first image frame with the first replacement image to obtain a third image frame, and replace the second image in the second image frame with the second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship.

[0135] S14, model training is performed on an initial model of the target space using the third image frame and the fourth image frame to obtain a corresponding three-dimensional model; the three-dimensional model can output a two-dimensional image corresponding to a target view angle under the condition of inputting any view angle.

[0136] Here, step S11 corresponds to the aforementioned step S101, and in implementation, the implementation manner of the aforementioned step S101 can be referred to; step S12 corresponds to the aforementioned steps S201 and S202, and in implementation, the implementation manner of the aforementioned steps S201 and S202 can be referred to; step S13 corresponds to the aforementioned step S102, and in implementation, the implementation manner of the aforementioned step S102 can be referred to; step S14 corresponds to the aforementioned step: model training is performed on an initial model of the target space using the third image frame and the fourth image frame to obtain a corresponding three-dimensional model; the three-dimensional model can output a two-dimensional image corresponding to a target view angle under the condition of inputting any view angle, and in implementation, the implementation manner of the aforementioned step can be referred to.

[0137] Another exemplary space modeling method is provided in the embodiments of the present application, and the implementation manner can include steps S21 to S24.

[0138] S21, obtain a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame includes a first image corresponding to a target object; and the second image frame includes a second image corresponding to the target object.

[0139] S22, find a target three-dimensional material matching the target object from a preset object material library; respectively perform three-dimensional pose discrimination on the target object in the first image and the second image to obtain corresponding first pose information and second pose information; render a first replacement image based on the first pose information and a second replacement image based on the second pose information by using the target three-dimensional material; wherein the preset object material library includes three-dimensional materials corresponding to the target object category; the three-dimensional material is provided with an initial orientation; the initial orientation is aligned with the camera coordinate system of the preset pose algorithm; the target object category includes at least a plurality of object categories; wherein the three-dimensional material corresponding to each object category is created based on the same category object data; the object data does not contain privacy information.

[0140] S23, identify object feature information of the target object from the first image and the second image; the object feature information includes at least one or more of the shape, size, and color of the target object; update the first replacement image and the second replacement image based on the object feature information

[0141] S24, replace the first image in the first image frame with the first replacement image to obtain a third image frame, and replace the second image in the second image frame with the second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship.

[0142] S25, model the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model; the three-dimensional model can output a two-dimensional image corresponding to a target view angle under the condition of inputting any view angle.

[0143] Here, step S21 corresponds to the aforementioned step S101, and in implementation, the implementation manner of the aforementioned step S101 can be referred to; step S22 corresponds to the aforementioned steps S201, S301 and S302, and in implementation, the implementation manner of the aforementioned steps S201, S301 and S302 can be referred to; step S23 corresponds to the aforementioned steps S401 and S402, and in implementation, the implementation manner of the aforementioned steps S401 and S402 can be referred to; step S24 corresponds to the aforementioned step S102, and in implementation, the implementation manner of the aforementioned step S102 can be referred to; step S25 corresponds to the aforementioned step S103, and in implementation, the implementation manner of the aforementioned step S103 can be referred to.

[0144] The application embodiment provides a space modeling method, which comprises the following steps: obtaining a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame comprises a first image corresponding to a target object; the second image frame comprises a second image corresponding to the target object; replacing the first image in the first image frame with a first replacement image to obtain a third image frame, and replacing the second image in the second image frame with a second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship; and modeling the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model. The space modeling method provided by the application can realize the desensitization of private information without damaging the integrity of a scene, thereby effectively protecting sensitive data and ensuring the quality and availability of the modeling result.

[0145] The application embodiment provides a space modeling device 6, as shown in the figure, which comprises the following parts: Figure 6

[0146] The obtaining module 61 is configured to obtain a first image frame and a second image frame; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame comprises a first image corresponding to a target object; the second image frame comprises a second image corresponding to the target object.

[0147] The replacing module 62 is configured to replace the first image in the first image frame with a first replacement image to obtain a third image frame, and replace the second image in the second image frame with a second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship.

[0148] The modeling module 63 is configured to model the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

[0149] In an embodiment of the application, the replacing module 62 is further configured to find a target three-dimensional material matching the target object from a preset object material library; and render the first replacement image and the second replacement image based on the target three-dimensional material.

[0150] In an embodiment of the application, the preset object material library comprises three-dimensional materials corresponding to each object category in a plurality of object categories; the object category is a category containing private information.

[0151] ​In an embodiment of the present application, the preset object material library includes three-dimensional materials corresponding to a target object category; the three-dimensional materials are provided with an initial orientation; the initial orientation is aligned with a camera coordinate system of the preset pose algorithm; the target object category includes at least a plurality of object categories; the three-dimensional material corresponding to each object category is created based on object data of the same category; and the object data does not contain private information.

[0152] In an embodiment of the present application, the replacement module 62 is further configured to perform three-dimensional pose discrimination on the target object in the first image and the second image respectively, to obtain corresponding first pose information and second pose information; and render a first replacement image based on the first pose information and a second replacement image based on the second pose information by using the target three-dimensional material.

[0153] In an embodiment of the present application, the replacement module 62 is further configured to identify object feature information of the target object from the first image and the second image; the object feature information includes at least one or more of shape, size, and color of the target object; and update the first replacement image and the second replacement image based on the object feature information.

[0154] In an embodiment of the present application, the modeling module 63 is further configured to perform model training on an initial model of the target space by using the third image frame and the fourth image frame, to obtain a corresponding three-dimensional model; and the three-dimensional model can output a two-dimensional image corresponding to a target view angle under the condition of inputting any view angle.

[0155] An embodiment of the present application provides a space modeling device, as shown in the accompanying drawings, the space modeling device includes a processor 71, a memory 72, and a communication bus 73. Figure 7

[0156] The communication bus 73 is configured to realize communication connection between the processor 71 and the memory 72.

[0157] The processor 71 is configured to execute a computer program stored in the memory 72, to realize the above-mentioned space modeling method.

[0158] ​The embodiment of the present application provides a space modeling device, a first image frame and a second image frame are obtained; the first image frame and the second image frame are image frames in an image set; the image set is obtained by image acquisition on a target space; the first image frame comprises a first image corresponding to a target object; the second image frame comprises a second image corresponding to the target object; the first image in the first image frame is replaced by a first replacement image to obtain a third image frame, and the second image in the second image frame is replaced by a second replacement image to obtain a fourth image frame; the first replacement image and the second replacement image have a corresponding relationship; the target space is modeled based on the third image frame and the fourth image frame, and a corresponding three-dimensional model is obtained. The space modeling device provided by the present application obtains the image frame containing the target object, updates the image frame by using the replacement image having the corresponding relationship, and then models the space based on the updated image frame, so that the desensitization processing of the private information can be realized without damaging the integrity of the scene, thereby effectively protecting the sensitive data and ensuring the quality and availability of the modeling result.

[0159] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the space modeling method. The computer readable storage medium can be a volatile memory (volatile memory), for example, a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), for example, a read-only memory (Read-Only Memory, ROM), a flash memory, a hard disk (Hard Disk Drive, HDD) or a solid state disk (Solid-State Drive, SSD); and can also be a respective device including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0160] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer usable program codes.

[0161] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0162] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0163] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0164] The above description is only a specific implementation of the present application. The protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A spatial modeling method, comprising: Obtain a first image frame and a second image frame; The first image frame and the second image frame are image frames in an image set; The image set is obtained by collecting images of the target space; The first image frame includes a first image corresponding to the target object; the second image frame includes a second image corresponding to the target object; replacing the first image in the first image frame with the first replacement image to obtain a third image frame, and replacing the second image in the second image frame with the second replacement image to obtain a fourth image frame; there is a corresponding relationship between the first replacement image and the second replacement image; The target space is modeled based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

2. The spatial modeling method according to claim 1, further comprising: Searching for a target three-dimensional material that matches the target object from a preset object material library; The first replacement image and the second replacement image are rendered based on the target three-dimensional material.

3. The spatial modeling method according to claim 2, wherein the preset object material library includes three-dimensional materials corresponding to each object category in a plurality of object categories; the object category is a category containing private information.

4. The spatial modeling method according to claim 3, wherein the preset object material library includes three-dimensional materials corresponding to the target object category; The three-dimensional material is provided with an initial orientation; The initial orientation is aligned with the camera coordinate system of the preset pose algorithm; The target object category includes at least the multiple object categories; wherein the three-dimensional material corresponding to each object category is created based on object data of the same category; and the object data does not contain privacy information.

5. The spatial modeling method according to claim 2, wherein rendering the first replacement image and the second replacement image based on the target three-dimensional material comprises: Performing three-dimensional posture discrimination on the target object in the first image and the second image respectively to obtain corresponding first posture information and second posture information; The first replacement image is rendered based on the first pose information using the target three-dimensional material, and the second replacement image is rendered based on the second pose information.

6. The spatial modeling method according to any one of claims 1 to 5, wherein before modeling the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model, the method further comprises: identifying object feature information of the target object from the first image and the second image; The object feature information includes at least one or more of the shape, size, and color of the target object; Based on the object feature information, the first replacement image and the second replacement image are updated.

7. The spatial modeling method according to claim 1, wherein modeling the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model comprises: Using the third image frame and the fourth image frame, performing model training on the initial model of the target space to obtain the corresponding three-dimensional model; The three-dimensional model can output a two-dimensional image corresponding to a target perspective when any perspective is input.

8. A spatial modeling device comprising: An acquisition module, configured to obtain a first image frame and a second image frame; The first image frame and the second image frame are image frames in an image set; The image set is obtained by collecting images of the target space; The first image frame includes a first image corresponding to the target object; the second image frame includes a second image corresponding to the target object; a replacement module, configured to replace the first image in the first image frame with the first replacement image to obtain a third image frame, and replace the second image in the second image frame with the second replacement image to obtain a fourth image frame; there being a corresponding relationship between the first replacement image and the second replacement image; A modeling module is used to model the target space based on the third image frame and the fourth image frame to obtain a corresponding three-dimensional model.

9. A spatial modeling device comprising: processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the spatial modeling method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more computer programs, wherein the one or more computer programs can be executed by one or more processors to implement the spatial modeling method according to any one of claims 1 to 7.

Images