3D vision processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a 3D vision processing method and device, electronic equipment and a storage medium, and relates to the technical field of 3D vision, and the method comprises the steps: carrying out the sampling of an original 3D vision scene, and obtaining a corresponding original image set; acquiring a language instruction which is input by a user and aims at the original 3D visual scene; editing the original image set according to the language instruction to obtain a target image edited according to the language instruction, and updating the original image set according to the target image to obtain a target image set edited according to the language instruction; and performing 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, thereby reducing the editing complexity, improving the editing flexibility, ensuring the realization of visual originality, and ensuring the uniformity of contents.

Description

Technical Field

[0001] The present invention relates to the field of 3D vision technology, and in particular to a 3D vision processing method, a 3D vision processing device, an electronic device and a computer-readable storage medium. Background Art

[0002] The purpose of 3D visual editing is to create and adjust complex three-dimensional virtual scenes and models to construct realistic 3D visual content. It is widely used in industrial manufacturing, simulation modeling, game animation, movie special effects and other fields. Among them, 3D visual editing relies on strong professional and design capabilities, so the task is very challenging. Especially when the complexity of the task to be handled is very high, it is necessary to adjust various details such as lighting and materials in the 3D scene, resulting in high human, economic and time costs for editing. In addition, during the editing process, there are also problems such as high complexity of the editing process, poor editing flexibility, inability to achieve visual creativity and inconsistent content. Summary of the invention

[0003] The embodiments of the present invention provide a 3D vision processing method, device, electronic device and computer-readable storage medium to solve or partially solve the problems of high editing complexity, poor editing flexibility, inability to realize visual creativity and inconsistent content in the 3D vision editing process.

[0004] The embodiment of the present invention discloses a 3D vision processing method, comprising:

[0005] Sampling the original 3D visual scene to obtain a corresponding original image set;

[0006] Acquire a language instruction for the original 3D visual scene input by a user, where the language instruction is text information containing a visual editing intention for the original 3D visual scene;

[0007] Editing the original image set according to the language instruction to obtain a target image edited according to the language instruction, and updating the original image set according to the target image to obtain a target image set edited according to the language instruction;

[0008] The target image set is subjected to 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene, wherein the target 3D visual scene is a scene obtained by editing the original 3D visual scene according to the language instruction.

[0009] In some feasible implementations, sampling the original 3D visual scene to obtain a corresponding original image set includes:

[0010] Acquire acquisition parameters for the original 3D visual scene, where the acquisition parameters include at least a camera position and a camera orientation;

[0011] The original 3D visual scene is sampled according to the camera position and the camera orientation to obtain a corresponding original image set.

[0012] In some feasible implementations, the original image set includes a plurality of original images, where the original images are images acquired from the original 3D visual scene at a specific viewpoint, and the editing of the original image set according to the language instruction to obtain a target image edited according to the language instruction includes:

[0013] Loading a conditional diffusion model for the original image set;

[0014] Acquiring noise information for the original image set;

[0015] The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model to edit the original image, so as to obtain a target image edited according to the language instruction.

[0016] In some feasible implementations, the inputting the original image, the language instruction corresponding to the original image, and the noise information into the conditional diffusion model to edit the original image to obtain a target image edited according to the language instruction includes:

[0017] The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model, the original image is forward diffused according to the noise information to obtain the corresponding noise image, and then the noise image is back-diffused according to the voice instruction to obtain the target image edited according to the language instruction.

[0018] In some feasible implementations, updating the original image set according to the target image to obtain the target image set edited according to the language instruction includes:

[0019] Acquire a first camera parameter corresponding to each original image in the original image set;

[0020] Acquire a second camera parameter corresponding to each target image in the target image set;

[0021] The original images having the same first camera parameters and second camera parameters are replaced with corresponding target images, and based on the replacement result, a target image set edited according to the language instruction is obtained.

[0022] In some feasible implementations, the first camera parameters include at least a first camera position and a first camera orientation, the second camera parameters include at least a second camera position and a second camera orientation, and replacing the original image having the same first camera parameters and second camera parameters with the corresponding target image, and obtaining the target image set edited according to the language instruction based on the replacement result, comprises:

[0023] Determine a corresponding first viewpoint using the first camera position and the first camera orientation;

[0024] Determine a corresponding second viewpoint using the second camera position and the second camera orientation;

[0025] The first viewpoint is matched with the second viewpoint, and an original image at the same viewpoint is replaced with a target image, and based on a replacement result, a target image set corresponding to the original 3D visual scene is obtained.

[0026] In some feasible implementations, performing 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene includes:

[0027] Extracting feature points from the target image;

[0028] Matching feature points of the target images to determine the corresponding relationship between the target images;

[0029] Predicting the camera pose corresponding to each of the target images using the correspondence between the target images;

[0030] The corresponding relationship and the camera posture are used to perform 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene.

[0031] The embodiment of the present invention further discloses a 3D vision processing device, including:

[0032] An image acquisition module is used to sample the original 3D visual scene to obtain a corresponding original image set;

[0033] An instruction acquisition module, used to acquire a language instruction input by a user for the original 3D visual scene, wherein the language instruction is text information containing a visual editing intention for the original 3D visual scene;

[0034] An image editing module, configured to edit the original image set according to the language instruction to obtain a target image edited according to the language instruction, and to update the original image set according to the target image to obtain a target image set edited according to the language instruction;

[0035] A scene reconstruction module is used to perform 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, wherein the target 3D visual scene is a scene after the original 3D visual scene is edited according to the language instruction.

[0036] In some feasible implementations, the image acquisition module is specifically used to:

[0037] Acquire acquisition parameters for the original 3D visual scene, where the acquisition parameters include at least a camera position and a camera orientation;

[0038] The original 3D visual scene is sampled according to the camera position and the camera orientation to obtain a corresponding original image set.

[0039] In some feasible implementations, the original image set includes a plurality of original images, where the original images are images acquired from the original 3D visual scene at a specific viewpoint, and the image editing module is specifically used for:

[0040] Loading a conditional diffusion model for the original image set;

[0041] Acquiring noise information for the original image set;

[0042] The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model to edit the original image, so as to obtain a target image edited according to the language instruction.

[0043] In some feasible implementations, the image editing module is specifically used to:

[0044] The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model, the original image is forward diffused according to the noise information to obtain the corresponding noise image, and then the noise image is back-diffused according to the voice instruction to obtain the target image edited according to the language instruction.

[0045] In some feasible implementations, the image editing module is specifically used to:

[0046] Acquire a first camera parameter corresponding to each original image in the original image set;

[0047] Acquire a second camera parameter corresponding to each target image in the target image set;

[0048] The original images having the same first camera parameters and second camera parameters are replaced with corresponding target images, and based on the replacement result, a target image set edited according to the language instruction is obtained.

[0049] In some feasible implementations, the first camera parameters include at least a first camera position and a first camera orientation, the second camera parameters include at least a second camera position and a second camera orientation, and the image editing module is specifically configured to:

[0050] Determine a corresponding first viewpoint using the first camera position and the first camera orientation;

[0051] Determine a corresponding second viewpoint using the second camera position and the second camera orientation;

[0052] The first viewpoint is matched with the second viewpoint, and an original image at the same viewpoint is replaced with a target image, and based on a replacement result, a target image set corresponding to the original 3D visual scene is obtained.

[0053] In some feasible implementations, the scene reconstruction module is specifically used to:

[0054] Extracting feature points from the target image;

[0055] Matching feature points of the target images to determine the corresponding relationship between the target images;

[0056] Predicting the camera pose corresponding to each of the target images using the correspondence between the target images;

[0057] The corresponding relationship and the camera posture are used to perform 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene.

[0058] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0059] The memory is used to store computer programs;

[0060] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0061] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0062] The embodiments of the present invention include the following advantages:

[0063] In an embodiment of the present invention, an original 3D visual scene is sampled to obtain a corresponding original image set, and a language instruction for the original 3D visual scene input by a user is obtained, where the language instruction is text information containing a visual editing intention for the original 3D visual scene. Then, the original image set is edited according to the language instruction to obtain a target image edited according to the language instruction, and the original image set is updated according to the target image to obtain a target image set edited according to the language instruction. Then, 3D visual reconstruction is performed on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, where the target 3D visual scene is edited according to the language instruction. The scene after editing the original 3D visual scene, by acquiring images of the 3D visual scene to obtain the corresponding original image set, and then editing the original images in combination with language instructions, can obtain a target image set edited according to the language instructions, so that the 3D visual scene constructed based on the target image set can be more coordinated and reasonable. At the same time, in the editing process, the editing method based on image + language instructions supports editing of the overall scene, local content, etc., which can effectively achieve the unification between local content and overall content, and improves the flexibility of editing, reduces the dependence on user expertise, and can meet the editing of different 3D visuals. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flowchart of a 3D vision processing method provided in an embodiment of the present invention;

[0065] Figure 2 is a processing flow chart of 3D visual editing provided in an embodiment of the present invention;

[0066] Figure 3 Schematic diagram of a 3D visual editing scene provided in an embodiment of the present invention

[0067] Figure 4 is a schematic diagram of the structure of an automatic editing system provided in an embodiment of the present invention;

[0068] Figure 5 is a structural block diagram of a 3D vision processing device provided in an embodiment of the present invention;

[0069] Figure 6 It is a block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] As an example, for the automatic editing process of 3D vision, in the related art, the 3D vision automatic editing methods mainly include methods based on physical simulation and generative methods based on short text prompts. Among them, the method based on physical simulation applies a physical-based inductive bias in its optimization process to achieve changes in materials or scene lighting. This method mainly changes the physical properties of 3D vision or performs physical simulation, which is not conducive to achieving visual creativity. The current generative method based on short text prompts can only convert short text into 3D visual parameters and then edit the overall scene, and it is impossible to add local editing to unify local content with the overall.

[0072] In this regard, in the present invention, by sampling the original 3D visual scene, a corresponding original image set is obtained, and a language instruction for the original 3D visual scene input by the user is obtained, the language instruction is text information containing the visual editing intention for the original 3D visual scene, and then the original image set is edited according to the language instruction to obtain a target image edited according to the language instruction, and the original image set is updated according to the target image to obtain a target image set edited according to the language instruction, and then 3D visual reconstruction is performed on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, and the target 3D visual scene is edited according to the language instruction. The scene after editing the original 3D visual scene, by acquiring images of the 3D visual scene to obtain the corresponding original image set, and then editing the original images in combination with language instructions, can obtain a target image set edited according to the language instructions, so that the 3D visual scene constructed based on the target image set can be more coordinated and reasonable. At the same time, in the editing process, the editing method based on image + language instructions supports editing of the overall scene, local content, etc., which can effectively achieve the unification between local content and overall content, and improves the flexibility of editing, reduces the dependence on user expertise, and can meet the editing of different 3D visuals.

[0073] Reference Figure 1 , shows a flowchart of a 3D vision processing method provided in an embodiment of the present invention, which may specifically include the following steps:

[0074] Step 101, sampling the original 3D visual scene to obtain a corresponding original image set;

[0075] In the process of editing 3D vision, the original 3D vision scene to be edited can be determined first, and the original 3D vision scene can be sampled to obtain a corresponding original image set. The original image set can include original images obtained by sampling the original 3D vision scene from different viewpoints, and the viewpoint can be determined by the corresponding camera position and camera orientation.

[0076] In some feasible implementations, after the original 3D visual scene is determined, acquisition parameters for the original 3D visual scene can be obtained, and the acquisition parameters include at least a camera position and a camera orientation. The original 3D visual scene is then sampled according to the camera position and the camera orientation to obtain a corresponding original image set, so that the 3D visual scene can be edited based on the original image set.

[0077] In one example, assuming that the original 3D visual scene is an indoor room with tables, chairs, windows, and some decorations, in order to obtain an image of this scene, the position and orientation of the camera can be determined first, and then sampling can be performed according to these parameters to obtain a series of original images. For example:

[0078] 1. Original 3D visual scene description:

[0079] Scene: An interior room

[0080] Main objects: tables, chairs, windows, decorations

[0081] Scene size: 5 meters (length) x 4 meters (width) x 3 meters (height)

[0082] 2. Determine the acquisition parameters:

[0083] Camera Position: Suppose we choose to place cameras at different locations in the room to obtain images from different perspectives.

[0084] Position 1: (2,2,1.5) meters (center of the room, slightly above the ground)

[0085] Position 2: (0,4,1.5) meters (upper left corner of the room, slightly above the ground)

[0086] Position 3: (5,0,1.5) meters (lower right corner of the room, slightly above the ground)

[0087] Camera orientation: Suppose we want the camera to face different directions in the room.

[0088] Facing 1: (0,0,-1) (towards the center of the room)

[0089] Facing 2: (-1,0,0) (towards the left wall of the room)

[0090] Facing 3: (1,0,0) (towards the right wall of the room)

[0091] 3. Sampling according to camera position and orientation:

[0092] Position 1 + Orientation 1: The camera is located at (2,2,1.5) meters and is facing the center of the room. The sampled image will show the table, chairs, and some decorations in the center of the room.

[0093] Position 1 + Orientation 2: The camera is at (2,2,1.5) meters, facing the left wall of the room. The sampled image will show the left wall, windows, and some decorations.

[0094] Position 2 + Orientation 1: The camera is at (0,4,1.5) meters, facing the center of the room. The sampled image will show the view of the upper left corner of the room, including part of the table, chairs, and decorations.

[0095] Position 3 + Face 3: The camera is at (5,0,1.5) meters, facing the right wall of the room. The sampled image will show the view of the lower right corner of the room, including part of the table, chairs, and decorations.

[0096] 4. Get the original image set:

[0097] Through the above steps, we obtain images from multiple perspectives, which together constitute the original image set of the original 3D visual scene.

[0098] The image set includes:

[0099] Image from the center of the room toward the center

[0100] Image from the center of the room toward the left wall

[0101] Image from the upper left corner of the room towards the center

[0102] Image from the bottom right corner of the room towards the right wall

[0103] Step 102, obtaining a language instruction for the original 3D visual scene input by a user, wherein the language instruction is text information containing a visual editing intention for the original 3D visual scene;

[0104] After completing the image acquisition of the original 3D visual scene, the language instruction for the original 3D visual scene can be further determined. Specifically, the language instruction for the original 3D visual scene input by the user can be obtained by responding to the user's input operation. The language instruction can be text information containing the visual editing intention for the original 3D visual scene.

[0105] For example, assuming that the 3D visual scene is an indoor scene, the language instructions may be "move the table to the upper right corner of the room", "change the color of the chair to red", "add a desk lamp on the table", "remove the decorations in the upper left corner of the room", "adjust the size of the window to 2 meters wide and 1.5 meters high", and "rotate the chair 90 degrees", etc., so that the original image is processed by inputting the corresponding language instructions to achieve the editing of the 3D visual scene, and the present invention is not limited to this.

[0106] Step 103, editing the original image set according to the language instruction to obtain a target image edited according to the language instruction, and updating the original image set according to the target image to obtain a target image set edited according to the language instruction;

[0107] After determining the language instructions and the original image set, each original image in the original image set can be edited according to the language instructions to obtain a target image edited according to the language instructions, and then the original image set is updated based on the target image to obtain a target image set corresponding to the original 3D visual scene, so as to perform 3D visual reconstruction based on the target image set to obtain a new 3D visual scene.

[0108] In some feasible implementations, the original image is an image captured from a specific viewpoint of the original 3D visual scene. In this case, the conditional diffusion model for the original image set can be loaded first, and the noise information for the original image set can be obtained. Then, the original image, the language instruction corresponding to the original image, and the noise information can be input into the conditional diffusion model to edit the original image, and the target image edited according to the language instruction can be obtained. In a specific implementation, the original image, the language instruction corresponding to the original image, and the noise information can be input into the conditional diffusion model, and the original image can be forward diffused according to the noise information to obtain the corresponding noise image. Then, the noise image can be back-diffused according to the voice instruction to obtain the target image edited according to the language instruction.

[0109] It should be noted that the conditional diffusion model refers to a multimodal model that can edit image content according to language instructions. In the process of editing the original image set according to the conditional diffusion model, the input of the image editing may include an input conditional image c I , a text instruction c T , and the noise input z t When editing the original image corresponding to the target viewpoint v in the original 3D visual scene, c I Use unedited images is usually the image originally captured at this viewpoint. For a noisy input z t , in optimization step i, the currently rendered noise version is input, that is, N(0,1) and Correspondingly, replace the image The process is represented as The noise level t can be a constant range [t min ,t max ] randomly selected noise information. In addition, for U θ, which can be defined as a DDIM (Denoising Diffusion Implicit Models) sampling process, taking a fixed number of intermediate steps s between the initial time step t and 0.

[0110] It should be noted that DDIM is a generative model, which is particularly suitable for image generation tasks. DDIM is based on the diffusion process and generates high-quality images by gradually denoising. The core idea of ​​DDIM is to gradually denoise a random noise image through a series of diffusion steps, and finally generate a clear image.

[0111] The following are the basic principles and steps of DDIM:

[0112] 1. Diffusion process

[0113] Initial Noise Image: Start with a random noise image, usually a Gaussian noise image.

[0114] Diffusion step: Through a series of diffusion steps, noise is gradually added to the image, and finally a completely noisy image is obtained.

[0115] 2. Reverse process (denoising process)

[0116] Reverse diffusion step: Starting from a completely noisy image, the noise is gradually removed to eventually generate a clear image.

[0117] Denoising model: Use a denoising model (usually a neural network) to predict the denoising operation at each step.

[0118] 3. Training process

[0119] Dataset: Use a dataset containing a large number of images for training.

[0120] Loss function: During training, a loss function is used to measure the difference between the generated images and the real images.

[0121] Optimizer: Use an optimizer (such as Adam) to minimize the loss function and update the parameters of the denoising model.

[0122] 4. Generation process

[0123] Initial Noise Image: Start with a random noise image.

[0124] Denoising step: Through a series of denoising steps, a clear image is gradually generated.

[0125] Generate image: Finally, a high-quality generated image is obtained.

[0126] After completing the image editing through the above process, the original image set can be updated according to the edited target image. Specifically, the first camera parameters corresponding to each original image in the original image set are obtained, and the second camera parameters corresponding to each target image in the target image set are obtained. Then, the original images with the same first camera parameters and second camera parameters are replaced with the corresponding target images. Based on the replacement results, the target image set edited according to the language instructions is obtained.

[0127] Among them, the first camera parameters include at least a first camera position and a first camera orientation, and the second camera parameters include at least a second camera position and a second camera orientation. Then, the first camera position and the first camera orientation can be used to determine the corresponding first viewpoint, and the second camera position and the second camera orientation can be used to determine the corresponding second viewpoint. Then, the first viewpoint is matched with the second viewpoint, and the original image at the same viewpoint is replaced with the target image. Based on the replacement result, a target image set corresponding to the original 3D visual scene is obtained.

[0128] It should be noted that the update process of the above-mentioned image can be implemented based on an iterative image editing and updating method. The iterative image editing and updating method refers to iteratively replacing the corresponding image in the original image set with the image edited by the language instruction; at the beginning of the image optimization, the image data set consists of the original captured images from a series of viewpoints, which are represented by v and the images are represented by These images are cached separately and used as conditions for the diffusion model in all stages. In each iteration, multiple image updates d can be performed, followed by multiple NeRF updates n. The image updates are performed sequentially in a random order of v at the beginning of training. NeRF updates always sample a set of random rays from the entire training dataset, so the supervision signal contains a mixture of old information and pixels from the most recently updated dataset images. Over time, as images are used to update NeRF and are gradually re-rendered and updated, they begin to converge on a globally consistent description of the edited scene, thereby supporting editing of the overall scene, local content, etc. based on the image + language instruction editing method, which can effectively achieve the unification between local content and overall content.

[0129] Step 104 , performing 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, wherein the target 3D visual scene is a scene obtained by editing the original 3D visual scene according to the language instruction.

[0130] After the update of the image set is completed, the target image set can be reconstructed in 3D vision to obtain a target 3D visual scene corresponding to the original 3D visual scene. After obtaining the corresponding original image set by acquiring images of the 3D visual scene, the original images can be edited in combination with language instructions to obtain a target image set edited according to the language instructions, so that the 3D visual scene constructed based on the target image set can be more coordinated and reasonable. At the same time, in the editing process, the editing method based on image + language instructions supports editing of the overall scene, local content, etc., which can effectively achieve the unification between local content and overall content, and improve the flexibility of editing, reduce the dependence on user expertise, and can meet the editing of different 3D visions.

[0131] In a specific implementation, feature points can be first extracted from the target image, and then the target image can be matched with feature points to determine the correspondence between the target images. The correspondence between the target images is then used to predict the camera posture corresponding to each of the target images, and the correspondence and the camera posture are used to perform 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene.

[0132] In one example, first, feature points are extracted from the target image. Feature points are points with significant features in the image, usually corner points, edge points or areas with rich textures. Among them, feature point detection algorithms include SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.

[0133] Next, the feature points in the target image can be matched. The purpose of feature point matching is to find corresponding feature points in two different target images. Optionally, the matching algorithm may include Brute-Force Matching and FLANN (Fast Library for Approximate Nearest Neighbors)-based matching. The matching result is usually a set of corresponding point pairs. Through feature point matching, the correspondence between the target images can be determined. Among them, the correspondence refers to the matching pairs of feature points in the two images, so as to determine the camera pose according to the correspondence and realize 3D reconstruction. Furthermore, using the correspondence between the target images, the camera pose corresponding to each target image can be predicted. The camera pose includes the position and orientation of the camera. Common methods include using epipolar geometry and PnP (Perspective-n-Point) algorithms to estimate the camera pose.

[0134] Finally, the correspondence and camera posture are used to perform 3D visual reconstruction. The purpose of 3D reconstruction is to restore the three-dimensional structure of the scene from multiple images. The corresponding three-dimensional structure can be constructed through Multi-View Stereo (MVS) and Structured Light. The reconstruction result is a target 3D visual scene corresponding to the original 3D visual scene. After the corresponding original image set is obtained by image acquisition of the 3D visual scene, the original image is edited in combination with language instructions to obtain the target image set edited according to the language instructions, so that the 3D visual scene constructed based on the target image set can be more coordinated and reasonable. At the same time, in the editing process, the editing method based on image + language instructions supports editing of the overall scene, local content, etc., which can effectively achieve the unity between local content and overall content, and improve the flexibility of editing, reduce the dependence on user expertise, and can meet the editing of different 3D vision.

[0135] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0136] In an embodiment of the present invention, an original 3D visual scene is sampled to obtain a corresponding original image set, and a language instruction for the original 3D visual scene input by a user is obtained, where the language instruction is text information containing a visual editing intention for the original 3D visual scene. Then, the original image set is edited according to the language instruction to obtain a target image edited according to the language instruction, and the original image set is updated according to the target image to obtain a target image set edited according to the language instruction. Then, 3D visual reconstruction is performed on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, where the target 3D visual scene is edited according to the language instruction. The scene after editing the original 3D visual scene, by acquiring images of the 3D visual scene to obtain the corresponding original image set, and then editing the original images in combination with language instructions, can obtain a target image set edited according to the language instructions, so that the 3D visual scene constructed based on the target image set can be more coordinated and reasonable. At the same time, in the editing process, the editing method based on image + language instructions supports editing of the overall scene, local content, etc., which can effectively achieve the unification between local content and overall content, and improves the flexibility of editing, reduces the dependence on user expertise, and can meet the editing of different 3D visuals.

[0137] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0138] Reference Figure 2 , shows a processing flow chart of 3D visual editing provided in an embodiment of the present invention, including:

[0139] Step 1: Image sampling S1, sampling the original 3D visual scene to obtain a sampled image set as the original image set; the user inputs the content automatically edited by the 3D vision as a language instruction;

[0140] Step 2: Conditional diffusion model processing S2, using the conditional diffusion model of 2D images to edit the images in the original image set according to the language instructions, and obtain the images edited by the language instructions;

[0141] Step 3: Iterative image set update S3, using the iterative image editing and updating method to update the original image set to form an image set edited by the language instruction;

[0142] Step 4: 3D reconstruction model processing S4, using the 3D reconstruction model to perform 3D visual reconstruction on the image set edited by the language instruction, and finally obtain the edited 3D visual scene.

[0143] For example, refer to Figure 3, showing a schematic diagram of a 3D visual editing scene provided in an embodiment of the present invention, including: S1: inputting a 3D visual data file, which is specifically embodied as a file in a format such as .obj, .3ds, etc.; after image sampling, N original image files containing information of each angle of the 3D scene are obtained; S2: the conditional diffusion model processes the N original image files, and generates N image files edited by language instructions; S3: using an iterative image editing and updating method to update the original image set containing N image files, a set of images edited by language instructions with the same number of images is obtained; S4: using a three-dimensional reconstruction model to perform 3D visual reconstruction on the image set edited by language instructions, and 1 edited 3D visual data file is obtained. It should be noted that Figure 3 The figure shows switching the color of the bear in the image, and the corresponding language instruction may be "change the bear into a grizzly bear", and the corresponding image editing operation may be performed based on the language instruction.

[0144] In addition, refer to Figure 4 , showing a schematic diagram of the structure of the automatic editing system provided in an embodiment of the present invention, the system can realize efficient automatic editing of 3D vision according to language instructions. Among them, the system may include an image sampling unit, a conditional diffusion model processing unit, an iterative image set updating unit, a three-dimensional reconstruction model processing unit and a 3D visual output unit; the image sampling unit is used to sample images of the 3D visual scene to obtain an original image set; the conditional diffusion model processing unit is used to edit the images in the original image set according to the language instructions to obtain the images edited by the language instructions; the iterative image set updating unit is used to form the image set edited by the language instructions; the three-dimensional reconstruction model processing unit is used to perform 3D visual reconstruction on the image set edited by the language instructions; the 3D visual output unit is used to output the edited 3D visual scene. Specifically, the output of the image sampling unit is used as the input of the conditional diffusion model processing unit; the output of the conditional diffusion model processing unit is used as the input of the iterative image set updating unit; the output of the iterative image set updating unit is used as the input of the three-dimensional reconstruction model processing unit; the output of the three-dimensional reconstruction model processing unit is used as the input of the 3D visual scene output unit.

[0145] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0146] Reference Figure 5, shows a structural block diagram of a 3D vision processing device provided in an embodiment of the present invention, which may specifically include the following modules:

[0147] The image acquisition module 501 is used to sample the original 3D visual scene to obtain a corresponding original image set;

[0148] An instruction acquisition module 502 is used to acquire a language instruction input by a user for the original 3D visual scene, where the language instruction is text information containing a visual editing intention for the original 3D visual scene;

[0149] The image editing module 503 is used to edit the original image set according to the language instruction to obtain a target image edited according to the language instruction, and to update the original image set according to the target image to obtain a target image set edited according to the language instruction;

[0150] The scene reconstruction module 504 is used to perform 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, wherein the target 3D visual scene is a scene after the original 3D visual scene is edited according to the language instruction.

[0151] In some feasible implementations, the image acquisition module 501 is specifically used for:

[0152] Acquire acquisition parameters for the original 3D visual scene, where the acquisition parameters include at least a camera position and a camera orientation;

[0153] The original 3D visual scene is sampled according to the camera position and the camera orientation to obtain a corresponding original image set.

[0154] In some feasible implementations, the original image set includes a plurality of original images, where the original images are images acquired from the original 3D visual scene at a specific viewpoint, and the image editing module 503 is specifically configured to:

[0155] Loading a conditional diffusion model for the original image set;

[0156] Acquiring noise information for the original image set;

[0157] The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model to edit the original image, so as to obtain a target image edited according to the language instruction.

[0158] In some feasible implementations, the image editing module 503 is specifically used for:

[0159] The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model, the original image is forward diffused according to the noise information to obtain the corresponding noise image, and then the noise image is back-diffused according to the voice instruction to obtain the target image edited according to the language instruction.

[0160] In some feasible implementations, the image editing module 503 is specifically used for:

[0161] Acquire a first camera parameter corresponding to each original image in the original image set;

[0162] Acquire a second camera parameter corresponding to each target image in the target image set;

[0163] The original images having the same first camera parameters and second camera parameters are replaced with corresponding target images, and based on the replacement result, a target image set edited according to the language instruction is obtained.

[0164] In some feasible implementations, the first camera parameters include at least a first camera position and a first camera orientation, the second camera parameters include at least a second camera position and a second camera orientation, and the image editing module 503 is specifically configured to:

[0165] Determine a corresponding first viewpoint using the first camera position and the first camera orientation;

[0166] Determine a corresponding second viewpoint using the second camera position and the second camera orientation;

[0167] The first viewpoint is matched with the second viewpoint, and an original image at the same viewpoint is replaced with a target image, and based on a replacement result, a target image set corresponding to the original 3D visual scene is obtained.

[0168] In some feasible implementations, the scene reconstruction module 504 is specifically used to:

[0169] Extracting feature points from the target image;

[0170] Matching feature points of the target images to determine the corresponding relationship between the target images;

[0171] Predicting the camera pose corresponding to each of the target images using the correspondence between the target images;

[0172] The corresponding relationship and the camera posture are used to perform 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene.

[0173] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0174] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned 3D vision processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0175] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned 3D vision processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] Figure 6 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0177] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and a power supply 611. Those skilled in the art will appreciate that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0178] It should be understood that in the embodiment of the present invention, the radio frequency unit 601 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 610 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 601 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 601 can also communicate with the network and other devices through a wireless communication system.

[0179] The electronic device provides users with wireless broadband Internet access through the network module 602, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0180] The audio output unit 603 can convert the audio data received by the RF unit 601 or the network module 602 or stored in the memory 609 into an audio signal and output it as sound. Moreover, the audio output unit 603 can also provide audio output related to a specific function performed by the electronic device 600 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 603 includes a speaker, a buzzer, a receiver, etc.

[0181] The input unit 604 is used to receive audio or video signals. The input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 606. The image frame processed by the graphics processor 6041 can be stored in the memory 609 (or other storage medium) or sent via the radio frequency unit 601 or the network module 602. The microphone 6042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 601 in the case of a telephone call mode.

[0182] The electronic device 600 also includes at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 6061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 6061 and / or the backlight when the electronic device 600 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 605 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0183] The display unit 606 is used to display information input by the user or information provided to the user. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0184] The user input unit 607 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 6071 or near the touch panel 6071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 6071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 610, receives the command sent by the processor 610 and executes it. In addition, the touch panel 6071 can be implemented in various types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 6071, the user input unit 607 may also include other input devices 6072. Specifically, other input devices 6072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0185] Further, the touch panel 6071 may be covered on the display panel 6061. When the touch panel 6071 detects a touch operation on or near it, it is transmitted to the processor 610 to determine the type of the touch event, and then the processor 610 provides a corresponding visual output on the display panel 6061 according to the type of the touch event. It can be understood that in one embodiment, the touch panel 6071 and the display panel 6061 are used as two independent components to implement the input and output functions of the electronic device, but in some embodiments, the touch panel 6071 and the display panel 6061 can be integrated to implement the input and output functions of the electronic device, which is not limited here.

[0186] The interface unit 608 is an interface for connecting an external device to the electronic device 600. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 608 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 600 or may be used to transmit data between the electronic device 600 and an external device.

[0187] The memory 609 can be used to store software programs and various data. The memory 609 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 609 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0188] The processor 610 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 609 and calling data stored in the memory 609, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 610 may include one or more processing units; preferably, the processor 610 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

[0189] The electronic device 600 may also include a power supply 611 (such as a battery) for supplying power to each component. Preferably, the power supply 611 may be logically connected to the processor 610 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0190] In addition, the electronic device 600 includes some functional modules not shown, which will not be described in detail here.

[0191] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0193] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0194] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0195] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0196] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0199] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0200] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A 3D vision processing method, characterized in that: include: Sampling the original 3D visual scene to obtain a corresponding original image set; Acquire a language instruction for the original 3D visual scene input by a user, where the language instruction is text information containing a visual editing intention for the original 3D visual scene; Editing the original image set according to the language instruction to obtain a target image edited according to the language instruction, and updating the original image set according to the target image to obtain a target image set edited according to the language instruction; The target image set is subjected to 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene, wherein the target 3D visual scene is a scene obtained by editing the original 3D visual scene according to the language instruction.

2. The method according to claim 1, characterized in that The sampling of the original 3D visual scene to obtain a corresponding original image set includes: Acquire acquisition parameters for the original 3D visual scene, where the acquisition parameters include at least a camera position and a camera orientation; The original 3D visual scene is sampled according to the camera position and the camera orientation to obtain a corresponding original image set.

3. The method according to claim 1, characterized in that The original image set includes a plurality of original images, wherein the original image is an image acquired from the original 3D visual scene at a specific viewpoint, and the editing of the original image set according to the language instruction to obtain a target image edited according to the language instruction includes: Loading a conditional diffusion model for the original image set; Acquiring noise information for the original image set; The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model to edit the original image, so as to obtain a target image edited according to the language instruction.

4. The method according to claim 3, characterized in that: The step of inputting the original image, the language instruction corresponding to the original image, and the noise information into the conditional diffusion model to edit the original image, and obtaining a target image edited according to the language instruction, comprises: The original image, the language instruction corresponding to the original image and the noise information are input into the conditional diffusion model, the original image is forward diffused according to the noise information to obtain the corresponding noise image, and then the noise image is back-diffused according to the voice instruction to obtain the target image edited according to the language instruction.

5. The method according to claim 1, characterized in that The updating of the original image set according to the target image to obtain the target image set edited according to the language instruction includes: Acquire a first camera parameter corresponding to each original image in the original image set; Acquire a second camera parameter corresponding to each target image in the target image set; The original images having the same first camera parameters and second camera parameters are replaced with corresponding target images, and based on the replacement result, a target image set edited according to the language instruction is obtained.

6. The method according to claim 5, characterized in that The first camera parameters include at least a first camera position and a first camera orientation, the second camera parameters include at least a second camera position and a second camera orientation, and the replacing of the original image having the same first camera parameters and second camera parameters with the corresponding target image, and obtaining the target image set edited according to the language instruction based on the replacement result, comprises: Determine a corresponding first viewpoint using the first camera position and the first camera orientation; Determine a corresponding second viewpoint using the second camera position and the second camera orientation; The first viewpoint is matched with the second viewpoint, and an original image at the same viewpoint is replaced with a target image, and based on a replacement result, a target image set corresponding to the original 3D visual scene is obtained.

7. The method according to claim 6, characterized in that The performing 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene includes: Extracting feature points from the target image; Matching feature points of the target images to determine the corresponding relationship between the target images; Predicting the camera pose corresponding to each of the target images using the correspondence between the target images; The corresponding relationship and the camera posture are used to perform 3D visual reconstruction to obtain a target 3D visual scene corresponding to the original 3D visual scene.

8. A 3D vision processing device, characterized in that: include: An image acquisition module is used to sample the original 3D visual scene to obtain a corresponding original image set; An instruction acquisition module, used to acquire a language instruction input by a user for the original 3D visual scene, wherein the language instruction is text information containing a visual editing intention for the original 3D visual scene; An image editing module, configured to edit the original image set according to the language instruction to obtain a target image edited according to the language instruction, and to update the original image set according to the target image to obtain a target image set edited according to the language instruction; A scene reconstruction module is used to perform 3D visual reconstruction on the target image set to obtain a target 3D visual scene corresponding to the original 3D visual scene, wherein the target 3D visual scene is a scene after the original 3D visual scene is edited according to the language instruction.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images