Image processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and device and electronic equipment. The method comprises the following steps: acquiring a plurality of images, wherein the plurality of images are images of different visual angles of the same target scene; obtaining a depth image corresponding to the target scene through a multi-view stereoscopic vision algorithm and the plurality of images; and inputting the depth image and the plurality of images into a trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network. Therefore, under the condition that the three-dimensional image needs to be established, a plurality of input images can be processed through a multi-view stereoscopic vision algorithm to obtain depth information required for establishing the three-dimensional image, and then the depth information is input into the trained neural mapping network to obtain the three-dimensional image required to be established. Therefore, the depth information can be acquired without using an active sensor, the dependence on the active sensor is reduced, and the hardware cost used for establishing the three-dimensional image is reduced.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 63 / 534,784, filed with the U.S. Patent Office on August 25, 2023, and entitled “METHOD FOR 3D RECONSTRUCTION FROM MULTIPLE-VIEWS WITH DENSE NEURAL SCENEMAPPING,” the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer technology, and more specifically, to an image processing method, device and electronic device. Background Art

[0004] 3D reconstruction is a technology that creates a mathematical model suitable for computer representation and processing of 3D objects. It is the basis for processing, operating and analyzing the properties of 3D objects in a computer environment, and is also a key technology for building virtual reality that expresses the objective world in computers. However, related 3D reconstruction methods are still relatively dependent on active sensors (such as lidar, structured light, etc.). Summary of the invention

[0005] In view of the above problems, the present application proposes an image processing method, device and electronic device to improve the above problems.

[0006] In a first aspect, the present application provides an image processing method, the method comprising: acquiring multiple images, the multiple images being images of the same target scene from different perspectives; obtaining a depth image corresponding to the target scene through a multi-view stereo vision algorithm and the multiple images; inputting the depth image and the multiple images into a trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network, and the depth image is used to supervise the trained neural mapping network to generate a three-dimensional image.

[0007] In a second aspect, the present application provides an image processing device, comprising: an image acquisition unit, used to acquire multiple images, wherein the multiple images are images of the same target scene from different perspectives; a depth information acquisition unit, used to obtain a depth image corresponding to the target scene through a multi-view stereo vision algorithm and the multiple images; a three-dimensional image creation unit, used to input the depth image and the multiple images into a trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network, and the depth image is used to supervise the trained neural mapping network to generate a three-dimensional image.

[0008] In a third aspect, the present application provides an electronic device, which includes at least a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium having program code stored therein, wherein the above method is executed when the program code is executed by a processor.

[0010] The present application provides an image processing method, device and electronic device, in which a plurality of images can be first obtained, wherein the plurality of images are images of different perspectives of the same target scene. Then, a depth image corresponding to the target scene can be obtained through a multi-view stereo vision algorithm and the input plurality of images, so that the depth image and the plurality of images can be input into a trained neural mapping network to obtain a three-dimensional image of the target scene. Thus, through the above-mentioned method, when a three-dimensional image needs to be established, the input plurality of images can be processed through a multi-view stereo vision algorithm to obtain the depth information required for establishing the three-dimensional image, and then the depth information is input into the trained neural mapping network to obtain the three-dimensional image to be established, thereby making it possible not to use active sensors (such as lidar, structured light, etc.) to collect depth information, thereby reducing the dependence on active sensors to reduce the hardware cost used for establishing the three-dimensional image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram showing an application scenario of the image processing method proposed in an embodiment of the present application;

[0013] Figure 2 A schematic diagram showing another application scenario of the image processing method proposed in an embodiment of the present application;

[0014] Figure 3 A flowchart of an image processing method proposed in an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing video acquisition by an electronic device in the present application during movement is shown;

[0016] Figure 5A schematic diagram showing a plurality of electronic devices synchronously performing video acquisition in the present application is shown;

[0017] Figure 6 A schematic diagram showing video frames included in videos captured by multiple electronic devices in the present application is shown;

[0018] Figure 7 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0019] Figure 8 A schematic diagram showing a data processing flow in an image processing method proposed in an embodiment of the present application;

[0020] Fig. 9 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0021] Fig.10 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0022] Fig.11 A schematic diagram showing a data processing flow in an image processing method proposed in an embodiment of the present application;

[0023] Fig.12 Schematic diagram showing multiple viewing angles in this application;

[0024] Fig.13 A schematic diagram showing pixels at the same position in the present application is shown;

[0025] Fig.14 A schematic diagram showing the calculation of color difference in this application is shown;

[0026] Fig.15 A structural block diagram of an image processing device proposed in an embodiment of the present application is shown;

[0027] Fig.16 A structural block diagram of an image processing device proposed in another embodiment of the present application is shown;

[0028] Fig.17 A structural block diagram of an electronic device for executing an image processing method according to an embodiment of the present application is shown;

[0029] Fig.18 It is a storage unit of an embodiment of the present application for storing or carrying a program code for implementing the image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] 3D reconstruction technology, as a core technology in the field of computer graphics and vision, aims to capture the geometric form, surface texture and even physical properties of 3D objects in the real world and convert them into digital models that can be directly recognized, processed and interacted with by computers. This process requires not only a high degree of accuracy, but also the integrity and authenticity of the model to support a wide range of application scenarios, such as industrial design, medical imaging, game development, movie special effects, augmented reality (AR), virtual reality (VR) and mixed reality (MR).

[0032] However, the inventors found in their research that the relevant three-dimensional reconstruction methods are still relatively dependent on active sensors (such as lidar, structured light, etc.). Among them, although active sensors (such as lidar, structured light) have certain advantages in three-dimensional reconstruction, their widespread application also faces certain challenges. For example, active sensors also have the disadvantages of high cost (high-end lidar and other equipment are expensive, which limits their popularity among ordinary users or small-scale projects), high environmental requirements (the performance of some active sensors in complex or extreme environments (such as strong light, rain and fog, strong electromagnetic interference) may be affected), and complex data post-processing (large amounts of raw data require efficient algorithms for rapid processing, which requires high computing resources).

[0033] Therefore, after the inventors discovered the above problems in the study, they proposed an image processing method, device and electronic device in the present application that can improve the above problems. In this method, multiple images can be acquired first, wherein the multiple images are images of different perspectives of the same target scene. Then, the depth image corresponding to the target scene can be obtained through a multi-view stereo vision algorithm and multiple input images, so that the depth image and the multiple images can be input into a trained neural mapping network to obtain a three-dimensional image of the target scene. Thus, through the above method, when it is necessary to establish a three-dimensional image, the input multiple images can be processed by a multi-view stereo vision algorithm to obtain the depth information required for establishing a three-dimensional image, and then the depth information is input into the trained neural mapping network to obtain the required three-dimensional image, so that active sensors (such as lidar, structured light, etc.) can be used to collect depth information, thereby reducing the dependence on active sensors to reduce the hardware cost used for establishing three-dimensional images.

[0034] Before further describing the embodiments of the present application in detail, an application environment involved in the embodiments of the present application is introduced.

[0035] The following first introduces the application scenarios involved in the embodiments of the present application.

[0036] In the embodiment of the present application, the image processing method provided can be executed by an electronic device. In this way, all steps in the image processing method provided by the embodiment of the present application can be executed by the electronic device. For example, Figure 1 As shown, in the case where all steps in the image processing method provided in the embodiment of the present application can be executed by an electronic device, all steps can be executed by the processor of the electronic device 100.

[0037] Furthermore, the image processing method provided in the embodiment of the present application can also be executed by a server. Correspondingly, in this mode of execution by the server, the server can start executing the steps in the image processing method provided in the embodiment of the present application in response to a trigger instruction. The trigger instruction can be sent by an electronic device used by the user, or can be triggered locally by the server in response to some automation events.

[0038] In addition, the image processing method provided in the embodiment of the present application can also be executed by the electronic device and the server in collaboration. In this way, some steps of the image processing method provided in the embodiment of the present application are executed by the electronic device, while other steps are executed by the server. For example, taking the image processing method in the present application as an example, Figure 2 As shown, the electronic device 100 can perform an image processing method including: obtaining multiple images. Then, the electronic device 100 can transmit the multiple images to the server 200, and then the server 200 executes the multi-view stereo vision algorithm and the multiple images to obtain the depth image corresponding to the target scene; and executes the input of the depth image and the multiple images into the trained neural mapping network to obtain the three-dimensional image of the target scene through the trained neural mapping network. Afterwards, the server 200 can also save the three-dimensional image locally, or return the three-dimensional image to the electronic device 100 for real-time display.

[0039] It should be noted that in this method of collaborative execution by the electronic device and the server, the steps respectively executed by the electronic device and the server are not limited to the methods introduced in the above examples. In actual applications, the steps respectively executed by the electronic device and the server can be dynamically adjusted according to actual conditions.

[0040] It should be noted that the electronic device 100 is not specifically limited in the embodiment of the present application. For example, the electronic device may be a head-mounted display device, a smart phone, a tablet computer, a vehicle-mounted terminal, or other devices. The server 200 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. In particular, in the case where the image processing method provided in the embodiment of the present application is executed by a server cluster or a distributed system composed of multiple physical servers, different steps in the image processing method may be executed by different physical servers respectively, or may be executed in a distributed manner by a server built based on a distributed system.

[0041] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0042] See also Figure 3 , an image processing method provided by an embodiment of the present application, the method comprising:

[0043] S110: Acquire multiple images, where the multiple images are images of the same target scene from different perspectives.

[0044] In the embodiment of the present application, the scene can be reconstructed in three dimensions through images of different viewing angles of the same scene, wherein the three-dimensional reconstruction of the scene can be understood as the three-dimensional reconstruction of the objects included in the scene.

[0045] In the embodiment of the present application, there are multiple ways to obtain multiple images used for each three-dimensional reconstruction.

[0046] As a method, multiple images captured adjacent to each other can be obtained from a video stream captured by an electronic device, wherein the video stream is captured by the electronic device while it is moving. In this method, the electronic device that captures the video can capture the video while moving. When the electronic device captures the video while moving, the multiple images captured adjacent to each other can have certain differences in viewing angles. For example, Figure 4 As shown, in Figure 4 In the scenario shown, the electronic device 10 can move along the direction indicated by the arrow in the figure and capture video during the movement.

[0047] As another method, multiple electronic devices can be used to capture video respectively, and the images captured at the same time in the videos captured by the multiple electronic devices are used as the multiple images to be acquired. In this method, each electronic device will capture video from a different perspective, so the multiple images to be acquired can come from different electronic devices. For example, Figure 5As shown, in the target scene, there are disposed electronic devices 10, 20 and 30, wherein the electronic devices 10, 20 and 30 can collect scene contents of the target scene from different perspectives.

[0048] Among them, based on Figure 5 In the case shown in FIG. 1 , the video frames included in the videos captured by the electronic device 10, the electronic device 20, and the electronic device 30 can be as follows: Figure 6 Based on Figure 6 In the case shown, the acquisition time of video frame 11, video frame 21 and video frame 31 is the same, then video frame 11, video frame 21 and video frame 31 can be used as the multiple images to be acquired. Similarly, the acquisition time of video frame 12, video frame 22 and video frame 32 is the same, then video frame 12, video frame 22 and video frame 32 can be used as the multiple images to be acquired. Similarly, the acquisition time of video frame 13, video frame 23 and video frame 33 is the same, then video frame 13, video frame 23 and video frame 33 can be used as the multiple images to be acquired.

[0049] S120: Obtain a depth image corresponding to the target scene through a multi-view stereo vision algorithm and multiple images.

[0050] The multi-view stereo vision algorithm in the embodiment of the present application may be an MVS (Multiple View Stereo) algorithm, wherein the MVS algorithm is a generalization of stereo vision, which can observe and obtain images of scenes from multiple perspectives (from outside to inside), and thereby complete matching and depth estimation.

[0051] In the process of obtaining the depth map of the target scene through the MVS algorithm, the same feature points can be found and matched in the images of multiple perspectives (i.e., the multiple images obtained in S110). These feature points can be corner points, edges, and other regions with significant features in the image. Optionally, the feature matching can be implemented by a variety of algorithms, such as SIFT (Scale Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Features) algorithm, etc. These algorithms can extract key points in the image and calculate their descriptors for matching between different images.

[0052] After that, the position of each feature point in three-dimensional space can be calculated by triangulation technology using the matched feature points and camera parameters (the parameters of the camera in the electronic device that collects multiple images). Among them, triangulation is based on geometric principles and determines the position of spatial points through observation data from two or more perspectives. In the MVS algorithm, the camera parameters and feature point matching results can be used to construct geometric constraint equations, and then the three-dimensional coordinates of the feature points can be solved.

[0053] According to the calculated feature point positions, the depth image of the target scene can be further generated by interpolation or filtering. Among them, the depth map can be generated by a variety of interpolation algorithms, such as linear interpolation, bilinear interpolation, etc. In addition, filtering techniques (such as Gaussian filtering, median filtering, etc.) can also be used to smooth the depth map and reduce noise and errors.

[0054] S130: Input the depth image and the multiple images into the trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network. The depth image is used to supervise the trained neural mapping network to generate the three-dimensional image.

[0055] In the embodiments of the present application, the neural mapping network can be understood as neural SLAM. Among them, neural SLAM can integrate the implicit neural representation of the three-dimensional world (for example, through NeRF) into the traditional SLAM system.

[0056] In the embodiment of the present application, the trained neural mapping network can establish a three-dimensional image of the target scene corresponding to the multiple images based on the input multiple images. When the multiple images are input into the trained neural mapping network, the depth image obtained in the above steps is also input synchronously, so that the depth image is used as supervision information in the process of generating the three-dimensional image.

[0057] Among them, the trained neural mapping network can also be understood as a special deep learning model, which can be used to extract three-dimensional information from two-dimensional image data (for example, multiple input images) and generate three-dimensional images. This network usually has powerful feature extraction and expression capabilities and can learn complex mapping relationships from two-dimensional to three-dimensional.

[0058] Among them, multiple images input into the trained neural mapping network provide visual information such as texture and color of the target scene, which is crucial for generating realistic three-dimensional images.

[0059] As mentioned above, a depth image is an image that contains information about the distance of each point in the scene from the camera (camera in the video acquisition device). In the MVS (multi-view stereo) algorithm, a depth image can be obtained by matching and triangulating multi-view images. The depth image provides the neural mapping network with important clues about the geometric structure of the target scene, which helps the network generate three-dimensional images more accurately.

[0060] When the depth image and multiple images are input to the trained neural mapping network, the trained neural mapping network will first perform feature extraction on the input multiple images and depth images. This process can be achieved through structures such as convolutional layers and pooling layers, which aim to extract key information in the image (such as edges, textures, etc.). After that, the trained neural mapping network will perform the fusion and mapping steps. In the fusion and mapping steps, the neural mapping network will combine the two-dimensional image features and depth information to construct structures and shapes in three-dimensional space. After feature extraction and mapping, the trained neural mapping network will output a three-dimensional image of the target scene. This three-dimensional image can be in various forms such as voxel grids, point clouds, mesh models, etc.

[0061] For example, when the electronic device is a head-mounted display device, a user wearing the head-mounted display device can move in a space (for example, in a room), and the head-mounted display device can perform real-time video capture during the user's movement. During the video capture process, the head-mounted display device can obtain multiple adjacent captured images from the captured video in real time, and then create a three-dimensional image based on the multiple images, and display the created three-dimensional image.

[0062] An image processing method provided in this embodiment enables, through the above-mentioned method, when it is necessary to create a three-dimensional image, multiple input images can be processed by a multi-view stereo vision algorithm to obtain the depth information required for creating the three-dimensional image, thereby making it possible to collect depth information without using active sensors (such as lidar, structured light, etc.), thereby reducing the dependence on active sensors and reducing the hardware cost used for creating the three-dimensional image.

[0063] See also Figure 7 , an image processing method provided by an embodiment of the present application, the method comprising:

[0064] S210: Acquire multiple images, where the multiple images are images of the same target scene from different perspectives.

[0065] S220: Obtain global scene information through the trained neural mapping network and multiple images.

[0066] In the embodiments of the present application, global scene information refers to a comprehensive and in-depth description of the entire scene (target scene), which goes beyond the limited information provided by a single image. Global scene information may include but is not limited to the following aspects:

[0067] Plane information: refers to the position and size of each plane in the scene, which is crucial for understanding the spatial layout and structure of the scene. Plane information can be used to build a 3D model of the scene.

[0068] Normal vector information: Normal vectors are vectors that point outwards from each surface point in 3D space. This information is very important for understanding the orientation of the surface, lighting effects, and shadow generation.

[0069] Semantic information: refers to the categories and attributes of each object in the scene (such as "person", "car", "tree", etc.). Semantic information helps with high-level understanding and analysis of the scene, such as object recognition in autonomous driving and obstacle detection in robot navigation.

[0070] Based on the above content, the global scene information may include: at least one of plane information, normal vector information and semantic information.

[0071] When multiple images are input into a trained neural mapping network, the neural mapping network can use the knowledge it has learned during training to perform in-depth analysis on each image and integrate information from different images. By comprehensively analyzing the information in multiple images, the neural mapping network can infer global scene information (including plane information, normal vector information, and semantic information). This global scene information can then be used in a variety of applications, such as 3D scene reconstruction.

[0072] S230: Obtain a depth image corresponding to the target scene through a multi-view stereo vision algorithm, global scene information, and multiple images.

[0073] Among them, Figure 8 As shown, in multi-view stereo vision algorithms, global scene information can be used as prior knowledge or constraints to guide the process of depth map generation.

[0074] S240: Input the depth image and the multiple images into the trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network. The depth image is used to supervise the trained neural mapping network to generate the three-dimensional image.

[0075] The present embodiment provides an image processing method, so that in the above manner, when it is necessary to establish a three-dimensional image, multiple input images can be processed by a multi-view stereo vision algorithm to obtain the depth information required for establishing the three-dimensional image, so that it is possible not to use active sensors (such as laser radar, structured light, etc.) to collect depth information, thereby reducing the dependence on active sensors to reduce the hardware cost used for establishing the three-dimensional image. In addition, in the present embodiment, global scene information can also be obtained through the trained neural mapping network, the first image and the second image. When the global scene information includes more detailed information about the target scene, the global scene information can be used to guide the multi-view stereo vision algorithm to generate a depth image to improve the accuracy of the generated depth image.

[0076] See also Fig. 9 , an image processing method provided by an embodiment of the present application, the method comprising:

[0077] S310: Acquire multiple images, where the multiple images are images of the same target scene from different perspectives.

[0078] S320: Obtain key features corresponding to each of the multiple images based on the epipolar attention mechanism.

[0079] It should be noted that in multi-view geometry, an epipolar line refers to a straight line determined by light rays from one camera's perspective through a point in another camera. When two cameras observe the same scene, a pixel in each camera corresponds to an epipolar line in the other camera, which may contain the corresponding point of the pixel in the other view. In multi-view stereo vision algorithms (MVS), epipolar geometry is a key tool for connecting different views.

[0080] The attention mechanism is a technique that simulates the human attention process, which allows the model to "focus" on more important parts when processing information. In deep learning, the attention mechanism can be implemented by weighting or selectively processing different parts of the input data.

[0081] In multi-view based image processing, the epipolar attention mechanism can be regarded as a special attention mechanism that focuses on searching and selecting key features along the epipolar direction. This helps the model to find the correspondence between different views more accurately and extract useful information for subsequent processing (such as 3D reconstruction, etc.).

[0082] As a way, in the process of obtaining key features based on the epipolar attention mechanism, features can be extracted from the image in each view first. For example, a deep learning model such as a convolutional neural network can be used to extract local and global features of the image.

[0083] Then, using the principle of epipolar geometry, possible corresponding points or regions are searched along the epipolar direction in each view. This can be achieved by calculating disparity maps or using other multi-view matching algorithms. During the epipolar search process, an attention mechanism is applied to weight the searched candidate points or regions. The attention weight can be determined based on the correlation, consistency or other metrics between the candidate point and the key features in the current view.

[0084] According to the results of attention weighting, the corresponding key features are extracted from each view. These key features may include edges, corners, textures and other information that are useful for subsequent processing. Finally, the key features in different views are fused and optimized. This can be achieved through feature splicing, pooling or other feature fusion techniques to obtain a more comprehensive and accurate description of the scene.

[0085] S330: Obtain corresponding depth images through key features corresponding to each of the multiple images and a deep learning model.

[0086] S340: Input the depth image and the multiple images into the trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network. The depth image is used to supervise the trained neural mapping network to generate the three-dimensional image.

[0087] As a way, in the embodiment of the present application, the backbone of the multi-view stereo vision algorithm can be an MVS2D network. In this case, the aforementioned S320 can be performed by the MVS2D network. In addition, as a way, the backbone of the multi-view stereo vision algorithm can be a monocular depth estimation network.

[0088] As a method, in an embodiment of the present application, the backbone of the neural mapping network can be Co-SLAM.

[0089] The present embodiment provides an image processing method, so that in the above manner, when it is necessary to establish a three-dimensional image, multiple input images can be processed by a multi-view stereo vision algorithm to obtain the depth information required for establishing the three-dimensional image, so that active sensors (such as laser radar, structured light, etc.) can be used to collect depth information, thereby reducing the dependence on active sensors to reduce the hardware cost used for establishing the three-dimensional image. In addition, in the present embodiment, in the process of generating a depth image corresponding to the first image, the key features corresponding to each of the multiple images can be obtained based on the epipolar attention mechanism, and then the key features and the deep learning model can be used to obtain the depth image corresponding to the first image.

[0090] See also Fig.10The present application provides an image processing method, which includes:

[0091] S410: The electronic device acquires multiple images and transmits the multiple images to a server, where the multiple images are images of the same target scene from different perspectives.

[0092] It should be noted that for some electronic devices, the data processing capability is relatively limited. However, the establishment of a three-dimensional image will have higher requirements on the data processing capability of the device. As a way, the step of establishing a three-dimensional image can be deployed and run in a server. In this way, when the electronic device detects that its data processing capability is limited, it can transmit the acquired multiple images to the server.

[0093] S420: The server obtains a depth image corresponding to the target scene through a multi-view stereo vision algorithm and multiple images.

[0094] S430: The server inputs the depth image and the multiple images into the trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network, and transmits the three-dimensional image to the electronic device.

[0095] S440: The electronic device displays the three-dimensional image.

[0096] For example, when the electronic device is a head-mounted display device, a user wearing the head-mounted display device can move in a space (for example, in a room), and the head-mounted display device can perform real-time video capture during the user's movement. During the video capture process, the head-mounted display device can obtain multiple adjacent captured images from the captured video in real time, and then transmit the multiple images to a server. After that, the head-mounted display device can display the three-dimensional image transmitted back by the server in real time.

[0097] The following is an introduction to the training process of the neural mapping network in the embodiment of the present application.

[0098] In an embodiment of the present application, as a method, a training set for training may be obtained, wherein the training set includes multiple groups of training images, each group of training images being images of the same target scene from different perspectives;

[0099] Then, through the multi-view stereo vision algorithm and each set of training images, the depth image of the target scene corresponding to each set of training images is obtained. After that, the neural mapping network to be trained is iteratively trained through multiple sets of training images and the depth images corresponding to each of the multiple sets of training images to obtain a trained neural mapping network.

[0100] Exemplarily, the acquired training set may include a first group of training images, a second group of training images, a third group of training images, ... an Nth group of training images.

[0101] Among them, the depth image of the target scene corresponding to the first group of training images can be obtained by the first group of training images and the multi-view stereo vision algorithm. Afterwards, the neural mapping network to be trained can be trained by the depth image of the target scene corresponding to the first group of training images and the first group of training images. Afterwards, the depth image of the target scene corresponding to the second group of training images can be obtained by the second group of training images and the multi-view stereo vision algorithm, and then the neural mapping network to be trained can be trained based on the depth image of the target scene corresponding to the second group of training images and the second group of training images, until the trained neural mapping network meets the end condition, and then the neural mapping network that meets the end condition is used as the trained neural mapping network.

[0102] In the above process, for each set of training images, the neural mapping network trained by the previous set of training images can be used as the current neural mapping network to be trained. For example, in the above process, based on the depth image of the target scene corresponding to the second set of training images, and the second set of training images, the neural mapping network to be trained is the neural mapping network trained by the first set of training images.

[0103] Optionally, iteratively training the neural mapping network to be trained using multiple groups of training images and depth images corresponding to the multiple groups of training images to obtain a trained neural mapping network may include: inputting a group of training images currently used for training into the neural mapping network to be trained, and obtaining corresponding estimated images and estimated depth images through the neural mapping network to be trained; obtaining a first loss value based on the target image and the estimated image, and obtaining a second loss value based on the depth images corresponding to the group of training images currently used for training and the estimated depth images; training the neural mapping network to be trained using the first loss value and the second loss value, and if the neural mapping network obtained by the current training does not meet the training stop condition, continuing the training using the next group of training images to obtain a trained neural mapping network.

[0104] For example, Fig.11As shown in the figure, in the process of processing by the multi-view stereo vision algorithm (MVS), for multiple input images, the corresponding depth image can be obtained through the epipolar attention mechanism (Epipolar Attention) and monocular feature extraction (Monocular Feature Extraction) and the depth network (Depth Regression). For the neural mapping network to be trained, the corresponding estimated image and estimated depth image can be obtained by volume rendering (Volumerendering). L1 represents the first loss value, and L2 represents the second loss value.

[0105] It should be noted that in the embodiment of the present application, when a 3D image is created, the 3D image may be created from a certain perspective. For example, the 3D image may be created from a main perspective. The main perspective may be the perspective of any image among the multiple images acquired. For example, Fig.12 As shown, the acquired multiple images may include Fig.12 The image corresponding to the perspective A, the image corresponding to the perspective B, and the image corresponding to the perspective C. Among them, the perspective A can be used as the main perspective, the perspective B can be used as the main perspective, and the perspective C can be used as the main perspective.

[0106] In this case, one image can be selected from the multiple images obtained as the main perspective image. In the process of obtaining the depth image, the depth image corresponding to the main perspective image can be obtained through a multi-view stereo vision algorithm and multiple images. Furthermore, when using the trained neural mapping network, the depth image and multiple images corresponding to the main perspective image will also be input into the trained neural mapping network, so that the three-dimensional image obtained by the trained neural mapping network is also related to the main perspective. Among them, the three-dimensional image is related to the main perspective, which can be understood as the content displayed by the three-dimensional image, which is the content seen when observing from the main perspective toward the target scene. For example, if the generated three-dimensional image is related to Fig.12 It is related to the viewing angle A in the figure, so it can be understood that the content included in the three-dimensional image is the content seen from viewing angle A.

[0107] As shown in the foregoing, in an embodiment of the present application, multiple images can be acquired by a single electronic device during movement. In this way, if the electronic device for image acquisition does not move, the image content of the multiple images acquired each time may be the same, or, in the scene for image acquisition, if there is no change in the scene content, the image content of the multiple images acquired by the electronic device each time will also be the same. In the case where the image content of multiple images acquired multiple times is the same, the image content of the three-dimensional image finally established will also be the same (equivalent to repeatedly establishing the same three-dimensional image), which may cause a waste of resources.

[0108] In order to improve the situation where the same three-dimensional image is repeatedly established, when multiple images are collected by a single electronic device, as a way, after acquiring multiple images, the image content of the multiple images with the multiple images acquired last time can be compared to see if they are the same. If they are the same, it means that the electronic device performing image acquisition is not currently moving, and the three-dimensional image generated last time can be directly output (for example, displayed) as the three-dimensional image established this time. If the image content of the images with the same perspective is different, it means that the electronic device is moving, and the method provided in the embodiment of the present application can be used to establish a three-dimensional image in real time.

[0109] In the embodiment of the present application, whether two images are identical can be determined through difference information between the two images.

[0110] The difference information between the two images may represent the difference between the two images. As a method, the difference information between the two images may be determined based on image parameters of the two images. The image parameters include at least one of a pixel value and a brightness value.

[0111] Optionally, take the case where the image parameters include pixel values, and the calculated difference information is the first difference value as an example. It should be noted that, for an image, each pixel included therein will have a corresponding pixel value. In this case, when calculating the first difference value of two images, the pixel value difference of two pixels at the same position in the two images can be first calculated to obtain the pixel value difference corresponding to all two pixels at the same position in the two images, and then the absolute values ​​of all the pixel value differences are taken and summed to obtain the first difference value of the two images.

[0112] For example, Fig.13As shown, the two images for which the first difference value calculation is performed include image 40 and image 41, wherein image 40 includes pixels P11, P12...P33. Correspondingly, image 41 includes pixels P11, P12...P33. Wherein, the position of pixel P11 in image 40 in image 40 is the first in the first row, and the position of pixel P11 in image 41 in image 41 is also the first in the first row, so pixel P11 in image 40 and pixel P11 in image 41 can be understood as two pixels in the same position. By analogy, two pixels in the same position in image 40 and image 41 can be obtained. Wherein, the pixel value difference between pixel P11 in image 40 and pixel P11 in image 41, the pixel value difference between pixel P12 in image 40 and pixel P12 in image 41...the pixel value difference between pixel P33 in image 40 and pixel P33 in image 41, can be taken as absolute values ​​and summed up respectively, and the first difference value between image 40 and image 41 can be obtained.

[0113] The pixel value difference between two pixels can be obtained by taking the absolute value of the difference between the two pixels in multiple color channels and then summing them up. For example, in the RGB color mode, Fig.14 As shown, the pixel value difference between the two pixels can be obtained by calculating the difference between the two pixels in the red channel, the green channel, and the blue channel, and then taking the absolute values ​​of the differences of these channels and summing them up.

[0114] Optionally, when the image parameters include pixel values ​​and the calculated difference information is the aforementioned first difference value, the difference between the two images on the specified color channel can be used as the corresponding first difference value. For example, for the two images for which the first difference value is calculated, the channel means of the two images on the specified color channel can be calculated, and then the difference of the respective channel means can be calculated to obtain the first difference value. In an embodiment of the present application, in the process of calculating the channel mean of an image, the value of each pixel in the image on the specified color channel can be first obtained, and then the value of each pixel in the image on the specified color channel can be summed to obtain the total color value. By comparing the total color value with the number of pixels in the image, the channel mean of the specified color channel can be obtained.

[0115] Exemplarily, if the pixels in the image include pixel P11, pixel P12, pixel P13, pixel P21, pixel P22 and pixel P23. Wherein, the designated color channel is the red channel. In this case, the values ​​of pixel P11, pixel P12, pixel P13, pixel P21, pixel P22 and pixel P23 on the red channel can be obtained, and the values ​​of pixel P11, pixel P12, pixel P13, pixel P21, pixel P22 and pixel P23 on the red channel can be summed to obtain the total color value. For example, the total color value can be V4. Then by dividing V4 / 6, the channel mean of the image on the red channel can be obtained.

[0116] Optionally, the acquired multiple frames of images can be converted into corresponding grayscale images, in which case the image parameters can be the brightness values ​​corresponding to the pixels. It should be noted that in the grayscale image, the brightness value of the pixel represents the grayscale level or brightness intensity of the pixel. A grayscale image is an image that contains only brightness information but not color information. In a grayscale image, each pixel is usually represented by a single value (for example, a brightness value between 0 and 255), where 0 represents black, 255 represents white, and the intermediate values ​​represent different degrees of gray. In this case, when calculating the difference between two pixels in the same position, only the difference in brightness value can be calculated without calculating the difference in each color channel, which can help reduce the amount of calculation and improve the final detection speed. It should be noted that the method of calculating the first difference value based on the brightness value is the same as the method of calculating the first difference value based on the pixel value. Therefore, the method of calculating the first difference value based on the brightness value can refer to the aforementioned content, and the method of calculating the first difference value based on the brightness value will not be introduced here.

[0117] After obtaining the first difference value, the first difference value can be compared with the difference threshold. If the first difference value is less than the difference threshold, it means that the two images are the same. If the first difference value is not less than the difference threshold, it means that the two images are not the same.

[0118] It should be noted that, when multiple images are obtained from a video stream captured by an electronic device, a currently captured frame of image and the first N frames of image adjacent to the currently captured frame of image can be taken together as the multiple images obtained.

[0119] The larger N is, the more images are acquired each time for building a three-dimensional image, and the richer the information is, which is conducive to more accurate building of the three-dimensional image. However, the larger N is, the greater the amount of data processing is, and the greater the consumption of processing resources is. In this case, in order to better balance the accuracy of building a three-dimensional image and the consumption of processing resources.

[0120] The value of N can be determined according to the actual situation of the device that creates the image.

[0121] For example, if the device creating the image is currently in a low-load period, N may be configured to be relatively higher. If the device creating the image is currently in a high-load period, N may be configured to be relatively lower.

[0122] In the embodiment of the present application, the low-load time period and the high-load time period can be divided according to the historical operation status of the device for establishing the image.

[0123] As one approach, the low-load time period and the high-load time period may be divided according to the number of programs run by the device that creates the image.

[0124] Optionally, multiple time periods may be pre-divided, and then the number of programs running in each time period may be counted, and each time period may be further divided into a low-load time period or a high-load time period according to the number of programs running in each time period. For example, a time period in which the number of programs running is less than a first quantity threshold may be determined as a low-load time period, and a time period in which the number of programs running is not less than the first quantity threshold may be determined as a high-load time period. For another example, a time period in which the occupancy rate of the processor is less than a first occupancy threshold may be determined as a low-load time period, and a time period in which the occupancy rate of the processor is not less than the first occupancy threshold may be determined as a high-load time period.

[0125] The length of each pre-divided time period may be of various types, for example, the length of each time period may be 1 hour, 2 hours, or 3 hours.

[0126] Optionally, the time period can be divided by counting the running conditions of the programs in the electronic device, and the time period divided can be determined synchronously to be a low-load time period or a high-load time period. In this way, the moment when the number of running programs changes compared to the first number threshold can be used as the moment when the low-load time period and the high-load time period intersect. After the electronic device is started, the number of currently running programs can be detected, and the moment of startup can be used as the starting moment of the low-load time period. If the number of running programs is detected to be greater than the first number threshold, the moment when the number of programs is detected to be greater than the first number threshold can be used as the starting moment of the high-load time period, and the starting moment is also the end moment of a low-load time period before the high-load time period. If after detecting that the number of running programs is greater than the first number threshold, it is detected that the number of programs is not greater than the first number threshold, the moment when the number of programs is detected to be less than the first number threshold will be used as the end moment of the current high-load time period, and the end moment is also the start moment of a new low-load time period entered. In this way, the electronic device does not need to divide multiple time periods in advance, and thus has better flexibility.

[0127] Optionally, the time period can be divided by counting the occupancy of the processor in the electronic device, and synchronously determining whether the divided time period is a low-load time period or a high-load time period. In this way, the moment when the current occupancy of the processor changes compared to the first occupancy threshold can be used as the moment when the low-load time period and the high-load time period intersect. Among them, after the electronic device is started, the current occupancy of the processor can be detected, and the moment when it is started can be used as the starting moment of the low-load time period. If the occupancy of the processor is detected to be greater than the first occupancy threshold, the moment when the occupancy is detected to be greater than the first occupancy threshold can be used as the starting moment of the high-load time period, and the starting moment is also the end moment of a low-load time period before the high-load time period. If after detecting that the occupancy is greater than the first occupancy threshold, it is detected that the occupancy is not greater than the first occupancy threshold, the moment when the occupancy is detected to be less than the first occupancy threshold will be used as the end moment of the current high-load time period, and the end moment is also the start moment of a new low-load time period entered.

[0128] In the case where there are multiple ways to divide the low-load time period and the high-load time period, the electronic device can flexibly determine which specific division method is currently used.

[0129] As a determination method, the user of the electronic device may determine the specific method of dividing the low-load time period and the high-load time period. As another method, the electronic device may determine the specific method of dividing the low-load time period and the high-load time period according to actual conditions.

[0130] Optionally, the method of dividing the low-load time period and the high-load time period may be determined according to the total running time of the electronic device after the electronic device is first started. Moreover, different division methods may result in different low-load time periods and high-load time periods.

[0131] It should be noted that in the early stage of using electronic devices, the user's program usage habits are relatively unstable. If the time periods are directly divided by counting the running status of the programs in the electronic devices, and the divided time periods are simultaneously determined to be low-load time periods or high-load time periods, the low-load time periods and high-load time periods obtained may be inaccurate. Correspondingly, after the user has used the electronic device for a period of time, the running rules of the programs in the electronic device will be relatively fixed. In this case, it will be relatively more accurate to divide the low-load time periods or high-load time periods in real time according to the running status.

[0132] Furthermore, when the total running time of the electronic device after the first startup is less than the first specified time, multiple time periods can be pre-divided, and then the number of programs running in each time period is counted, and then each time period is divided into a low-load time period or a high-load time period according to the number of programs running in each time period. Correspondingly, when the total running time of the electronic device after the first startup is not less than the first specified time, the time period can be divided by counting the running conditions of the programs in the electronic device, and simultaneously determining whether the divided time period is a low-load time period or a high-load time period.

[0133] See also Fig.15 , an image processing device 500 provided in an embodiment of the present application, the device 500 includes:

[0134] The image acquisition unit 510 is used to acquire multiple images, where the multiple images are images of the same target scene from different perspectives.

[0135] As a method, the image acquisition unit 510 is specifically used to acquire a plurality of adjacent images from a video stream acquired by the electronic device, wherein the video stream is acquired by the electronic device during movement.

[0136] The depth information acquisition unit 520 is used to obtain a depth image corresponding to the target scene through a multi-view stereo vision algorithm and multiple images.

[0137] The three-dimensional image creation unit 530 is used to input the depth image and multiple images into the trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network. The depth image is used to supervise the trained neural mapping network to generate the three-dimensional image.

[0138] As a method, the depth information acquisition unit 520 is specifically used to obtain global scene information through a trained neural mapping network and multiple images; and obtain a depth image corresponding to the target scene through a multi-view stereo vision algorithm, global scene information, and multiple images.

[0139] As a method, the depth information acquisition unit 520 is specifically used to obtain the key features corresponding to each of the multiple images based on the epipolar attention mechanism; and obtain the corresponding depth image through the key features corresponding to each of the multiple images and the deep learning model.

[0140] As a method, the image processing device 500 also includes a training unit 540, which is used to obtain a training set for training, wherein the training set includes multiple groups of training images, each group of training images is an image of the same target scene from different perspectives; through a multi-view stereo vision algorithm and each group of training images, a depth image of the target scene corresponding to each group of training images is obtained; and the neural mapping network to be trained is trained through the depth image, the third image and the fourth image to obtain a trained neural mapping network.

[0141] Among them, the training unit 540 is specifically used to input a set of training images currently used for training into the neural mapping network to be trained, and obtain the corresponding estimated image and the estimated depth image through the neural mapping network to be trained; obtain a first loss value according to the target image and the estimated image, and obtain a second loss value according to the depth image corresponding to the set of training images currently used for training and the estimated depth image; train the neural mapping network to be trained through the first loss value and the second loss value to obtain a trained neural mapping network.

[0142] The present embodiment provides an image processing device, so that when it is necessary to create a three-dimensional image, the image processing device can be used to process multiple input images through a multi-view stereo vision algorithm to obtain depth information required for creating the three-dimensional image, thereby making it possible to collect depth information without using active sensors (such as lidar, structured light, etc.), thereby reducing dependence on active sensors and reducing the hardware cost used for creating the three-dimensional image.

[0143] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0144] The following will be combined Fig.16 An electronic device provided by the present application is described.

[0145] See also Fig.16 Based on the above image processing method, image generation method, and device, the embodiment of the present application also provides another electronic device 100 that can execute the above image processing method. The electronic device 2000 includes one or more (only one is shown in the figure) processors 202, a memory 204, a network module 206, a sensor module 208, and an audio acquisition device 210 that are coupled to each other. Among them, the memory 204 stores a program that can execute the content of the above embodiments, and the processor 202 can execute the program stored in the memory 204.

[0146] Among them, the processor 202 may include one or more processing cores. The processor 202 uses various interfaces and lines to connect various parts of the entire electronic device 2000, and executes various functions and processes data of the electronic device 2000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 204, and calling data stored in the memory 204. Optionally, the processor 202 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 202 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 202, but may be implemented separately through a communication chip.

[0147] The memory 204 may include a random access memory (RAM) or a read-only memory (ROM). The memory 204 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 204 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.

[0148] The network module 206 is used to implement information interaction between the electronic device 2000 and other devices, for example, transmission of device control instructions, operation request instructions, and status information acquisition instructions, etc. When the electronic device 2000 is specifically a different device, the corresponding network module 206 may be different.

[0149] The sensor module 208 may include at least one sensor. Specifically, the sensor module 208 may include but is not limited to: a light sensor, a motion sensor, a pressure sensor, an infrared heat sensor, a distance sensor, an acceleration sensor, and other sensors.

[0150] Among them, the pressure sensor can be a sensor that detects the pressure generated by pressing on the electronic device 2000. That is, the pressure sensor detects the pressure generated by the contact or pressing between the user and the electronic device, such as the pressure generated by the contact or pressing between the user's ear and the mobile terminal. Therefore, the pressure sensor can be used to determine whether contact or pressing occurs between the user and the electronic device 2000, and the magnitude of the pressure.

[0151] Among them, the acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that recognize the posture of the electronic device 2000 (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. In addition, the electronic device 2000 can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, etc., which will not be repeated here.

[0152] The audio acquisition device 210 is used to collect audio signals. Optionally, the audio acquisition device 210 includes a plurality of audio acquisition devices. The audio acquisition device may be a microphone. For example, in one embodiment, the audio acquisition device 210 may include two microphones, in which one of the microphones may correspond to an analog-to-digital converter, and the other microphone may correspond to an analog-to-digital converter with two different analog gains. In another embodiment, the audio acquisition device 210 may include three microphones. In this embodiment, two of the microphones (e.g., a main microphone and a secondary microphone) may each correspond to an analog-to-digital converter, and the other microphone (e.g., a camera microphone) may correspond to an analog-to-digital converter with two different analog gains.

[0153] As a method, the network module of the electronic device 2000 is a radio frequency module, which is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The radio frequency module may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. For example, the radio frequency module can exchange information with an external device through the electromagnetic waves sent or received, and then receive audio signals sent by the external device.

[0154] Furthermore, the electronic device 2000 may also include an image acquisition device for image acquisition, for example, a video, a static picture or a dynamic picture may be taken by the image acquisition device.

[0155] Please refer to Fig.17 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0156] The computer-readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0157] In summary, the present application provides an image processing method, device and electronic device, in which a plurality of images can be first acquired, wherein the plurality of images are images of different perspectives of the same target scene. Then, a depth image corresponding to the target scene can be obtained through a multi-view stereo vision algorithm and the input of the plurality of images, so that the depth image and the plurality of images can be input into a trained neural mapping network to obtain a three-dimensional image of the target scene. Thus, through the above-mentioned method, when it is necessary to establish a three-dimensional image, the plurality of input images can be processed by a multi-view stereo vision algorithm to obtain the depth information required for establishing the three-dimensional image, thereby making it possible not to use active sensors (such as lidar, structured light, etc.) to collect the depth information, thereby reducing the dependence on active sensors to reduce the hardware cost used for establishing the three-dimensional image.

[0158] This application combines MVS depth estimation (i.e., obtaining a depth image through a multi-view stereo vision algorithm in this application) and global optimization through dense neural mapping into an end-to-end process. In this way, the estimated depth map from MVS can be used to supervise the dense neural mapping to achieve real-time 3D reconstruction. This configuration reduces or eliminates the reliance on true depth at inference time.

[0159] This application combines a lightweight MVS backbone (i.e., an MVS2D network) with a real-time neural SLAM system (i.e., the aforementioned trained neural mapping network) to achieve dense mapping. It can run in real time and achieve dense 3D reconstruction with accurate geometric priors. The method of this application is very practical for mixed reality (MR) applications because we can reconstruct 3D scenes from RGB images (e.g., multiple images required in this application) captured by a head-mounted display (HMD).

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire multiple images, where the multiple images are images of the same target scene from different perspectives; Obtaining a depth image corresponding to the target scene through a multi-view stereo vision algorithm and the multiple images; The depth image and the multiple images are input into a trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network, and the depth image is used to supervise the trained neural mapping network to generate a three-dimensional image.

2. The method according to claim 1, characterized in that The step of obtaining a depth image corresponding to the target scene through a multi-view stereo vision algorithm and the multiple images includes: Obtaining global scene information through the trained neural mapping network and the multiple images; A depth image corresponding to the target scene is obtained through the multi-view stereo vision algorithm, the global scene information, and the multiple images.

3. The method according to claim 2, characterized in that The global scene information includes: at least one of plane information, normal vector information and semantic information in the target scene.

4. The method according to claim 1, characterized in that: The step of obtaining a depth image corresponding to the target scene through a multi-view stereo vision algorithm and the multiple images includes: Acquire key features corresponding to each of the multiple images based on an epipolar attention mechanism; The corresponding depth image is obtained through the key features corresponding to each of the multiple images and the deep learning model.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a training set for training, wherein the training set includes multiple groups of training images, each group of training images is an image of a same target scene from a different perspective; Through the multi-view stereo vision algorithm and each set of training images, a depth image of the target scene corresponding to each set of training images is obtained; The neural mapping network to be trained is iteratively trained through multiple groups of training images and depth images corresponding to each of the multiple groups of training images to obtain a trained neural mapping network.

6. The method according to claim 5, characterized in that The neural mapping network to be trained is iteratively trained by using multiple sets of training images and depth images corresponding to each of the multiple sets of training images to obtain a trained neural mapping network, including: Inputting a set of training images currently used for training into the neural mapping network to be trained, and obtaining corresponding estimated images and estimated depth images through the neural mapping network to be trained; Obtaining a first loss value according to the target image and the estimated image, and obtaining a second loss value according to a depth image corresponding to a set of training images currently used for training and the estimated depth image, wherein the target image is any image in the set of training images currently used for training; The neural mapping network to be trained is trained by using the first loss value and the second loss value. If the neural mapping network currently obtained by training does not meet the training stop condition, the next set of training images is used to continue training to obtain the trained neural mapping network.

7. The method according to any one of claims 1 to 6, characterized in that: The acquiring of multiple images comprises: A plurality of adjacently captured images are acquired from a video stream captured by an electronic device, wherein the video stream is captured by the electronic device during movement.

8. An image processing device, characterized in that: The device comprises: An image acquisition unit, used to acquire a plurality of images, wherein the plurality of images are images of the same target scene from different perspectives; A depth information acquisition unit, configured to obtain a depth image corresponding to the target scene through a multi-view stereo vision algorithm and the multiple images; The three-dimensional image establishment unit is used to input the depth image and the multiple images into a trained neural mapping network to obtain a three-dimensional image of the target scene through the trained neural mapping network, and the depth image is used to supervise the trained neural mapping network to generate a three-dimensional image.

9. An electronic device, characterized in that: The method comprises a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement any one of the methods of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images