Image processing method and related device

Abstract

The application relates to the AR field, in particular to an image processing method and related equipment. The method comprises the following steps: acquiring a current image, calculating a first depth map corresponding to the current image according to the current image, obtaining a second depth map corresponding to the current image from a server according to the current image, the second depth map representing more abundant depth information of feature points in the current image than the first depth map, and the first depth map containing depth map information of feature points not possessed by the second depth map; obtaining a target depth map of the current image according to the current image, the first depth map and the second depth map; acquiring a virtual object image and a depth map of the virtual object; and obtaining a target image according to the target depth map, the depth map of the virtual object, the current image and the virtual object image. The problem that the virtual-real occlusion effect has scale ambiguity and inconsistency is solved.

Description

[0001] This application claims priority to Chinese Patent Application No. 202010950951.0, filed on September 10, 2020, entitled "Image Processing Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of augmented reality (AR), and more particularly to an image processing method and related equipment for achieving realistic virtual-real occlusion effects. Background Technology

[0003] AR (Augmented Reality) uses computer-generated virtual information to supplement the real world, making virtual information and the real world appear to coexist in the same space. Currently, most AR applications simply overlay virtual objects onto a real scene without properly handling the occlusion relationship between virtual and real objects. This easily causes spatial disorientation in the user's perception and fails to surpass the sensory experience of reality. An effect without virtual-real occlusion is like... Figure 1a As shown. In AR applications, it is necessary to handle the realistic relationship between virtual objects and the real scene, i.e., occlusion. The effect of occlusion is as follows: Figure 1b As shown. Correct occlusion relationships enable users to have a natural and accurate spatial perception in AR applications; incorrect occlusion relationships reduce the realism of AR applications.

[0004] When incorrect occlusion relationships exist in the fused image, it becomes difficult for the observer to correctly judge the relative positions of real and virtual objects, hindering the achievement of a realistic fusion effect. Incorrect occlusion relationships can easily lead to disorientation and spatial confusion for the observer, resulting in an unrealistic fusion outcome. Therefore, addressing the occlusion problem in AR is crucial for enhancing the realism of virtual objects within real-world scenes and achieving a truly immersive fusion effect. AR occlusion processing focuses on ensuring that virtual objects are correctly occluded by objects in the real-world scene in front of them. It typically utilizes techniques such as depth extraction or scene modeling to obtain the occlusion relationships between real and virtual objects in the fused scene, extracting the occlusion edges of foreground objects in the real-world image, and ultimately generating a fused image with correct occlusion relationships.

[0005] Apple's iPad Pro 2020 uses RGB images and depth maps captured by a direct time-of-flight (dToF) camera to generate a depth map using machine learning methods to enhance the occlusion effect, achieving realistic occlusion. However, because it only uses monocular depth estimation from RGB images, the occlusion effect suffers from scale discrepancies and inconsistencies. Summary of the Invention

[0006] This application provides an image processing method and related equipment based on edge-cloud joint depth estimation, which solves the problems of scale ambiguity and inconsistency in virtual-real occlusion effects.

[0007] In a first aspect, embodiments of this application provide an image processing method based on edge-cloud joint depth estimation, comprising:

[0008] The process involves: acquiring the current image; calculating a first depth map corresponding to the current image; obtaining a second depth map corresponding to the current image from the server; the second depth map representing more depth information of feature points in the current image than the first depth map, and the first depth map containing depth information of feature points not present in the second depth map; obtaining a target depth map of the current image based on the current image, the first depth map, and the second depth map; the target depth map representing more depth information of feature points in the current image than the second depth map; acquiring a virtual object image and its depth map; and obtaining the target image, i.e., an image with virtual-real occlusion effect, based on the target depth map, the virtual object's depth map, the current image, and the virtual object image.

[0009] It should be noted that the feature points in this application refer to points where the grayscale value of the image changes drastically or points with large curvature on the edge of the image.

[0010] The second depth map represents more depth information of feature points in the current image than the first depth map, specifically in the following three aspects:

[0011] First, the number of pixels in the second depth map is greater than the number of pixels in the first depth map.

[0012] The first depth map is obtained by the terminal device based on feature points extracted from the image. Some pixels of the second depth map are obtained based on offline maps pre-stored in the server, while other pixels are also obtained based on feature points extracted from the image.

[0013] Due to the difference in computing power between terminal devices and servers, for the same image, the number of 2D feature points extracted by the terminal device is lower than that extracted by the server. Generally, the terminal device extracts a few hundred 2D feature points, while the server extracts tens of thousands. For 3D features that correspond one-to-one with the 2D feature points, the number of 3D points obtained by the terminal device based on the extracted 2D feature points is less than the number of 3D points obtained by the server. Therefore, the number of pixels in the depth map obtained by projecting the 3D points determined by the server is greater than the number of pixels in the depth map obtained by projecting the 3D points determined by the terminal device. Furthermore, the second depth map also includes pixels obtained from offline maps pre-stored in the server. Thus, it can be seen that the number of pixels in the second depth map is greater than the number of pixels in the first depth map.

[0014] Second, the pixel precision in the second depth map is higher than that in the first depth map.

[0015] The second depth map is obtained from the offline map pre-stored on the server. The 3D points in the offline map are determined based on the calibrated laser equipment, while the first depth map is obtained by matching feature points between the current image and historical images. Therefore, the pixel accuracy in the second depth map is higher than that in the first depth map.

[0016] Third, the distribution of pixels in the second depth map is more uniform than that in the first depth map.

[0017] For the same image, i.e. the current image, since the number of pixels in the second depth image is greater than the number of pixels in the first depth image, the distribution of pixels in the second depth image is more uniform than the distribution of pixels in the first depth image.

[0018] Specifically, the first depth map contains depth information of feature points that the second depth map does not have. This means that the number of pixels in the union of the first and second depth maps is greater than the number of pixels in the second depth map. The pixels in the target depth map can be regarded as the union of the pixels in the first and second depth maps. The number of pixels in the target depth map is greater than the number of pixels in the second depth map. Therefore, the target depth map represents the depth information of feature points in the current image more comprehensively than the second depth map.

[0019] Optionally, for any pixel position S, when the pixel value at pixel position S in the first depth map is 0 or the pixel value at pixel position S in the second depth map is 0, the pixel value at pixel position S in the target depth map is the maximum of the pixel values ​​at pixel position S in the first depth map and the pixel values ​​at pixel position S in the second depth map; when neither the pixel value at pixel position S in the first depth map nor the pixel value at pixel position S in the second depth map is 0, the pixel value at pixel position S in the target depth map is the average of the sum of the pixel values ​​at pixel position S in the first depth map and the pixel values ​​at pixel position S in the second depth map, or the weighted average of the two.

[0020] The target depth map is obtained based on the first and second depth maps. In contrast, existing technologies use depth maps obtained from RGB images and dToF cameras to enhance the virtual-real occlusion effect. It is evident that the target depth map represents richer depth information than the depth map used to enhance the virtual-real occlusion effect. Therefore, the target image obtained based on the target depth map, the virtual object's depth map, the current image, and the virtual object image does not suffer from inter-frame flickering or instability in its virtual-real occlusion effect. The second depth map is obtained from an offline map stored on the server based on the current image. Since the offline map includes a panoramic map composed of multiple frames of base maps, the target depth map obtained from the current image and the second depth map can be considered as being based on multi-view depth estimation. Compared to monocular depth estimation in existing technologies, the target image obtained based on this target depth map, the virtual object's depth map, the current image, and the virtual object image does not suffer from scale discrepancies or inconsistencies in its virtual-real occlusion effect.

[0021] In one feasible embodiment, the target depth map and the depth map of the virtual object are used to determine the presentation and distribution of pixels of the virtual object image and pixels of the current image in the target image.

[0022] Specifically, for any pixel P in the target image, if the first depth value is greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the virtual object image; if the first depth value is not greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the current image; wherein, the first depth value and the second depth value are the depth values ​​corresponding to pixel P in the target depth map and the depth map of the virtual object, respectively.

[0023] In one feasible embodiment, obtaining the second depth map corresponding to the current image from the server based on the current image includes:

[0024] Send a first acquisition request to the server to obtain the 3D point cloud corresponding to the local map; receive a first response message from the server in response to the first acquisition request, the first response message carrying the 3D point cloud corresponding to the local map, the local map including the scene data corresponding to the current image; acquire the second pose of the current image, the second pose of the current image being the pose of the terminal device when capturing the current image; project the current image according to the second pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map corresponding to the current image.

[0025] The local map is the area indicated by the first geographic location information in the offline map stored on the server; the first geographic location information is the geographic location information in the first pose of the current image, and the first pose of the current image is the pose of the current image in the coordinate system of the offline map. The first pose of the current image is obtained by processing the current image based on VPS technology.

[0026] Specifically, the process of projecting the current image based on its second pose and the corresponding 3D point cloud of the local map to obtain the second depth map refers to projecting the 3D point cloud of the local map onto the imaging plane of the current image based on its second pose to obtain the second depth map.

[0027] The offline map includes a panoramic map, 2D feature points, and corresponding 3D points. The 2D feature points are obtained by feature extraction from the panoramic map. The panoramic map is based on multiple frames of a base map, such as by stitching together multiple base frames. Optionally, the base map is an RGB image; other map formats are also possible and not limited here. The offline map is acquired using a calibrated laser panoramic camera.

[0028] It should be noted that the pose in this application refers to a 6-DOF pose, which includes geographic location information and orientation information.

[0029] In one feasible embodiment, obtaining the second depth map corresponding to the current image from the server based on the current image further includes:

[0030] Feature extraction is performed on the current image to obtain the first 2D feature points of the current image. The first 2D feature points of the current image are then matched with the eighth 2D feature points in multiple historical images to obtain the second 2D feature points of the current image. The second 2D feature points are the 2D feature points in the first 2D feature points of the current image that match the eighth 2D feature points in the historical images. Based on the second pose of the current image, the second 2D feature points of the current image, the 2D feature points in the eighth 2D feature points of the historical images that match the second 2D feature points, and the second pose of the historical images, the 3D points corresponding to the second 2D feature points of the current image are obtained. The second pose of the current image is the pose of the terminal device when the current image was captured.

[0031] The second depth map is obtained by projecting the second pose of the current image and the corresponding 3D point cloud of the local map onto the imaging plane of the current image, including:

[0032] The current image is projected based on its second pose, the 3D point cloud corresponding to the local map, and the 3D points corresponding to the second 2D feature points of the current image to obtain a second depth map.

[0033] Specifically, the second depth map is obtained by projecting the current image onto the imaging plane of the current image based on the second pose of the current image, the 3D point cloud corresponding to the local map, and the 3D points corresponding to the second 2D feature points of the current image.

[0034] Optionally, the first 2D feature point of the current image and the eighth 2D feature point in the historical image can be Oriented FAST and Rotated BRIEF (ORB) feature points, where FAST stands for features from accelerated segment test and BRIEF stands for binary robust independent elementary features; they can also be Accelerated-KAZE (AKAZE) feature points, ASL Eat or SuperPoint feature points obtained from deep learning using knowledge distillation or network search, or Difference of Gaussian (DOG) feature points, histogram of oriented gradient (HOG) feature points, BRIEF feature points, BRISK feature points, or FREAK feature points.

[0035] Due to limitations in edge computing power and the need for real-time performance, it is necessary to quickly extract features from images, and the number of features to be extracted is relatively small. Generally, the ORB method or an improved ORB method is used to extract features from regions with rich texture in the image. The features extracted by this method are ORB feature points, and the number of these feature points can be preset, usually several hundred.

[0036] It should be noted that AKAZE feature points can be obtained by extracting features from the image using the AKAZE feature extraction method. Other feature points in this embodiment can be obtained by extracting features from the image using corresponding methods.

[0037] By introducing 3D points obtained from 2D feature points of the current image and 2D feature points of historical images, and projecting these 3D points together with the 3D point cloud corresponding to the local map to obtain a second depth map, the depth information of feature points in the current image represented by the second depth map is further enriched, thereby also enriching the depth information of feature points in the current image represented by the target depth map.

[0038] In one feasible embodiment, obtaining a second depth map corresponding to the current image based on the current image includes:

[0039] Send a second retrieval request to the server, the second retrieval request carrying the current image, the second retrieval request instructing the server to obtain a second depth map based on the current image and the offline map stored by the server; receive a second response message from the server in response to the second retrieval request, the second response message carrying the second depth map.

[0040] The second depth map is obtained from the server. Since the second depth map is calculated by the server, it does not require the terminal device to consume computing resources to calculate it, thus reducing the resource consumption and power consumption of the terminal device.

[0041] In one feasible embodiment, calculating the first depth map corresponding to the current image based on the current image includes:

[0042] The process involves: obtaining the first pose of the current image, which is the pose of the current image in a first coordinate system, where the offline map stored on the server is located; processing the current image based on the first pose to obtain a first image; the first image is obtained by transforming the second pose of the current image into the first pose; the second pose of the current image is the pose of the terminal device when it captured the current image; extracting features from the first image to obtain a third 2D feature point; matching the third 2D feature point with the eighth 2D feature points of multiple historical images to obtain a fourth 2D feature point; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the eighth 2D feature points of multiple historical images; obtaining the 3D point corresponding to the fourth 2D feature point based on the fourth 2D feature point, the first pose of the current image, the 2D feature point among the eighth 2D feature points of multiple historical images that matches the fourth 2D feature point, and the first pose of the historical image to which the 2D feature point belongs; and projecting the current image based on the first pose of the current image and the 3D point corresponding to the fourth 2D feature point to obtain a first depth map.

[0043] Specifically, the projection of the current image onto the imaging plane of the current image based on the first pose of the current image and the 3D points corresponding to the fourth 2D feature points to obtain the first depth map means: projecting the 3D points corresponding to the fourth 2D feature points onto the imaging plane of the current image based on the first pose of the current image to obtain the first depth map.

[0044] Optionally, the third 2D feature point can be an ORB feature point, an AKAZE feature point, an ASLFeat feature point or a superpoint feature point obtained by knowledge distillation or network search based on deep learning, or a DOG feature point, HOG feature point, BRIEF feature point, BRISK feature point or FREAK feature point.

[0045] In one feasible embodiment, obtaining the first pose of the current image includes:

[0046] Send a third retrieval request to the server, the third retrieval request carrying the current image, the third retrieval request is used to request the first pose of the current image; receive a third response message sent by the server in response to the third retrieval request, the third response message carrying the first pose of the current image.

[0047] In one feasible embodiment, obtaining the first pose of the current image includes:

[0048] A fourth acquisition request is sent to the server, which carries the current image and the second pose of the current image. A fourth response message is received from the server in response to the fourth acquisition request. The fourth response message carries pose transformation information, which is used to transform between the second pose and the first pose of the current image. The second pose of the current image is transformed according to the pose transformation information to obtain the first pose of the current image.

[0049] The terminal device acquires the first pose or pose transformation information of the current image to align the coordinate system of the current image with that of the offline map, facilitating subsequent calculations.

[0050] In a feasible embodiment, obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map specifically involves:

[0051] A fifth acquisition request is sent to the server; the fifth acquisition request carries the terminal's geographical location information; a fifth response message is received from the server in response to the fifth acquisition request, the fifth response message carrying a depth estimation model; the depth estimation model is a neural network model corresponding to the terminal's geographical location information; the first depth map and the second depth map are stitched together to obtain a third depth map; the current image and the third depth map are input into the depth estimation model to obtain the target depth map.

[0052] Multiple depth estimation models are stored on the server, and each of these multiple depth estimation models corresponds one-to-one with multiple geographic location information.

[0053] In a feasible embodiment, obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map specifically involves:

[0054] The initial convolutional neural network model is trained to obtain a depth estimation model; the first depth map and the second depth map are concatenated to obtain a third depth map; the current image and the third depth map are input into the depth estimation model to obtain the target depth map;

[0055] The initial convolutional neural network is trained to obtain a depth estimation model, including:

[0056] Multiple image samples and their corresponding depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps. A loss value is calculated based on the multiple predicted depth maps, the corresponding ground truth depth maps, and a loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain a depth estimation model for the current image. The loss function is determined based on the error between the predicted and ground truth depth maps, the error between the gradients of the predicted and ground truth depth maps, and the error between the normal vectors of the predicted and ground truth depth maps.

[0057] Since the images are different for different locations, in order to improve the accuracy of the target depth map, the server determines a depth estimation model for each geographical location. When the depth estimation model is needed, the server determines the corresponding depth estimation model based on the geographical location of the terminal device, and obtains the target depth map based on the depth estimation model, which further improves the density of pixels in the target depth map of the current image, that is, enriches the depth information of the feature points in the current image represented by the target depth map.

[0058] In one feasible embodiment, obtaining the target depth map of the current image based on the current image and the second depth map includes:

[0059] Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps. The resolution of each of the T first feature maps is different, and the resolution of each of the T second feature maps is different; T is an integer greater than 1. The first and second feature maps with the same resolution from the T first feature maps and T second feature maps are superimposed to obtain T third feature maps. The T third feature maps are upsampled and fused to obtain the target depth map of the current image. The third depth map is obtained by stitching together the first depth map and the second depth map.

[0060] In one feasible embodiment, obtaining the target depth map of the current image based on the current image and the second depth map includes:

[0061] Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the second depth map to obtain T second feature maps. Multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps. The resolutions of each of the T first feature maps, the T second feature maps, and the T fourth feature maps are all different. The reference depth map is obtained from the depth map captured by the time-of-flight (TOF) camera, where T is an integer greater than 1. The first, second, and fourth feature maps with the same resolution from the T first, T second, and T fourth feature maps are superimposed to obtain T fifth feature maps. The T fifth feature maps are upsampled and fused to obtain the target depth map of the current image. The third depth map is obtained by stitching together the first and second depth maps.

[0062] In the process of calculating the target depth map of the current image, a reference depth map based on the depth map acquired by the TOF camera is introduced, which further enriches the depth information of the current image represented by the target depth map.

[0063] In one feasible embodiment, the reference depth map is obtained based on images captured by a TOF camera, specifically including:

[0064] The depth map acquired by the TOF camera is projected into 3D space based on the pose of the current image to obtain the fourth depth map; the fourth depth map is back-projected onto the reference image based on the pose of the reference image to obtain the reference depth map; the reference image is the image adjacent to the current image in terms of acquisition time.

[0065] In one feasible embodiment, the upsampling and fusion process includes:

[0066] For feature map P' j Upsampling is performed to obtain the feature map P” j The feature map P” j The resolution and the (j+1)th feature map P in the processing object j+1 The resolutions are the same; the width of the (j+1)th feature map is j+1 times the width of the feature map with the smallest resolution in the processing object, where j is greater than or equal to 1 and less than or equal to T-1; the feature map P” j With feature map P j+1 The feature map P' is obtained by fusion. j+1 Let j = j + 1, and repeat the above steps until j = T - 1; T is the number of feature maps in the object being processed; where, when j = 1, feature map P' j To process the feature map with the lowest resolution in the object, when j = T-1, feature map P'j+1 This is the result of upsampling and fusion processing.

[0067] In one feasible embodiment, the current image and the third depth map are input into the depth estimation model of the current image for processing to obtain the target depth map of the current image; wherein, the depth estimation model is implemented based on a convolutional neural network, and the third depth map is obtained by stitching together the first depth map and the second depth map.

[0068] In one feasible embodiment, obtaining the target image based on the target depth map, the virtual object depth map, the current image, and the virtual object image includes:

[0069] Edge optimization is performed on the target depth map based on the current image to obtain an optimized depth map; the accuracy of this optimized depth map is higher than that of the target depth map. The optimized depth map is then segmented to obtain foreground and background depth maps of the current image. The background depth map is the optimized depth map containing the background region, and the foreground depth map is the optimized depth map containing the foreground region. The L background depth maps are fused based on the L first poses corresponding to each of the L background depth maps to obtain a fused 3D scene. The L background depth maps include the background depth maps of pre-stored images and the background depth map of the current image; the L first poses include the first poses of pre-stored images and the current image; L is an integer greater than 1. The current image's pose is then used to... The fused 3D scene is back-projected to obtain a fused background depth map; the fused background depth map and the foreground depth map of the current image are stitched together to obtain an updated depth map; the virtual object and the current image are processed according to the updated depth map and the depth map of the virtual object to obtain the target image; for any pixel P in the target image, if the first depth value is greater than the second depth value, the pixel value of pixel P is the pixel value of the corresponding pixel P in the virtual object image; if the first depth value is not greater than the second depth value, the pixel value of pixel P is the pixel value of the corresponding pixel P in the current image; wherein, the first depth value and the second depth value are the depth values ​​corresponding to pixel P in the updated depth map and the depth map of the virtual object, respectively.

[0070] By performing edge optimization on the target depth map of the current image, a depth map with sharp edges is obtained. Then, by comparing the depth values ​​at the same position in the depth map of the sharp-edge depth map and the depth map of the virtual object, the pixel values ​​in the target image are determined. Thus, the occlusion between people or objects and virtual objects in the current image not only solves the problem of inter-frame flickering and instability in the target image, but also solves the problem of unsharp occlusion edges.

[0071] In one feasible embodiment, the current image includes the target person, and the method of this application further includes:

[0072] Target person detection is performed on the optimized depth map to obtain detection results;

[0073] The optimized depth map is segmented to obtain the foreground depth map and background depth map of the current image, including:

[0074] Based on the detection results, the optimized depth map is segmented to obtain the foreground depth map and background depth map of the current image. The foreground depth map of the current image includes the depth map corresponding to the target person.

[0075] The foreground depth map including the target task is determined from the optimized depth map, and the updated depth map is obtained based on the foreground depth map. The target image is obtained based on the depth map, the depth map of the virtual object, the image of the virtual object and the current image. This enhances the virtual and real occlusion effect between the virtual object and the person in the target image, resulting in a strong overall sense of immersion and an excellent user experience.

[0076] Secondly, embodiments of this application provide an image processing method based on edge-cloud joint depth estimation, including:

[0077] The system receives a depth estimation model request message from a terminal device, which carries the terminal device's geographic location information; it retrieves the depth estimation model for the current image from multiple depth estimation models stored in the server based on the terminal device's geographic location information; the depth estimation model for the current image is the depth estimation model corresponding to the terminal device's geographic location information among the multiple depth estimation models; these multiple depth estimation models correspond one-to-one with multiple geographic location information; and it sends a response message to the terminal device in response to the depth estimation model request message, which carries the depth estimation model for the current image.

[0078] Since the images differ depending on the location, the server determines a depth estimation model for each location to improve the accuracy of the target depth map. When the depth estimation model is needed, the server determines the corresponding depth estimation model based on the location of the terminal device, and obtains the target depth map based on the depth estimation model. This enriches the depth information of the feature points in the current image represented by the target depth map of the current image. The target image is obtained based on the target depth map, the depth map of the virtual object, the current image, and the image of the virtual object, thus solving the problem of inter-frame flickering and instability in the target image.

[0079] In one feasible embodiment, the method of this application further includes:

[0080] For multiple geographic location information, depth estimation models corresponding to each geographic location information are trained separately. Specifically, for any geographic location information S among the multiple geographic location information, the following steps are performed to train the depth estimation model corresponding to geographic location information S:

[0081] Multiple image samples and their corresponding depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps. The multiple image samples are acquired by the terminal device based on the geographic location information S. The loss value is calculated based on the multiple predicted depth maps, the corresponding true depth maps, and the loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain a depth estimation model corresponding to the geographic location information S. The loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0082] In one feasible embodiment, the method of this application further includes:

[0083] The system receives a first acquisition request from a terminal device; obtains a local map from an offline map based on the position in the first pose of the current image, wherein the local map is the area indicated by the pose in the first pose of the current image in the offline map; acquires the 3D point cloud corresponding to the local map; wherein the first pose of the current image is the pose of the current image in a first coordinate system, and the first coordinate system is the coordinate system in which the offline map is located; and sends a first response message to the terminal device in response to the first acquisition request, wherein the first response message carries the 3D point cloud corresponding to the local map.

[0084] In one feasible embodiment, the method of this application further includes:

[0085] The system receives a second acquisition request sent by a terminal device, the second acquisition request carrying the current image; obtains a second depth map corresponding to the current image based on the current image and an offline map; and sends a second response message to the terminal device in response to the second acquisition request, the second response message carrying the second depth map.

[0086] The offline map includes a panoramic map, 2D feature points, and corresponding 3D points. The 2D feature points are obtained by feature extraction from the panoramic map. The panoramic map is based on multiple frames of a base map, such as by stitching together multiple base frames. Optionally, the base map is an RGB image; other map formats are also possible and not limited here. The offline map is acquired using a calibrated laser panoramic camera.

[0087] In one feasible embodiment, obtaining a second depth map corresponding to the current image based on the current image and the offline map includes:

[0088] The first pose of the current image is determined based on the current image. The first pose of the current image is the pose of the current image in the first coordinate system, which is the coordinate system of the offline map. Based on the position in the first pose of the current image, the 3D point cloud corresponding to the local map is obtained from the 3D point cloud corresponding to the offline map. The local map is the area indicated by the first position in the offline map stored on the server. The current image is projected based on the first pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map.

[0089] Specifically, the process of projecting the current image onto the imaging plane of the current image based on the first pose of the current image and the corresponding 3D point cloud of the local map to obtain the second depth map refers to: projecting the 3D point cloud of the local map onto the imaging plane of the current image based on the first pose of the current image to obtain the second depth map.

[0090] In one feasible embodiment, obtaining a second depth map corresponding to the current image based on the current image and the offline map further includes:

[0091] Feature extraction is performed on the current image to obtain the ninth 2D feature point of the current image; the ninth 2D feature point of the current image is matched with the eleventh 2D feature point in multiple historical images to obtain the eleventh 2D feature point of the current image, which is the 2D feature point among the ninth 2D feature points of the current image that matches the eleventh 2D feature point in the historical images; based on the first pose of the current image, the eleventh 2D feature point of the current image, the 2D feature point among the eleventh 2D feature points in the historical images that matches the eleventh 2D feature point, and the first pose of the historical images, the 3D point corresponding to the eleventh 2D feature point of the current image is obtained;

[0092] The current image is projected based on its first pose and the corresponding 3D point cloud of the local map to obtain a second depth map, including:

[0093] The current image is projected based on the first pose of the current image, the 3D point cloud corresponding to the local map, and the 3D point corresponding to the eleventh 2D feature point of the current image to obtain a second depth map.

[0094] Specifically, the second depth map is obtained by projecting the current image onto the imaging plane of the current image based on the first pose of the current image, the 3D point cloud corresponding to the local map, and the 3D point corresponding to the eleventh 2D feature point of the current image.

[0095] Optionally, the ninth and tenth 2D feature points can be SIFT feature points, or SURF feature points (speeded-up robust features), or SuperPoint, ASLFeat, R2D2, or D2Net feature points obtained from deep learning.

[0096] Because servers have strong computing power, they can extract feature points from both texture-rich and texture-weak regions when extracting image features. The feature extraction method can be SIFT or a method based on SIFT; the extracted feature points can be called SIFT feature points, and the number can be preset, for example, approximately 10,000. Alternatively, the extraction method can be SURF or a method based on SURF; the extracted feature points can be called SURF feature points.

[0097] This shows that the number of SIFT feature points is higher than the number of ORB feature points.

[0098] By introducing a 3D point obtained from the ninth 2D feature point of the current image and the tenth 2D feature point of the historical image, and projecting this 3D point together with the 3D point cloud corresponding to the local map to obtain a second depth map, the depth information representing the current image described in the second depth map is further enriched.

[0099] In one feasible embodiment, obtaining a second depth map corresponding to the current image based on the current image and the offline map further includes:

[0100] The ninth 2D feature point of the current image is matched with the 2D feature point in the local map to obtain the sixth 2D feature point of the local map; the 3D point corresponding to the sixth 2D feature point is obtained based on the sixth 2D feature point, the ninth 2D feature point in the current image that matches the sixth 2D feature point, the pose of the local map and the first pose of the current image.

[0101] The current image is projected based on its first pose, the corresponding 3D point cloud of the local map, and the 3D point corresponding to the eleventh 2D feature point of the current image to obtain a second depth map, including:

[0102] The current image is projected based on the first pose of the current image, the 3D point cloud corresponding to the local map, the 3D point corresponding to the eleventh 2D feature point of the current image, and the 3D point corresponding to the sixth 2D feature point of the current image to obtain a second depth map.

[0103] The second depth map is obtained by projecting the current image onto its first pose, the corresponding 3D point cloud of the local map, the 3D point corresponding to the eleventh 2D feature point, and the 3D point corresponding to the sixth 2D feature point. Specifically, this means:

[0104] Based on the first pose of the current image, the 3D point cloud corresponding to the local map, the 3D point corresponding to the eleventh 2D feature point of the current image, and the 3D point corresponding to the sixth 2D feature point are projected onto the imaging plane of the current image to obtain the second depth map.

[0105] By introducing 3D points obtained from the ninth 2D feature point of the current image and the tenth 2D feature point of the historical image, and 3D points obtained from the ninth 2D feature point of the current image and the sixth 2D feature point of the local map, the second depth map is obtained by projecting the 3D points obtained from the 2D feature points of the current image and the tenth 2D feature point of the historical image, and the 3D points obtained from the ninth 2D feature point of the current image and the sixth 2D feature point of the local map together with the corresponding 3D point cloud of the local map, thereby further enriching the depth information representing the current image described in the second depth map.

[0106] In one feasible embodiment, the offline map includes multiple frames of base map, and the method of this application further includes:

[0107] Based on the current image, M base maps are obtained from multiple base maps, where the similarity between each base map and the current image is greater than a first threshold. The ninth 2D feature point of the current image is matched with the twelfth 2D feature point in the M base maps to obtain the seventh 2D feature point in the base maps. From the 3D point cloud corresponding to the local map, the 3D points corresponding to the seventh 2D feature point are selected to obtain the processed 3D points. The ninth 2D feature point of the current image is matched with the tenth 2D feature point in multiple historical images to obtain the eleventh 2D feature point of the current image. The eleventh 2D feature point is the 2D feature point in the current image that matches the eleventh 2D feature point in the historical image among the ninth 2D feature points. Based on the eleventh 2D feature point, the first pose of the current image, the 2D feature point in multiple historical images that matches the eleventh 2D feature point, and the first pose of the historical image to which the 2D feature point belongs, the 3D point corresponding to the eleventh 2D feature point is obtained. The processed 3D point and the 3D point corresponding to the eleventh 2D feature point are then processed based on the first pose of the current image to obtain the updated 3D point cloud corresponding to the local map.

[0108] The current image is projected onto the first pose of the current image and the corresponding 3D point cloud of the local map to obtain a second depth map, including:

[0109] The current image is projected using the first pose of the current image and the 3D point cloud corresponding to the updated local map to obtain a second depth map.

[0110] Specifically, the second depth map is obtained by projecting the current image onto the imaging plane of the current image based on the first pose of the current image and the 3D point cloud corresponding to the updated local map.

[0111] Optionally, the eleventh 2D feature point can be a SIFT feature point, a SURF feature point, or a SuperPoint feature point, ASLFeat feature point, R2D2 feature point, or D2Net feature point obtained based on deep learning.

[0112] Because the offline map on the server is collected offline, it may differ from the actual environment. For example, a large billboard in a shopping mall might have been present during offline data collection, but has been removed by the time the user captures the current image. This results in the server-deployed map containing inconsistent 3D point cloud information. Furthermore, the images received by the server may have undergone privacy processing, also leading to inconsistent 3D point cloud information in the deployed map. To address these issues, the server updates the 3D point cloud corresponding to the deployed local map, obtaining an updated 3D point cloud. Then, based on this updated 3D point cloud, a target depth map is generated, further enriching the depth information representing feature points in the current image as described in the second depth map.

[0113] In one feasible embodiment, the method of this application further includes:

[0114] The system receives a third acquisition request from the terminal device, which carries the current image; performs feature point matching based on the current image and the offline map to determine the first pose of the current image, which is the pose of the current image in a first coordinate system, which is the coordinate system of the offline map; and sends a third response message to the terminal device in response to the third acquisition request, which carries the first pose of the current image.

[0115] In one feasible embodiment, the method of this application further includes:

[0116] The system receives a fourth acquisition request from a terminal device, the fourth acquisition request carrying the current image and the second pose of the current image; the second pose of the current image is the pose of the terminal device when capturing the current image; feature point matching is performed based on the current image and an offline map to determine the first pose of the current image, the first pose being the pose of the current image in a first coordinate system, the first coordinate system being the coordinate system of the offline map; pose transformation information is determined based on the pose of the current image and the first pose, the pose transformation information being used for the transformation between the pose of the current image and the first pose; and a fourth response message is sent to the terminal device in response to the fourth acquisition request, the fourth response message carrying the pose transformation information.

[0117] In one feasible embodiment, feature point matching is performed based on the current image and an offline map to determine the first pose of the current image, including:

[0118] A local map is obtained from the offline map based on the position in the second pose of the current image; the local map is the region indicated by the position in the second pose of the current image in the offline map; feature point matching is performed based on the current image and the local map to determine the first pose.

[0119] After acquiring the first pose or position transformation information, the terminal device obtains the first pose or pose transformation information of the current image to align the coordinate system of the current image with the coordinate system of the offline map, facilitating subsequent calculations.

[0120] Thirdly, embodiments of this application provide an electronic device, including a memory and one or more processors; wherein one or more programs are stored in the memory; and when the one or more processors execute the one or more programs, the electronic device causes the electronic device to implement part or all of the methods described in the first or second aspect.

[0121] Fourthly, embodiments of this application provide a computer storage medium, characterized in that it includes computer instructions that, when executed on an electronic device, cause the electronic device to perform part or all of the method described in the first or second aspect.

[0122] Fifthly, embodiments of this application provide a computer program product, characterized in that, when the computer program product is run on a computer, it causes the computer to perform part or all of the methods as described in the first or second aspect.

[0123] In a sixth aspect, a terminal device is provided, the terminal device including a module for performing the method in the first aspect.

[0124] In a seventh aspect, a server is provided, the server including a module for performing the methods of the second aspect.

[0125] It should be understood that any of the above possible implementation methods can be freely combined without violating the laws of nature, and will not be elaborated upon in this application.

[0126] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0127] Figure 1a This is a schematic diagram illustrating the effect without any obstruction between solid and void.

[0128] Figure 1b This is a schematic diagram illustrating the effect of occlusion.

[0129] Figure 1c A system architecture diagram is provided for an embodiment of this application;

[0130] Figure 1d This is a schematic diagram of a CNN structure;

[0131] Figure 1e This application provides a schematic diagram of a chip hardware structure.

[0132] Figure 1f Another system architecture diagram is provided for the embodiments of this application;

[0133] Figure 2 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0134] Figure 3 A schematic flowchart illustrating an image processing method provided in an embodiment of this application;

[0135] Figure 3A A schematic diagram of a first depth map, a second depth map, and a target depth map provided for embodiments of this application;

[0136] Figure 4 This is a schematic diagram illustrating the relationship between the base map, the local map, and the processed local map.

[0137] Figure 5 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0138] Figure 6 This is a schematic diagram illustrating the effect of virtual and real occlusion using an embodiment of this application;

[0139] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0140] Figure 8 This is a schematic diagram of the structure of another terminal device provided in an embodiment of this application;

[0141] Figure 9 A schematic diagram of a system structure provided in an embodiment of this application;

[0142] Figure 10 Another system structure diagram provided for an embodiment of this application;

[0143] Figure 11 This is a schematic diagram of the structure of another terminal device provided in an embodiment of this application;

[0144] Figure 12 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0145] Figure 13 This is a schematic diagram of the structure of another terminal device provided in an embodiment of this application;

[0146] Figure 14 This is a schematic diagram of another server structure provided in an embodiment of this application. Detailed Implementation

[0147] The technical solutions in the embodiments of this application will now be described clearly and in detail with reference to the accompanying drawings.

[0148] In the following descriptions, terms such as "first," "second," and similar words are used for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0149] The pose in this application refers to a 6-degree-of-freedom (DOF) pose, including 3 rotation angles (pitch, roll, and yaw) and 3 position-related degrees of freedom. The 3 position-related degrees of freedom can be collectively referred to as geographic location information, or simply position. This position can be obtained based on GPS, or based on BeiDou or other positioning systems. The 3 rotation angles can be collectively referred to as orientation information.

[0150] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0151] (1) Neural Network

[0152] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0153]

[0154] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to perform a non-linear transformation on the features acquired in the neural network, converting the input signal of the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0155] (2) Deep Neural Networks

[0156] A deep neural network (DNN) can be understood as a neural network with many hidden layers. There's no specific metric for "many" layers; the commonly used terms "multi-layer neural network" and "deep neural network" are essentially the same thing. DNNs can be categorized into three layers based on their position: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs seem complex, the operation of each layer is actually not complicated; it can be simply described by the following linear relationship expression: in It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number is quite large. So, how are the specific parameters defined in a DNN? First, let's look at the definition of the coefficient W. Taking a three-layer DNN as an example, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as... Note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks.

[0157] (3) Convolutional Neural Network

[0158] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0159] (4) Loss Function

[0160] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0161] The system architecture provided in the embodiments of this application is described below.

[0162] See appendix Figure 1cThis application provides a system architecture. As shown in the system architecture, the data acquisition device 160 is used to collect training data. Exemplarily, the training data in this application embodiment may include: image samples, depth map samples, and real depth maps; after collecting the training data, the data acquisition device 160 stores this training data in the database 130, and the training device 120 trains the depth estimation model 101 based on the training data maintained in the database 130.

[0163] The following describes the depth estimation model 101 obtained by the training device 120 based on the training data. For example, the training device 120 processes image samples and depth map samples, calculates the loss value based on the output predicted depth map and the true depth map, and the loss function, until the calculated loss value converges, thereby completing the training of the depth estimation model 101.

[0164] The depth estimation model 101 can be used to implement the image processing method provided in the embodiments of this application. That is, the current image, the first depth map, and the second depth map are preprocessed and then input into the depth estimation model 101, which is the target depth map of the current image. Specifically, the depth estimation model 101 in the embodiments of this application can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the depth estimation model 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0165] The depth estimation model 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1c The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, AR virtual reality (VR) terminal, vehicle terminal, etc., or it can be a server or cloud, etc. (See attached...) Figure 1c In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include: the current image, or the current image and the first depth map.

[0166] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0167] Finally, I / O interface 112 returns the processing result, such as the target depth map of the current image obtained above, to client device 140, thereby providing it to the user.

[0168] It is worth noting that the training device 120 can generate corresponding depth estimation models 101 based on different training data for different objectives or tasks. The corresponding depth estimation model 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0169] In the appendix Figure 1c In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0170] It is worth noting that, attached Figure 1c This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 1c In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0171] like Figure 1cAs shown, a depth estimation model 101 is obtained by training the training device 120. In this embodiment of the application, the depth estimation model 101 can be the neural network in this application. Specifically, the neural network in this application can include CNN or deep convolutional neural networks (DCNN), etc.

[0172] Since CNN is a common type of neural network, the following will combine... Figure 1d This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0173] like Figure 1d As shown, a convolutional neural network (CNN) may include an input layer 11, a convolutional / pooling layer 12 (where the pooling layer is optional), a neural network layer 13, and an output layer 14.

[0174] Convolutional / pooling layer 12:

[0175] Convolutional layers:

[0176] like Figure 1d The convolutional / pooling layer 12 shown may include layers as in Examples 121-126. For example, in one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0177] The following section will use convolutional layer 121 as an example to introduce the internal working principle of a convolutional layer.

[0178] Convolutional layer 121 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During convolution, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0179] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 10 to make correct predictions.

[0180] When a convolutional neural network 10 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 10 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0181] Pooling layer:

[0182] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 1dLayers 121-126 in Example 12 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input pooling layer image. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input pooling layer image.

[0183] Neural network layer 13:

[0184] After processing by the convolutional / pooling layers 12, the convolutional neural network 10 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 12 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 10 needs to utilize neural network layer 13 to generate one or a set of required class numbers of output. Therefore, neural network layer 13 can include multiple hidden layers (such as...). Figure 1d As shown in 131, 132 to 13n), the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0185] After the multiple hidden layers in neural network layer 13, the final layer of the entire convolutional neural network 10 is the output layer 14. This output layer 14 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 10 has propagated forward (e.g., ... Figure 1d Propagation from directions 11 to 14 is forward propagation, and backward propagation (such as...) is completed. Figure 1d The propagation from 14 to 11 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 10 and the error between the output of the convolutional neural network 10 through the output layer and the ideal result.

[0186] It should be noted that, as Figure 1dThe convolutional neural network 10 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, for example, including only... Figure 1d As shown in the network structure, for example, the convolutional neural network used in the embodiments of this application may only include an input layer 11, a convolutional / pooling layer 12, and an output layer 14.

[0187] The following describes a chip hardware structure provided by an embodiment of this application.

[0188] Figure 1e This application provides a chip hardware structure, which includes a neural network processor 30. The chip can be configured as follows: Figure 1c The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 1c The training device 120 shown is used to complete the training work of the training device 120 and output the depth map estimation model 101. For example... Figure 1d The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 1f The image fusion method and the training method of the image fusion model in the embodiments of this application can both be implemented in, for example, the chip shown. Figure 1f This is achieved in the chip shown.

[0189] The neural network processor 30 can be any processor suitable for large-scale XOR operations, such as a neural network processing unit (NPU), tensor processing unit (TPU), or graphics processing unit (GPU). Taking an NPU as an example: the neural network processor NPU30 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the main CPU. The core of the NPU is the arithmetic circuit 303, and the controller 304 controls the arithmetic circuit 303 to retrieve data from memory (weight memory or input memory) and perform operations. The TPU is a Google-customized application-specific integrated circuit for machine learning artificial intelligence accelerators.

[0190] In some implementations, the arithmetic circuit 303 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0191] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 303 retrieves the weight data of matrix B from the weight memory 302 and caches it in each PE (Engineer Component) of the arithmetic circuit 303. The arithmetic circuit 303 retrieves the input data of matrix A from the input memory 301, performs matrix operations based on the input data of matrix A and the weight data of matrix B, and stores the partial or final result of the obtained matrix in the accumulator 308.

[0192] The vector computation unit 307 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 307 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0193] In some implementations, vector computation unit 307 can store the processed output vector into unified memory 306. For example, vector computation unit 307 can apply a nonlinear function to the output of arithmetic circuit 303, such as an accumulated value vector, to generate activation values. In some implementations, vector computation unit 307 generates normalized values, merged values, or both. In some implementations, vector computation unit 307 stores the processed vector into unified memory 306. In some implementations, the vector processed by vector computation unit 307 can be used as activation input to arithmetic circuit 303, for example, for use in subsequent layers of a neural network. Figure 1d As shown, if the current processing layer is hidden layer 1 (131), the vector processed by the vector calculation unit 307 can also be used in the calculation of hidden layer 2 (132).

[0194] The unified memory 306 is used to store input data and output data.

[0195] Weight data is stored directly into weight memory 302 via direct memory access controller (DMAC) 305. Input data is also stored into unified memory 306 via DMAC.

[0196] The bus interface unit (BIU) 310 is used for interaction between the DMAC and the instruction fetch buffer 309; the bus interface unit 310 is also used for the instruction fetch buffer 309 to fetch instructions from external memory; the bus interface unit 310 is also used for the memory access controller 305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0197] The DMAC 305 is mainly used to store input data from external memory DDR into unified memory 306, or to store weight data into weight memory 302, or to store input data into input memory 301.

[0198] The instruction fetch memory 309, connected to the controller 304, is used to store the instructions used by the controller 304.

[0199] The controller 304 is used to call the instructions cached in the instruction fetch memory 309 to control the operation of the computing accelerator.

[0200] Generally, the unified memory 306, input memory 301, weight memory 302, and instruction fetch memory 309 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0201] in, Figure 1d The operations of each layer in the convolutional neural network shown can be performed by the computation circuit 303 or the vector calculation unit 307. For example, the training method of the depth estimation model and the related methods for determining the target depth map in the embodiments of this application can both be performed by the computation circuit 303 or the vector calculation unit 307.

[0202] like Figure 1f As shown, this application embodiment provides another system architecture. This system architecture includes local device 401, local device 402, and... Figure 1cThe execution device 110 and data storage system 150 shown are connected to the execution device 110 via a communication network.

[0203] The execution device 110 can be implemented by one or more servers. Optionally, the execution device 110 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 110 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 110 can use data in the data storage system 150 or call program code in the data storage system 150 to implement the training method of the time series prediction model of this application embodiment.

[0204] Specifically, in one implementation, the execution device 110 can perform the following process:

[0205] Multiple depth map samples corresponding to multiple image samples are input into an initial convolutional neural network for processing to obtain multiple first predicted depth maps. A first loss value is calculated based on the multiple first predicted depth maps, the corresponding ground truth depth maps of the multiple image samples, and a loss function. The parameters in the initial convolutional neural network are adjusted based on the first loss value to obtain a first convolutional neural network. Then, multiple depth map samples corresponding to multiple image samples are input into the first convolutional neural network for processing to obtain multiple second predicted depth maps. A second loss value is obtained based on the multiple second predicted depth maps, the corresponding ground truth depth maps of the multiple image samples, and a loss function. It is determined whether the second loss value converges. If it converges, the first convolutional neural network is determined as the depth estimation model for the current image. If it does not converge, the parameters in the first convolutional neural network are adjusted based on the second loss value to obtain a second convolutional neural network. This process is repeated until the obtained loss value converges, and the convolutional neural network with converged loss value is determined as the depth estimation model for the current image.

[0206] The device 110 can obtain a depth estimation model through the above process, which can be used to obtain a target depth map of the current image.

[0207] Users can interact with execution device 110 by operating their respective user devices (e.g., local device 401 and local device 402). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0208] Each user's local device can interact with the execution device 410 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0209] In one implementation, local devices 401 and 402 obtain a depth estimation model from execution device 110, deploy the depth estimation model on local devices 401 and 402, and use the depth estimation model to perform depth estimation.

[0210] In another implementation, a depth estimation model can be directly deployed on execution device 110. Execution device 410 obtains the current image and the first depth map from local devices 401 and 402, and uses the depth estimation model to estimate the depth of the current image and the first depth map to obtain the target depth map of the current image.

[0211] The aforementioned execution device 110 can also be a cloud device, in which case the execution device 110 can be deployed in the cloud; or, the aforementioned execution device 110 can also be a terminal device, in which case the execution device 110 can be deployed on the user terminal side. This application embodiment does not limit this.

[0212] The application scenarios of this application are described below. For example... Figure 2 As shown, this application scenario includes terminal device 100 and server 200.

[0213] Among them, the terminal device 100 can be a smartphone, tablet, AR glasses or other smart device.

[0214] Server 200 can be a desktop server, rack server, cabinet server, blade server, or other types of server.

[0215] Terminal device 100 acquires the current image and calculates a first depth map corresponding to the current image; it then acquires a second depth map corresponding to the current image from the server; the second depth map represents more depth information of feature points in the current image than the first depth map; based on the current image, the first depth map, and the second depth map, a target depth map of the current image is obtained, which represents more depth information of feature points in the current image than the second depth map; finally, a target image is obtained based on the current image, the target depth map, a virtual object image, and the depth map of the virtual object, wherein the target depth map and the depth map of the virtual object are used to determine the presentation and distribution of pixels in the current image and the virtual object image in the target image, thereby achieving virtual-real occlusion.

[0216] The following details how the aforementioned terminal device 100 and server 200 achieve virtual-real occlusion.

[0217] See Figure 3 , Figure 3 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 3 As shown, the method includes:

[0218] S301. Obtain the current image, calculate the first depth map corresponding to the current image based on the current image, and obtain the second depth map corresponding to the current image from the server based on the current image.

[0219] The second depth map represents more depth information of feature points in the current image than the first depth map, and the first depth map contains depth information of feature points that are not present in the second depth map.

[0220] Optionally, the current image can be an RGB image, a grayscale image, or an image of other forms.

[0221] The current image is acquired in real time from the camera of the terminal device, or from images stored in the terminal device, or from other devices; no specific limitation is made here.

[0222] Optionally, a second depth map corresponding to the current image is obtained from the server based on the current image, including:

[0223] Method 1: Send a first acquisition request to the server to obtain the 3D point cloud corresponding to the local map; receive a first response message from the server in response to the first acquisition request, the first response message carrying the 3D point cloud corresponding to the local map, the local map including scene data corresponding to the current image; acquire the second pose of the current image, the second pose of the current image being the pose of the terminal device when capturing the current image; project the current image according to the second pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map corresponding to the current image.

[0224] Wherein, the local map is the area indicated by the first geographical location information in the offline map stored on the server; the first geographical location information is whether the geographical location is required in the first pose of the current image, the first pose of the current image is the pose of the current image in the first coordinate system, the first pose of the current image is obtained by processing the current image based on VPS technology; the first coordinate system is the coordinate system where the offline map is located.

[0225] Specifically, the server, based on VPS technology, obtains the first pose of the current image in the coordinate system of the offline map. The server then retrieves a local map from the offline map based on the position in the first pose. In one example, the local map is a region within a certain range (e.g., a radius of 50 meters) centered on that position in the offline map. Since the offline map includes 2D feature points and their corresponding feature descriptors, the local map also includes 2D feature points and their corresponding feature descriptors. The server also stores 3D points corresponding to the 2D feature points. The server retrieves the 3D points corresponding to the 2D feature points in the local map, i.e., the 3D point cloud corresponding to the local map. The server sends a first response message to the terminal device, which carries the 3D point cloud corresponding to the local map. The terminal device projects the 3D point cloud corresponding to the local map onto the imaging plane of the current image based on the second pose of the current image to obtain a second depth map.

[0226] It should be noted that the server stores offline maps, which include panoramic maps, 2D feature points in the panoramic maps, and 3D features corresponding to the 2D feature points. The panoramic maps are obtained based on multiple frames of base maps.

[0227] Because 3D points also exist in non-feature point areas in offline maps, the 3D point cloud in offline maps is dense. For example, if an offline map is acquired using a calibrated laser + panoramic camera device, the 3D points in the offline map come from the laser point cloud, and the 2D feature points in the offline map come from the panoramic image.

[0228] It should be noted that the feature points in this application refer to points where the grayscale value of the image changes drastically or points with large curvature on the edge of the image (i.e., the intersection of two edges).

[0229] Optionally, a second depth map corresponding to the current image is obtained from the server based on the current image, including:

[0230] Method 2: Send a first acquisition request to the server to obtain the 3D point cloud corresponding to the local map; receive a first response message from the server in response to the first acquisition request, the first response message carrying the 3D point cloud corresponding to the local map, the local map being the area indicated by the first position in the offline map stored by the server; the first position is the position of the first pose of the current image, the first pose of the current image is the pose of the current image in the first coordinate system, the first pose of the current image is obtained based on the current image; the first coordinate system is the coordinate system of the offline map; perform feature extraction on the current image to obtain the first 2D feature points of the current image; compare the first 2D feature points of the current image with the eighth 2D feature points in multiple historical images. Points are matched to obtain the second 2D feature points of the current image. The second 2D feature points are the 2D feature points of the first 2D feature points of the current image that match the eighth 2D feature point in the historical image. Based on the second pose of the current image, the second 2D feature points of the current image, the 2D feature points of the eighth 2D feature points in the historical image that match the second 2D feature points, and the second pose of the historical image, the 3D points corresponding to the second 2D feature points of the current image are obtained. The second pose of the current image is the pose of the terminal device when capturing the current image. Based on the second pose of the current image, the 3D point cloud corresponding to the local map, and the 3D points corresponding to the second 2D feature points of the current image, the current image is projected to obtain the second depth map.

[0231] Specifically, the process by which the server obtains the 3D point cloud corresponding to the local map can be found in the relevant description above, and will not be repeated here. In order to obtain a second depth map that represents the depth information of the current image more richly, the terminal device matches the first 2D feature points of the current image with the eighth 2D feature points of multiple historical images to obtain the second 2D feature points of the current image. The second 2D feature points are the 2D feature points of the first 2D feature points of the current image that match the eighth 2D feature points in the historical images. The terminal device determines the relative pose between the second pose of the current image and the second pose of the historical images. Based on the relative pose, the second 2D feature points of the current image and the eighth 2D feature points in the historical images that match the second 2D feature points are triangulated to obtain the 3D points corresponding to the second 2D feature points of the current image. Based on the second pose of the current image, the 3D point cloud corresponding to the local map and the 3D points corresponding to the second 2D feature points are projected onto the imaging plane of the current image to obtain the second depth map.

[0232] Optionally, the first 2D feature point of the current image and the eighth 2D feature point in the historical image can be ORB feature points, AKAZE feature points, ASLFeat feature points or superpoint feature points obtained by knowledge distillation or network search based on deep learning, or DOG feature points, HOG feature points, BRIEF feature points, BRISK feature points or FREAK feature points.

[0233] Due to limitations in edge computing power and the need for real-time performance, it is necessary to quickly extract features from images, and the number of features to be extracted is relatively small. Generally, the ORB method or an improved ORB method is used to extract features from regions with rich texture in the image. The features extracted by this method are ORB feature points, and the number is usually several hundred.

[0234] It should be noted that AKAZE feature points can be obtained by extracting features from the image using the AKAZE feature extraction method. Other feature points in this application can be obtained by extracting features from the image using corresponding methods, which will not be described further here.

[0235] The aforementioned historical images are images uploaded to the server by the terminal device before the current image; optionally, the first response message also carries the first pose of the current image; similarly, the terminal device can also obtain the first poses of multiple historical images; when determining the 3D point corresponding to the second 2D feature point of the current image, the terminal device determines the relative pose between the first pose of the current image and the first pose of the historical image; based on the relative pose, the second 2D feature point of the current image and the eighth 2D feature point in the historical image that matches the second 2D feature point are triangulated to obtain the 3D point corresponding to the second 2D feature point of the current image.

[0236] As can be seen from the above, when determining the second depth map, the 3D points corresponding to the second 2D feature points of the current image are introduced. Compared with the first method, more 3D points are introduced. Therefore, the second depth map obtained by the second method represents more depth information of the current image than the second depth map obtained by the first method.

[0237] The matching of 2D feature points as mentioned in this application specifically means that the similarity between two matching 2D feature points is higher than a preset similarity; or that the Euclidean distance between two matching 2D feature points is lower than a preset distance.

[0238] It should be noted that the 3D point cloud corresponding to the local map obtained by the terminal device can be obtained by the server according to Method 1, or it can be the updated 3D point cloud corresponding to the local map obtained by the server after updating the 3D point cloud obtained by the server according to Method 1. For details of the update process, please refer to the relevant description of the server-side embodiment.

[0239] Optionally, a second depth map is obtained from the server based on the current image, including:

[0240] Method 3: Send a second acquisition request to the server, the second acquisition request carrying the current image, the second acquisition request instructing the server to obtain the second depth map based on the current image and the offline map stored by the server; receive a second response message from the server in response to the second acquisition request, the second response message carrying the second depth map.

[0241] In method three, the second depth map is calculated by the server, which does not require the terminal device to consume computing resources to calculate it, thus reducing the resource consumption and power consumption of the terminal device.

[0242] Optionally, a first depth map corresponding to the current image is calculated based on the current image, including:

[0243] The process involves: obtaining a first pose, which is the pose of the current image in a first coordinate system; processing the current image based on the first pose to obtain a first image; transforming the second pose of the current image into the first pose; extracting features from the first image to obtain a third 2D feature point; the second pose of the current image is the pose of the terminal device when capturing the current image; matching the third 2D feature point with a pre-stored eighth 2D feature point to obtain a fourth 2D feature point; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the pre-stored eighth 2D feature points from multiple historical images; obtaining a 3D point corresponding to the fourth 2D feature point based on the fourth 2D feature point, the first pose, the 2D feature point among the pre-stored eighth 2D feature points that matches the fourth 2D feature point, and the first pose of the image to which the 2D feature point belongs; and projecting the current image based on the first pose of the current image and the 3D point corresponding to the fourth 2D feature point to obtain a first depth map.

[0244] The pre-stored 2D feature points can be 2D feature points from at least one historical image. The at least one historical image packet consists of one or more images whose timestamps differ from the timestamp of the terminal device and the current image are small, or one or more images whose timestamps precede the timestamp of the current image. Furthermore, the multiple images can be consecutive frames or non-consecutive frames. In one example, the timestamp of an image can be the time when the image was captured; however, it can also be other times, which are not limited here.

[0245] Specifically, after obtaining the third 2D feature point of the first image, the third 2D feature point is matched with pre-stored 2D feature points to obtain the fourth 2D feature point of the first image; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the pre-stored 2D feature points; the fourth 2D feature point can be from the same image or from different images; for the fourth 2D feature point and the 2D feature point among the pre-stored 2D feature points that matches the fourth 2D feature point, firstly, the pose of the image to which the fourth 2D feature point belongs is determined, then the relative pose between the pose and the first pose is determined, and finally, based on the phase pose, the fourth 2D feature point and the 2D feature point among the pre-stored 2D feature points that matches the fourth 2D feature point are triangulated to obtain the 3D point corresponding to the fourth 2D feature point; according to the first pose of the current image, the 3D point corresponding to the fourth 2D feature point is projected onto the imaging plane of the current image to obtain the first depth map.

[0246] Optionally, the third 2D feature point can be an ORB feature point, an AKAZE feature point, an ASLFeat feature point or superpoint feature point obtained by knowledge distillation or network search based on deep learning, or a DOG feature point, HOG feature point, BRIEF feature point, BRISK feature point or FREAK feature point.

[0247] Optionally, obtaining the first pose includes:

[0248] Send a third acquisition request to the server, the third acquisition request including the current image, so that the server obtains the first pose based on the current image; receive a third response message sent by the server in response to the third acquisition request, the third response message carrying the first pose.

[0249] Optionally, obtaining the first pose includes:

[0250] A fourth acquisition request is sent to the server, which carries the current image and the second pose of the current image. A fourth response message is received from the server in response to the fourth acquisition request. The fourth response message carries pose transformation information, which is used for the transformation between the second pose and the first pose of the current image. The second pose of the current image is transformed according to the pose transformation information to obtain the first pose.

[0251] It should be noted that pose transformation information is essentially a matrix.

[0252] Optionally, some or all of the first, second, third, and fourth fetch requests can be the same fetch request; that is, one fetch request implements some or all of the functions of the first, second, third, and fourth fetch requests. Correspondingly, some or all of the first, second, third, and fourth response messages can be the same response message.

[0253] The second depth map represents more depth information of feature points in the current image than the first depth map, specifically in the following three aspects:

[0254] First, the number of pixels in the second depth map is greater than the number of pixels in the first depth map.

[0255] The first depth map is obtained by the terminal device based on feature points extracted from the image. Some pixels of the second depth map are obtained based on offline maps pre-stored in the server, while other pixels are also obtained based on feature points extracted from the image.

[0256] Due to the difference in computing power between terminal devices and servers, for the same image, the number of 2D feature points extracted by the terminal device is lower than that extracted by the server. Generally, the terminal device extracts a few hundred 2D feature points, while the server extracts tens of thousands. For 3D features that correspond one-to-one with the 2D feature points, the number of 3D points obtained by the terminal device based on the extracted 2D feature points is less than the number of 3D points obtained by the server. Therefore, the number of pixels in the depth map obtained by projecting the 3D points determined by the server is greater than the number of pixels in the depth map obtained by projecting the 3D points determined by the terminal device. Furthermore, the second depth map also includes pixels obtained from offline maps pre-stored in the server. Thus, it can be seen that the number of pixels in the second depth map is greater than the number of pixels in the first depth map.

[0257] Second, the pixel precision in the second depth map is higher than that in the first depth map.

[0258] The second depth map is obtained from the offline map pre-stored on the server. The 3D points in the offline map are determined based on the calibrated laser equipment, while the first depth map is obtained by matching feature points between the current image and historical images. Therefore, the pixel accuracy in the second depth map is higher than that in the first depth map.

[0259] Third, the distribution of pixels in the second depth map is more uniform than that in the first depth map.

[0260] For the same image, i.e. the current image, since the number of pixels in the second depth image is greater than the number of pixels in the first depth image, the distribution of pixels in the second depth image is more uniform than the distribution of pixels in the first depth image.

[0261] Specifically, the first depth map contains depth information of feature points that the second depth map does not have. This means that the number of pixels in the union of the first and second depth maps is greater than the number of pixels in the second depth map. The pixels in the target depth map can be regarded as the union of the pixels in the first and second depth maps. The number of pixels in the target depth map is greater than the number of pixels in the second depth map. Therefore, the target depth map represents the depth information of feature points in the current image more comprehensively than the second depth map.

[0262] Optionally, for any pixel position S, when the pixel value at pixel position S in the first depth map is 0 or the pixel value at pixel position S in the second depth map is 0, the pixel value at pixel position S in the target depth map is the maximum of the pixel values ​​at pixel position S in the first depth map and the pixel values ​​at pixel position S in the second depth map; when neither the pixel value at pixel position S in the first depth map nor the pixel value at pixel position S in the second depth map is 0, the pixel value at pixel position S in the target depth map is the average of the sum of the pixel values ​​at pixel position S in the first depth map and the pixel values ​​at pixel position S in the second depth map, or the weighted average of the two.

[0263] S302. Obtain the target depth map of the current image based on the current image, the first depth map, and the second depth map.

[0264] The target depth map represents more depth information of feature points in the current image than the second depth map.

[0265] like Figure 3AAs shown, image a is the first depth map, image b is the second depth map, and image c is the target depth map. The four-pointed stars in the images represent pixels in the depth maps. The gray four-pointed stars in image a do not exist at the corresponding positions in image b, indicating that the first depth map contains depth information that is not present in the second depth map. Image c can be seen as a superposition of images a and b, meaning that the number of pixels in the target depth map is greater than the number of pixels in the second depth map.

[0266] In a feasible embodiment, obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map specifically involves:

[0267] A fifth acquisition request is sent to the server; the fifth acquisition request carries the terminal's geographical location information; a fifth response message is received from the server in response to the fifth acquisition request, the fifth response message carrying a depth estimation model; the depth estimation model is a neural network model corresponding to the terminal's geographical location information; the first depth map and the second depth map are stitched together to obtain a third depth map; the current image and the third depth map are input into the depth estimation model to obtain the target depth map.

[0268] Since the images are different at different locations, in order to improve the accuracy of the target depth map, the server determines a depth estimation model for each location. When the depth estimation model is needed, the server determines the corresponding depth estimation model based on the location of the terminal device, and obtains the target depth map based on the depth estimation model. This further improves the density of pixels in the target depth map of the current image, that is, it enriches the depth information of the feature points in the current image represented by the target depth map.

[0269] It should be noted that the specific implementation process of obtaining the target depth map of the current image using the depth estimation model can be found in the relevant descriptions above, and will not be repeated here. The aforementioned depth estimation model is based on a neural network, such as a convolutional neural network, which is not limited to this model.

[0270] Since the images differ depending on the location, the server determines a depth estimation model for each location to improve the accuracy of the target depth map. This depth estimation model can be trained by the server or trained on other devices and then obtained by the server from those devices.

[0271] By obtaining the depth estimation model from the server, the terminal device does not need to train a depth estimation model, which reduces the power consumption of the terminal device and improves the real-time performance of virtual and real occlusion.

[0272] In a feasible embodiment, obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map specifically involves:

[0273] The initial convolutional neural network model is trained to obtain a depth estimation model; the first depth map and the second depth map are concatenated to obtain a third depth map; the current image and the third depth map are input into the depth estimation model to obtain the target depth map;

[0274] The initial convolutional neural network is trained to obtain a depth estimation model, including:

[0275] Multiple image samples and their corresponding depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps. A loss value is calculated based on the multiple predicted depth maps, the corresponding ground truth depth maps, and a loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain a depth estimation model for the current image. The loss function is determined based on the error between the predicted and ground truth depth maps, the error between the gradients of the predicted and ground truth depth maps, and the error between the normal vectors of the predicted and ground truth depth maps.

[0276] The loss function is as follows:

[0277]

[0278] in The depth map d output by the depth estimation model at scale i. i Corresponding ground truth depth map g i The error, The depth map gradient dg output by the depth estimation model at scale i represents the depth map gradient. i Gradient of the corresponding ground truth depth map (gg) i The error, The depth estimation model at scale i outputs a depth map normal vector dn. i With the corresponding ground truth depth map normal vector gn i The error.

[0279] Alternatively, the depth estimation model can employ network structures such as DiverseDepth, SARPN, or CSPN.

[0280] The feature extraction function in the aforementioned depth estimation model can be implemented using network structures such as VGGNet, ResNet, ResNeXt, and DenseNet.

[0281] VGGNet: Uses 3x3 convolutional kernels and 2x2 pooling kernels throughout, improving performance by progressively deepening the network structure. For VGG-16, the input is a 224x224 RGB image. During preprocessing, the average value of the three channels is calculated, and this average value is subtracted from each pixel (resulting in fewer iterations and faster convergence). The image is processed through a series of convolutional layers, using very small 3x3 kernels. 3x3 kernels are chosen because it is the smallest size capable of capturing 8-neighborhood information. The stride of the convolutional layers is set to 1 pixel, and the padding of the 3x3 convolutional layers is set to 1 pixel. Max pooling is used in five layers. After some convolutional layers, the max-pooling window is 2x2 with a stride of 2. Following the convolutional layers are three fully-connected (FC) layers. The first two fully connected layers each have 4096 channels, and the third fully connected layer has 1000 channels, used for classification. All fully connected layers in the network have the same configuration. Following the fully connected layers is a Softmax layer, also used for classification. All hidden layers (between each convolutional layer) use ReLU as the activation function.

[0282] ResNet is a type of residual network. It can be understood as a subnetwork that can be stacked to form a very deep network. Residual networks are characterized by their ease of optimization and the ability to improve accuracy by significantly increasing depth. The residual blocks within them use skip connections, mitigating the vanishing gradient problem that arises when increasing depth in deep neural networks.

[0283] ResNeXt builds upon the ideas of ResNet, proposing a structure that improves accuracy without increasing parameter complexity while reducing the number of hyperparameters. It borrows the idea of ​​Inception to extend network width, using multiple branches to learn different features. It replaces the original three-layer convolutional blocks of ResNet with parallel stacked blocks of the same topology. This improves model accuracy without significantly increasing the number of parameters. Furthermore, because the topology remains the same, the number of hyperparameters is reduced, making the model easier to port and leading to its popularity as a framework for recognition tasks.

[0284] DenseNet (Densely Connected Network): In traditional convolutional networks, each layer uses only the output features of the previous layer as its input. In DenseNet, each layer uses the features of all preceding layers as its input, and its own features as the input of all subsequent layers. DenseNet has several advantages: it alleviates the problem of gradient vanishing, enhances feature propagation, encourages feature reuse, and significantly reduces the number of parameters.

[0285] By adopting a multi-scale feature fusion strategy, incorporating temporal consistency and scale consistency constraints, and using joint training with multiple datasets, the generalization ability of the model in different scenarios is improved.

[0286] Optionally, a target depth map of the current image is obtained based on the current image, the first depth map, and the second depth map, including:

[0287] Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; the resolution of each of the T first feature maps is different, and the resolution of each of the T second feature maps is different; T is an integer greater than 1; the first and second feature maps with the same resolution from the T first feature maps and T second feature maps are superimposed to obtain T third feature maps; the T third feature maps are upsampled and fused to obtain the target depth map of the current image, and the third depth map is obtained by stitching together the first depth map and the second depth map.

[0288] Among them, for any one of the T first feature maps, there is a second feature map in the T second feature maps with a unique resolution that is the same as that first feature map.

[0289] The multi-scale feature extraction in this application specifically refers to the operation of convolving an image using multiple different convolution kernels.

[0290] In this application, "overlay" specifically refers to processing the overlaid images at the pixel level. For example, if two overlaid images each have a size of H*W, the resulting overlaid image will have a size of H*2W or 2H*W; or if three overlaid images each have a size of H*W, the resulting overlaid image will have a size of H*3W or 3H*W.

[0291] Optionally, a target depth map of the current image is obtained based on the current image, the first depth map, and the second depth map, including:

[0292] Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps. Multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps. The resolutions of each of the T first feature maps, the T second feature maps, and the T fourth feature maps are all different. The reference depth map is obtained based on the depth map acquired by the TOF camera, where T is an integer greater than 1. The first, second, and fourth feature maps with the same resolution from the T first, T second, and T fourth feature maps are superimposed to obtain T fifth feature maps. The T fifth feature maps are upsampled and fused to obtain the target depth map of the current image. The third depth map is obtained by stitching together the first and second depth maps.

[0293] Among them, for any first feature map in the T first feature maps, there is a second feature map in the T second feature maps that has the same resolution as the first feature map; and there is a fourth feature map in the T fourth feature maps that has the same resolution as the first feature map.

[0294] Optionally, the aforementioned reference depth map is the depth map captured by the aforementioned TOF camera.

[0295] Optionally, the depth map acquired by the TOF camera is projected into three-dimensional space according to the second pose of the current image to obtain a fourth depth map; the fourth depth map is back-projected onto the reference image according to the pose of the reference image to obtain a reference depth map; wherein, the reference image is an image adjacent to the current image in terms of acquisition time, the resolution of the depth map acquired by the TOF camera is lower than the preset resolution, and the frame rate of the TOF acquiring the depth map is lower than the preset frame rate.

[0296] Optionally, the preset frame rate can be 1fps, 2fps, 5fps or other frame rates, and the preset resolution can be 240*180, 120*90, 60*45, 20*15 or other resolutions.

[0297] In one example, the TOF camera captures a depth map at a frame rate of 1 fps, with a resolution of 20*15.

[0298] Optionally, the above upsampling and fusion processing specifically includes:

[0299] S1: For feature map P' j Upsampling is performed to obtain the feature map P” j The feature map P” j The resolution and the (j+1)th feature map P in the processing object j+1With the same resolution, feature map P j+1 The width is j+1 times the width of the feature map with the smallest resolution in the processing object, where j is an integer greater than 0 and less than T; T is the number of feature maps in the processing object.

[0300] S2: Transfer feature map P” j With feature map P j+1 The fusion process yields the third feature map P'. j+1 ,

[0301] S3: Let j = j + 1, and repeat S1-S3 until j = T-1;

[0302] When j=1, the third feature map P' j To process the feature map with the lowest resolution in the object, when j = T-1, the third feature map P' j+1 This is the result of upsampling and fusion processing.

[0303] The objects to be processed include the aforementioned T third feature maps or T fifth feature maps.

[0304] For example, given five third feature maps: feature map P1, feature map P2, feature map P3, feature map P4, and feature map P5, with resolution increasing sequentially; feature map P1 is upsampled to obtain feature map P”1 with the same resolution as feature map P2, and feature map P”1 is fused with feature map P2 to obtain feature map P'2; feature map P'2 is upsampled to obtain feature map P”2 with the same resolution as feature map P3, and feature map P”2 is fused with feature map P3 to obtain feature map P'3; feature map P'3 is upsampled to obtain feature map P”3 with the same resolution as feature map P4, and feature map P”3 is fused with feature map P4 to obtain feature map P'4; feature map P'4 is upsampled to obtain feature map P”4 with the same resolution as feature map P5, and feature map P”4 is fused with feature map P5 to obtain the target depth map of the current image.

[0305] Optionally, the above upsampling is deconvolution upsampling.

[0306] S303. Obtain the image of the virtual object and the depth map of the virtual object; obtain the target image based on the target depth map, the depth map of the virtual object, the current image, and the image of the virtual object.

[0307] In one feasible embodiment, the target depth map and the depth map of the virtual object are used to determine the presentation and distribution of pixels of the virtual object image and pixels of the current image in the target image.

[0308] Alternatively, the virtual object image can be obtained by projecting the 3D model of the virtual object into a renderer in the terminal device; or it can be obtained from other devices.

[0309] Optionally, for any pixel P in the target image, if the first depth value is greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the virtual object image; if the first depth value is not greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the current image; wherein, the first depth value and the second depth value are the depth values ​​corresponding to pixel P in the target image and the depth map of the virtual object, respectively.

[0310] By determining the pixel values ​​of each pixel in the target image in the manner described above, the target image is obtained.

[0311] In a feasible embodiment, before obtaining the target image based on the target depth map, the depth map of the virtual object, the current image, and the virtual object image, edge optimization is performed on the target depth map based on the current image to obtain an optimized depth map; the accuracy of the optimized depth map is higher than that of the target depth map; then the target image is determined according to the above method, wherein the first depth value is the depth value corresponding to pixel point P in the target image in the optimized depth map.

[0312] In one feasible embodiment, obtaining the target image based on the target depth map, the virtual object depth map, the current image, and the virtual object image includes:

[0313] The target depth map is segmented to obtain the foreground depth map and background depth map of the current image. The background depth map is the depth map containing the background region in the target depth map, and the foreground depth map is the depth map containing the foreground region in the target depth map. The L background depth maps are fused according to the L poses corresponding to each of the L background depth maps to obtain the fused 3D scene. The L background depth maps include the background depth maps of the pre-stored image and the background depth map of the current image, and the L poses include the first pose of the pre-stored image and the current image; L is an integer greater than 1.

[0314] The fused 3D scene is back-projected based on the background depth map of the current image to obtain a fused background depth map; the fused background depth map and the foreground depth map of the current image are then stitched together to obtain an updated depth map; the target image is obtained based on the updated depth map, the depth map of the virtual object, the current image, and the image of the virtual object. The specific process is described above and will not be repeated here.

[0315] The aforementioned pre-stored image was received by the server prior to the current image.

[0316] In one feasible embodiment, displaying a virtual object image and the current image overlaid based on the target depth map of the current image includes:

[0317] Edge optimization is performed on the target depth map based on the current image to obtain an optimized depth map. The accuracy of the optimized depth map is higher than that of the target depth map. The optimized depth map is then segmented to obtain a foreground depth map and a background depth map of the current image. The background depth map is the depth map containing the background region in the optimized depth map, and the foreground depth map is the depth map containing the foreground region in the optimized depth map. The optimized depth map is obtained by edge optimization of the target depth map of the current image. The L background depth maps are fused according to the L poses corresponding to the L background depth maps respectively to obtain a fused 3D scene. The L background depth maps include the background depth maps of the pre-stored image and the background depth map of the current image. The L poses include the first pose of the pre-stored image and the first pose of the current image. L is an integer greater than 1. The fused 3D scene is back-projected according to the first pose of the current image to obtain a fused background depth map. The fused background depth map and the foreground depth map of the current image are then stitched together to obtain an updated depth map. The target image is obtained based on the updated depth map, the depth map of the virtual object, the current image, and the virtual object image. The specific process is described above and will not be repeated here.

[0318] It should be noted that the foreground region is the area where the object of interest is located, such as prominent objects like people, cars, plants, and animals; the background region is the area in the image other than the foreground region.

[0319] Optionally, the current image can be inspected, such as for people, cars, plants, animals, or other objects, to identify them.

[0320] Specifically, the optimized depth map is used for human image segmentation. Specifically, human image segmentation is performed on the target depth map of the current image based on the human image mask to obtain the foreground depth map and background depth map of the current image. The foreground depth map is a depth map that includes the human image.

[0321] In one example, the current image includes the target task. After obtaining the optimized depth map, task detection is performed on the optimized depth map to obtain the detection result. Then, based on the detection result, the optimized depth map is segmented to obtain the foreground depth map and background depth map of the optimized depth map. The foreground depth map is the depth map that includes the target person.

[0322] Optionally, the pose can be obtained from the corresponding image, or it can be a SLAM pose, a pose obtained through deep learning methods, or a pose obtained through other methods, which is not limited here.

[0323] Optionally, the specific fusion method used for the above fusion can be either the truncated signed distance function (TSDF) fusion method or the surfel fusion method.

[0324] In an optional embodiment, edge optimization is performed on the target depth map based on the current image to obtain an optimized depth map, including:

[0325] Based on the current image and the target depth map, an offset map of the target depth map is obtained. Specifically, the current image and the target depth map can be processed through a neural network to obtain the offset map of the target depth map. The corresponding pixels of the target depth map and the offset map of the target depth map are superimposed to obtain a depth map with sharp edges. This depth map is the optimized depth map mentioned above.

[0326] As can be seen, in the embodiments of this application, the second depth map is obtained by projecting the 3D point cloud obtained from the offline map stored on the server. Since the offline map includes a panoramic map composed of multiple frames of base maps, the 2D feature points of the panoramic map and the corresponding 3D points are different in each of the multiple frames of base maps. Therefore, the 3D point cloud of this application can be seen as being obtained based on multiple frames of images. The second depth map obtained by projecting the 3D point cloud can also be seen as being obtained based on multiple frames of images. The target depth map obtained based on the first depth map and the second depth map of the current image can be seen as being obtained based on multi-view depth estimation. Therefore, there is no scale discrepancy or inconsistency problem in the virtual and real occlusion effect of the target image obtained based on the target depth map, the current image, the depth map of the virtual object, and the image of the virtual object. By performing depth estimation on the current image, the first depth map, and the second depth map, a target depth map with richer depth information representing the feature points in the current image is obtained, thus eliminating the problems of inter-frame flickering and instability in the virtual-real occlusion effect of the target image. The depth map captured by the TOF camera is introduced during depth estimation, which further enriches the depth information of the current image represented by the target depth map. By performing edge optimization on the target depth map of the current image, an optimized depth map is obtained. Then, the depth maps of multiple frames are fused to obtain a depth map with sharper edges, which is conducive to further improving the virtual-real occlusion effect.

[0327] It should be noted that, on the terminal side, the method of this application is repeated. Therefore, when obtaining the 3D point cloud corresponding to the local map, it can be regarded as traversing the 3D points corresponding to the 2D feature points of the panoramic map in the offline map. Since the 2D feature points in the offline map are the 2D feature points in the multi-frame base map that constitute the panoramic map, in the process of repeatedly executing the method of this application, the 3D point cloud corresponding to the local map obtained comes from the 3D points corresponding to the 2D feature points of different base maps. The second depth map is obtained based on the 3D projection corresponding to the local map, and the target depth map is obtained based on the current image, the first depth map and the second depth map. It can be regarded as the target depth map is obtained based on multi-view depth estimation. Therefore, virtual and real occlusion is performed based on the target depth map, which solves the problem of scale discrepancies and inconsistencies in monocular depth estimation.

[0328] See Figure 5 , Figure 5 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 5 As shown, the method includes:

[0329] S501, Receive a depth estimation model request message sent by the terminal device, the request message carrying the geographical location information of the terminal device.

[0330] The geographic location information of the terminal device refers to the geographic location information of the terminal device when it acquires the current image.

[0331] The aforementioned geographic location information refers to coordinates in a world coordinate system, which can be a UTM coordinate system, a GPS coordinate system, or other world coordinate systems.

[0332] S502. Obtain the depth estimation model of the current image from multiple depth estimation models stored in the server based on the geographical location information of the terminal device.

[0333] The depth estimation model for the current image is the depth estimation model corresponding to the terminal device among multiple depth estimation models stored on the server. The multiple depth estimation models stored on the server correspond one-to-one with multiple geographic location information.

[0334] To improve the accuracy of depth estimation, the server stores a depth estimation model trained separately for each geographic location.

[0335] In one feasible embodiment, the method of this embodiment further includes:

[0336] For multiple geographic location information, depth estimation models corresponding one-to-one with each geographic location information are trained. Specifically, for any geographic location information S among the multiple locations, the depth estimation model corresponding to that geographic location information S can be obtained by training according to the following steps:

[0337] Multiple image samples and their corresponding depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps. These multiple image samples are collected by the terminal device from the geographic location information S. The loss value is calculated based on the multiple predicted depth maps, the corresponding real depth maps, and the loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain the depth estimation model corresponding to the geographic location information S.

[0338] The loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0339] It should be noted that the above is only one training process; in practical applications, the process will be iterated multiple times in the same manner until the calculated loss value converges. The convolutional neural network model at which the loss value converges is then determined as the depth estimation model corresponding to the geographic location information S. The specific training process described above can be found in the relevant description in S302, and will not be repeated here.

[0340] S503. Send a response message to the terminal device in response to the depth estimation model request message, the response message carrying the depth estimation model of the current image.

[0341] It should be noted here that, in order to... Figure 3 The illustrated embodiments are consistent, and the depth estimation model request message can be viewed as... Figure 3 In the corresponding embodiment, the fifth acquisition request, the response message to the depth estimation model request message, can be regarded as the fifth response message.

[0342] Optionally, the method in this embodiment further includes:

[0343] The system receives a first acquisition request from a terminal device; obtains a local map from an offline map based on the position in the first pose of the current image, wherein the local map is the area indicated by the pose in the first pose of the current image in the offline map; acquires the 3D point cloud corresponding to the local map; wherein the first pose of the current image is the pose of the current image in a first coordinate system, and the first coordinate system is the coordinate system in which the offline map is located; and sends a first response message to the terminal device in response to the first acquisition request, wherein the first response message carries the 3D point cloud corresponding to the local map.

[0344] Specifically, the server stores an offline map and 3D points corresponding to the 2D feature points in the offline map. The server obtains a local map from the offline map based on the position in the first pose of the current image. This local map is the area indicated by the position in the first pose of the offline map. In one example, the local map is an area in the offline map centered on this position within a certain range (e.g., with a radius of 50 meters). This local map includes multiple 2D feature points. Based on the correspondence between the 2D feature points and 3D points, the 3D points corresponding to the 2D feature points in the local map are obtained. The 3D points corresponding to the 2D feature points in the local map are the 3D point cloud corresponding to the local map.

[0345] The first pose of the current image is obtained by the server based on VPS technology; this first pose is the pose in the coordinate system of the offline map, that is, the first pose of the current image is consistent with the coordinate system of the offline map.

[0346] Optionally, the first response message also carries the first pose of the current image.

[0347] Optionally, the method in this embodiment further includes:

[0348] The system receives a second acquisition request sent by a terminal device, the second acquisition request carrying the current image; obtains a second depth map corresponding to the current image based on the current image and an offline map; and sends a second response message to the terminal device for the second acquisition request, the second response message carrying the second depth map.

[0349] The second depth map is determined in the server and then sent to the terminal device, so that the terminal device does not need to calculate the second depth map, thus reducing the terminal device's resource consumption and power consumption.

[0350] Optionally, a second depth map is obtained based on the current image and the offline map, including:

[0351] The first pose of the current image is determined based on the current image. The first pose of the current image is the pose of the current image in the first coordinate system, which is the coordinate system of the offline map. Based on the position in the first pose of the current image, the 3D point cloud corresponding to the local map is obtained from the 3D point cloud corresponding to the offline map. The local map is the area indicated by the first position in the offline map stored on the server.

[0352] The current image is projected based on the first pose of the current image and the corresponding 3D point cloud of the local map to obtain the second depth map.

[0353] Specifically, the process of projecting the current image onto the imaging plane of the current image based on the first pose of the current image and the corresponding 3D point cloud of the local map to obtain the second depth map refers to: projecting the 3D point cloud of the local map onto the imaging plane of the current image based on the first pose of the current image to obtain the second depth map.

[0354] Optionally, obtaining a second depth map based on the current image and the offline map also includes:

[0355] Feature extraction is performed on the current image to obtain the ninth 2D feature point of the current image. The ninth 2D feature point of the current image is matched with the eleventh 2D feature points in multiple historical images to obtain the eleventh 2D feature point of the current image. The eleventh 2D feature point is the 2D feature point in the current image that matches the eleventh 2D feature point in the historical images. The eleventh 2D feature point is a SIFT feature point. Based on the first pose of the current image, the eleventh 2D feature point of the current image, the 2D feature point in the historical images that matches the eleventh 2D feature point, and the first pose of the historical images, the 3D point corresponding to the eleventh 2D feature point of the current image is obtained.

[0356] The current image is projected based on its first pose and the corresponding 3D point cloud of the local map to obtain a second depth map, including:

[0357] The current image is projected based on the first pose of the current image, the 3D point cloud corresponding to the local map, and the 3D point corresponding to the eleventh 2D feature point of the current image to obtain a second depth map.

[0358] Specifically, after obtaining the eleventh 2D feature point of the current image, the relative pose between the first pose of the current image and the first pose of the historical image is determined; based on the relative pose, the eleventh 2D feature point of the current image and the eleventh 2D feature point in the historical image that matches the eleventh 2D feature point are triangulated to obtain the 3D point corresponding to the eleventh 2D feature point of the current image; based on the first pose of the current image, the 3D point cloud corresponding to the local map and the 3D point corresponding to the eleventh 2D feature point of the current image are projected onto the imaging plane of the current image to obtain the second depth map.

[0359] Optionally, the ninth and tenth 2D feature points can be SIFT feature points, SURF feature points, or SuperPoint, ASLFeat, R2D2, or D2Net feature points obtained based on deep learning.

[0360] Because servers possess strong computing power, they can extract feature points from both texture-rich and texture-weak regions when extracting image features. The feature extraction method can be SIFT or a method based on SIFT; the extracted feature points can be called SIFT feature points, and the number is approximately 10,000. Alternatively, SURF or a method based on SURF can be used, and the extracted feature points can be called SURF feature points.

[0361] As can be seen from the foregoing, the number of SIFT feature points is higher than the number of ORB feature points.

[0362] By introducing 3D points obtained from the ninth 2D feature point of the current image and the tenth 2D feature point of the historical image based on the 3D point cloud corresponding to the local map, and projecting these 3D points together with the 3D point cloud corresponding to the local map to obtain a second depth map, the depth information of the current image represented by the second depth map is further enriched, thereby enriching the depth information of the current image represented by the target depth map.

[0363] Optionally, obtaining a second depth map corresponding to the current image based on the current image and the offline map further includes:

[0364] The ninth 2D feature point of the current image is matched with the 2D feature point in the local map to obtain the sixth 2D feature point of the local map; the 3D point corresponding to the sixth 2D feature point is obtained based on the sixth 2D feature point, the ninth 2D feature point in the current image that matches the sixth 2D feature point, the pose of the local map and the first pose of the current image.

[0365] The current image is projected based on its first pose, the corresponding 3D point cloud of the local map, and the 3D point corresponding to the eleventh 2D feature point of the current image to obtain a second depth map, including:

[0366] The current image is projected based on the first pose of the current image, the 3D point cloud corresponding to the local map, the 3D point corresponding to the eleventh 2D feature point of the current image, and the 3D point corresponding to the sixth 2D feature point of the current image to obtain a second depth map.

[0367] Since the local map is an offline map, and the area within a certain range centered on the location of the terminal device is a certain region of the local map, the 2D feature point that matches the ninth 2D feature point in the current image is a 2D feature point of a certain region of the local map. Therefore, in order to improve the matching efficiency, a reference map is obtained from the local map based on the orientation information in the first pose of the current image. The reference map is the region indicated by the orientation information of the first pose in the local map. The 2D feature point in the reference map is matched with the ninth 2D feature point of the current image to obtain the sixth 2D feature point of the reference map. The sixth 2D feature point of the reference map is the sixth 2D feature point of the aforementioned local map.

[0368] like Figure 4 As shown, the square area is the offline map, the circular area is the local map, and the center point of the circular area is the position of the terminal device in the offline map; the fan-shaped area with an angle range of [-45°, 45°] is the target map, where the angle range of [-45°, 45°] is the yaw angle range in the orientation information of the terminal device in the first pose.

[0369] After obtaining the sixth 2D feature point of the local map, the relative pose between the first pose of the current image and the pose of the local map is determined. Then, based on the relative pose, the sixth 2D feature point of the local map and the first 2D feature point in the current image that matches the sixth 2D feature point are triangulated to obtain the 3D point corresponding to the sixth 2D feature point. Based on the first pose of the current image, the 3D point cloud corresponding to the local map, the 3D point corresponding to the eleventh 2D feature point of the current image, and the 3D point corresponding to the sixth 2D feature point are projected onto the imaging plane of the current image to obtain the second depth map.

[0370] By introducing the 3D point corresponding to the sixth 2D feature point of the local map, based on the 3D point cloud corresponding to the local map and the 3D point corresponding to the eleventh 2D feature point of the current image, the depth information of the current image represented by the second depth map is further enriched, thereby enriching the depth information of the current image represented by the target depth map.

[0371] Optionally, the offline map includes multiple frames of base map, and the method of this application also includes:

[0372] Based on the current image, M base maps are obtained from multiple base maps, where the similarity between each base map frame and the current image is greater than a first threshold. The ninth 2D feature point of the current image is matched with the twelfth 2D feature point in the M base maps to obtain the seventh 2D feature point in the base maps; this ninth 2D feature point is a SIFT feature point. From the 3D point cloud corresponding to the local map, the 3D points corresponding to the seventh 2D feature point are selected to obtain processed 3D points. The ninth 2D feature point of the current image is matched with the tenth 2D feature point in multiple historical images to obtain the current... The eleventh 2D feature point of the image is the 2D feature point that matches the 2D feature point in the historical image among the ninth 2D feature points of the current image. Based on the eleventh 2D feature point, the first pose of the current image, the 2D feature point that matches the eleventh 2D feature point among the eleventh 2D feature points of multiple historical images, and the first pose of the historical image to which the 2D feature point belongs, the 3D point corresponding to the eleventh 2D feature point is obtained. Based on the first pose of the current image, the processed 3D point and the 3D point corresponding to the eleventh 2D feature point are processed to obtain the 3D point cloud corresponding to the updated local map.

[0373] The current image is projected onto the first pose of the current image and the corresponding 3D point cloud of the local map to obtain a second depth map, including:

[0374] The current image is projected using the first pose of the current image and the 3D point cloud corresponding to the updated local map to obtain a second depth map.

[0375] Specifically, the second depth map is obtained by projecting the current image onto the imaging plane of the current image based on the first pose of the current image and the 3D point cloud corresponding to the updated local map.

[0376] Optionally, the eleventh 2D feature point can be a SIFT feature point, a SURF feature point, or a SuperPoint feature point, ASLFeat feature point, R2D2 feature point, or D2Net feature point obtained based on deep learning.

[0377] Because the offline maps on the server are collected offline, they may differ from the actual environment. For example, a large billboard in a shopping mall might have existed during offline data collection, but has been removed by the time the user captures the current image. This results in the server-delivered map containing inconsistent 3D point cloud information. Furthermore, the images received by the server may have undergone privacy processing, which can also lead to inconsistencies in the 3D point cloud information in the delivered map.

[0378] For the reasons mentioned above, the server updates the 3D point cloud corresponding to the local map it sends out, and obtains the updated 3D point cloud corresponding to the local map.

[0379] The server extracts features from the current image to obtain the ninth 2D feature point. The methods for feature extraction include, but are not limited to, scale-invariant feature transform (SIFT) and methods improved upon SIFT. The extracted 2D feature point can also be called a SIFT feature point. The server matches the ninth 2D feature point in the current image with pre-stored 2D feature points to obtain the eleventh 2D feature point in the current image. Optionally, the pre-stored 2D feature point is the eleventh 2D feature point from N historical images. These N historical images are acquired by the terminal device and are among the images whose timestamps are prior to the timestamp of the current image. N is an integer greater than 0. Optionally, the server removes noise from the eleventh 2D feature point of the current image. Specifically, it calculates the verification value of each eleventh 2D feature point in the current image using the homography matrix, the fundamental matrix, and the essential matrix. If the verification value of the eleventh 2D feature point is lower than the second preset threshold, the eleventh 2D feature point is determined to be noise and thus deleted.

[0380] The server determines the relative pose between the first pose of the current image and the first pose of the historical image. Based on the relative pose, it performs triangulation on the eleventh 2D feature point of the current image and the eleventh 2D feature point in the historical image that matches the eleventh 2D feature point to obtain the 3D point corresponding to the eleventh 2D feature point in the current image.

[0381] The server retrieves M base maps from multiple base maps using an image retrieval method. Each of these M base maps has a similarity greater than a first preset threshold with the current image, where M is an integer greater than 0. The image retrieval method includes, but is not limited to, the bag-of-words tree method or the NetVlad method based on deep learning. The server matches the eleventh 2D feature point in the current image with the twelfth 2D feature point in the M base maps to obtain the seventh 2D feature point of the base map. From the 3D point cloud corresponding to the local map, the server selects the 3D points corresponding to the seventh 2D feature point to obtain the processed 3D points.

[0382] Based on the first pose of the current image, the 3D points corresponding to the processed 3D points and the eleventh 2D feature points are processed to obtain the 3D point cloud corresponding to the updated local map, thereby realizing the update of the 3D point cloud corresponding to the local map in the server.

[0383] Because the server has strong computing power, it can extract feature points from both texture-rich and texture-weak regions when extracting image features. The feature extraction method can be SIFT or a method based on SIFT. The extracted feature points can be called SIFT feature points, and the number is about 10,000.

[0384] The above method ensures that the 3D point cloud corresponding to the offline map stored on the server is continuously updated by images uploaded from the terminal device. This maintains consistency between the 3D point cloud of the offline map and the images captured by the terminal device, providing high-precision 3D point cloud information for depth estimation. Furthermore, by ensuring consistency between the 3D point cloud of the offline map and the images captured by the terminal device, the more images uploaded by the terminal device, the more thorough the update of the 3D point cloud of the offline map, resulting in more accurate depth estimation results.

[0385] Optionally, the method of this application further includes:

[0386] The system receives a third acquisition request from the terminal device, the third acquisition request carrying the current image; performs feature point matching based on the current image and the offline map to determine the first pose of the current image, the first pose being the pose of the current image in a first coordinate system, the first coordinate system being the coordinate system of the offline map; and sends a third response message to the terminal device in response to the third acquisition request, the third response message carrying the first pose.

[0387] Optionally, the method of this application further includes:

[0388] The system receives a fourth acquisition request from the terminal device, which carries the current image and the second pose of the current image. The second pose of the current image is the pose of the terminal device when the current image was captured. Feature point matching is performed based on the current image and the offline map to determine the first pose of the current image. The first pose is the pose of the current image in a first coordinate system, which is the coordinate system of the offline map. Pose transformation information is determined based on the second pose and the first pose of the current image. The pose transformation information is used for the transformation between the second pose and the first pose of the current image. A fourth response message is sent to the terminal device in response to the fourth acquisition request. The fourth response message carries the pose transformation information.

[0389] Optionally, feature point matching is performed based on the current image and the offline map to determine the first pose of the current image, including:

[0390] A local map is obtained from the offline map based on the position in the second pose of the current image; the local map is the region indicated by the position in the second pose of the current image in the offline map; feature point matching is performed based on the current image and the local map to determine the first pose.

[0391] By sending the first pose or pose transformation information of the current image to the terminal device, the coordinate system of the current image is kept consistent with that of the offline map stored in the server, which facilitates subsequent processing.

[0392] As can be seen, in this application, a second depth map is obtained by projecting a 3D point cloud obtained from an offline map stored on a server. Since the offline map includes a panoramic map composed of multiple frames of base maps, and the 2D feature points and their corresponding 3D points are different in each frame of the panoramic map, the 3D point cloud in this application can be seen as being obtained based on multiple frames of images. The second depth map obtained by projecting this 3D point cloud can also be seen as being based on multiple frames of images. The target depth map obtained based on the current image, the first depth map, and the second depth map can be seen as being obtained based on multi-view depth estimation. Therefore, the target depth map obtained based on the current image, the first depth map, and the second depth map can be seen as being obtained based on multi-view depth estimation. Thus, virtual-real occlusion is performed based on this target depth map, solving the scale discrepancies and inconsistencies in monocular depth estimation. The server updates the 3D point cloud corresponding to the local map based on the current image, avoiding the problem of errors when directly using the 3D point cloud corresponding to the local map to participate in the estimation of the target depth map due to changes in the current scene.

[0393] like Figure 6 As shown, by accurately estimating the depth map of the bushes, the virtual panda can be seen through the gaps in the real bushes. The panda can also be obscured by buildings such as walls. Of course, the solution of this application also supports virtual and real occlusion between people and virtual objects. As can be seen from the last figure, the method based on this application can accurately estimate the depth map of the entire scene, so that the virtual panda can be within the person's arm, with a strong overall sense of immersion and excellent user experience.

[0394] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 7 As shown, the terminal device 100 includes:

[0395] The 2D-3D matching module 102 is used to match the first 2D feature points of the current image with the eighth 2D feature points of multiple historical images to obtain the second 2D feature points of the current image. The second 2D feature points are the 2D feature points of the first 2D feature points of the current image that match the eighth 2D feature points of multiple historical images. The module determines the relative pose between the second pose of the current image and the second pose of the historical images. Based on the relative pose, the module performs triangulation on the eighth 2D feature points of the second 2D feature points of the current image and the historical images that match the second 2D feature points to obtain the corresponding 3D points. The module projects the corresponding 3D points onto the plane where the current image is located to obtain the first depth map.

[0396] The depth estimation module 104 is used to process the current image and the first depth map through a depth estimation model to obtain the target depth map of the current image. Specifically, it performs feature extraction on the current image to obtain T first feature maps, each of which has a different resolution. It also performs feature extraction on the first depth map to obtain T ninth feature maps, each of which has a different resolution. The T first feature maps and the T ninth feature maps are then superimposed with the first and ninth feature maps of the same resolution to obtain T tenth feature maps. Finally, the T tenth feature maps are upsampled and fused to obtain the target depth map of the current image.

[0397] It should be noted that the specific process of upsampling and fusing T tenth feature maps to obtain the target depth map of the current image can be found in the relevant description of step S302, and will not be described here again.

[0398] The segmentation module 108 is used to process the current image through a segmentation network to obtain a segmentation result image, which can be called a mask image, such as a portrait mask image. The segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network extracts features from the current image, then performs bilinear upsampling on the features to obtain a feature map of the same size as the input. Finally, the softmax classifier obtains the label for each pixel, thus obtaining the segmentation result image. For example, the current image is input into BiSeNet to obtain a feature map, and then the softmax classifier classifies each pixel in this feature map to obtain a label for each pixel. Based on the label, the pixel value of each pixel is determined, where pixels belonging to the foreground region have a value of 255, and pixels belonging to the background region have a value of 0. For example, the foreground region can include areas containing portraits, and the background region can be areas that do not contain portraits.

[0399] The depth map edge optimization module 105 is used to extract the edge structure of the current image and its target depth map respectively, to obtain the edge structure information of the current image and the edge structure information of the target depth map; using the edge structure information of the current image as a reference, the difference between the edge structure of the target depth map and the edge structure of the current image is calculated, and then the edge position of the target depth map is modified by this difference, thereby optimizing the edge of the depth map and obtaining the optimized depth map, which is a sharp depth map corresponding to the edge of the current image.

[0400] The virtual-real occlusion application module 107 is used to segment the optimized depth map according to the segmentation result map to obtain a foreground depth map and a background depth map; to obtain a target image based on the foreground depth map, the background depth map, the depth map of the virtual object, the current image, and the virtual object image; for any pixel P in the target image, if the first depth value is greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the virtual object image; if the first depth value is not greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the current image; wherein, the first depth value is the depth value corresponding to pixel P in the target image in the foreground depth map or the background depth map, and the second depth value is the depth value corresponding to pixel P in the target image in the depth map of the virtual object; and the target image is displayed on the terminal device.

[0401] See Figure 8 , Figure 8 This is a schematic diagram of the structure of another terminal device provided in an embodiment of this application. For example... Figure 8 As shown, the terminal device 100 includes:

[0402] The 2D-3D matching module 102 is used to match the first 2D feature points of the current image with the eighth 2D feature points of multiple historical images to obtain the second 2D feature points of the current image. The second 2D feature points are the 2D feature points of the first 2D feature points of the current image that match the eighth 2D feature points of multiple historical images. The module determines the relative pose between the second pose of the current image and the second pose of the historical images. Based on the relative pose, the module performs triangulation on the eighth 2D feature points of the second 2D feature points of the current image and the historical images that match the second 2D feature points to obtain the corresponding 3D points. The module projects the corresponding 3D points onto the plane where the current image is located to obtain the first depth map.

[0403] The depth estimation module 104 is used to process the current image and the first depth map through a depth estimation model to obtain the target depth map of the current image. Specifically, it performs feature extraction on the current image to obtain T first feature maps, each of which has a different resolution. It also performs feature extraction on the first depth map to obtain T ninth feature maps, each of which has a different resolution. The T first feature maps and the T ninth feature maps are then superimposed with the first and ninth feature maps of the same resolution to obtain T tenth feature maps. Finally, the T tenth feature maps are upsampled and fused to obtain the target depth map of the current image.

[0404] It should be noted that the specific process of upsampling and fusing T tenth feature maps to obtain the target depth map of the current image can be found in the relevant description of step S302, and will not be described here again.

[0405] The segmentation module 108 is used to process the current image through a segmentation network to obtain a segmentation result image, which can be called a mask image, such as a portrait mask image. The segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network extracts features from the current image, then performs bilinear upsampling on the features to obtain a feature map of the same size as the input. Finally, the softmax classifier obtains the label for each pixel, thus obtaining the segmentation result image. For example, the current image is input into BiSeNet to obtain a feature map, and then the softmax classifier classifies each pixel in this feature map to obtain a label for each pixel. Based on the label, the pixel value of each pixel is determined, where pixels belonging to the foreground region have a value of 255, and pixels belonging to the background region have a value of 0. For example, the foreground region can include areas containing portraits, and the background region can be areas that do not contain portraits.

[0406] The depth map edge optimization module 105 is used to extract the edge structure of the current image and its target depth map respectively, to obtain the edge structure information of the current image and the edge structure information of the target depth map; using the edge structure information of the current image as a reference, the difference between the edge structure of the target depth map and the edge structure of the current image is calculated, and then the edge position of the target depth map is modified by this difference, thereby optimizing the edge of the depth map and obtaining an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0407] The multi-view fusion module 106 is used to segment the optimized depth map according to the segmentation result map to obtain a foreground depth map and a background depth map. The background depth map is the depth map containing the background region in the optimized depth map, and the foreground depth map is the depth map containing the foreground region in the optimized depth map. The L background depth maps are fused according to the L second poses corresponding to the L background depth maps respectively to obtain a fused 3D scene. The L background depth maps include the background depth map of the pre-stored image and the background depth map of the current image. The L second poses include the second pose of the pre-stored image and the second pose of the current image. L is an integer greater than 1. The fused 3D scene is back-projected according to the second pose of the current image to obtain a fused background depth map. The fused background depth map and the foreground depth map of the current image are stitched together to obtain an updated depth map.

[0408] The virtual-real occlusion application module 107 is used to obtain a target image based on the updated depth map, the depth map of the virtual object, the current image, and the virtual object image. For any pixel P in the target image, if the first depth value is greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel in the virtual object image; if the first depth value is not greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel in the current image; wherein, the first depth value and the second depth value are the depth values ​​corresponding to pixel P in the target image in the updated depth map and the depth map of the virtual object, respectively; and the target image is displayed on the terminal device.

[0409] See Figure 9 , Figure 9 This is a schematic diagram of a system structure provided for an embodiment of this application. For example... Figure 9 As shown, the system includes a terminal device 100 and a server 200. The terminal device 100 includes a 2D-3D matching module 102, a depth estimation module 104, a depth map edge optimization module 105, a segmentation module 108, and a virtual-real occlusion application module 107. The server 200 includes a VPS positioning and map distribution module 101 and a local map update module 103.

[0410] The VPS positioning and map distribution module 101 is used to calculate the first pose of the current image based on the VPS after receiving the current image. The first pose is the pose in the coordinate system of the offline map. Based on the position of the first pose, it obtains a local map from the offline map. For example, the local map is an area within a certain range (e.g., a radius of 50 meters) around the position in the first pose in the offline map. The offline map includes 2D feature points of objects and their corresponding feature descriptors. Based on the relationship between 2D feature points and 3D points, it obtains the 3D point cloud corresponding to the local map. It then sends the 3D point cloud corresponding to the local map and the first pose of the current image to the 2D-3D matching module 102.

[0411] Optionally, the second pose of the current image is received, and the pose transformation information between the first pose and the second pose of the current image is determined; the 3D point cloud and pose transformation information corresponding to the local map are sent to the 2D-3D matching module 102.

[0412] The 2D-3D matching module 102 is used to process the current image information according to the first pose of the current image to obtain a first image, which is an image obtained by transforming the second pose of the current image into the first pose; after receiving pose transformation information, it processes the second pose of the current image according to the pose transformation information to obtain the first pose of the current image; then processes the current image information according to the first pose of the current image to obtain the first image; performs feature extraction on the first image to obtain the third 2D feature point of the first image; matches the third 2D feature point with the eighth 2D feature points of multiple historical images to obtain the fourth 2D feature point of the first image; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the pre-stored 2D feature points; obtains the 3D point corresponding to the fourth 2D feature point according to the fourth 2D feature point, the first pose, the 2D feature point among the eighth 2D feature points that matches the fourth 2D feature point, and the pose of the image to which the 2D feature point belongs; projects the 3D point corresponding to the fourth 2D feature point onto the imaging plane of the current image to obtain a first depth map.

[0413] The depth estimation module 104 is used to process the current image and the first depth map through a depth estimation model to obtain the target depth map of the current image. Specifically, it performs feature extraction on the current image to obtain T first feature maps, each of which has a different resolution. It also performs feature extraction on the first depth map to obtain T ninth feature maps, each of which has a different resolution. The T first feature maps and the T ninth feature maps are then superimposed with the first and ninth feature maps of the same resolution to obtain T tenth feature maps. Finally, the T tenth feature maps are upsampled and fused to obtain the target depth map of the current image.

[0414] It should be noted that the specific process of upsampling and fusing T tenth feature maps to obtain the target depth map of the current image can be found in the relevant description of step S302, and will not be described here again.

[0415] In an optional embodiment, the local map update module 103 is configured to: obtain M frames of base maps from multiple base maps based on the current image, wherein the similarity between each base map in the M frames and the current image is greater than a first threshold; match the ninth 2D feature point of the current image with the twelfth 2D feature point in the M frames of base maps to obtain the seventh 2D feature point in the base maps; filter out the 3D points corresponding to the seventh 2D feature point from the 3D point cloud corresponding to the local map to obtain the processed 3D points; and match the ninth 2D feature point of the current image with the eleventh 2D feature point in multiple historical images to obtain the eleventh 2D feature point of the current image, wherein the eleventh 2D feature point is the ninth 2D feature point of the current image. The 2D feature points that match the 11th 2D feature point in the historical image are identified. Based on the 11th 2D feature point, the first pose of the current image, the 2D feature points that match the 11th 2D feature point among the 11th 2D feature points in the multiple historical images, and the first pose of the historical image to which the 2D feature point belongs, a 3D point corresponding to the 11th 2D feature point is obtained. The processed 3D point and the 3D point corresponding to the 11th 2D feature point are processed based on the first pose of the current image to obtain an updated 3D point cloud corresponding to the local map, thereby updating the 3D points corresponding to the local map. The updated 3D point cloud corresponding to the local map is projected onto the imaging plane of the current image to obtain a second depth map.

[0416] The depth estimation module 104 is used to process the current image and the third depth map using the depth estimation model of the current image to obtain the target depth map of the current image. Specifically, multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps. The resolution of each of the T first feature maps is different, and the resolution of each of the T second feature maps is different. T is an integer greater than 1. The first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps are superimposed to obtain T third feature maps. The T third feature maps are upsampled and fused to obtain the target depth map of the current image. The third depth map is obtained by stitching together the first depth map and the second depth map.

[0417] Optionally, the depth estimation module 104 is used to process the current image, the reference depth map, and the third depth map using a depth estimation model for the current image to obtain the target depth map of the current image. Specifically, multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps, where each of the T first feature maps has a different resolution, each of the T second feature maps has a different resolution, and each of the T fourth feature maps has a different resolution, where T is an integer greater than 1; the first, second, and fourth feature maps with the same resolution from the T first, T second, and T fourth feature maps are superimposed to obtain T fifth feature maps; the T fifth feature maps are upsampled and fused to obtain the target depth map of the current image, and the third depth map is obtained by stitching together the first and second depth maps.

[0418] The segmentation module 108 is used to process the current image through a segmentation network to obtain a segmentation result image, which can be called a mask image, such as a portrait mask image. The segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network extracts features from the current image, then performs bilinear upsampling on the features to obtain a feature map of the same size as the input. Finally, the softmax classifier obtains the label for each pixel, thus obtaining the segmentation result image. For example, the current image is input into BiSeNet to obtain a feature map, and then the softmax classifier classifies each pixel in this feature map to obtain a label for each pixel. Based on the label, the pixel value of each pixel is determined, where pixels belonging to the foreground region have a value of 255, and pixels belonging to the background region have a value of 0. For example, the foreground region can include areas containing portraits, and the background region can be areas that do not contain portraits.

[0419] The depth map edge optimization module 105 is used to extract the edge structure of the current image and its target depth map respectively, to obtain the edge structure information of the current image and the edge structure information of the target depth map; using the edge structure information of the current image as a reference, the difference between the edge structure of the target depth map and the edge structure of the current image is calculated, and then the edge position of the target depth map is modified by this difference, thereby optimizing the edge of the depth map and obtaining the optimized depth map, which is a sharp depth map corresponding to the edge of the current image.

[0420] The virtual-real occlusion application module 107 is used to segment the optimized depth map according to the segmentation result map to obtain a foreground depth map and a background depth map; to obtain a target image based on the foreground depth map, the background depth map, the depth map of the virtual object, the current image, and the virtual object image; for any pixel P in the target image, if the first depth value is greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the virtual object image; if the first depth value is not greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel P in the current image; wherein, the first depth value is the depth value corresponding to pixel P in the target image in the foreground depth map or the background depth map, and the second depth value is the depth value corresponding to pixel P in the target image in the depth map of the virtual object; and the target image is displayed on the terminal device.

[0421] See Figure 10 , Figure 10 This is a schematic diagram of another system structure provided for an embodiment of this application. For example... Figure 10 As shown, the system includes a terminal device 100 and a server 200. The terminal device 100 includes a 2D-3D matching module 102, a depth estimation module 104, a depth map edge optimization module 105, a segmentation module 108, and a virtual-real occlusion application module 107. The server 200 includes a VPS positioning and map distribution module 101 and a local map update module 103.

[0422] The VPS positioning and map distribution module 101 is used to calculate the first pose of the current image based on the VPS after receiving the current image. The first pose is the pose in the coordinate system of the offline map. Based on the position of the first pose, it obtains a local map from the offline map. For example, the local map is an area within a certain range (e.g., a radius of 50 meters) around the position in the first pose in the offline map. The offline map includes 2D feature points of objects and their corresponding feature descriptors. Based on the relationship between 2D feature points and 3D points, it obtains the 3D point cloud corresponding to the local map. It then sends the 3D point cloud corresponding to the local map and the first pose of the current image to the 2D-3D matching module 102.

[0423] Optionally, the second pose of the current image is received, and the pose transformation information between the first pose and the second pose of the current image is determined; the 3D point cloud and pose transformation information corresponding to the local map are sent to the 2D-3D matching module 102.

[0424] The 2D-3D matching module 102 is used to process the current image information according to the first pose of the current image to obtain a first image, which is an image obtained by transforming the second pose of the current image into the first pose; after receiving pose transformation information, it processes the second pose of the current image according to the pose transformation information to obtain the first pose of the current image; then processes the current image information according to the first pose of the current image to obtain the first image; performs feature extraction on the first image to obtain the third 2D feature point of the first image; matches the third 2D feature point with the eighth 2D feature point of multiple historical images to obtain the fourth 2D feature point of the first image; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the eighth 2D feature point; obtains the 3D point corresponding to the fourth 2D feature point according to the fourth 2D feature point, the first pose, the 2D feature point among the eighth 2D feature points that matches the fourth 2D feature point, and the pose of the image to which the 2D feature point belongs; projects the 3D point corresponding to the fourth 2D feature point onto the imaging plane of the current image to obtain a first depth map.

[0425] The depth estimation module 104 is used to process the current image and the third depth map through a depth estimation model to obtain the target depth map of the current image. Specifically, it performs feature extraction on the current image to obtain T first feature maps, each of which has a different resolution; it performs feature extraction on the third depth map to obtain T second feature maps, each of which has a different resolution; it then superimposes the first and second feature maps with the same resolution from the T first and T second feature maps to obtain T third feature maps; and it performs upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image. The third depth map is obtained by stitching together the first and second depth maps.

[0426] It should be noted that the specific process of upsampling and fusing T tenth feature maps to obtain the target depth map of the current image can be found in the relevant description of step S302, and will not be described here again.

[0427] In an optional embodiment, the local map update module 103 is configured to: obtain M frames of base maps from multiple base maps based on the current image, wherein the similarity between each base map in the M frames and the current image is greater than a first threshold; match the ninth 2D feature point of the current image with the twelfth 2D feature point in the M frames of base maps to obtain the seventh 2D feature point in the base maps; filter out the 3D points corresponding to the seventh 2D feature point from the 3D point cloud corresponding to the local map to obtain the processed 3D points; and match the ninth 2D feature point of the current image with the eleventh 2D feature point in multiple historical images to obtain the eleventh 2D feature point of the current image, wherein the eleventh 2D feature point is the th feature point of the current image. The 2D feature point that matches the 11th 2D feature point in the historical image is obtained from the 9th 2D feature point; the 3D point corresponding to the 11th 2D feature point is obtained based on the 11th 2D feature point, the first pose of the current image, the 2D feature point that matches the 11th 2D feature point among the 2D feature points of the multiple historical images, and the first pose of the historical image to which the 2D feature point belongs; the processed 3D point and the 3D point corresponding to the 11th 2D feature point are processed based on the first pose of the current image to obtain the updated 3D point cloud corresponding to the local map, thereby realizing the update of the 3D points corresponding to the local map; the updated 3D point cloud corresponding to the local map is projected onto the imaging plane of the current image to obtain the second depth map.

[0428] The depth estimation module 104 is used to process the current image and the third depth map using the depth estimation model of the current image to obtain the target depth map of the current image. Specifically, multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps. The resolution of each of the T first feature maps is different, and the resolution of each of the T second feature maps is different. T is an integer greater than 1. The first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps are superimposed to obtain T third feature maps. The T third feature maps are upsampled and fused to obtain the target depth map of the current image. The third depth map is obtained by stitching together the first depth map and the second depth map.

[0429] Optionally, the depth estimation module 104 is used to process the current image, the reference depth map, and the third depth map using a depth estimation model for the current image to obtain the target depth map of the current image. Specifically, multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps, where each of the T first feature maps has a different resolution, each of the T second feature maps has a different resolution, and each of the T fourth feature maps has a different resolution, where T is an integer greater than 1; the first, second, and fourth feature maps with the same resolution from the T first, T second, and T fourth feature maps are superimposed to obtain T fifth feature maps; the T fifth feature maps are upsampled and fused to obtain the target depth map of the current image, and the third depth map is obtained by stitching together the first and second depth maps.

[0430] The segmentation module 108 is used for the current image. Through the segmentation network, it obtains a segmentation result image, which can be called a mask image, such as a portrait mask image. The segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network extracts features from the current image, then performs bilinear upsampling on these features to obtain a feature map of the same size as the input. Finally, the softmax classifier obtains the label for each pixel, thus producing the segmentation result image. For example, the current image is input into BiSeNet to obtain a feature map, and then the softmax classifier classifies each pixel in this feature map to obtain a label for each pixel. Based on the label, the pixel value of each pixel is determined, where pixels belonging to the foreground region have a value of 255, and pixels belonging to the background region have a value of 0. For example, the foreground region can include areas containing portraits, and the background region can be areas that do not contain portraits.

[0431] The depth map edge optimization module 105 is used to extract the edge structure of the current image and its target depth map respectively, to obtain the edge structure information of the current image and the edge structure information of the target depth map; using the edge structure information of the current image as a reference, the difference between the edge structure of the target depth map and the edge structure of the current image is calculated, and then the edge position of the target depth map is modified by this difference, thereby optimizing the edge of the depth map and obtaining an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0432] The depth map edge optimization module 105 is used to extract the edge structure of the current image and its target depth map respectively, to obtain the edge structure information of the current image and the edge structure information of the target depth map; using the edge structure information of the current image as a reference, the difference between the edge structure of the target depth map and the edge structure of the current image is calculated, and then the edge position of the target depth map is modified by this difference, thereby optimizing the edge of the depth map and obtaining an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0433] The multi-view fusion module 106 is used to segment the optimized depth map according to the segmentation result map to obtain a foreground depth map and a background depth map. The background depth map is the depth map containing the background region in the optimized depth map, and the foreground depth map is the depth map containing the foreground region in the optimized depth map. The L background depth maps are fused according to the L second poses corresponding to the L background depth maps respectively to obtain a fused 3D scene. The L background depth maps include the background depth map of the pre-stored image and the background depth map of the current image. The L second poses include the second pose of the pre-stored image and the second pose of the current image. L is an integer greater than 1. The fused 3D scene is back-projected according to the second pose of the current image to obtain a fused background depth map. The fused background depth map and the foreground depth map of the current image are stitched together to obtain an updated depth map.

[0434] The virtual-real occlusion application module 107 is used to obtain a target image based on the updated depth map, the depth map of the virtual object, the current image, and the virtual object image. For any pixel P in the target image, if the first depth value is greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel in the virtual object image; if the first depth value is not greater than the second depth value, then the pixel value of pixel P is the pixel value of the corresponding pixel in the current image; wherein, the first depth value and the second depth value are the depth values ​​corresponding to pixel P in the target image in the updated depth map and the depth map of the virtual object, respectively; and the target image is displayed on the terminal device.

[0435] Optionally, the reference depth map is a depth map acquired by a TOF camera.

[0436] To reduce the power consumption of the terminal device, the aforementioned TOF camera acquires depth maps at a frame rate lower than the preset frame rate, and the resolution of the depth map is lower than the preset resolution; the terminal device 100 projects the depth map acquired by the TOF camera into three-dimensional space according to the second pose of the current image to obtain a fourth depth map; the fourth depth map is projected onto the reference image according to the pose of the reference image to obtain a reference depth map, wherein the reference image is an image adjacent to the current image in terms of acquisition time.

[0437] Optionally, the TOF camera can be a camera on terminal device 100 or a camera on other terminal devices. After the TOF camera on another terminal device acquires the depth map, the other terminal device sends the depth map acquired by the TOF camera to terminal device 100. This method allows the use of TOF-acquired depth maps even when terminal device 100 does not include a TOF camera, thereby improving the accuracy of the target depth map of the current image.

[0438] See Figure 11 , Figure 11 A schematic diagram of the terminal device is provided for an embodiment of this application. For example... Figure 11 As shown, the terminal device 1100 includes:

[0439] Acquisition unit 1101 is used to acquire the current image.

[0440] The calculation unit 1102 is used to calculate the first depth map corresponding to the current image based on the current image;

[0441] The acquisition unit 1101 is further configured to obtain a second depth map corresponding to the current image from the server based on the current image. The second depth map represents more depth information of the current image than the first depth map represents the current image, and the first depth map contains depth information of feature points that the second depth map does not have.

[0442] The depth estimation unit 1103 is used to obtain a target depth map of the current image based on the current image, the first depth map, and the second depth map. The target depth map represents more depth information of feature points in the current image than the second depth map represents.

[0443] The acquisition unit 1101 is also used to acquire virtual object images and virtual object depth maps;

[0444] The determining unit 1104 is used to obtain the target image based on the target depth map, the depth map of the virtual object, the current image, and the virtual object image.

[0445] In an optional embodiment, the target depth map and the depth map of the virtual object are used to determine the presentation and distribution of pixels of the virtual object image and pixels of the current image in the target image.

[0446] In an optional embodiment, the terminal device 1100 further includes:

[0447] The sending unit 1105 is used to send a first acquisition request to the server to obtain the 3D point cloud corresponding to the local map;

[0448] The receiving unit 1106 is used to receive a first response message sent by the server in response to the first acquisition request. The first response message carries a 3D point cloud corresponding to a local map, and the local map includes scene data corresponding to the current image.

[0449] In terms of obtaining the second depth map corresponding to the current image from the server based on the current image, the acquisition unit 1101 is specifically used for:

[0450] Obtain the second pose of the current image, wherein the second pose of the current image is the pose of the terminal device when the current image is captured;

[0451] The current image is projected based on its second pose and the corresponding 3D point cloud of the local map to obtain a second depth map.

[0452] In an optional embodiment, in obtaining the second depth map from the server based on the current image, the acquisition unit 1101 is further specifically configured to:

[0453] Feature extraction is performed on the current image to obtain the first 2D feature point of the current image. The first 2D feature point of the current image is matched with 2D feature points in multiple historical images to obtain the second 2D feature point of the current image. The second 2D feature point is the 2D feature point in the first 2D feature point of the current image that matches the eighth 2D feature point in the historical image. Based on the second pose of the current image, the second 2D feature point of the current image, the 2D feature point in the eighth 2D feature point in the historical image that matches the second 2D feature point, and the second pose of the historical images, the 3D point corresponding to the second 2D feature point of the current image is obtained. The second pose is the pose of the terminal device 1100 when capturing the current image.

[0454] In terms of projecting the current image based on the second pose of the current image and the corresponding 3D point cloud of the local map to obtain a second depth map, the acquisition unit 1101 is specifically used for:

[0455] The current image is projected based on its second pose, the 3D point cloud corresponding to the local map, and the 3D points corresponding to the second 2D feature points of the current image to obtain a second depth map.

[0456] Optionally, the first 2D feature point and the eighth 2D feature point include ORB feature points, AKAZE feature points, DOG feature points, HOG feature points, BRIEF feature points, BRISK feature points, FREAK feature points, ASLFeat feature points, or SuperPoint feature points.

[0457] In an optional embodiment, the terminal device 1100 further includes:

[0458] The sending unit 1105 is used to send a second acquisition request to the server. The second acquisition request carries the current image and instructs the server to obtain a second depth map based on the current image and the offline map stored by the server.

[0459] The receiving unit 1106 is used to receive a second response message sent by the server in response to the second acquisition request, the second response message carrying a second depth map.

[0460] In an optional embodiment, the computing unit 1102 is specifically used for:

[0461] The process involves: obtaining the first pose of the current image, which is the pose of the current image in a first coordinate system (the coordinate system of the offline map stored on the server); processing the current image based on the first pose to obtain a first image; the first image is obtained by transforming the second pose of the current image into the first pose; the second pose of the current image is the pose of the terminal device when capturing the current image; extracting features from the first image to obtain a third 2D feature point; matching the third 2D feature point with the eighth 2D feature points of multiple historical images to obtain a fourth 2D feature point; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the eighth 2D feature points of multiple historical images; obtaining the 3D point corresponding to the fourth 2D feature point based on the fourth 2D feature point, the first pose of the current image, the 2D feature point among the eighth 2D feature points of multiple historical images that matches the fourth 2D feature point, and the first pose of the historical image to which the 2D feature point belongs; and projecting the current image based on the first pose of the current image and the 3D point corresponding to the fourth 2D feature point to obtain a first depth map.

[0462] Optionally, the third 2D feature point includes ORB feature points, AKAZE feature points, DOG feature points, HOG feature points, BRIEF feature points, BRISK feature points, FREAK feature points, ASLFeat feature points, or SuperPoint feature points.

[0463] In an optional embodiment, the terminal device 1100 includes:

[0464] The sending unit 1105 is used to send a third acquisition request to the server. The third acquisition request carries the current image and is used to request the acquisition of the first pose of the current image.

[0465] The receiving unit 1106 is used to receive a third response message sent by the server in response to a third acquisition request, the third response message carrying the first pose of the current image.

[0466] In an optional embodiment, the terminal device 1100 includes:

[0467] Sending unit 1105 is used to send a fourth acquisition request to the server. The fourth acquisition request carries the current image and the second pose of the current image.

[0468] The receiving unit 1106 is configured to receive a fourth response message sent by the server in response to the fourth acquisition request. The fourth response message carries pose transformation information, which is used for the transformation between the second pose and the first pose of the current image.

[0469] In acquiring the first pose of the current image, the acquisition unit 1101 is specifically used for:

[0470] The second pose of the current image is transformed based on the pose transformation information to obtain the first pose of the current image.

[0471] In one feasible embodiment, the terminal device includes:

[0472] The sending unit 1105 is used to send a fifth acquisition request to the server; the fifth acquisition request carries the geographical location information of the terminal.

[0473] The receiving unit 1106 is used to receive a fifth response message sent by the server in response to the fifth acquisition request. The fifth response message carries a depth estimation model. The depth estimation model is a neural network model corresponding to the geographical location information of the terminal.

[0474] The determining unit 1104 is specifically used for:

[0475] The first and second depth maps are stitched together to obtain the third depth map; the current image and the third depth map are then input into the depth estimation model to obtain the target depth map.

[0476] In one feasible embodiment, the determining unit 1104 is specifically used for:

[0477] The initial convolutional neural network model is trained to obtain a depth estimation model; the first depth map and the second depth map are concatenated to obtain a third depth map; the current image and the third depth map are input into the depth estimation model to obtain the target depth map;

[0478] The initial convolutional neural network is trained to obtain a depth estimation model, including:

[0479] Multiple image samples and their corresponding depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps. A loss value is calculated based on the multiple predicted depth maps, the corresponding ground truth depth maps, and a loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain a depth estimation model for the current image. The loss function is determined based on the error between the predicted and ground truth depth maps, the error between the gradients of the predicted and ground truth depth maps, and the error between the normal vectors of the predicted and ground truth depth maps.

[0480] In an optional embodiment, the depth estimation unit 1103 is specifically used for:

[0481] Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps. The resolution of each of the T first feature maps is different, and the resolution of each of the T second feature maps is different; T is an integer greater than 1. The third depth map is obtained by stitching the first depth map and the second depth map. The first feature maps and the second feature maps with the same resolution from the T first feature maps and the T second feature maps are superimposed to obtain T third feature maps. The T third feature maps are upsampled and fused to obtain the target depth map of the current image.

[0482] In an optional embodiment, the depth estimation unit 1103 is specifically used for:

[0483] Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps. Multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps. The resolutions of each of the T first feature maps, the T second feature maps, and the T fourth feature maps are all different. The reference depth map is obtained from the depth map captured by the TOF camera, where T is an integer greater than 1. The third depth map is obtained by stitching together the first and second depth maps. The first, second, and fourth feature maps with the same resolution from the T first, T second, and T fourth feature maps are superimposed to obtain T fifth feature maps. The T fifth feature maps are then upsampled and fused to obtain the target depth map of the current image.

[0484] In an optional embodiment, the reference depth map is obtained from images captured by a TOF camera, specifically including:

[0485] The depth map acquired by the TOF camera is projected into three-dimensional space based on the second pose of the current image to obtain the fourth depth map; the fourth depth map is back-projected onto the reference image based on the pose of the reference image to obtain the reference depth map; the reference image is the image adjacent to the current image in terms of acquisition time.

[0486] In an optional embodiment, the upsampling and fusion process includes:

[0487] For feature map P' j Upsampling is performed to obtain the feature map P” j The feature map P” j The resolution and the (j+1)th feature map P in the processing object j+1 The resolutions are the same; the width of the (j+1)th feature map is j+1 times the width of the feature map with the smallest resolution in the processing object, where j is greater than or equal to 1 and less than or equal to T-1; the feature map P” j With feature map P j+1 The feature map P' is obtained by fusion. j+1 Let j = j + 1, and repeat the above steps until j = T - 1; T is the number of feature maps in the object being processed; where, when j = 1, feature map P' j To process the feature map with the lowest resolution in the object, when j = T-1, feature map P' j+1 This is the result of upsampling and fusion processing.

[0488] In an optional embodiment, the determining unit 1104 is specifically used for:

[0489] Edge optimization is performed on the target depth map based on the current image to obtain an optimized depth map; the accuracy of the optimized depth map is higher than that of the target depth map. The optimized depth map is then segmented to obtain a foreground depth map and a background depth map of the current image. The background depth map is the depth map containing the background region in the optimized depth map, and the foreground depth map is the depth map containing the foreground region in the optimized depth map. The L background depth maps are fused according to the L poses corresponding to the L background depth maps respectively to obtain a fused 3D scene. The L background depth maps include the background depth maps of the pre-stored image and the background depth map of the current image, and the L poses include the first pose of the pre-stored image and the current image; L is an integer greater than 1. The fused 3D scene is back-projected according to the first pose of the current image to obtain a fused background depth map. The fused background depth map and the foreground depth map of the current image are then stitched together to obtain an updated depth map. The virtual object and the current image are processed according to the updated depth map and the depth map of the virtual object to obtain the target image.

[0490] In an optional embodiment, the current image includes a target person, and the terminal device 1100 further includes:

[0491] The detection unit 1107 is used to detect the target person in the optimized depth map to obtain the detection result;

[0492] In segmenting the optimized depth map to obtain the foreground depth map and background depth map of the current image, the determining unit 1104 is specifically used for:

[0493] Based on the detection results, the optimized depth map is segmented to obtain the foreground depth map and background depth map of the current image. The foreground depth map of the current image includes the depth map corresponding to the target person.

[0494] It should be noted that the above-mentioned units (acquisition unit 1101, calculation unit 1102, depth estimation unit 1103, determination unit 1104, transmission unit 1105, reception unit 1106, and detection unit 1107) are used to perform the relevant steps of the above method.

[0495] In this embodiment, the terminal device 1100 is presented in the form of a unit. Here, "unit" can refer to an application-specific integrated circuit (ASIC), a processor and memory executing one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the above-mentioned functions. Furthermore, the acquisition unit 1101, calculation unit 1102, depth estimation unit 1103, determination unit 1104, and detection unit 1107 can be... Figure 13 The processor 1301 of the terminal device shown is used for implementation.

[0496] See Figure 12 , Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 12 As shown, the server 1200 includes:

[0497] The receiving unit 1201 is used to receive a depth estimation model request message sent by the terminal device, the request message carrying the geographical location information of the terminal device;

[0498] The acquisition unit 1202 is used to acquire the depth estimation model of the current image from multiple depth estimation models stored in the server 1200 based on the geographical location information of the terminal device; the depth estimation model of the current image is the depth estimation model corresponding to the geographical location information of the terminal device among the multiple depth estimation models; the multiple depth estimation models correspond one-to-one with multiple geographical location information.

[0499] The sending unit 1203 is used to send a response message to the terminal device in response to the depth estimation model request message, the response message carrying the depth estimation model of the current image.

[0500] In an optional embodiment, server 1200 further includes:

[0501] Training unit 1204 is used to train depth estimation models corresponding to multiple geographic location information for each geographic location information. Specifically, for any geographic location information S among the multiple geographic location information, training unit 1204 trains according to the following steps to obtain the depth estimation model corresponding to geographic location information S:

[0502] Multiple image samples and their corresponding depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps. The multiple image samples are acquired by the terminal device based on the geographic location information S. The loss value is calculated based on the multiple predicted depth maps, the corresponding true depth maps, and the loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain the depth estimation model corresponding to the geographic location information S. The loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0503] In an optional embodiment, the receiving unit 1201 is further configured to receive a first acquisition request sent by the terminal device;

[0504] The acquisition unit 1202 is further configured to obtain a local map from the offline map based on the position in the first pose of the current image, wherein the local map is the area indicated by the pose in the first pose of the current image in the offline map; and to acquire the 3D point cloud corresponding to the local map; wherein the first pose of the current image is the pose of the current image in a first coordinate system, and the first coordinate system is the coordinate system in which the offline map is located.

[0505] The sending unit 1203 is also configured to send a first response message to the terminal device in response to the first acquisition request, the first response message carrying a 3D point cloud corresponding to the local map.

[0506] In an optional embodiment, the receiving unit 1201 is further configured to receive a second acquisition request sent by the terminal device, the second acquisition request carrying the current image;

[0507] Server 1200 also includes:

[0508] The determining unit 1205 is used to obtain the second depth map corresponding to the current image based on the current image and the offline map;

[0509] The sending unit 1203 is further configured to send a second response message to the terminal device in response to the second acquisition request, the second response message carrying the second depth map.

[0510] In an optional embodiment, the determining unit 1205 is specifically used for:

[0511] The first pose of the current image is determined based on the current image. The first pose of the current image is the pose of the current image in the first coordinate system, which is the coordinate system of the offline map. Based on the position in the first pose of the current image, the 3D point cloud corresponding to the local map is obtained from the 3D point cloud corresponding to the offline map. The local map is the area indicated by the first position in the offline map stored on server 1200. The current image is projected based on the first pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map.

[0512] In an optional embodiment, the determining unit 1205 is further configured to:

[0513] Feature extraction is performed on the current image to obtain the ninth 2D feature point of the current image; the ninth 2D feature point of the current image is matched with the eleventh 2D feature point in multiple historical images to obtain the eleventh 2D feature point of the current image, which is the 2D feature point among the ninth 2D feature points of the current image that matches the eleventh 2D feature point in the historical images; based on the first pose of the current image, the eleventh 2D feature point of the current image, the 2D feature point among the eleventh 2D feature points in the historical images that matches the eleventh 2D feature point, and the first pose of the historical images, the 3D point corresponding to the eleventh 2D feature point of the current image is obtained;

[0514] The current image is projected based on its first pose and the corresponding 3D point cloud of the local map to obtain a second depth map, including:

[0515] The current image is projected based on the first pose of the current image, the 3D point cloud corresponding to the local map, and the 3D point corresponding to the eleventh 2D feature point of the current image to obtain a second depth map.

[0516] In an optional embodiment, the determining unit 1205 is further configured to:

[0517] The ninth 2D feature point of the current image is matched with the 2D feature point in the local map to obtain the sixth 2D feature point of the local map; the 3D point corresponding to the sixth 2D feature point is obtained based on the sixth 2D feature point, the ninth 2D feature point in the current image that matches the sixth 2D feature point, the pose of the local map and the first pose of the current image.

[0518] Based on the first pose of the current image, the 3D point cloud corresponding to the local map and the 3D points corresponding to the eleventh 2D feature point of the current image are projected onto the current image to obtain the aspect of the second depth map. The determination unit 1205 is specifically used for:

[0519] The current image is projected based on the first pose of the current image, the 3D point cloud corresponding to the local map, the 3D point corresponding to the eleventh 2D feature point of the current image, and the 3D point corresponding to the sixth 2D feature point of the current image to obtain a second depth map.

[0520] In an optional embodiment, the offline map includes multiple frames of base map, and the server 1200 further includes an update unit 1206, which is specifically used for:

[0521] Based on the current image, M base maps are obtained from multiple base maps, where the similarity between each base map and the current image is greater than a first threshold. The ninth 2D feature point of the current image is matched with the twelfth 2D feature point in the M base maps to obtain the seventh 2D feature point in the base maps. From the 3D point cloud corresponding to the local map, the 3D points corresponding to the seventh 2D feature point are selected to obtain the processed 3D points. The ninth 2D feature point of the current image is matched with the tenth 2D feature point in multiple historical images to obtain the eleventh 2D feature point of the current image. The eleventh 2D feature point is the 2D feature point in the current image that matches the eleventh 2D feature point in the historical image among the ninth 2D feature points. Based on the eleventh 2D feature point, the first pose of the current image, the 2D feature point in multiple historical images that matches the eleventh 2D feature point, and the first pose of the historical image to which the 2D feature point belongs, the 3D point corresponding to the eleventh 2D feature point is obtained. The processed 3D point and the 3D point corresponding to the eleventh 2D feature point are then processed based on the first pose of the current image to obtain the updated 3D point cloud corresponding to the local map.

[0522] In terms of projecting the current image based on the first pose of the current image and the corresponding 3D point cloud of the local map to obtain a second depth map, the determining unit 1205 is specifically used for:

[0523] The current image is projected based on the first pose of the current image and the 3D point cloud corresponding to the updated local map to obtain a second depth map.

[0524] Optionally, the ninth, eleventh, and twelfth 2D feature points all include SIFT feature points, SURF feature points, ASLFeat feature points, SuperPoint feature points, R2D2 feature points, or D2Net feature points.

[0525] In an optional embodiment, the receiving unit 1201 is further configured to receive a third acquisition request from the terminal device, the third acquisition request carrying the current image;

[0526] The determining unit 1205 is also used to perform feature point matching based on the current image and the offline map to determine the first pose of the current image, where the first pose is the pose of the current image in the first coordinate system, and the first coordinate system is the coordinate system of the offline map.

[0527] The sending unit 1203 is further configured to send a third response message to the terminal device in response to the third acquisition request, the third response message carrying the first pose.

[0528] In an optional embodiment, the receiving unit 1201 is further configured to receive a fourth acquisition request from the terminal device, the fourth acquisition request carrying a current image and a second pose of the current image; the second pose of the current image is the pose of the terminal device when capturing the current image;

[0529] The determining unit 1205 is also used to perform feature point matching based on the current image and the offline map to determine the first pose of the current image, where the first pose is the pose of the current image in the first coordinate system, and the first coordinate system is the coordinate system of the offline map; and to determine pose transformation information based on the second pose and the first pose of the current image, where the pose transformation information is used for the transformation between the second pose and the first pose of the current image.

[0530] The sending unit 1203 is further configured to send a fourth response message to the terminal device in response to the fourth acquisition request, the fourth response message carrying the pose transformation information.

[0531] It should be noted that the aforementioned units (receiving unit 1201, acquiring unit 1202, sending unit 1203, training unit 1204, determining unit 1205, and updating unit 1206) are used to execute the relevant steps of the above method. For example, receiving unit 1201 is used to execute the relevant content of step S501, acquiring unit 1202, training unit 1204, determining unit 1205, and updating unit 1206 are used to execute the relevant content of step S502, and sending unit 1203 is used to execute the relevant content of step S503.

[0532] In this embodiment, the server 1200 is presented in the form of a unit. Here, "unit" can refer to an application-specific integrated circuit (ASIC), a processor and memory executing one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the above-mentioned functions. Furthermore, the acquisition unit 1202, training unit 1204, determination unit 1205, and update unit 1206 can be... Figure 14 The processor 1401 of the server shown is used to implement this.

[0533] like Figure 13 The terminal device 1300 shown can Figure 13 The terminal device 1300 is implemented using the structure described above. It includes at least one processor 1301, at least one memory 1302, and at least one communication interface 1303. The processor 1301, the memory 1302, and the communication interface 1303 are connected through the communication bus and communicate with each other.

[0534] Processor 1301 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs in the above scheme.

[0535] The communication interface 1303 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.

[0536] The memory 1302 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0537] The memory 1302 stores the application code for executing the above scheme, and its execution is controlled by the processor 1301. The processor 1301 executes the application code stored in the memory 1302.

[0538] The code stored in memory 1302 can execute an image processing method provided above, such as: acquiring the current image; calculating a first depth map based on the current image; acquiring a second depth map from the server based on the current image, wherein the density of pixels in the second depth map is higher than that in the first depth map, the second depth map represents more depth information of the current image than the first depth map, and the number of pixels in the union of the first and second depth maps is greater than the number of pixels in the second depth map; obtaining a target depth map of the current image based on the current image, the first depth map, and the second depth map, wherein the target depth map represents more depth information of the current image than the second depth map; acquiring a virtual object image and a depth map of the virtual object; obtaining a target image, i.e., an image with virtual and real occlusion effects, based on the target depth map, the depth map of the virtual object, the current image, and the virtual object image; the target depth map and the depth map of the virtual object are used to determine the pixel corresponding to position P in the target image from the pixels at any position P in the current image and the pixels corresponding to position P in the virtual object image.

[0539] like Figure 14 The server 1400 shown can Figure 14Implemented using the structure described above, the server 1400 includes at least one processor 1401, at least one memory 1402, and at least one communication interface 1403. The processor 1401, the memory 1402, the display 1404, and the communication interface 1403 are connected via the communication bus and communicate with each other.

[0540] Processor 1401 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs in the above scheme.

[0541] The communication interface 1403 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.

[0542] Memory 1402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processor via a bus. Memory may also be integrated with the processor.

[0543] The memory 1402 stores the application code for executing the above scheme, and its execution is controlled by the processor 1401. The processor 1401 executes the application code stored in the memory 1402.

[0544] The code stored in memory 1402 can execute an image processing method provided above, such as: receiving a depth estimation model request message sent by a terminal device, the request message carrying the location of the terminal device; obtaining the depth estimation model of the current image from multiple depth estimation models stored in the server according to the location of the terminal device; the depth estimation model of the current image is the depth estimation model corresponding to the location of the terminal device among the multiple depth estimation models; the multiple depth estimation models correspond one-to-one with multiple locations; and sending a response message to the terminal device in response to the depth estimation model request message, the response message carrying the depth estimation model of the current image.

[0545] This application also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, includes some or all of the steps of any of the image processing methods described in the above method embodiments.

[0546] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0547] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0548] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0549] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0550] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0551] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0552] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0553] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Get the current image; A first depth map corresponding to the current image is calculated based on the current image; The second depth map corresponding to the current image is obtained from the server based on the current image. The second depth map represents more depth information of feature points in the current image than the first depth map represents depth information of feature points in the current image. The first depth map contains depth information of feature points that the second depth map does not have. A target depth map of the current image is obtained based on the current image, the first depth map, and the second depth map. The target depth map represents more depth information of feature points in the current image than the second depth map represents depth information of feature points in the current image. Obtain the image of the virtual object and the depth map of the virtual object; The target image is obtained based on the target depth map, the depth map of the virtual object, the current image, and the virtual object image.

2. The method according to claim 1, characterized in that, The target depth map and the virtual object depth map are used to determine the presentation and distribution of pixels of the virtual object image and pixels of the current image in the target image.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the second depth map corresponding to the current image from the server based on the current image includes: Send a first acquisition request to the server; The system receives a first response message from the server in response to the first acquisition request. The first response message carries a 3D point cloud corresponding to a local map. The local map includes scene data corresponding to the current image. Obtain the second pose of the current image, which is the pose of the terminal device when it captured the current image; The current image is projected based on its second pose and the 3D point cloud corresponding to the local map to obtain the second depth map.

4. The method according to claim 3, characterized in that, The step of obtaining the second depth map corresponding to the current image from the server based on the current image further includes: Feature extraction is performed on the current image to obtain the first 2D feature points of the current image. The first 2D feature point of the current image is matched with the eighth 2D feature point in multiple historical images to obtain the second 2D feature point of the current image. The second 2D feature point is the 2D feature point in the first 2D feature point of the current image that matches the eighth 2D feature point in the historical image. Based on the second pose of the current image, the second 2D feature point of the current image, the eighth 2D feature point in the historical image that matches the second 2D feature point, and the second pose of the historical image, the 3D point corresponding to the second 2D feature point of the current image is obtained; the second pose of the historical image is the pose of the terminal device when the historical image was captured. The step of projecting the current image based on the second pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map includes: The current image is projected based on the second pose of the current image, the 3D point cloud corresponding to the local map, and the 3D points corresponding to the second 2D feature points of the current image to obtain the second depth map.

5. The method according to claim 4, characterized in that, Both the first 2D feature point and the eighth 2D feature point include ORB feature points, AKAZE feature points, DOG feature points, HOG feature points, BRIEF feature points, BRISK feature points, FREAK feature points, ASLFeat feature points, or SuperPoint feature points.

6. The method according to claim 1 or 2, characterized in that, The step of obtaining the second depth map of the current image from the server based on the current image includes: A second acquisition request is sent to the server, the second acquisition request carrying the current image, the second acquisition request instructing the server to obtain the second depth map based on the current image and the offline map stored by the server; The server receives a second response message in response to the second acquisition request, the second response message carrying the second depth map.

7. The method according to claim 1, characterized in that, The step of calculating the first depth map of the current image based on the current image includes: Obtain the first pose of the current image, where the first pose is the pose of the current image in a first coordinate system; the first coordinate system is the coordinate system of the offline map stored on the server. The current image is processed according to the first pose to obtain a first image; the first image is an image obtained by transforming the second pose of the current image into the first pose; the second pose of the current image is the pose when the terminal device captured the current image; Feature extraction is performed on the first image to obtain a third 2D feature point of the first image; the third 2D feature point is matched with an eighth 2D feature point of multiple historical images to obtain a fourth 2D feature point of the first image; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the eighth 2D feature point of the multiple historical images. The 3D point corresponding to the fourth 2D feature point is obtained based on the fourth 2D feature point, the first pose of the current image, the 2D feature point that matches the fourth 2D feature point among the eighth 2D feature points of the multiple historical images, and the first pose of the historical image to which the 2D feature point belongs. The current image is projected based on the first pose of the current image and the 3D point corresponding to the fourth 2D feature point to obtain the first depth map.

8. The method according to claim 7, characterized in that, The third 2D feature point includes ORB feature point, AKAZE feature point, DOG feature point, HOG feature point, BRIEF feature point, BRISK feature point, FREAK feature point, ASLFeat feature point, or SuperPoint feature point.

9. The method according to claim 7 or 8, characterized in that, Obtaining the first pose of the current image includes: A third acquisition request is sent to the server, the third acquisition request carrying the current image, the third acquisition request being used to request the acquisition of the first pose of the current image; The server sends a third response message in response to the third acquisition request, the third response message carrying the first pose of the current image.

10. The method according to claim 7 or 8, characterized in that, Obtaining the first pose of the current image includes: A fourth acquisition request is sent to the server, the fourth acquisition request carrying the current image and the second pose of the current image. The system receives a fourth response message from the server in response to the fourth acquisition request. This fourth response message carries pose transformation information used for the transformation between the second pose and the first pose of the current image. The second pose of the current image is transformed according to the pose transformation information to obtain the first pose of the current image.

11. The method according to claim 1, characterized in that, The specific steps for obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map are as follows: A fifth acquisition request is sent to the server; the fifth acquisition request carries the terminal's geographical location information; The system receives a fifth response message sent by the server in response to the fifth acquisition request. The fifth response message carries a depth estimation model, which is a neural network model corresponding to the geographical location information of the terminal. The first depth map and the second depth map are stitched together to obtain the third depth map; The current image and the third depth map are input into the depth estimation model to obtain the target depth map.

12. The method according to claim 1, characterized in that, The specific steps for obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map are as follows: The initial convolutional neural network model is trained to obtain a depth estimation model; The first depth map and the second depth map are stitched together to obtain the third depth map; The current image and the third depth map are input into the depth estimation model to obtain the target depth map; The step of training the initial convolutional neural network to obtain the depth estimation model includes: Multiple image samples and their corresponding multiple depth map samples are input into the initial convolutional neural network for processing to obtain multiple predicted depth maps; The loss value is calculated based on the multiple predicted depth maps, the real depth maps corresponding to the multiple image samples, and the loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain the depth estimation model of the current image; The loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

13. The method according to claim 1, characterized in that, The step of obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map includes: Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; each of the T first feature maps has a different resolution, and each of the T second feature maps has a different resolution; T is an integer greater than 1; the third depth map is obtained by stitching together the first depth map and the second depth map; The first and second feature maps with the same resolution from the T first feature maps and the T second feature maps are superimposed to obtain T third feature maps; The T third feature maps are upsampled and fused to obtain the target depth map of the current image.

14. The method according to claim 1, characterized in that, The step of obtaining the target depth map of the current image based on the current image, the first depth map, and the second depth map includes: Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps. Each of the T first feature maps has a different resolution, each of the T second feature maps has a different resolution, and each of the T fourth feature maps has a different resolution. The reference depth map is obtained based on the depth map acquired by the Time-of-Flight (TOF) camera, where T is an integer greater than 1. The third depth map is obtained by stitching together the first and second depth maps. The first feature map, the second feature map, and the fourth feature map with the same resolution among the T first feature maps, the T second feature maps, and the T fourth feature maps are superimposed to obtain T fifth feature maps; The T fifth feature maps are upsampled and fused to obtain the target depth map of the current image.

15. The method according to claim 14, characterized in that, The reference depth map is obtained from images captured by a Time-of-Flight (TOF) camera, and specifically includes: Based on the second pose of the current image, the depth map captured by the TOF camera is projected into three-dimensional space to obtain a fourth depth map; The fourth depth map is back-projected onto the reference image based on the pose of the reference image to obtain the reference depth map; the reference image is an image that is adjacent to the current image in terms of acquisition time.

16. The method according to any one of claims 13-15, characterized in that, The upsampling and fusion processing includes: For feature map P' j Upsampling is performed to obtain the feature map P'' j The feature map P'' j The resolution and the (j+1)th feature map P in the processing object j+1 The resolutions are the same; the width of the (j+1)th feature map is j+1 times the width of the feature map with the smallest resolution in the processing object, where j is greater than or equal to 1 and less than or equal to T-1; The feature map P'' j With the feature map P j+1 The feature map P' is obtained by fusion. j+1 , Let j = j + 1, and repeat the above steps until j = T - 1; where T is the number of feature maps in the processing object; Wherein, when j=1, the feature map P' j For the feature map with the smallest resolution among the processed objects, when j=T-1, the feature map P' j+1 This is the result of the upsampling and fusion processing.

17. The method according to claim 1, characterized in that, The step of obtaining the target image based on the target depth map, the depth map of the virtual object, the current image, and the virtual object image includes: The target depth map is edge-optimized based on the current image to obtain an optimized depth map; the accuracy of the optimized depth map is higher than that of the target depth map. The optimized depth map is segmented to obtain a foreground depth map and a background depth map of the current image. The background depth map is the depth map containing the background region in the optimized depth map, and the foreground depth map is the depth map containing the foreground region in the optimized depth map. The L background depth maps are fused according to the L poses corresponding to the L background depth maps respectively to obtain the fused 3D scene; the L background depth maps include the background depth maps of the pre-stored image and the background depth map of the current image, and the L poses include the first pose of the pre-stored image and the current image; L is an integer greater than 1; The fused 3D scene is back-projected based on the first pose of the current image to obtain a fused background depth map; the fused background depth map and the foreground depth map of the current image are then stitched together to obtain an updated depth map. The virtual object and the current image are processed based on the updated depth map and the depth map of the virtual object to obtain the target image.

18. The method according to claim 17, characterized in that, The current image includes the target person, and the method further includes: The optimized depth map is then used to detect the target person to obtain the detection results; The step of segmenting the optimized depth map to obtain the foreground depth map and background depth map of the current image includes: The optimized depth map is segmented based on the detection results to obtain the foreground depth map and background depth map of the current image, wherein the foreground depth map of the current image includes the depth map corresponding to the target person.

19. A terminal device, characterized in that, include: The acquisition unit is used to acquire the current image; The calculation unit is used to calculate a first depth map corresponding to the current image based on the current image; The acquisition unit is further configured to obtain a second depth map corresponding to the current image from the server based on the current image, wherein the second depth map represents more depth information of feature points in the current image than the first depth map represents depth information of feature points in the current image; and the first depth map contains depth information of feature points not present in the second depth map; A depth estimation unit is configured to obtain a target depth map of the current image based on the current image, the first depth map, and the second depth map, wherein the target depth map represents more depth information of feature points in the current image than the second depth map represents depth information of feature points in the current image. The acquisition unit is also used to acquire virtual object images and depth maps of the virtual objects; The determining unit is used to obtain a target image based on the target depth map, the depth map of the virtual object, the current image, and the virtual object image.

20. The terminal device according to claim 19, characterized in that, The target depth map and the virtual object depth map are used to determine the presentation and distribution of pixels of the virtual object image and pixels of the current image in the target image.

21. The terminal device according to claim 19 or 20, characterized in that, The terminal device also includes: The sending unit is used to send a first acquisition request to the server to obtain the 3D point cloud corresponding to the local map; The receiving unit is configured to receive a first response message sent by the server in response to the first acquisition request. The first response message carries a 3D point cloud corresponding to the local map, and the local map includes scene data corresponding to the current image. In the aspect of obtaining the second depth map corresponding to the current image from the server based on the current image, the acquisition unit is specifically used for: Obtain the second pose of the current image, wherein the second pose of the current image is the pose of the terminal device when capturing the current image; The current image is projected based on the second pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map.

22. The terminal device according to claim 21, characterized in that, In terms of obtaining the second depth map corresponding to the current image from the server based on the current image, the acquisition unit is further configured to: Feature extraction is performed on the current image to obtain the first 2D feature points of the current image. The first 2D feature point of the current image is matched with the eighth 2D feature point in multiple historical images to obtain the second 2D feature point of the current image. The second 2D feature point is the 2D feature point in the first 2D feature point of the current image that matches the eighth 2D feature point in the historical image. Based on the second pose of the current image, the second 2D feature point of the current image, the eighth 2D feature point in the historical image that matches the second 2D feature point, and the second pose of the historical image, the 3D point corresponding to the second 2D feature point of the current image is obtained; the second pose of the historical image is the pose of the terminal device when the historical image was captured. In the aspect of projecting the current image based on the second pose of the current image and the 3D point cloud corresponding to the local map to obtain the second depth map, the acquisition unit is specifically used for: The current image is projected based on the second pose of the current image, the 3D point cloud corresponding to the local map, and the 3D points corresponding to the second 2D feature points of the current image to obtain the second depth map.

23. The terminal device according to claim 22, characterized in that, Both the first 2D feature point and the eighth 2D feature point include ORB feature points, AKAZE feature points, DOG feature points, HOG feature points, BRIEF feature points, BRISK feature points, FREAK feature points, ASLFeat feature points, or SuperPoint feature points.

24. The terminal device according to claim 19 or 20, characterized in that, The terminal device also includes: The sending unit is configured to send a second acquisition request to the server, the second acquisition request carrying the current image, the second acquisition request instructing the server to obtain the second depth map based on the current image and the offline map stored by the server; The receiving unit is configured to receive a second response message sent by the server in response to the second acquisition request, the second response message carrying the second depth map.

25. The terminal device according to claim 19, characterized in that, The computing unit is specifically used for: Obtain the first pose of the current image, where the first pose is the pose of the current image in a first coordinate system; the first coordinate system is the coordinate system of the offline map stored on the server. The current image is processed according to the first pose to obtain a first image; the first image is an image obtained by transforming the second pose of the current image into the first pose; the second pose of the current image is the pose when the terminal device captured the current image; Feature extraction is performed on the first image to obtain a third 2D feature point of the first image; the third 2D feature point is matched with an eighth 2D feature point of multiple historical images to obtain a fourth 2D feature point of the first image; the fourth 2D feature point is the 2D feature point among the third 2D feature points that matches the eighth 2D feature point of the multiple historical images. The 3D point corresponding to the fourth 2D feature point is obtained based on the fourth 2D feature point, the first pose of the current image, the 2D feature point that matches the fourth 2D feature point among the eighth 2D feature points of the multiple historical images, and the first pose of the historical image to which the 2D feature point belongs. The first depth map is obtained by projecting the current image onto the first pose of the current image and the 3D point corresponding to the fourth 2D feature point.

26. The terminal device according to claim 25, characterized in that, The third 2D feature point includes ORB feature point, AKAZE feature point, DOG feature point, HOG feature point, BRIEF feature point, BRISK feature point, FREAK feature point, ASLFeat feature point, or SuperPoint feature point.

27. The terminal device according to claim 25 or 26, characterized in that, The terminal device includes: The sending unit is used to send a third acquisition request to the server, the third acquisition request carrying the current image, the third acquisition request being used to request the acquisition of the first pose of the current image; The receiving unit is configured to receive a third response message sent by the server in response to the third acquisition request, wherein the third response message carries the first pose of the current image.

28. The terminal device according to claim 25 or 26, characterized in that, The terminal device includes: The sending unit is configured to send a fourth acquisition request to the server, the fourth acquisition request carrying the current image and the second pose of the current image. The receiving unit is configured to receive a fourth response message sent by the server in response to the fourth acquisition request. The fourth response message carries pose transformation information, which is used for the transformation between the second pose and the first pose of the current image. In the aspect of obtaining the first pose of the current image, the computing unit is specifically used for: The second pose of the current image is transformed according to the pose transformation information to obtain the first pose of the current image.

29. The terminal device according to claim 19, characterized in that, The terminal device includes: The sending unit is used to send a fifth acquisition request to the server; the fifth acquisition request carries the geographical location information of the terminal. The receiving unit is configured to receive a fifth response message sent by the server in response to a fifth acquisition request, wherein the fifth response message carries a depth estimation model; the depth estimation model is a neural network model corresponding to the geographical location information of the terminal. The determining unit is specifically used for: The first depth map and the second depth map are stitched together to obtain the third depth map; The current image and the third depth map are input into the depth estimation model to obtain the target depth map.

30. The terminal device according to claim 19, characterized in that, The determining unit is specifically used for: The initial convolutional neural network model is trained to obtain a depth estimation model; The first depth map and the second depth map are stitched together to obtain the third depth map; The current image and the third depth map are input into the depth estimation model to obtain the target depth map; The step of training the initial convolutional neural network to obtain the depth estimation model includes: Multiple image samples and their corresponding multiple depth map samples are input into the initial convolutional neural network for processing to obtain multiple predicted depth maps; The loss value is calculated based on the multiple predicted depth maps, the real depth maps corresponding to the multiple image samples, and the loss function. The parameters in the initial convolutional neural network are adjusted based on the loss value to obtain the depth estimation model of the current image; The loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

31. The terminal device according to claim 19, characterized in that, The depth estimation unit is specifically used for: Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; each of the T first feature maps has a different resolution, and each of the T second feature maps has a different resolution; T is an integer greater than 1; the third depth map is obtained by stitching together the first depth map and the second depth map; The first and second feature maps with the same resolution from the T first feature maps and the T second feature maps are superimposed to obtain T third feature maps; The T third feature maps are upsampled and fused to obtain the target depth map of the current image.

32. The terminal device according to claim 19, characterized in that, The depth estimation unit is specifically used for: Multi-scale feature extraction is performed on the current image to obtain T first feature maps, and multi-scale feature extraction is performed on the third depth map to obtain T second feature maps; multi-scale feature extraction is performed on the reference depth map to obtain T fourth feature maps. The resolution of each of the T first feature maps, the resolution of each of the T second feature maps, and the resolution of each of the T fourth feature maps are all different. The reference depth map is obtained based on the depth map captured by the time-of-flight (TOF) camera, where T is an integer greater than 1; the third depth map is obtained by stitching together the first depth map and the second depth map. The first feature map, the second feature map, and the fourth feature map with the same resolution among the T first feature maps, the T second feature maps, and the T fourth feature maps are superimposed to obtain T fifth feature maps; The T fifth feature maps are upsampled and fused to obtain the target depth map of the current image.

33. The terminal device according to claim 32, characterized in that, The reference depth map is obtained from images captured by a Time-of-Flight (TOF) camera, and specifically includes: Based on the second pose of the current image, the depth map captured by the TOF camera is projected into three-dimensional space to obtain a fourth depth map; The fourth depth map is back-projected onto the reference image based on the pose of the reference image to obtain the reference depth map; the reference image is an image that is adjacent to the current image in terms of acquisition time.

34. The terminal device according to any one of claims 31-33, characterized in that, The upsampling and fusion processing includes: For feature map P' j Upsampling is performed to obtain the feature map P'' j The feature map P'' j The resolution and the (j+1)th feature map P in the processing object j+1 The resolutions are the same; the width of the (j+1)th feature map is j+1 times the width of the feature map with the smallest resolution in the processing object, where j is greater than or equal to 1 and less than or equal to T-1; The feature map P'' j With the feature map P j+1 The feature map P' is obtained by fusion. j+1 , Let j = j + 1, and repeat the above steps until j = T - 1; where T is the number of feature maps in the processing object; Wherein, when j=1, the feature map P' j For the feature map with the smallest resolution among the processed objects, when j=T-1, the feature map P' j+1 This is the result of the upsampling and fusion processing.

35. The terminal device according to claim 19, characterized in that, The determining unit is specifically used for: The target depth map is edge-optimized based on the current image to obtain an optimized depth map; the accuracy of the optimized depth map is higher than that of the target depth map. The optimized depth map is segmented to obtain a foreground depth map and a background depth map of the current image. The background depth map is the depth map containing the background region in the optimized depth map, and the foreground depth map is the depth map containing the foreground region in the optimized depth map. The L background depth maps are fused according to the L poses corresponding to the L background depth maps respectively to obtain the fused 3D scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the first pose of the pre-stored images and the current image; L is an integer greater than 1; The fused 3D scene is back-projected based on the first pose of the current image to obtain a fused background depth map; the fused background depth map and the foreground depth map of the current image are then stitched together to obtain an updated depth map. The virtual object and the current image are processed based on the updated depth map and the depth map of the virtual object to obtain the target image.

36. The terminal device according to claim 35, characterized in that, The current image includes the target person, and the terminal device further includes: The detection unit is used to detect target people in the optimized depth map to obtain detection results; In the aspect of segmenting the optimized depth map to obtain the foreground depth map and background depth map of the current image, the determining unit is specifically used for: The optimized depth map is segmented based on the detection results to obtain the foreground depth map and background depth map of the current image, wherein the foreground depth map of the current image includes the depth map corresponding to the target person.

37. An electronic device comprising a communication interface, a memory, and one or more processors; wherein, One or more programs are stored in the memory; characterized in that, when the one or more processors execute the one or more programs, the electronic device causes the electronic device to implement the method as described in any one of claims 1 to 18.

38. A computer storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 18.

39. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 18.

Images