Method for processing an image and related device

Abstract

This application relates to the field of AR, and specifically relates to a method and device for processing images. The method includes: obtaining a current image and a virtual object image, and obtaining a first depth map and a second depth map of the current image according to the current image, where the second depth map is obtained from a server; performing depth estimation according to the current image, the first depth map and the second depth map to obtain a target depth map of the current image; superimposing and displaying the virtual object image and the current image according to the target depth map of the current image. By performing depth estimation based on the current image, the first depth map and the second depth map, a high-precision target depth map is obtained, thereby solving the problem that there will be inter-frame flicker and instability in the subsequent virtual-real occlusion effect.

Description

Technical Field

[0001] This application relates to the field of Augmented Reality (AR), and particularly to an image processing method and related devices. Background Art

[0002] AR uses virtual information generated by a computer to supplement the real world, making the virtual information and the real world seem to coexist in the same space. Currently, most AR applications simply superimpose virtual objects in front of the real scene without correctly handling the occlusion relationship between virtual objects and the real world, which easily causes a disorder in the spatial position in the user's senses and cannot exceed the sensory experience of reality. The effect without virtual-real occlusion is as Figure 1a shown. In AR applications, it is necessary to process the real relationship between virtual objects and the real scene, that is, virtual-real occlusion. The effect with virtual-real occlusion is as Figure 1b shown. The correct occlusion relationship can enable users to have a natural and correct spatial perception in AR applications; the wrong occlusion relationship will reduce the realism of AR applications.

[0003] When there is an incorrect virtual-real occlusion relationship in the fused image, it is very difficult for the observer to correctly judge the relative position relationship between virtual and real objects, and it is difficult to obtain a realistic virtual-real fusion effect. The wrong occlusion relationship is likely to cause the observer to get lost in the sensory direction and be disordered in the spatial position, and the virtual-real fusion result is not real. Therefore, in order to enhance the realism of virtual objects in the real scene, obtain a realistic virtual-real fusion effect, and solve the occlusion problem in AR is of great significance. The virtual-real occlusion processing in AR focuses on enabling virtual objects to be correctly occluded by the objects in front of them in the real scene. It generally uses means such as depth extraction or scene modeling to obtain the occlusion relationship between real objects and virtual objects in the virtual-real fusion scene, extract the occlusion edges of foreground objects in the real scene image, and finally generate a virtual-real fusion image with a correct occlusion relationship.

[0004] Apple released the latest iPad Pro 2020, which uses RGB images and depth maps collected by a direct time of flight (dToF) camera as input to achieve full-scene virtual-real occlusion. Based on the depth map collected by the dToF camera, portrait segmentation, and monocular depth estimation for virtual-real occlusion, but there are problems such as scale ambiguity and inconsistency in monocular depth estimation, resulting in frame-to-frame flicker, instability, and unsharp occlusion edges in the final virtual-real occlusion effect. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method and related devices. In the present application, the cloud local map update algorithm can solve the problem that directly using offline map points to participate in depth map estimation introduces errors due to changes in the offline map and the current usage scenario, and at the same time solve the problems of scale ambiguity and inter-frame instability in monocular depth estimation, and further solve the problems of virtual-real occlusion flicker and inconsistent inter-frame occlusion phenomena; by optimizing the edges of the target depth map of the current image to obtain an optimized depth map, and then fusing multiple frames of depth maps, a depth map with sharper portrait edges is obtained, which is beneficial to improving the virtual-real occlusion effect.

[0006] In a first aspect, the embodiments of the present application provide an image processing method, including:

[0007] Obtain a current image and a virtual object image, and obtain a first depth map and a second depth map of the current image according to the current image, where the second depth map is obtained from a server; perform feature extraction according to the current image, the first depth map and the second depth map of the current image, and obtain a target depth map of the current image according to the result of the feature extraction.

[0008] Perform depth estimation on the current image, the first depth map and the second depth map to obtain a high-precision target depth map.

[0009] In a feasible embodiment, the method of this embodiment further includes:

[0010] Overlay and display the virtual object image and the current image according to the target depth map of the current image.

[0011] By introducing a high-precision target depth map, the problem of frame-interval flicker and instability in the virtual-real occlusion effect is solved.

[0012] In a feasible embodiment, obtaining the first depth map of the current image according to the current image includes:

[0013] Perform feature extraction on the current image to obtain the first 2D feature points of the current image; match the first 2D feature points of the current image with the pre-stored 2D feature points to obtain the second 2D feature points in the current image; according to the correspondence between the second 2D feature points in the current image and the pre-stored 2D feature points and 3D points, obtain the 3D points corresponding to the second 2D feature points in the current image; where the first depth map of the current image includes the 3D points corresponding to the second 2D feature points in the current image.

[0014] Among them, the feature points refer to the points where the image gray value changes violently or the points with a large curvature on the image edge (i.e., the intersection of two edges).

[0015] The above-mentioned corresponding relationship between the preselected stored 2D feature points and 3D points means that for each preselected stored 2D feature point, there is a corresponding 3D point.

[0016] The matching of the 2D feature points mentioned in this application specifically means that the similarity between two matching 2D feature points is higher than the preset similarity.

[0017] By matching the 2D feature points of the current image with the pre-stored 2D feature points, a first depth map is obtained according to the matching result, and then depth estimation can be performed based on the first depth map to obtain a high-precision target depth map.

[0018] In a feasible embodiment, obtaining the first depth map of the current image according to the current image includes:

[0019] Performing feature extraction on the current image to obtain the first 2D feature points of the current image; matching the first 2D feature points of the current image with the 2D feature points of the local map obtained from the server to obtain the third 2D feature points in the local map; obtaining the 3D points corresponding to the third 2D feature points in the local map according to the corresponding relationship between the second 2D feature points in the local map and the 2D feature points and 3D points in the local map; wherein, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

[0020] By performing 2D feature point matching between the current image and the local map, a first depth map is obtained according to the matching result, and then depth estimation can be performed based on the first depth map to obtain a high-precision target depth map.

[0021] In a feasible embodiment, matching the first 2D feature points of the current image with the 2D feature points of the local map to obtain the third 2D feature points in the local map includes:

[0022] Obtaining a target map from the local map according to the first pose, where the position of the target map in the local map is associated with the position indicated by the angle information in the first pose; the first pose is the pose obtained by the terminal device according to the current image converted into the pose in the world coordinate system, matching the 2D feature points in the target map with the first 2D feature points of the current image to obtain the third 2D feature points of the target map, and the third 2D feature points of the local map include the third 2D feature points of the target map.

[0023] Among them, the world coordinate system is the absolute coordinate system of the system, and the coordinates of all points on the screen are determined by the origin of this coordinate system before the user coordinate system is established.

[0024] Determining the target map from the local map through the first pose reduces the matching range and improves the efficiency of determining the first depth map.

[0025] In a feasible embodiment, feature extraction is performed based on the current image, the first depth map, and the second depth map, and a target depth map of the current image is obtained according to the result of the feature extraction, including:

[0026] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform feature extraction on the third depth map to obtain T second feature maps; the resolutions of each of the T first feature maps are different from each other, and the resolutions of each of the T second feature maps are different from each other; T is an integer greater than 1; superimpose the first feature map and the second feature map with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image; wherein, the third depth map is the above-mentioned first depth map, or the third depth map is obtained by splicing the first depth map and the second depth map.

[0027] The multi-scale feature extraction specifically refers to the operation of convolving an image with multiple different convolutional kernels.

[0028] The "superimposition" in this application specifically refers to processing the images to be superimposed at the pixel level. For example, the two images to be superimposed include a size of H*W, and the size of the superimposed image is H*2W, or 2H*W; for another example, the three images to be superimposed include a size of H*W, and the size of the superimposed image is H*3W, or 3H*W.

[0029] Perform depth estimation based on the first depth map and the second depth map to obtain a high-precision target depth map, thereby solving the problems of inter-frame flicker and instability in the subsequent virtual-real occlusion effect.

[0030] In a feasible embodiment, feature extraction is performed based on the current image, the first depth map, and the second depth map, and a target depth map of the current image is obtained according to the result of the feature extraction, including:

[0031] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; perform multi-scale feature extraction on the reference depth map to obtain T fourth feature maps. The resolutions of each of the T first feature maps are different, the resolutions of each of the T second feature maps are different, and the resolutions of each of the T fourth feature maps are different; the reference depth map is obtained based on the depth map collected by a time of flight (TOF) camera, and T is an integer greater than 1; Superimpose the first feature map, the second feature map, and the fourth feature map with the same resolution among the T first feature maps, the T second feature maps, and the T fourth feature maps to obtain T fifth feature maps; perform upsampling and fusion processing on the T fifth feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by stitching the first depth map and the second depth map, or the third depth map is the above-mentioned first depth map.

[0032] Introducing the depth map collected by the TOF camera during depth estimation further improves the accuracy of the target depth map of the current image, thereby solving the problems of inter-frame flicker and instability in the subsequent virtual-real occlusion effect.

[0033] In a feasible embodiment, the reference depth map is obtained based on the image collected by the TOF camera, specifically including:

[0034] Project the depth map collected by the TOF camera into three-dimensional space according to the pose of the current image to obtain a corresponding fourth depth map; back-project the fourth depth map onto the reference image according to the pose of the reference image to obtain the reference depth map; the reference image is an image adjacent to the current image in terms of acquisition time; wherein, the resolution of the depth map collected by the TOF camera is lower than the preset resolution, and the frame rate of the TOF camera when collecting this depth map is lower than the preset frame rate.

[0035] To reduce the power consumption of the terminal device, reduce the frame rate and the resolution of the depth map collected by the TOF camera.

[0036] In a feasible embodiment, the above-mentioned upsampling and fusion processing includes:

[0037] S1: Upsample the feature map P’ j to obtain a feature map P” j , and the resolution of this feature map P” j is the same as that of the (j + 1)-th feature map P j+1 in the processing object. The width of the feature map P j+1 is (j + 1) times the width of the feature map with the smallest resolution in the processing object, where j is an integer greater than 0 and less than T; T is the number of feature maps in the processing object;

[0038] S2: Merge the feature map P” j with the feature map P j+1 to obtain the third feature map P’ j+1 ,

[0039] S3: Let j = j + 1, and repeat S1 - S3 until j = T - 1;

[0040] where, when j = 1, the third feature map P’ j is the feature map with the smallest resolution in the processing object, and when j = T - 1, the third feature map P’ j+1 is the result of upsampling and fusion processing.

[0041] In a feasible embodiment, feature extraction is performed based on the current image, the first depth map, and the second depth map, and the target depth map of the current image is obtained according to the result of the feature extraction, including:

[0042] Input the current image and the third depth map into the depth estimation model of the current image for feature extraction, and obtain the target depth map of the current image according to the result of the feature extraction, where the depth estimation model is implemented based on a convolutional neural network.

[0043] In a feasible embodiment, the method of the present application further includes:

[0044] Send a depth estimation model acquisition request to the server, where the depth estimation model acquisition request carries the current image and the location of the terminal device; receive the response message corresponding to the depth estimation model acquisition request sent by the server, and the response message carries the depth estimation model of the current image, and the depth estimation model of the current image is obtained by the server according to the current image and the location of the terminal device in the world coordinate system.

[0045] Optionally, the world coordinate system can be a universal transverse mercator grid system (UTM) coordinate system or a GPS coordinate system, etc.

[0046] By obtaining the depth estimation model from the server, the terminal device does not need to train the depth estimation model by itself, reducing the power consumption of the terminal device and improving the real-time performance of virtual-real occlusion at the same time.

[0047] In a feasible embodiment, the method of the present application further includes:

[0048] Train the initial convolutional neural network model to obtain the depth estimation model;

[0049] Among them, training the initial convolutional neural network to obtain a depth estimation model includes:

[0050] Inputting multiple image samples and their corresponding multiple depth map samples into the initial convolutional neural network for processing to obtain multiple predicted depth maps; calculating a loss value based on the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; adjusting the parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of the current image; wherein, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0051] It should be noted here that the above is only one training process; in actual applications, it will be iterated multiple times in the above manner until the calculated loss value converges; the convolutional neural network model when the loss value converges is determined as the depth estimation model of the current image.

[0052] In a feasible embodiment, superimposing and displaying a virtual object image and the current image according to the target depth map of the current image includes:

[0053] Performing edge optimization on the target depth map of the current image to obtain an optimized depth map; superimposing and displaying the virtual object image and the current image according to the optimized depth map.

[0054] By performing edge optimization on the target depth map of the current image, a depth map with sharp human edges is obtained, which is beneficial to improving the virtual-real occlusion effect.

[0055] In a feasible embodiment, superimposing and displaying a virtual object image and the current image according to the target depth map of the current image includes:

[0056] Segmenting the optimized depth map to obtain the foreground depth map and the background depth map of the current image, where the background depth map is the depth map containing the background area in the optimized depth map, and the foreground depth map is the depth map containing the foreground area in the optimized depth map. The optimized depth map is obtained by performing edge optimization on the target depth map of the current image; fusing the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; performing back-projection on the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map; splicing the fused background depth map and the foreground depth map of the current image to obtain an updated depth map; superimposing and displaying the virtual object image and the current image according to the updated depth map.

[0057] It should be noted here that the foreground area refers to the area where the object of interest is located, such as significant objects like people, cars, animals, and plants; the background area is the area in the image other than the foreground area.

[0058] By performing edge optimization on the target depth map of the current image to obtain an optimized depth map, and then fusing multiple frames of depth maps, a depth map with sharper edges is obtained, which is beneficial to further enhancing the virtual-real occlusion effect.

[0059] In a second aspect, an embodiment of the present application provides another method for processing an image, including:

[0060] Receiving a depth estimation model acquisition request sent by a terminal device, where the depth estimation model acquisition request carries the current image collected by the terminal device and the location of the terminal device; obtaining the depth estimation model of the current image from multiple depth estimation models stored in the server according to the location of the current image; sending a response message in response to the depth estimation model acquisition request to the terminal device, where the response message carries the depth estimation model of the current image.

[0061] To improve the accuracy of depth estimation, in the server, a depth estimation model will be separately trained for each location; when performing depth estimation, the depth estimation model of the current image is obtained from the server based on the current image and the location of the terminal device.

[0062] In a feasible embodiment, obtaining the depth estimation model of the current image from multiple depth estimation models stored in the server includes:

[0063] Obtaining multiple frames of first images according to the location of the terminal device, where the multiple frames of first images are images within a preset range centered on the location of the terminal device in the base map, and obtaining a target image from the multiple frames of first images, where the target image is the image with the highest similarity to the current image among the multiple frames of first images; determining the depth estimation model corresponding to the target image as the depth estimation model of the current image.

[0064] Specifically, obtaining multiple frames of first images according to the location of the terminal device, where the multiple frames of first images are images within a preset range centered on the location of the terminal device in the base map, and obtaining a target image from the multiple frames of first images, where the target image is the image with the highest similarity to the current image among the multiple frames of first images; obtaining the pose of the current image according to the pose of the target image; determining the depth estimation model corresponding to this location from the server according to the location in the pose of the current image, and this depth estimation model is the depth estimation model of the current image.

[0065] In a feasible embodiment, the method of the present application further includes:

[0066] For multiple frames of first images, depth estimation modules for each frame of the multiple frames of first images are trained separately.

[0067] Among them, for each frame of the multiple frames of first images, the following steps are performed for training to obtain the depth estimation model of each first image:

[0068] Multiple image samples and their corresponding multiple depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps; a loss value is calculated based on the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; the parameters in the initial convolutional neural network are adjusted according to the loss value to obtain the depth estimation model of each frame of the first image; among them, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0069] It should be noted here that the above is only one training process; in actual applications, it will be iterated multiple times in the above manner until the calculated loss value converges; the convolutional neural network model when the loss value converges is determined as the depth estimation model of the current image.

[0070] In a feasible embodiment, the method of this embodiment further includes:

[0071] An initial depth map of the current image is obtained according to the current image and the pre-stored image; a fifth depth map is obtained according to the current image and the 3D points corresponding to the local map; the initial depth map and the fifth depth map are optimized according to the pose of the current image to obtain a second depth map.

[0072] Optionally, the pre-stored image is uploaded by the terminal device, and the timestamp of the pre-stored image is before the timestamp of the current image.

[0073] In a feasible embodiment, obtaining the initial depth map of the current image according to the current image and the pre-stored image includes:

[0074] The first 2D feature points of the current image are matched with the 2D feature points of the pre-stored image to obtain the sixth 2D feature points of the current image; the noise points in the sixth 2D feature points of the current image are removed to obtain the seventh 2D feature points of the current image; triangulation calculation is performed on each 2D feature point in the seventh 2D feature points of the current image to obtain the initial 3D points in space of the seventh 2D feature points of the current image; the initial depth map of the current image includes the initial 3D points in space of the seventh 2D feature points of the current image.

[0075] In a feasible embodiment, obtaining a fifth depth map based on the current image and the 3D points corresponding to the local map includes

[0076] Obtaining M maps from multiple frames of the base map, where each of the M maps has a similarity greater than a first preset threshold with the current image; M is an integer greater than 0; Matching the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; Each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, where the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; According to the correspondence between each fifth 2D feature point in the M maps and the 3D points, obtaining the 3D points corresponding to each fourth 2D feature point in the multiple feature point matching pairs; According to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, obtaining a fifth depth map, where the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs.

[0077] Through the above method, the offline map in the server can be continuously updated by the images uploaded by the terminal device, so that the offline map is consistent with the surrounding environment when the user uses it, thereby providing high-precision 3D point cloud information. The cloud local map obtained in this way can be kept consistent with the environment used by the user, and the more images the user uploads, the more thorough the update is, and it has the characteristic of becoming more accurate with use, thus solving the problem of frame-by-frame flickering and instability in the subsequent virtual-real occlusion effect.

[0078] In a third aspect, an embodiment of the present application provides a terminal device, including:

[0079] An acquisition unit, configured to acquire a current image and obtain a first depth map and a second depth map of the current image according to the current image, where the second depth map is acquired from the server;

[0080] An estimation unit, configured to perform feature extraction according to the current image, the first depth map of the current image, and the second depth map, and obtain a target depth map of the current image according to the result of the feature extraction.

[0081] In a feasible embodiment, the acquisition unit is further configured to acquire a virtual object image;

[0082] The terminal device further includes:

[0083] An overlay display unit, configured to overlay and display the virtual object image and the current image according to the target depth map of the current image.

[0084] In a feasible embodiment, in terms of obtaining a first depth map of the current image based on the current image, the obtaining unit is specifically configured to:

[0085] Extract features from the current image to obtain first 2D feature points of the current image; match the first 2D feature points of the current image with pre-stored 2D feature points to obtain second 2D feature points in the current image; according to the correspondence between the second 2D feature points in the current image and the pre-stored 2D feature points and 3D points, obtain 3D points corresponding to the second 2D feature points in the current image; wherein, the first depth map of the current image includes the 3D points corresponding to the second 2D feature points in the current image.

[0086] In a feasible embodiment, in terms of obtaining a first depth map of the current image based on the current image, the obtaining unit is specifically configured to:

[0087] Extract features from the current image to obtain first 2D feature points of the current image; match the first 2D feature points of the current image with the 2D feature points of the local map obtained from the server to obtain third 2D feature points in the local map; according to the correspondence between the second 2D feature points in the local map and the 2D feature points and 3D points in the local map, obtain 3D points corresponding to the third 2D feature points in the local map; wherein, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

[0088] In a feasible embodiment, in terms of matching the first 2D feature points of the current image with the 2D feature points of the local map to obtain third 2D feature points in the local map, the obtaining unit is specifically configured to:

[0089] Obtain a target map from the local map according to a first pose, and the position of the target map in the local map is associated with the position indicated by the angle information in the first pose; the first pose is the pose obtained by the terminal device based on the current image converted into the pose in the world coordinate system, match the 2D feature points in the target map with the first 2D feature points of the current image to obtain third 2D feature points of the target map, and the third 2D feature points of the local map include the third 2D feature points of the target map.

[0090] In a feasible embodiment, the estimating unit is specifically configured to:

[0091] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform feature extraction on the third depth map to obtain T second feature maps; the resolutions of each of the T first feature maps are different from each other, and the resolutions of each of the T second feature maps are different from each other; T is an integer greater than 1; superimpose the first feature map and the second feature map with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image; wherein, the third depth map is the above-mentioned first depth map or is obtained by splicing the first depth map and the second depth map.

[0092] In a feasible embodiment, the estimation unit is specifically configured to:

[0093] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; perform multi-scale feature extraction on the reference depth map to obtain T fourth feature maps, the resolutions of each of the T first feature maps are different from each other, the resolutions of each of the T second feature maps are different from each other, and the resolutions of each of the T fourth feature maps are different from each other; the reference depth map is obtained from the depth map collected by the TOF camera, and T is an integer greater than 1; superimpose the first feature map, the second feature map and the fourth feature map with the same resolution among the T first feature maps, the T second feature maps and the T fourth feature maps to obtain T fifth feature maps; perform upsampling and fusion processing on the T fifth feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by splicing the first depth map and the second depth map, or the third depth map is the above-mentioned first depth map.

[0094] In a feasible embodiment, the reference depth map is obtained from the image collected by the TOF camera, specifically including:

[0095] Project the depth map collected by the TOF camera into the three-dimensional space according to the pose of the current image to obtain a fourth depth map; back-project the fourth depth map onto the reference image according to the pose of the reference image to obtain the reference depth map; the reference image is an image adjacent to the current image in terms of acquisition time; wherein, the resolution of the depth map collected by the TOF camera is lower than the preset resolution, and the frame rate when the TOF camera collects this depth map is lower than the preset frame rate.

[0096] In a feasible embodiment, the above-mentioned upsampling and fusion processing includes:

[0097] S1: Perform upsampling on the feature map P’ j to obtain the feature map P” j such that the feature map P”j has the same resolution as the (j + 1)-th feature map P in the object to be processed j+1 in the object to be processed, and the width of the feature map P j+1 is (j + 1) times the width of the feature map with the smallest resolution in the object to be processed, where j is an integer greater than 0 and less than T; T is the number of feature maps in the object to be processed;

[0098] S2: Fuse the feature map P” j with the feature map P j+1 to obtain a third feature map P’ j+1 ,

[0099] S3: Let j = j + 1, and repeat S1 - S3 until j = T - 1;

[0100] Among them, when j = 1, the third feature map P’ j is the feature map with the smallest resolution in the object to be processed, and when j = T - 1, the third feature map P’ j+1 is the result of upsampling and fusion processing.

[0101] In a feasible embodiment, the estimation unit is specifically configured to:

[0102] Input the current image and the third depth map into the depth estimation model of the current image for feature extraction, and obtain the target depth map of the current image according to the result of feature extraction, where the depth estimation model is implemented based on a convolutional neural network.

[0103] In a feasible embodiment, the terminal device further includes:

[0104] A sending unit, configured to send a depth estimation model acquisition request to the server, where the depth estimation model acquisition request carries the current image and the location of the terminal device;

[0105] A receiving unit, configured to receive a response message corresponding to the depth estimation model acquisition request sent by the server, where the response message carries the depth estimation model of the current image, and the depth estimation model of the current image is obtained by the server according to the current image and the location of the terminal device in the world coordinate system.

[0106] In a feasible embodiment, the terminal device further includes:

[0107] A training unit, configured to train an initial convolutional neural network model to obtain a depth estimation model; among them, the training unit is specifically configured to:

[0108] Input multiple image samples and their corresponding multiple depth map samples into an initial convolutional neural network for processing to obtain multiple predicted depth maps; calculate a loss value based on the multiple predicted depth maps, the ground truth depth maps corresponding to the multiple image samples, and a loss function; adjust the parameters in the initial convolutional neural network according to the loss value to obtain a depth estimation model for the current image; wherein, the loss function is determined based on the error between the predicted depth map and the ground truth depth map, the error between the gradient of the predicted depth map and the gradient of the ground truth depth map, and the error between the normal vector of the predicted depth map and the normal vector of the ground truth depth map.

[0109] In a feasible embodiment, the superimposed display unit is specifically configured to:

[0110] Perform edge optimization on the target depth map of the current image to obtain an optimized depth map; superimpose and display a virtual object image and the current image according to the optimized depth map.

[0111] In a feasible embodiment, the superimposed display unit is specifically configured to:

[0112] Segment the optimized depth map to obtain a foreground depth map and a background depth map of the current image, where the background depth map is the depth map of the optimized depth map containing the background region, and the foreground depth map is the depth map of the optimized depth map containing the foreground region, and the optimized depth map is obtained by performing edge optimization on the target depth map of the current image; fuse the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; perform back-projection on the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map; splice the fused background depth map and the foreground depth map of the current image to obtain an updated depth map; superimpose and display a virtual object image and the current image according to the updated depth map.

[0113] In a fourth aspect, an embodiment of the present application provides a server, including:

[0114] A receiving unit, configured to receive a depth estimation model acquisition request sent by a terminal device, where the depth estimation model acquisition request carries the current image collected by the terminal device and the location of the terminal device;

[0115] An acquisition unit, configured to acquire a depth estimation model of the current image from multiple depth estimation models stored in the server according to the location of the current image;

[0116] A sending unit, configured to send a response message in response to a depth estimation model acquisition request to a terminal device, where the response message carries a depth estimation model of a current image.

[0117] In a feasible embodiment, the acquisition unit is specifically configured to:

[0118] Acquire multiple first images according to the position of the terminal device, where the multiple first images are images within a preset range centered on the position of the terminal device in a base map, and acquire a target image from the multiple first images, where the target image is the image with the highest similarity to the current image among the multiple first images; determine the depth estimation model corresponding to the target image as the depth estimation model of the current image.

[0119] In a feasible embodiment, the server further includes:

[0120] A training unit, configured to respectively train depth estimation modules for each of the multiple first images in the multiple first images.

[0121] Wherein, for each of the multiple first images in the multiple first images, the following steps are performed for training to obtain a depth estimation model for each first image:

[0122] Input multiple image samples and their corresponding multiple depth map samples into an initial convolutional neural network for processing to obtain multiple predicted depth maps; calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; adjust parameters in the initial convolutional neural network according to the loss value to obtain a depth estimation model for each first image.

[0123] Wherein, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0124] In a feasible embodiment, the acquisition unit is further configured to obtain an initial depth map of the current image according to the current image and a pre-stored image; obtain a fifth depth map according to the current image and 3D points corresponding to a local map.

[0125] The server further includes:

[0126] An optimization unit, configured to optimize the initial depth map and the fifth depth map according to the pose of the current image to obtain a second depth map.

[0127] In a feasible embodiment, in terms of obtaining the initial depth map of the current image according to the current image and a pre-stored image, the acquisition unit is specifically configured to:

[0128] Match the first 2D feature points of the current image with the 2D feature points of the pre-stored image to obtain the sixth 2D feature points of the current image; remove the noise points in the sixth 2D feature points of the current image to obtain the seventh 2D feature points of the current image; perform triangulation calculation on each 2D feature point in the seventh 2D feature points of the current image to obtain the initial 3D points in space of the seventh 2D feature points of the current image; the initial depth map of the current image includes the initial 3D points in space of the seventh 2D feature points of the current image.

[0129] In a feasible embodiment, in terms of obtaining the fifth depth map according to the 3D points corresponding to the current image and the local map, the obtaining unit is specifically configured to:

[0130] Obtain M maps from multiple frames of base maps, where the similarity between each of the M maps and the current image is greater than a first preset threshold; M is an integer greater than 0; match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; according to the correspondence between each fifth 2D feature point in the M maps and the 3D points, obtain the 3D points corresponding to each fourth 2D feature point in the multiple feature point matching pairs; according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, obtain the fifth depth map, and the fifth depth map includes, among the 3D points corresponding to the local map, the 3D points that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs.

[0131] In a fifth aspect, an embodiment of the present application provides a terminal device, including a memory and one or more processors; wherein, one or more programs are stored in the memory; when the one or more processors execute the one or more programs, the terminal device implements part or all of the method described in the first aspect.

[0132] In a sixth aspect, an embodiment of the present application provides a server, including a memory and one or more processors; wherein, one or more programs are stored in the memory; when the one or more processors execute the one or more programs, the server implements part or all of the method described in the second aspect.

[0133] In a seventh aspect, an embodiment of the present application provides a computer storage medium, characterized in that it includes computer instructions, and when the computer instructions run on an electronic device, the electronic device executes part or all of the method described in the first aspect or the second aspect.

[0134] In an eighth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a computer, the computer is caused to execute part or all of the method described in the first aspect or the second aspect.

[0135] It should be understood that any of the above possible implementation manners can be freely combined without violating the laws of nature, and will not be elaborated in the present application.

[0136] It should be understood that the description of technical features, technical solutions, beneficial effects or similar languages in the present application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that an embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. Description of the Drawings

[0137] Figure 1a It is a schematic diagram of the effect without virtual-real occlusion;

[0138] Figure 1b It is a schematic diagram of the effect with virtual-real occlusion;

[0139] Figure 1c The present application provides an embodiment with a schematic diagram of a system architecture;

[0140] Figure 1d It is a schematic diagram of the structure of a CNN;

[0141] Figure 1e It is a schematic diagram of the hardware structure of a chip provided by an embodiment of the present application;

[0142] Figure 1f The present application provides another embodiment with a schematic diagram of a system architecture;

[0143] Figure 2 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0144] Figure 3 It is a schematic diagram of the flow of a method for processing an image provided by an embodiment of the present application;

[0145] Figure 4Schematic diagram of the relationship among the base map, local map, and processed local map;

[0146] Figure 5 Schematic flowchart of another image processing method provided by an embodiment of the present application;

[0147] Figure 6 Schematic diagram of the virtual-real occlusion effect using an embodiment of the present application;

[0148] Figure 7 Schematic diagram of the structure of a terminal device provided by an embodiment of the present application;

[0149] Figure 8 Schematic diagram of the structure of another terminal device provided by an embodiment of the present application;

[0150] Figure 9 Schematic diagram of the structure of a system provided by an embodiment of the present application;

[0151] Figure 10 Schematic diagram of the structure of another system provided by an embodiment of the present application;

[0152] Figure 11 Schematic diagram of the structure of another terminal device provided by an embodiment of the present application;

[0153] Figure 12 Schematic diagram of the structure of a server provided by an embodiment of the present application;

[0154] Figure 13 Schematic diagram of the structure of another terminal device provided by an embodiment of the present application;

[0155] Figure 14 Schematic diagram of the structure of another server provided by an embodiment of the present application. Detailed implementation manners

[0156] Next, the technical solutions in the embodiments of the present application will be described clearly and in detail with reference to the accompanying drawings.

[0157] Hereinafter, in some cases, terms such as "first" and "second" are only used for descriptive purposes and cannot be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0158] Since the embodiments of the present application involve a large number of neural network applications, for the convenience of understanding, relevant terms and related concepts such as neural networks involved in the embodiments of the present application will be introduced below.

[0159] (1) Neural network

[0160] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs and an intercept 1 as inputs. The output of this operation unit can be:

[0161]

[0162] where s = 1, 2,..., n, n is a natural number greater than 1, W s is the weight of x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to perform a non-linear transformation on the features obtained in the neural network, converting the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many such single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region composed of several neural units.

[0163] (2) Deep neural network

[0164] A deep neural network (DNN) can be understood as a neural network with many hidden layers. Here, "many" does not have a specific measurement standard. Generally speaking, a multi-layer neural network and a deep neural network are essentially the same thing. Dividing DNN according to the positions of different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i + 1-th layer. Although DNN seems very complex, in terms of the work of each layer, it is actually not complex. Simply speaking, it is the following linear relationship expression: where is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called the coefficient), and α() is the activation function. Each layer simply performs such a simple operation on the input vector to obtain the output vector Since DNN has many layers, the coefficient W and the offset vector The quantity is already quite large. Then, how are the specific parameters defined in the DNN? First, let's take a look at the definition of the coefficient W. Taking a three-layer DNN as an example, for instance, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer where the coefficient W is located, and the subscripts correspond to the index 2 of the output third layer and the index 4 of the input second layer. To summarize, the coefficient from the kth neuron in the (L - 1)th layer to the jth neuron in the Lth layer is defined as Note that there is no W parameter in the input layer. In a deep neural network, more hidden layers enable the network to better depict complex situations in the real world. Theoretically speaking, the more parameters a model has, the higher its complexity and the larger its "capacity", which means it can complete more complex learning tasks.

[0165] (3) Convolutional Neural Network

[0166] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor composed of convolutional layers and subsampling layers, and this feature extractor can be regarded as a filter. A convolutional layer refers to the neuron layer in a convolutional neural network that performs convolutional processing on the input signal. In the convolutional layer of a convolutional neural network, a neuron can only be connected to some neighboring layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can consist of some neurons arranged in a rectangle. The neurons in the same feature plane share weights, and the shared weight here is the convolutional kernel. Sharing weights can be understood as a way of extracting features that is independent of position. The convolutional kernel can be formalized in the form of a matrix of random size, and during the training process of the convolutional neural network, the convolutional kernel can obtain reasonable weights through learning. Additionally, the direct benefit brought by sharing weights is to reduce the connections between the layers of the convolutional neural network while also reducing the risk of overfitting.

[0167] (4) Loss Function

[0168] During the process of training a deep neural network, since we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the real target value, and then update the weight vector of each layer of the neural network according to the difference between the two. (Of course, there is usually a process of parameter initialization before the first update, that is, configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is too high, we adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the real target value or a value very close to the real target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function. They are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0169] The following describes the system architecture provided by the embodiments of the present application.

[0170] See the appendix Figure 1c , the embodiments of the present application provide a system architecture. As shown in the system architecture, the data acquisition device 160 is used to acquire training data. Exemplarily, in the embodiments of the present application, the training data may include: image samples, depth map samples, and real depth maps; after acquiring the training data, the data acquisition device 160 stores these training data in the database 130, and the training device 120 trains a depth estimation model 101 based on the training data maintained in the database 130.

[0171] The following describes how the training device 120 obtains the depth estimation model 101 based on the training data. Exemplarily, the training device 120 processes the image samples and depth map samples, calculates the loss value according to the output predicted depth map, the real depth map, and the loss function, until the calculated loss value converges, thereby completing the training of the depth estimation model 101.

[0172] The depth estimation model 101 can be used to implement the image processing method provided by the embodiments of the present application. That is, the current image, the first depth map, and the second depth map are input into the depth estimation model 101 after relevant preprocessing, which is the target depth map of the current image. The depth estimation model 101 in the embodiments of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data acquisition device 160, and it may also be received from other devices. Additionally, it should be noted that the training device 120 may not necessarily train the depth estimation model 101 entirely based on the training data maintained in the database 130, and it may also obtain training data from the cloud or other places for model training. The above descriptions should not be regarded as limitations on the embodiments of the present application.

[0173] The depth estimation model 101 trained according to the training device 120 can be applied to different systems or devices, such as being applied to Figure 1c the execution device 110 shown in the figure. The execution device 110 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR (augmented reality), VR (virtual reality), a vehicle-mounted terminal, etc., or it can also be a server or the cloud, etc. In the attached Figure 1c figure, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. A user can input data to the I / O interface 112 through the client device 140. The input data in the embodiments of the present application can include: the current image, or the current image and the first depth map.

[0174] During the preprocessing of the input data by the execution device 110, or during the relevant processing such as calculation by the calculation module 111 of the execution device 110, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0175] Finally, the I / O interface 112 returns the processing result, such as the target depth map of the current image obtained above, to the client device 140, so as to be provided to the user.

[0176] It is worth noting that the training device 120 can generate corresponding depth estimation models 101 based on different training data for different targets or different tasks. The corresponding depth estimation model 101 can be used to achieve the above targets or complete the above tasks, so as to provide the required results for users.

[0177] In the attached Figure 1cIn the case shown, the user can manually provide input data, and this manual provision can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and user authorization is needed, the user can set the corresponding permissions in the client device 140. The user can view the results output by the execution device 110 in the client device 140, and the specific presentation form can be specific ways such as display, sound, and action. The client device 140 can also be used as a data acquisition end to collect the input data input to the I / O interface 112 and the output results of the output I / O interface 112 shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but directly store the input data input to the I / O interface 112 and the output results of the output I / O interface 112 shown in the figure by the I / O interface 112 as new sample data in the database 130.

[0178] It should be noted that the appendix Figure 1c is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the appendix Figure 1c the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0179] As Figure 1c shown, the depth estimation model 101 is trained according to the training device 120. The depth estimation model 101 in the embodiment of the present application can be the neural network in the present application. Specifically, the neural network in the present application can include CNN or deep convolutional neural networks (DCNN), etc.

[0180] Since CNN is a common neural network, the following will combine Figure 1d to introduce the structure of CNN in detail. As described in the basic concept introduction above, the convolutional neural network is a deep neural network with a convolutional structure and is a deep learning architecture. The deep learning architecture refers to performing multiple levels of learning at different abstraction levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.

[0181] As Figure 1dAs shown, a convolutional neural network (CNN) may include an input layer 11, a convolutional layer / pooling layer 12 (where the pooling layer is optional), a neural network layer 13, and an output layer 14.

[0182] Convolutional layer / pooling layer 12:

[0183] Convolutional layer:

[0184] As Figure 1d shown, the convolutional layer / pooling layer 12 may include layers such as example layers 121 - 126. For example: In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer or as the input to another convolutional layer to continue the convolution operation.

[0185] Next, taking convolutional layer 121 as an example, the internal working principle of one convolutional layer will be introduced.

[0186] The convolutional layer 121 may include a number of convolutional operators, also known as kernels, which act as filters for extracting specific information from the input image matrix in image processing. Essentially, a convolutional operator can be a weight matrix, which is usually predefined. During the process of performing convolution operations on an image, the weight matrix typically processes the input image pixel by pixel (or two pixels at a time... depending on the value of the stride) along the horizontal direction, thereby completing the task of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as that of the input image, and during the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolving with a single weight matrix will produce a convolved output with a single depth dimension. However, in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns), i.e., multiple matrices of the same type, are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image, where the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and yet another weight matrix is used to blur the unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same. Then, the multiple feature maps of the same size that are extracted are combined to form the output of the convolution operation.

[0187] The weight values in these weight matrices need to be obtained through a large amount of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 10 to make correct predictions.

[0188] When the convolutional neural network 10 has multiple convolutional layers, the initial convolutional layer (such as 121) often extracts more general features, which can also be referred to as low-level features; as the depth of the convolutional neural network 10 increases, the features extracted by the subsequent convolutional layers (such as 126) become more and more complex, such as high-level semantic features. The higher the semantic features, the more suitable they are for the problem to be solved.

[0189] Pooling layer:

[0190] Since it is often necessary to reduce the number of training parameters, a pooling layer is often introduced periodically after the convolutional layer. Figure 1dFor each of the layers 121 - 126 shown in [example 12], it can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. During the image processing, the sole purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a max pooling operator for sampling the input image to obtain a smaller-sized image. The average pooling operator can calculate the average value of the pixel values in the image within a specific range as the result of average pooling. The max pooling operator can take the pixel with the maximum value within the specific range as the result of max pooling. Additionally, just as the size of the weight matrix in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after being processed by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0191] Neural network layer 13:

[0192] After being processed by the convolutional layer / pooling layer 12, the convolutional neural network 10 is still not sufficient to output the required output information. As mentioned before, the convolutional layer / pooling layer 12 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other relevant information), the convolutional neural network 10 needs to use the neural network layer 13 to generate one or a group of outputs with the number of classes required. Therefore, the neural network layer 13 can include multiple hidden layers (such as Figure 1d 131, 132 to 13n shown), and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type can include image recognition, image classification, image super-resolution reconstruction, and so on...

[0193] After the multiple hidden layers in the neural network layer 13, that is, the last layer of the entire convolutional neural network 10 is the output layer 14. The output layer 14 has a loss function similar to categorical cross-entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 10 (such as Figure 1d the propagation from 11 to 14 is the forward propagation) is completed, the backpropagation (such as Figure 1d the propagation from 14 to 11 is the backpropagation) will start to update the weight values and biases of the previously mentioned layers to reduce the loss of the convolutional neural network 10, that is, the error between the result output by the convolutional neural network 10 through the output layer and the ideal result.

[0194] It should be noted that as Figure 1dThe convolutional neural network 10 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models. For example, it may only include Figure 1d a part of the network structure shown in Figure 1d . For instance, the convolutional neural network adopted in the embodiments of the present application may only include an input layer 11, a convolutional / pooling layer 12, and an output layer 14.

[0195] The following introduces a chip hardware structure provided by the embodiments of the present application.

[0196] Figure 1f A chip hardware structure provided by the embodiments of the present application. The chip includes a neural network processor 30. The chip may be disposed in an execution device 110 as shown in Figure 1c Figure 1c to complete the computing work of the computing module 111. The chip may also be disposed in a training device 120 as shown in Figure 1c Figure 1c to complete the training work of the training device 120 and output a depth map estimation model 101. The algorithms of each layer in the convolutional neural network shown in Figure 1d Figure 1d can all be implemented in the chip shown in Figure 1f Figure 1f . The image fusion method and the training method of the image fusion model in the embodiments of the present application can both be implemented in the chip shown in Figure 1f Figure 1f .

[0197] The neural network processor 30 may be a neural-network processing unit (NPU), a tensor processing unit (TPU), a graphics processing unit (GPU), or any other processor suitable for large-scale exclusive OR operation processing. Taking the NPU as an example: The neural network processor NPU 30 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and tasks are assigned by the main CPU. The core part of the NPU is the arithmetic circuit 303, and the controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform operations. Among them, the TPU is an application-specific integrated circuit for artificial intelligence accelerator fully customized by Google.

[0198] In some implementations, the arithmetic circuit 303 includes multiple processing units (process engines, PEs) internally. In some implementations, the arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general matrix processor.

[0199] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 303 fetches the weight data of matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit 303. The arithmetic circuit 303 fetches the input data of matrix A from the input memory 301, performs matrix operations based on the input data of matrix A and the weight data of matrix B, and saves the partial results or final results of the obtained matrix in the accumulator 308.

[0200] The vector calculation unit 307 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. For example, the vector calculation unit 307 can be used for network calculations in non-convolutional / non-FC layers of a neural network, such as pooling, batch normalization, local response normalization, etc.

[0201] In some implementations, the vector calculation unit 307 can store the processed output vector in the unified buffer 306. For example, the vector calculation unit 307 can apply a non-linear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 307 generates normalized values, combined values, or both. In some implementations, the vector calculation unit 307 stores the processed vector in the unified memory 306. In some implementations, the vector processed by the vector calculation unit 307 can be used as the activation input of the arithmetic circuit 303, for example, for use in subsequent layers of a neural network, such as Figure 1d As shown, if the current processing layer is the hidden layer 1 (131), the vector processed by the vector calculation unit 307 can also be used in the calculation in the hidden layer 2 (132).

[0202] The unified memory 306 is used to store input data and output data.

[0203] The weight data is directly stored into the weight memory 302 through a direct memory access controller (DMAC) 305. The input data is also stored into the unified memory 306 through the DMAC.

[0204] A bus interface unit (BIU) 310 is used for the interaction between the DMAC and an instruction fetch buffer 309; the bus interface unit 310 is also used for the instruction fetch buffer 309 to obtain instructions from an external memory; the bus interface unit 301 is also used for the memory access controller 305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0205] The DMAC 305 is mainly used to store the input data in the external DDR memory into the unified memory 306, or store the weight data into the weight memory 302, or store the input data into the input memory 301.

[0206] An instruction fetch memory 309 connected to the controller 304 is used to store the instructions used by the controller 304;

[0207] The controller 304 is used to call the instructions cached in the instruction fetch memory 309 to control the working process of the arithmetic accelerator.

[0208] Generally, the unified memory 306, the input memory 301, the weight memory 302, and the instruction fetch memory 309 are all on-chip memories, and the external memory is the memory outside the NPU. The external memory can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM), or other readable and writable memories.

[0209] Among them, Figure 1d The operations of each layer in the shown convolutional neural network can be executed by an arithmetic circuit 303 or a vector calculation unit 307. Exemplarily, the training method of the depth estimation model and the related method for determining the target depth map in the embodiments of the present application can both be executed by the arithmetic circuit 303 or the vector calculation unit 307.

[0210] Such as Figure 1e As shown, the embodiments of the present application provide another system architecture. The system architecture includes a local device 401, a local device 402, and Figure 1cThe execution device 110 and the data storage system 150 shown in the figure, where the local device 401 and the local device 402 are connected to the execution device 110 through a communication network.

[0211] The execution device 110 can be implemented by one or more servers. Optionally, the execution device 110 can be used in cooperation with other computing devices, such as data storage devices, routers, load balancers, and other devices. The execution device 110 can be arranged at a physical site or distributed across multiple physical sites. The execution device 110 can use the data in the data storage system 150 or call the program code in the data storage system 150 to implement the training method of the time series prediction model in the embodiments of the present application.

[0212] Specifically, in one implementation, the execution device 110 can perform the following process:

[0213] Input multiple depth map samples corresponding to multiple image samples into the initial convolutional neural network for processing to obtain multiple first predicted depth maps; calculate a first loss value based on the multiple first predicted depth maps, the true depth maps corresponding to the multiple image samples, and the loss function; adjust the parameters in the initial convolutional neural network according to the first loss value; obtain a first convolutional neural network; then input the multiple depth map samples corresponding to the multiple image samples into the first convolutional neural network for processing to obtain multiple second predicted depth maps; obtain a second loss value based on the multiple second predicted depth maps, the true depth maps corresponding to the multiple image samples, and the loss function; determine whether the second loss value converges; if it converges, determine the first convolutional neural network as the depth estimation model of the current image; if it does not converge, adjust the parameters in the first convolutional neural network according to the second loss value to obtain a second convolutional neural network, and repeat the above process until the obtained loss value converges, and determine the convolutional neural network when the loss value converges as the depth estimation model of the current image.

[0214] Through the above process, the execution device 110 can obtain a depth estimation model, which can be used to obtain the target depth map of the current image.

[0215] Users can operate their respective user devices (such as the local device 401 and the local device 402) to interact with the execution device 110. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car, or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0216] The local device of each user can interact with the execution device 410 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0217] In one implementation, the local devices 401 and 402 obtain a depth estimation model from the execution device 110, deploy the depth estimation model on the local devices 401 and 402, and use the depth estimation model to perform depth estimation.

[0218] In another implementation, the depth estimation model can be directly deployed on the execution device 110. The execution device 410 obtains the current image and the first depth map from the local devices 401 and 402, and uses the depth estimation model to perform depth estimation on the current image and the first depth map to obtain the target depth map of the current image.

[0219] The above-mentioned execution device 110 can also be a cloud device. In this case, the execution device 110 can be deployed in the cloud; or, the above-mentioned execution device 110 can also be a terminal device. In this case, the execution device 110 can be deployed on the user terminal side. The embodiments of the present application do not limit this.

[0220] The application scenarios of the present application will be described below. As Figure 2 shown, this application scenario includes a terminal device 100 and a server 200.

[0221] Among them, the terminal device 100 can be a smart phone, a tablet computer, AR glasses or other smart devices.

[0222] The server 200 can be a desktop server, a rack server, a cabinet server, a blade server or other types of servers.

[0223] The terminal device 100 obtains the current image and the virtual object image, and obtains the first depth map and the second depth map of the current image according to the current image. Among them, as Figure 2 shown, the second depth map is obtained by the terminal device 100 from the server 200; depth estimation is performed according to the current image, the first depth map and the second depth map to obtain the target depth map of the current image; the virtual object image and the current image are superimposed and displayed according to the target depth map of the current image, so as to realize virtual occlusion.

[0224] The following details how the above-mentioned terminal device 100 and server 200 implement virtual-real occlusion.

[0225] See Figure 3 , Figure 3 is a schematic flowchart of a method for processing an image provided by an embodiment of the present application. As Figure 3As shown, the method includes:

[0226] S301. Obtain the current image and the virtual object image, and obtain the first depth map of the current image according to the current image.

[0227] Optionally, the current image can be an RGB image, a grayscale image, or an image in other forms.

[0228] Among them, the current image is obtained in real time from the camera of the terminal device, or obtained from the images stored in the terminal device, or obtained from other devices, and no specific limitation is made here.

[0229] Optionally, the virtual object image can be obtained by projecting the three-dimensional model of the virtual object by the renderer in the terminal device; or obtained from other devices.

[0230] Optionally, obtaining the first depth map of the current image according to the current image includes:

[0231] Extract features from the current image to obtain the first 2D feature points of the current image; match the first 2D feature points of the current image with the pre-stored 2D feature points to obtain the second 2D feature points in the current image; according to the second 2D feature points in the current image and the corresponding relationship between the pre-stored 2D feature points and 3D points, obtain the 3D points corresponding to the second 2D feature points in the current image; among them, the first depth map of the current image includes the 3D points corresponding to the second 2D feature points in the current image.

[0232] Feature points refer to points where the image gray value changes violently or points with a large curvature on the image edge (i.e., the intersection of two edges).

[0233] The above-mentioned corresponding relationship between the pre-stored 2D feature points and 3D points means that for each pre-stored 2D feature point, there is a corresponding 3D point.

[0234] For the matching of the 2D feature points mentioned in this application, specifically, it means that the similarity of the two matching 2D feature points is higher than the preset similarity.

[0235] It should be noted here that after obtaining the 3D points corresponding to the second 2D feature points in the current image, project the 3D points onto a two-dimensional plane to obtain the above-mentioned first depth map, and the first depth map includes the 3D point information corresponding to the second 2D feature points in the current image.

[0236] Among them, the above local map is obtained by the server from the base map according to the pose of the terminal device.

[0237] Among them, the pre-stored 2D feature points can be the 2D feature points in at least one historical image. The at least one historical image includes one or more images with a relatively small time difference between the time stamps of the above terminal device and the current image, or one or more images with time stamps before the time stamp of the current image. Further, the multiple images can be consecutive frame images or non-consecutive frame images.

[0238] Optionally, obtaining the first depth map of the current image according to the current image includes:

[0239] Performing feature extraction on the current image to obtain the first 2D feature points of the current image; matching the first 2D feature points of the current image with the 2D feature points of the local map to obtain the third 2D feature points in the local map; the local map is obtained according to the current image by the server and retrieved from the server; according to the third 2D feature points in the local map and the correspondence between the 2D feature points and 3D points in the local map, obtaining the 3D points corresponding to the third 2D feature points in the local map; among them, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

[0240] After obtaining the 3D points corresponding to the third 2D feature points in the local map, projecting the 3D points onto a two-dimensional plane to obtain the above first depth map, and the first depth map includes the 3D point information corresponding to the third 2D feature points in the local map.

[0241] In an optional embodiment, the terminal device sends a local map acquisition request to the server. The local map is a base map that includes information about the objects around the location point of the terminal device. The local map acquisition request carries the current image. Among them, the base map is a map that includes 3D point cloud information of the object, 2D feature points, and their feature descriptors. After receiving the local map acquisition request, the server calculates the position and angle information of the current image in the global map according to the visual positioning system (VPS). The position and angle information are collectively referred to as pose information, and the pose information is based on the pose in the world coordinate system; obtaining the local map from the base map according to the pose information. The local map is an area within a certain range (such as a radius of 50 meters) centered on this position in the base map; the base map includes 2D feature points of the object and their corresponding feature descriptors, and 3D point cloud information. Therefore, the local map also includes 2D feature points of the object and their corresponding feature descriptors and 3D point cloud information. The server sends a response message for responding to the local map acquisition request to the terminal device, and the response message carries the above local map.

[0242] The terminal device extracts features from the current image to obtain the first 2D feature points of the current image and their corresponding feature descriptors; matches the first 2D feature points of the current image with the 2D feature points in the local map to obtain the third 2D feature points in the local map, and obtains the 3D points corresponding to the third 2D feature points in the local map according to the correspondence between the 2D feature points and the 3D points in the local map. The first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

[0243] In an optional embodiment, matching the first 2D feature points of the current image with the 2D feature points of the local map to obtain the third 2D feature points in the local map includes:

[0244] Obtain a target map from the local map according to the first pose. The position of the target map in the local map is associated with the position indicated by the angle information in the first pose; the first pose is the pose obtained by the terminal device according to the current image converted into the pose in the world coordinate system. Match the 2D feature points in the target map with the first 2D feature points of the current image to obtain the third 2D feature points of the target map. The third 2D feature points of the local map include the third 2D feature points of the target map.

[0245] Wherein, the world coordinate system is the absolute coordinate system of the system, and the coordinates of all points on the screen are determined by the origin of this coordinate system before the user coordinate system is established.

[0246] In an optional embodiment, in order to improve the above matching efficiency, the terminal device uses a simultaneous localization and mapping (SLAM) system to calculate the pose of the current image in a preset coordinate system. The preset coordinate system is a coordinate system with the current position of the terminal device as the origin. Align and transform the pose of the current image and the pose obtained by the server based on VPS to obtain the pose of the terminal device in the world coordinate system. This pose is the above-mentioned first pose; process the local map according to the pose of the terminal device in the world coordinate system to obtain a target map. The position of the target map in the local map is consistent with the position indicated by the angle information in the pose of the terminal device in the world coordinate system.

[0247] As Figure 4 shown, the square area is the base map, the circular area is the local map, and the center point of the circular area is the coordinate point in the pose of the terminal device in the world coordinate system; the fan-shaped area with an angle range of [-45°, 45°] is the processed local map, where the angle range of [-45°, 45°] is the angle information in the pose of the terminal device in the world coordinate system.

[0248] Then, the first 2D feature points of the current image are matched with the 2D feature points in the target map to obtain the third 2D feature points of the target map. According to the correspondence between the 2D feature points and the 3D point cloud information in the target map, the 3D point cloud corresponding to the third 2D feature points in the target map is obtained. The first depth map of the current image includes the 3D point cloud corresponding to the third 2D feature points in the target map.

[0249] Here, it should be noted that since the base map in the server is collected offline, there are differences between the base map in the server and the current actual environment. For example, a large billboard in the mall existed during offline collection. After a period of time, when the user captures the current image, the billboard has been removed, which results in inconsistent 3D point cloud information between the map sent by the server and the current environment. In addition, the images received by the server can be images after privacy processing, which also leads to inconsistent 3D point cloud information between the map sent by the server and the current environment.

[0250] For the above reasons, the server updates the sent local map to obtain the second depth map.

[0251] Specifically, the server extracts features from the current image to obtain 2D feature points of the current image. The methods for extracting features from the current image include, but are not limited to, the scale invariant feature transform (SIFT) method, the Oriented FAST and Rotated BRIEF (ORB) method, the speeded up robust features (SURF) method, and the super point method. Among them, FAST stands for Features from Accelerated Segment Test, and BRIEF stands for Binary Robust Independent Elementary Features. The server matches the first 2D feature points in the current image with the pre-stored 2D feature points to obtain the sixth 2D feature points in the current image. Optionally, the pre-stored 2D feature points are the 2D feature points in N historical images. The above N historical images are acquired by the terminal device and are images with timestamps before the timestamp of the current image and with timestamps relatively close to the timestamp of the current image. N is an integer greater than 0. Then the server removes the noise points in the sixth 2D feature points of the current image. Specifically, the verification value of each sixth 2D feature point in the current image is calculated through the homography matrix, the fundamental matrix, and the essential matrix. If the verification value of the sixth 2D feature point is lower than the second preset threshold, it is determined that the sixth 2D feature point is a noise point, and thus the sixth 2D feature point is deleted to obtain the seventh 2D feature points of the current image.

[0252] The server performs triangulation calculation on the seventh 2D feature points of the current image to obtain the initial positions of the seventh 2D feature points in the current image in space, and further obtains the initial depth map corresponding to the seventh 2D feature points in the current image.

[0253] The server obtains M images from multiple frame base maps according to an image retrieval method, where the similarity between each of the M maps and the current image is greater than a first preset threshold, and M is an integer greater than 0. The image retrieval method includes, but is not limited to, the bag-of-words tree method or the NetVlad method based on deep learning; match the first 2D feature points in the current image with the 2D feature points in the M images to obtain multiple pairs of matching feature points. Each pair of matching feature points in the multiple pairs of matching feature points includes a fourth 2D feature point and a fifth 2D feature point. The fourth 2D feature point and the fifth 2D feature point are feature points that match each other. The fourth 2D feature point is the first 2D feature point in the current image, and the fifth 2D feature point is the feature point in the M images; since each 2D feature point in the M images corresponds to a 3D point, the correspondence between the fifth 2D feature points and the 3D points in the M images and the multiple pairs of matching feature points determine the 3D point corresponding to the fourth 2D feature point in the current image; obtain a fifth depth map for the 3D points corresponding to the above local map. The fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple pairs of feature point matches.

[0254] Combine the above initial depth map and the fifth depth map, and use bundle adjustment (BA) in combination with the initial pose information of each image in the current image and the pre-stored images to adjust and optimize the position information of the point cloud, and calculate the reprojection error of each 3D point in the optimized point cloud. Delete the 3D points whose reprojection error exceeds the error threshold; and repeat the above BA optimization and point cloud deletion steps multiple times. Finally, obtain an optimized high-precision depth map, which is the above second depth map, thereby realizing the update of the base map in the server.

[0255] Through the above method, the base map stored in the server can be continuously updated by the images uploaded by the terminal device, so that the base map is consistent with the image content collected by the terminal device, thereby providing high-precision 3D point cloud information during depth estimation. And through the above method, the content in the base map is made consistent with the content in the images collected by the terminal device. The more images are uploaded by the terminal device, the more thorough the update of the map, so that the result of depth estimation is more accurate.

[0256] S302. Determine the target depth map of the current image according to the current image and the first depth map of the current image.

[0257] In an optional embodiment, determining the target depth map of the current image according to the current image and the first depth map of the current image includes:

[0258] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps. Among them, the resolutions of each of the T first feature maps are different, and the resolutions of each of the T second feature maps are different. T is an integer greater than 1. Superimpose the first feature map and the second feature map with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps. Perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image. Among them, the third depth map is the above-mentioned first depth map, or the third depth map is obtained by stitching the first depth map and the second depth map.

[0259] Among them, for any one of the T first feature maps, there is a second feature map in the T second feature maps whose resolution is uniquely the same as that of the first feature map.

[0260] Multi-scale feature extraction specifically refers to the operation of convolving an image with multiple different convolutional kernels.

[0261] The "superimposition" in this application specifically refers to processing the images to be superimposed at the pixel level. For example, the two images to be superimposed include a size of H*W, and the size of the superimposed image is H*2W, or 2H*W. Another example is that the three images to be superimposed include a size of H*W, and the size of the superimposed image is H*3W, or 3H*W.

[0262] In an optional embodiment, determining the target depth map of the current image according to the current image and the first depth map of the current image includes:

[0263] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and the resolutions of each of the T first feature maps are different; perform multi-scale feature extraction on the third depth map to obtain T second feature maps, and the resolutions of each of the T second feature maps are different; perform multi-scale feature processing on the reference depth map to obtain T fourth feature maps, and the resolutions of each of the T fourth feature maps are different. Among them, the reference depth map is obtained from the depth map collected by the TOF camera. T is an integer greater than 1. Superimpose the first feature map, the second feature map, and the fourth feature map with the same resolution among the T first feature maps, the T second feature maps, and the T fourth feature maps to obtain T fifth feature maps.

[0264] Perform upsampling and fusion processing on the T fifth feature maps to obtain the target depth map of the current image. Among them, the third depth map is the above-mentioned first depth map, or the third depth map is the above-mentioned first depth map obtained by stitching the first depth map and the second depth map.

[0265] Optionally, the above-mentioned reference depth map is the depth map collected by the above-mentioned TOF camera.

[0266] Optionally, the depth map collected by the TOF camera is projected into a three-dimensional space according to the pose of the current image to obtain a fourth depth map; the fourth depth map is back-projected onto the reference image according to the pose of the reference image to obtain a reference depth map; wherein, the reference image is an image adjacent to the current image in terms of acquisition time, the resolution of the depth map collected by the TOF camera is lower than a preset resolution, and the frame rate at which the TOF collects this depth map is lower than a preset frame rate.

[0267] Optionally, the above-mentioned preset frame rate can be 1fps, 2fps, 5fps or other frame rates, and the preset resolution can be 240*180, 120*90, 60*45, 20*15, or other resolutions.

[0268] Optionally, the TOF collects a depth map at a frame rate of 1fps, and the resolution of this depth map is 20*15.

[0269] Optionally, the above-mentioned third depth map is the first depth map of the above-mentioned current image, or the third depth map is obtained by splicing the first depth map of the above-mentioned current image and the above-mentioned second depth map.

[0270] Specifically, the above-mentioned upsampling and fusion processing specifically includes:

[0271] S1: Upsample the feature map P' j to obtain a feature map P'', j the resolution of this feature map P'' j is the same as that of the (j + 1)-th feature map P j+1 in the processing object, the width of the feature map P j+1 is (j + 1) times the width of the feature map with the smallest resolution in the processing object, where j is an integer greater than 0 and less than T; T is the number of feature maps in the processing object;

[0272] S2: Fuse the feature map P'' j with the feature map P j+1 to obtain a third feature map P' j+1 ,

[0273] S3: Let j = j + 1, and repeat steps S1 - S3 until j = T - 1;

[0274] where, when j = 1, the third feature map P' j is the feature map with the smallest resolution in the processing object, and when j = T - 1, the third feature map P' j+1 is the result of the upsampling and fusion processing.

[0275] Among them, the above-mentioned processing objects include the above-mentioned T third feature maps or T fifth feature maps.

[0276] For example, there are 5 third feature maps, namely feature map P1, feature map P2, feature map P3, feature map P4, and feature map P5, and their resolutions increase in sequence; perform upsampling on feature map P1 to obtain feature map P”1 with the same resolution as feature map P2, and fuse feature map P”1 and feature map P2 to obtain feature map P’2; perform upsampling on feature map P’2 to obtain feature map P”2 with the same resolution as feature map P3, and fuse feature map P”2 and feature map P3 to obtain feature map P’3; perform upsampling on feature map P’3 to obtain feature map P”3 with the same resolution as feature map P4, and fuse feature map P”3 and feature map P4 to obtain feature map P’4; perform upsampling on feature map P’4 to obtain feature map P”4 with the same resolution as feature map P5, and fuse feature map P”4 and feature map P5 to obtain the target depth map of the current image.

[0277] Optionally, the above-mentioned upsampling is deconvolution upsampling.

[0278] In an optional embodiment, determining the target depth map of the current image according to the current image and the first depth map of the current image includes:

[0279] Method 1: Input the current image and the first depth map of the current image into a depth estimation model for feature extraction, and obtain the target depth map of the current image according to the result of the feature extraction, or,

[0280] Method 2: Concatenate the first depth map and the second depth map of the current image to obtain a third depth map, then input the current image and the third depth map of the current image into a depth estimation model for feature extraction, and obtain the target depth map of the current image according to the result of the feature extraction, or,

[0281] Method 3: Concatenate the first depth map and the second depth map of the current image to obtain a third depth map, then input the current image, the third depth map of the current image, and the above-mentioned reference depth map into a depth estimation model for feature extraction, and obtain the target depth map of the current image according to the result of the feature extraction.

[0282] It should be noted here that the specific implementation process of obtaining the target depth map of the current image by using the depth estimation model can refer to the relevant description of “determining the target depth map of the current image according to the current image and the first depth map of the current image” above, and will not be described here again.

[0283] In an alternative embodiment, before obtaining the target depth map of the current image using the depth estimation model, a depth estimation model acquisition request is sent to the server. The depth estimation model acquisition request carries the current image and the location of the terminal device. A response message for responding to the depth estimation model acquisition request is received from the server. The response message carries the depth estimation model of the current image, and the depth estimation model of the current image is obtained by the server according to the current image and the location of the terminal device.

[0284] Wherein, the location of the terminal device is the location where the terminal device is located when collecting the current image, and this location is the coordinate in the world coordinate system.

[0285] By obtaining the depth estimation model from the server, the terminal device does not need to train the depth estimation model by itself, reducing the power consumption of the terminal device and improving the real-time performance of virtual-real occlusion at the same time.

[0286] In an alternative embodiment, before obtaining the target depth map of the current image using the depth estimation model, the method of this embodiment further includes:

[0287] Training the initial convolutional neural network model to obtain the depth estimation model of the current image.

[0288] Specifically, training the initial convolutional neural network to obtain the depth estimation model of the current image includes:

[0289] Inputting multiple depth map samples corresponding to multiple image samples into the initial convolutional neural network for processing to obtain multiple first predicted depth maps; calculating a first loss value according to the multiple first predicted depth maps, the true depth maps corresponding to the multiple image samples, and the loss function; adjusting the parameters in the initial convolutional neural network according to the first loss value; obtaining the first convolutional neural network;

[0290] Then inputting multiple depth map samples corresponding to multiple image samples into the first convolutional neural network for processing to obtain multiple second predicted depth maps; obtaining a second loss value according to the multiple second predicted depth maps, the true depth maps corresponding to the multiple image samples, and the loss function; determining whether the second loss value converges; if it converges, determining the first convolutional neural network as the depth estimation model of the current image; if it does not converge, adjusting the parameters in the first convolutional neural network according to the second loss value to obtain the second convolutional neural network, and repeating the above process until the obtained loss value converges, and determining the convolutional neural network when the loss value converges as the depth estimation model of the current image.

[0291] The above loss function is:

[0292]

[0293] Wherein Denote the depth map \(d\) output by the depth estimation model at scale \(i\). i And the corresponding ground truth depth map \(g\). i The error of Denote the depth map gradient \(dg\) output by the depth estimation model at scale \(i\). i And the corresponding ground truth depth map gradient \(gg\). i The error of Denote the depth map normal vector \(dn\) output by the depth estimation model at scale \(i\). i And the corresponding ground truth depth map normal vector \(gn\). i The error of

[0294] Optionally, the depth estimation model can adopt network structures such as DiverseDepth, SARPN, or CSPN.

[0295] Among them, the feature extraction function in the above depth estimation model can be implemented by network structures such as VGGNet, ResNet, ResNeXt, DenseNet, etc.

[0296] VGGNet: All use 3*3 convolutional kernels and 2*2 pooling kernels, and improve performance by continuously deepening the network structure. For VGG-16, its input is an RGB image of size 224*224. When preprocessing, calculate the average value of the three channels and subtract the average value from each pixel (less iteration and faster convergence after processing). The image is processed through a series of convolutional layers. In the convolutional layers, very small 3*3 convolutional kernels are used. The reason for choosing 3*3 convolutional kernels is that 3*3 is the smallest size that can capture the information of the 8-neighborhood of pixels. The stride of the convolutional layer is set to 1 pixel, and the padding of the 3*3 convolutional layer is set to 1 pixel. The pooling layer uses max pooling, and there are 5 layers in total. After some convolutional layers, the window of max-pooling is 2*2, and the stride is set to 2. After the convolutional layers are three fully-connected layers (FC). The first two fully-connected layers each have 4096 channels, and the third fully-connected layer has 1000 channels for classification. The fully-connected layer configurations of all networks are the same. After the fully-connected layer is Softmax for classification. ReLU is used as the activation function for all hidden layers (in the middle of each conv layer).

[0297] ResNet: It is a residual network, which can be understood as a sub-network, and this sub-network can be stacked to form a very deep network. The characteristic of the residual network is that it is easy to optimize and can improve the accuracy by increasing a considerable depth. The internal residual blocks use skip connections, which alleviate the problem of gradient disappearance caused by increasing depth in deep neural networks.

[0298] Based on the idea of ResNet, ResNeXt proposes a structure that can improve accuracy without increasing the parameter complexity and at the same time reduce the number of hyperparameters. That is, drawing on the idea of Inception to expand the network width, it uses multiple branches to learn different features and replaces the three-layer convolutional block in the original ResNet with blocks of the same topological structure stacked in parallel. Without significantly increasing the order of the number of parameters, it improves the accuracy of the model. At the same time, due to the same topological structure, the hyperparameters are also reduced, facilitating model transplantation, and it has become a popular framework for recognition tasks.

[0299] DenseNet (Dense Connection Network): In traditional convolutional networks, each layer only uses the output features of the previous layer as its input. In DenseNet, each layer uses the features of all previous layers as input and its own features as input for all subsequent layers. DenseNet has the following advantages: alleviating the problem of gradient dispersion, enhancing feature propagation, encouraging feature reuse, and greatly reducing the number of parameters.

[0300] By adopting a multi-scale feature fusion strategy, adding temporal consistency and scale consistency constraints, and using multi-dataset joint training to improve the generalization ability of the model in different scenarios.

[0301] S303. Overlay and display the virtual object image and the current image according to the target depth map of the current image.

[0302] In a feasible embodiment, overlaying and displaying the virtual object image and the current image according to the target depth map of the current image includes:

[0303] Perform edge optimization on the target depth map of the current image to obtain the optimized depth map of the current image;

[0304] Overlay and display the virtual object image and the current image according to the optimized depth map.

[0305] Among them, overlaying and displaying the virtual object image and the current image according to the optimized depth map includes:

[0306] Judge the relationship between the depth value corresponding to each pixel of the current image and the depth value of the virtual object according to the optimized depth map. If the depth value corresponding to any pixel point A in the virtual object image is greater than the depth value of the pixel point B at the same position in the current image corresponding to the virtual object image, then display the color of the current image; otherwise, display the color of the virtual object image. After traversing all pixel points according to this method, an occlusion effect is displayed on the terminal device.

[0307] In a feasible embodiment, overlaying and displaying the virtual object image and the current image according to the target depth map of the current image includes:

[0308] Segment the target depth map of the current image to obtain the foreground depth map and the background depth map of the current image. The background depth map is the depth map containing the background area in the target depth map of the current image, and the foreground depth map is the depth map containing the foreground area in the target depth map of the current image. Then, fuse the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene. The L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image. L is an integer greater than 1.

[0309] Back-project the fused three-dimensional scene according to the background depth map of the current image to obtain a fused background depth map. Then, splice the fused background depth map and the foreground depth map of the current image to obtain an updated depth map. Finally, superimpose and display the virtual object image and the current image according to the updated depth map.

[0310] It should be noted here that the foreground area refers to the area where the object of interest is located, such as significant objects like people, cars, animals, and plants. The background area is the area in the image other than the foreground area.

[0311] Specifically, when the object of interest is a person, segmenting the target depth map of the current image is specifically to perform human segmentation on the target depth map of the current image according to the human mask to obtain the foreground depth map and the background depth map of the current image.

[0312] In a feasible embodiment, superimposing and displaying the virtual object image and the current image according to the target depth map of the current image includes:

[0313] Segment the optimized depth map to obtain the foreground depth map and the background depth map of the current image. The background depth map is the depth map containing the background area in the optimized depth map, and the foreground depth map is the depth map containing the foreground area in the optimized depth map. The optimized depth map is obtained by performing edge optimization on the target depth map of the current image. Then, fuse the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene. The L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image. L is an integer greater than 1. Back-project the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map. Then, splice the fused background depth map and the foreground depth map of the current image to obtain an updated depth map. Finally, superimpose and display the virtual object image and the current image according to the updated depth map.

[0314] It should be noted here that the foreground region refers to the region where the object of interest is located, such as prominent objects like people, cars, animals, and plants; the background region is the region in the image other than the foreground region.

[0315] Specifically, perform human segmentation on the optimized depth map of the current image. Specifically, perform human segmentation on the target depth map of the current image according to the human mask to obtain the third depth map and the fourth depth map of the current image.

[0316] Optionally, the above pose can be obtained according to the corresponding image, or it can be a SLAM pose, a pose obtained by a deep learning method, or a pose obtained by other methods.

[0317] Optionally, the specific fusion method used for the above fusion can be the truncated signed distance function (TSDF) fusion method, or it can also be the surfel fusion method.

[0318] In an optional embodiment, perform edge optimization on the target depth map of the current image to obtain an optimized depth map, including:

[0319] Extract the edge structure information of the current image and the edge structure information of the target depth map of the current image; calculate the difference between the edge structure information of the target depth map of the current image and the edge structure information of the current image, and then modify the edge position of the target depth map of the current image based on this difference to obtain the edge of the optimized depth map; according to the edge of the optimized depth map, obtain a sharp depth map corresponding to the edge of the current image, and this depth map is the above-mentioned optimized depth map.

[0320] Overlay and display the virtual object image and the current image according to the updated depth map, specifically including:

[0321] Judge the relationship between the depth value corresponding to each pixel of the current image and the depth value of the virtual object image according to the updated depth map. If the depth value corresponding to any pixel point A in the virtual object is greater than the depth value of the pixel point B in the current image at the same position as the pixel point A in the virtual object, then display the color of the current image; otherwise, display the color of the virtual object; after traversing all pixel points according to this method, display the occlusion effect on the terminal device.

[0322] It can be seen that in the embodiments of the present application, by performing depth estimation on the current image, the first depth map, and the second depth map of the current image, a target depth map with higher accuracy is obtained, thereby solving the problem that there will be frame - to - frame flicker and instability in the subsequent virtual - real occlusion effect; when performing depth estimation, the depth map collected by the TOF camera is introduced, further improving the accuracy of the target depth map of the current image, thereby solving the problem that there will be frame - to - frame flicker and instability in the subsequent virtual - real occlusion effect; by performing edge optimization on the target depth map of the current image to obtain an optimized depth map, and then fusing multiple frames of depth maps, a depth map with sharper portrait edges is obtained, which is beneficial to further improving the virtual - real occlusion effect.

[0323] See Figure 5 , Figure 5 is a schematic flowchart of another image processing method provided by the embodiments of the present application. As Figure 5 shown, the method includes:

[0324] S501. Receive a depth estimation model request message sent by a terminal device. The request message carries the current image collected by the terminal device and the position of the terminal device when collecting the current image.

[0325] Among them, the above - mentioned position is the coordinate in the world coordinate system, and the world coordinate system can be a UTM coordinate system, a GPS coordinate system, or other world coordinate systems, etc.

[0326] S502. Obtain the depth estimation model of the current image from multiple depth estimation models stored in the server according to the current image and the position of the terminal device.

[0327] In a feasible embodiment, obtaining the depth estimation model of the current image from multiple depth estimation models stored in the server according to the current image and the position of the terminal device includes:

[0328] Obtain multiple frames of first images according to the position of the terminal device. The multiple frames of first images are images within a preset range centered on the position of the terminal device in the base map; obtain a target image from the multiple frames of first images. The target image is the image with the highest similarity to the current image among the multiple frames of first images; determine the depth estimation model corresponding to the target image as the depth estimation model of the current image.

[0329] Specifically, in order to improve the accuracy of depth estimation, in the server, a depth estimation model is separately trained for each position; after receiving a depth estimation model acquisition request from a terminal device, the server obtains multiple first images according to the position of the terminal device. The multiple first images are images within a preset range centered on the position of the terminal device in the base map. A target image is obtained from the multiple first images. The target image is the image with the highest similarity to the current image among the multiple first images; the pose of the current image is obtained according to the pose of the target image; a depth estimation model corresponding to this position is determined from the server according to the position in the pose of the current image, and this depth estimation model is the depth estimation model of the current image.

[0330] In a feasible embodiment, the method of this embodiment further includes:

[0331] For multiple first images, depth estimation modules for each of the multiple first images are separately trained.

[0332] Among them, for each of the multiple first images, the following steps are performed for training to obtain the depth estimation model of each first image:

[0333] Multiple image samples and their corresponding multiple depth map samples are input into an initial convolutional neural network for processing to obtain multiple predicted depth maps; a loss value is calculated according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; the parameters in the initial convolutional neural network are adjusted according to the loss value to obtain the depth estimation model of each first image.

[0334] Among them, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0335] It should be pointed out here that the above is only one training process; in actual applications, the above method will be iterated multiple times until the calculated loss value converges; the convolutional neural network model when the loss value converges is determined as the depth estimation model of the current image. The specific training process above can refer to the relevant description in S302 and will not be described here again.

[0336] S503. Send a response message in response to the depth estimation model request message to the terminal device. The response message carries the depth estimation model of the current image.

[0337] In an optional embodiment, a local map acquisition request sent by a receiving terminal device is received, and the local map acquisition request carries a current image; a local map is acquired from a base map stored in a server according to the pose of the current image; a response message for responding to the local map acquisition request is sent to the server, and the response message carries the local map.

[0338] Specifically, after receiving the local map request message, the server calculates the position and angle information of the current image in the global map according to VPS. This position and angle information are collectively referred to as pose information, and this pose information is based on the pose in the world coordinate system; a local map is acquired from the base map according to the pose information. The local map is an area within a certain range (such as a radius of 50 meters) centered on this position in the base map; the base map includes 2D feature points of objects and their corresponding feature descriptors, and 3D point cloud information. Therefore, the local map also includes 2D feature points of objects and their corresponding feature descriptors and 3D point cloud information.

[0339] In an optional embodiment, the method of this application further includes:

[0340] An initial depth map of the current image is acquired according to the current image and a pre-stored image, and a fifth depth map is obtained according to the current image and 3D points corresponding to the local map; the initial depth map and the fifth depth map are optimized according to the pose of the current image to obtain a second depth map.

[0341] Optionally, the pre-stored image is uploaded by the terminal device, and the timestamp of the pre-stored image is before the timestamp of the current image.

[0342] Specifically, the first 2D feature points of the current image are matched with the 2D feature points of the pre-stored image to obtain the sixth 2D feature points of the current image; the noise points in the sixth 2D feature points of the current image are removed to obtain the seventh 2D feature points of the current image; triangulation calculations are performed on each 2D feature point in the seventh 2D feature points of the current image to obtain the initial 3D points in space of the seventh 2D feature points of the current image; the initial depth map of the current image includes the initial 3D points in space of the seventh 2D feature points of the current image.

[0343] In a feasible embodiment, obtaining a fifth depth map according to the current image and 3D points corresponding to the local map includes

[0344] From multiple frame-based maps, obtain M maps, where the similarity between each of the M maps and the current image is greater than a first preset threshold; M is an integer greater than 0; match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; according to the correspondence between each fifth 2D feature point in the M maps and the 3D points, obtain the 3D point corresponding to each fourth 2D feature point in the multiple feature point matching pairs; according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, obtain a fifth depth map, where the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs.

[0345] Since the base maps in the server are collected offline, there are differences between the base maps in the server and the current actual environment. For example, a large billboard in a shopping mall existed during offline collection, but after a period of time, when the user captures the current image, the billboard has been removed, which results in inconsistent 3D point cloud information between the maps sent by the server and the current environment. In addition, the images received by the server can be images after privacy processing, which also leads to inconsistent 3D point cloud information between the maps sent by the server and the current environment.

[0346] For the above reasons, the server updates the sent local map to obtain a second depth map.

[0347] Specifically, the server extracts features from the current image to obtain the first 2D feature points of the current image; the methods for extracting features from the current image include but are not limited to the SIFT method, the ORB method, the SURF method, and the super integral (spuerpoint) method. The server matches the 2D feature points in the current image with the 2D feature points of the pre-stored images to obtain the sixth 2D feature points in the current image. Optionally, the above pre-stored images are images obtained by the terminal device and with timestamps before the timestamp of the current image, and at least one image with a timestamp relatively close to the timestamp of the current image. Then the server removes the noise points in the sixth 2D feature points of the current image. Specifically, the verification value of each sixth 2D feature point in the current image is calculated through the homography matrix, the fundamental matrix, and the essential matrix; if the verification value of the sixth 2D feature point is lower than the second preset threshold, it is determined that the sixth 2D feature point is a noise point, and thus the sixth 2D feature point is deleted to obtain the seventh 2D feature points of the current image.

[0348] The server performs triangulation calculation on the seventh 2D feature points of the current image to obtain the initial positions of the seventh 2D feature points in the current image in space, and further obtains the initial depth map corresponding to the seventh 2D feature points in the current image.

[0349] The server uses image retrieval methods to obtain M images from multiple frame base maps, and the similarity between each of the M maps and the current image is greater than the first preset threshold. The image retrieval methods include but are not limited to the bag-of-words tree method or the NetVlad method based on deep learning; the first 2D feature points in the current image are matched with the 2D feature points in the M images to obtain multiple matching feature point pairs. Each matching feature point pair in the multiple matching feature point pairs includes a fourth 2D feature point and a fifth 2D feature point. The fourth 2D feature point and the fifth 2D feature point are mutually matching feature points. The fourth 2D feature point is a 2D feature point in the current image, and the fifth 2D feature point is a feature point in the M images; since each 2D feature point in the M images corresponds to a 3D point, the correspondence between the fifth 2D feature points and the 3D points in the M images and the multiple matching feature point pairs determine the 3D points corresponding to the fourth 2D feature points in the current image; according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, a fifth depth map is obtained. The fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs.

[0350] Combining the above initial depth map and the fifth depth map, and combining the initial pose information of each image in the current image and the historical images, using BA to adjust and optimize the position information of the 3D points in the depth map, and calculating the reprojection error of each 3D point in the optimized point cloud, and deleting the 3D points whose reprojection error exceeds the error threshold; and repeating the above BA optimization and point cloud deletion steps multiple times, finally obtaining an optimized high-precision depth map. The optimized high-precision depth map is the above second depth map, thereby realizing the update of the base map in the server.

[0351] Through the above method, the base map stored in the server can be continuously updated by the images uploaded by the terminal device, so that the base map is consistent with the image content collected by the terminal device, thereby providing high-precision 3D point cloud information during depth estimation. And through the above method, the content in the base map is made consistent with the content in the images collected by the terminal device. The more images are uploaded by the terminal device, the more thorough the update of the map, so that the result of depth estimation is more accurate.

[0352] It can be seen that in this application, the cloud-based local map update algorithm can solve the problem that directly using offline map points to participate in depth map estimation introduces errors due to changes in the offline map and the current usage scenario. At the same time, it can solve the scale ambiguity and inter-frame instability problems of monocular depth estimation, and further solve the problems of virtual-real occlusion flicker and inconsistent inter-frame occlusion phenomena.

[0353] In a specific example, in step S10, a color image collected by a terminal device and information measured by an inertial measurement unit (IMU) are obtained; SLAM system initialization is performed according to the color image and the information measured by the IMU; after the SLAM system initialization is successful, the terminal device calculates the pose of the terminal device in its own local coordinate system in real time according to the SLAM system; after the terminal device obtains the current image, face detection is performed on the current image, and privacy processing is performed on the face area in the current image, such as filling the color of the portrait area with black; then the current image after privacy processing is transmitted to the server; the server calls the VPS algorithm to locate the current image. If the positioning is successful, the pose of the current image in the base map stored by the server is obtained; the server obtains a local map from the base map according to the pose of the current image in the base image. The local map is an area with a radius of 150 m centered on the position of the current image in the base map; and the 3D point cloud, 2D feature points, and feature descriptors of the local map are transmitted to the terminal device; and the current valid upload image counter is set to 1, and step S20 is entered; if the positioning fails, the current valid upload image counter is set to 0, and a failure message is returned, and step S10 is re-entered.

[0354] Step S20:

[0355] Step S20.1: If the current valid upload image counter is 0, directly proceed to Step S20.2; otherwise, the server extracts features from the current image to obtain the first 2D feature points of the current image. The methods for extracting features from the current image include, but are not limited to, the SIFT method, the ORB method, the SURF method, and the super point method. The server matches the first 2D feature points in the current image with the 2D feature points of the pre-stored images to obtain the sixth 2D feature points in the current image. Optionally, the pre-stored images are images obtained by the terminal device and with timestamps before the timestamp of the current image, and at least one image with a timestamp relatively close to the timestamp of the current image. Then the server removes the noise points in the sixth 2D feature points of the current image. Specifically, it calculates the verification value of each sixth 2D feature point in the current image through the homography matrix, the fundamental matrix, and the essential matrix; if the verification value of the sixth 2D feature point is lower than the second preset threshold, it determines that the sixth 2D feature point is a noise point, and thus deletes the sixth 2D feature point to obtain the seventh 2D feature points of the current image. The server performs triangulation calculation on the seventh 2D feature points of the current image to obtain the initial position in space of the fourth 2D feature points in the current image, and further obtains the initial depth map corresponding to the seventh 2D feature points in the current image.

[0356] Step S20.2: The server obtains the 10 closest images from the local map obtained in Step S10 according to the NetVlad algorithm. Extract features from the current image uploaded by the terminal device according to the 2D feature point categories of the local map; in this example, the 2D feature points of the local map include, but are not limited to, SIFT feature points, SURF feature points, ORB feature points, superpoint feature points, D2Net feature points, ASLFeat feature points, R2D2 feature points, etc. Through the brute-force matching method, match the first 2D feature points in the current image with the 2D feature points in the above 10 images to obtain the 2D-2D feature point matching relationship; since there are corresponding 3D points for each 2D feature point in the local map, the 3D points corresponding to the matching relationship with the first 2D feature points of the current image can be filtered out from the 3D points of the local image according to the obtained 2D-2D feature point matching relationship;

[0357] Step S20.3: Combine the above initial depth map and the depth map obtained from the 3D points in the local image that match the 2D feature points of the current image, and use BA to adjust and optimize the position information of the 3D points in combination with the initial pose information of each image in the current image and the historical images, and calculate the reprojection error of the optimized 3D points. Delete the 3D points with reprojection error exceeding the error threshold; and repeat the above BA optimization and deletion steps multiple times. Finally, obtain the optimized high-precision depth map, thereby realizing the update of the basic map in the server;

[0358] Step S20.3: Set the current valid upload image counter to 1.

[0359] Step S30: Based on the current image, the above-optimized high-precision depth map, and the depth map obtained from the 3D points corresponding to the 2D feature points in the local map that match the 2D feature points in the current image, perform depth estimation to obtain the target depth map of the current image. The specific implementation process can refer to the relevant description in S302 and will not be elaborated here.

[0360] Step S40: Input the current image into a human segmentation network to obtain a human segmentation result map. The human segmentation network includes a feature extraction network and a softmax classifier. The structure of the feature extraction network can be network structures such as FCN, ParseNet, DeepLabv1, DeepLab v2, DeepLab v3, RefineNet, SegNet, PSPNet, ENet, ICNet, BiSeNet, etc. The feature extraction network is used to extract the features of the current image, then bilinearly upsample the features of the current image to obtain a feature map with the same size as the input, and finally obtain the label of each pixel through the softmax classifier, thereby obtaining the human segmentation result map, also known as the human mask. For example, input the current image into BiSeNet to obtain a feature map, and then classify each pixel of this feature map through the Softmax classifier. Among them, the pixels in the human region are classified as 255, and the pixels in the non-human region are classified as 0.

[0361] It should be noted here that the segmentation of the current image in this application is not limited to human segmentation, but can also be object segmentation, such as the segmentation of objects like cars, airplanes, kittens, etc. The segmentation method can refer to human segmentation and will not be elaborated here.

[0362] Step S50: The terminal device performs human segmentation on the target depth map of the current image according to the human mask to obtain the foreground depth map and the background depth map of the current image. The background depth map of the current image is the depth map containing the background area in the target depth map of the current image, and the foreground depth map of the current image is the depth map containing the foreground area in the target depth map of the current image; perform truncated signed distance function (TSDF) fusion on the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain the fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; perform back-projection on the fused three-dimensional scene according to the pose of the current image to obtain the fused background depth map; splice the fused background depth map and the foreground depth map of the current image to obtain the updated depth map

[0363] Step S60: Send the depth map obtained in step 50 and the current image into the renderer. In the renderer, judge the relationship between the depth value of each pixel of the current image and the depth map of the virtual object. If the depth value of the virtual object is greater than the depth map, display the color of the current image, otherwise display the color of the virtual object. After traversing each pixel point one by one, display the occlusion effect on the terminal device.

[0364] As Figure 6 shown, by accurately estimating the depth map of the tree grove, the virtual panda can be seen through the gaps in the real tree grove, and the panda can also be occluded by buildings such as walls. Of course, the algorithm also supports the occlusion between people and virtual objects. It can be seen from the last figure that the algorithm can accurately estimate the depth map of the entire scene, enabling the virtual panda to be between people's arms, with a strong overall sense of immersion and excellent user experience.

[0365] See Figure 7 , Figure 7 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As Figure 7 shown, the terminal device 100 includes:

[0366] A 2D-3D matching module 102, configured to match the first 2D feature points of the current image with the pre-stored 2D feature points to obtain the second 2D feature points of the current image, and according to the second 2D feature points in the current image and the relationship between the pre-stored 2D feature points and 3D points, obtain the 3D points corresponding to the second 2D feature points in the current image; the 3D points corresponding to the second 2D feature points constitute the first depth map;

[0367] The depth estimation module 104 is configured to perform feature extraction on the current image to obtain T first feature maps, where the resolution of each of the T first feature maps is different; perform feature extraction on the first depth map to obtain T second feature maps, where the resolution of each of the T second feature maps is different; superimpose the first feature maps and the second feature maps with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image.

[0368] It should be noted here that the specific process of performing upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image can refer to the relevant description in step S302 and will not be elaborated here.

[0369] The human segmentation module 108 is configured to obtain a human segmentation result map of the current image through a human segmentation network. The human segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network is used to extract the features of the current image, then perform bilinear upsampling on the features of the current image to obtain a feature map with the same size as the input, and finally obtain the label of each pixel through the softmax classifier, thereby obtaining the human segmentation result map, also known as the human mask. For example, input the current image into BiSeNet to obtain a feature map, and then classify each pixel of this feature map through the Softmax classifier. Among them, the pixels in the human region are classified as 255, and the pixels in the non-human region are classified as 0.

[0370] The depth map edge optimization module 105 is configured to respectively perform edge structure extraction on the current image and its target depth map to obtain the edge structure information of the current image and the edge structure information of the target depth map; calculate the difference between the edge structure of the target depth map and the edge structure of the current image with the edge structure information of the current image as a reference, and then modify the edge position of the target depth map through this difference, thereby optimizing the edge of the depth map to obtain an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0371] The virtual-real occlusion application module 107 is used to perform human segmentation on the optimized depth map according to the human portrait mask, so as to obtain a depth map including the human portrait area and a depth map of the non-human portrait area; then, according to the relationship between the depth values in the depth map including the human portrait area and the depth map including the non-human portrait area and the depth value of the virtual object, the current image and the virtual object image are superimposed and displayed; if the depth value corresponding to any pixel point A in the virtual object is greater than the depth value of the pixel point B at the same position as the pixel point A in the virtual object in the current image, the color of the current image is displayed; otherwise, the color of the virtual object image is displayed; after traversing all pixel points according to this method, the occlusion effect is displayed on the terminal device. Since the optimized depth map is segmented for the human portrait, when the current image includes a human portrait, the occlusion effect between the person and the virtual object can be displayed on the terminal device.

[0372] See Figure 8 , Figure 8 which is a schematic structural diagram of another terminal device provided by an embodiment of the present application. As Figure 8 shown, the terminal device 100 includes:

[0373] The 2D-3D matching module 102 is used to match the first 2D feature points of the current image with the pre-stored 2D feature points to obtain the second 2D feature points of the current image; according to the second 2D feature points in the current image and the relationship between the pre-stored 2D feature points and 3D points, the 3D points corresponding to the second 2D feature points in the current image are obtained; the 3D points corresponding to the second 2D feature points constitute the first depth map;

[0374] The depth estimation module 104 is used to perform feature extraction on the current image to obtain T first feature maps, and the resolution of each first feature map in the T first feature maps is different; perform feature extraction on the first depth map to obtain T second feature maps, and the resolution of each second feature map in the T second feature maps is different; superimpose the first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image.

[0375] It should be noted here that the specific process of performing upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image can be referred to the relevant description of step S302 and will not be described here.

[0376] The human portrait segmentation module 108 is used to obtain a human portrait segmentation result map from the current image through a human portrait segmentation network. The human portrait segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network is used to extract the features of the current image, then perform bilinear upsampling on the features of the current image to obtain a feature map with the same size as the input, and finally obtain the label of each pixel through the softmax classifier, so as to obtain the human portrait segmentation result map, also known as the human portrait mask. For example, input the current image into BiSeNet to obtain a feature map, and then classify each pixel of this feature map through the Softmax classifier. Among them, the pixels in the human portrait area are classified as 255, and the pixels in the non-human portrait area are classified as 0.

[0377] The depth map edge optimization module 105 is used to respectively extract the edge structures of the current image and its target depth map to obtain the edge structure information of the current image and the edge structure information of the target depth map; taking the edge structure information of the current image as a reference, calculate the difference between the edge structure of the target depth map and the edge structure of the current image, and then modify the edge position of the target depth map through this difference, so as to optimize the edge of the depth map and obtain an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0378] The multi-view fusion module 106 is used to perform human portrait segmentation on the optimized depth map according to the human portrait mask to obtain a foreground depth map and a background depth map. The background depth map is the depth map containing the non-human area in the optimized depth map, and the foreground depth map is the depth map containing the human portrait area in the optimized depth map; fuse the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; perform back-projection on the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map; splice the fused background depth map and the foreground depth map of the current image to obtain an updated depth map;

[0379] The virtual-real occlusion application module 107 is used to judge the relationship between the depth value corresponding to each pixel of the current image and the depth value of the virtual object according to the updated depth map. If the depth value corresponding to any pixel point A in the virtual object is greater than the depth value corresponding to the pixel point B at the same position in the current image as the pixel point A in the virtual object, then display the color of the current image; otherwise, display the color of the virtual object; after traversing all pixel points according to this method, display the occlusion effect on the terminal device.

[0380] See Figure 9 , Figure 9A schematic diagram of a system structure provided by an embodiment of the present application. As Figure 9 shown, the system includes a terminal device 100 and a server 200. Among them, the terminal device 100 includes: a 2D-3D matching module 102, a depth estimation module 104, a depth map edge optimization module 105, a human segmentation module 108, and a virtual-real occlusion application module 107; the server 200 includes a VPS positioning and map distribution module 101 and a local map update module 103;

[0381] The VPS positioning and map distribution module 101 is configured to, after receiving the current image, calculate the position and angle information of the current image in the global map according to VPS. The position and angle information are collectively referred to as pose information, and the pose information is based on the pose in the world coordinate system; obtain a local map from the base map according to the pose information. For example, the local map is an area within a certain range (such as a radius of 50 meters) centered on the position in the above pose information in the base map; the base map includes 2D feature points of objects and their corresponding feature descriptors, and 3D point cloud information. Therefore, the local map also includes 2D feature points of objects and their corresponding feature descriptors and 3D point cloud information;

[0382] The 2D-3D matching module 102 is configured to extract features from the current image to obtain the first 2D feature points of the current image; match the first 2D feature points of the current image according to the 2D feature points of the local map to obtain the third 2D feature points in the local map; the local map is obtained by the server according to the current image; obtain the 3D points corresponding to the third 2D feature points in the local map according to the second 2D feature points in the local map and the correspondence between the 2D feature points and 3D points in the local map; among them, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map;

[0383] The depth estimation module 104 is configured to extract features from the current image to obtain T first feature maps, and the resolution of each first feature map in the T first feature maps is different; extract features from the first depth map to obtain T second feature maps, and the resolution of each second feature map in the T second feature maps is different; superimpose the first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image;

[0384] In an optional embodiment, the local map update module 103 is configured to match the first 2D feature points of the current image with the 2D feature points of the pre-stored image to obtain the sixth 2D feature points of the current image; remove the noise points from the sixth 2D feature points of the current image to obtain the seventh 2D feature points of the current image; perform triangulation calculation on each 2D feature point in the seventh 2D feature points of the current image to obtain the initial 3D points of the seventh 2D feature points of the current image in space; the initial depth map of the current image includes the initial 3D point cloud of the seventh 2D feature points of the current image in space; obtain M maps from multiple base maps, where the similarity between each of the M maps and the current image is greater than a first preset threshold; match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; obtain the 3D point corresponding to each fourth 2D feature point in the multiple feature point matching pairs according to the correspondence between the fifth 2D feature point and the 3D point in the M maps; obtain a fifth depth map according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, where the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs; combine the above initial depth map and the third depth map, combine the above initial depth map and the fifth depth map, and use BA to adjust and optimize the position information of the 3D points in the depth map in combination with the initial pose information of each image in the current image and the pre-stored image, and calculate the reprojection error of each 3D point in the optimized point cloud, and delete the 3D points whose reprojection error exceeds the error threshold; and repeat the above BA optimization and point cloud deletion steps multiple times, and finally obtain an optimized high-precision depth map, which is the above second depth map, so as to realize the update of the base map in the server;

[0385] The depth estimation module 104 is configured to perform feature extraction on the current image to obtain T first feature maps, where the resolution of each first feature map in the T first feature maps is different; perform feature extraction on the third depth map to obtain T second feature maps, where the resolution of each second feature map in the T second feature maps is different, and the third depth map is obtained by stitching the first depth map and the second depth map; superimpose the first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image;

[0386] It should be noted here that the specific process of upsampling and fusing the T third feature maps to obtain the target depth map of the current image can be referred to the relevant description in step S302, which will not be elaborated here.

[0387] The human segmentation module 108 is used to obtain the human segmentation result map of the current image through the human segmentation network. The human segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network is used to extract the features of the current image, then perform bilinear upsampling on the features of the current image to obtain a feature map with the same size as the input, and finally obtain the label of each pixel through the softmax classifier, so as to obtain the human segmentation result map, also called the human mask. For example, input the current image into BiSeNet to obtain a feature map, and then classify each pixel of this feature map through the Softmax classifier. Among them, the pixels in the human region are classified as 255, and the pixels in the non-human region are classified as 0.

[0388] The depth map edge optimization module 105 is used to extract the edge structures of the current image and its target depth map respectively to obtain the edge structure information of the current image and the edge structure information of the target depth map; taking the edge structure information of the current image as a reference, calculate the difference between the edge structure of the target depth map and the edge structure of the current image, and then modify the edge position of the target depth map through this difference, so as to optimize the edge of the depth map and obtain an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0389] The virtual-real occlusion application module 107 is used to perform human segmentation on the optimized depth map according to the human mask to obtain a depth map including the human region and a depth map including the non-human region; then superimpose and display the current image and the virtual object image according to the relationship between the depth values in the depth map including the human region and the depth map including the non-human region and the depth value of the virtual object; if the depth value corresponding to any pixel point A in the virtual object image is greater than the depth value corresponding to the pixel point B at the same position as the pixel point A in the current image in the virtual object image, then display the color of the current image; otherwise, display the color of the virtual object image; after traversing all pixel points according to this method, the occlusion effect is displayed on the terminal device. Since the optimized depth map is subjected to human segmentation, when the current image includes a human, the occlusion effect between the human and the virtual object can be displayed on the terminal device.

[0390] See Figure 10 , Figure 10 which is another schematic diagram of the system structure provided by the embodiment of the present application. As Figure 10As shown, the system includes a terminal device 100 and a server 200. Among them, the terminal device 100 includes: a 2D-3D matching module 102, a depth estimation module 104, a depth map edge optimization module 105, a portrait segmentation module 108, and a virtual-real occlusion application module 107; the server 200 includes a VPS positioning and map distribution module 101 and a local map update module 103;

[0391] The VPS positioning and map distribution module 101 is configured to, after receiving the current image, calculate the position and angle information of the current image in the global map according to VPS. The position and angle information are collectively referred to as pose information, and the pose information is based on the pose in the world coordinate system; obtain a local map from the base map according to the pose information. For example, the local map is an area in the base map centered on the position in the above pose information and within a certain range around it (such as within a radius of 50 meters); the base map includes 2D feature points of objects and their corresponding feature descriptors, and 3D point cloud information. Therefore, the local map also includes 2D feature points of objects and their corresponding feature descriptors and 3D point cloud information;

[0392] The 2D-3D matching module 102 is configured to extract features from the current image to obtain the first 2D feature points of the current image; match the first 2D feature points of the current image according to the 2D feature points of the local map to obtain the third 2D feature points in the local map; the local map is obtained by the server according to the current image; obtain the 3D points corresponding to the third 2D feature points in the local map according to the second 2D feature points in the local map and the correspondence between the 2D feature points and 3D points in the local map; among them, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map;

[0393] The depth estimation module 104 is configured to extract features from the current image to obtain T first feature maps, and the resolution of each first feature map in the T first feature maps is different; extract features from the first depth map to obtain T second feature maps, and the resolution of each second feature map in the T second feature maps is different; superimpose the first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image;

[0394] In an optional embodiment, the local map update module 103 is configured to match the first 2D feature points of the current image with the 2D feature points of the pre-stored image to obtain the sixth 2D feature points of the current image; remove the noise points from the sixth 2D feature points of the current image to obtain the seventh 2D feature points of the current image; perform triangulation calculation on each 2D feature point in the seventh 2D feature points of the current image to obtain the initial 3D points of the seventh 2D feature points of the current image in space; the initial depth map of the current image includes the initial 3D point cloud of the seventh 2D feature points of the current image in space; obtain M maps from multiple base maps, where the similarity between each of the M maps and the current image is greater than a first preset threshold; match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the sixth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; obtain the 3D point corresponding to each fourth 2D feature point in the multiple feature point matching pairs according to the correspondence between the fifth 2D feature points and the 3D points in the M maps; obtain a fifth depth map according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, where the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs; combine the above initial depth map and the third depth map, combine the above initial depth map and the fifth depth map, and use BA to adjust and optimize the position information of the 3D points in the depth map in combination with the initial pose information of each image in the current image and the pre-stored image, and calculate the reprojection error of each 3D point in the optimized point cloud, and delete the 3D points whose reprojection error exceeds the error threshold; and repeat the above BA optimization and point cloud deletion steps multiple times, and finally obtain an optimized high-precision depth map, which is the above second depth map, so as to realize the update of the base map in the server;

[0395] The depth estimation module 104 is configured to perform feature extraction on the current image to obtain T first feature maps, where the resolution of each first feature map in the T first feature maps is different; perform feature extraction on the third depth map to obtain T second feature maps, where the resolution of each second feature map in the T second feature maps is different, and the third depth map is obtained by splicing the first depth map and the second depth map; superimpose the first feature maps and the second feature maps with the same resolution in the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image;

[0396] It should be noted here that the specific process of upsampling and fusing the T third feature maps to obtain the target depth map of the current image can be referred to the relevant description in step S302 and will not be elaborated here.

[0397] The human segmentation module 108 is used to obtain a human segmentation result map from the current image through a human segmentation network. The human segmentation network includes a feature extraction network and a softmax classifier. The feature extraction network is used to extract the features of the current image, then bilinearly upsample the features of the current image to obtain a feature map with the same size as the input, and finally obtain the label of each pixel through the softmax classifier, so as to obtain the human segmentation result map, also known as the human mask. For example, input the current image into BiSeNet to obtain a feature map, and then classify each pixel of this feature map through the Softmax classifier. Among them, the pixels in the human region are classified as 255, and the pixels in the non-human region are classified as 0.

[0398] The depth map edge optimization module 105 is used to respectively extract the edge structures of the current image and its target depth map to obtain the edge structure information of the current image and the edge structure information of the target depth map; taking the edge structure information of the current image as a reference, calculate the difference between the edge structure of the target depth map and the edge structure of the current image, and then modify the edge position of the target depth map through this difference, so as to optimize the edge of the depth map and obtain an optimized depth map, which is a sharp depth map corresponding to the edge of the current image;

[0399] The multi-view fusion module 106 is used to perform human segmentation on the optimized depth map according to the human mask to obtain a foreground depth map and a background depth map. The background depth map is the depth map containing the non-human region in the optimized depth map, and the foreground depth map is the depth map containing the human region in the optimized depth map; fuse the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; back-project the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map; splice the fused background depth map and the foreground depth map of the current image to obtain an updated depth map;

[0400] The virtual-real occlusion application module 107 is used to determine the relationship between the depth value corresponding to each pixel of the current image and the depth value of the virtual object according to the updated depth map. If the depth value corresponding to any pixel point A in the virtual object is greater than the depth value corresponding to the pixel point B at the same position as pixel point A in the current image, the color of the current image is displayed; otherwise, the color of the virtual object is displayed. After traversing all pixel points according to this method, the occlusion effect is displayed on the terminal device.

[0401] Here it should be noted that in order to improve the accuracy of the target depth map of the current image, for Figures 7 - 10 the depth estimation module 104 in, the depth map collected by the TOF camera is introduced; at this time, the depth estimation module 104 is used to perform feature extraction on the current image to obtain T first feature maps, and perform feature extraction on the third depth map to obtain T second feature maps; perform feature extraction on the reference depth map to obtain T fourth feature maps. The resolution of each first feature map in the T first feature maps is different, the resolution of each second feature map in the T second feature maps is different, and the resolution of each fourth feature map in the T fourth feature maps is different; the reference depth map is obtained according to the depth map collected by the time-of-flight (TOF) camera, and T is an integer greater than 1; the first feature maps, second feature maps, and fourth feature maps with the same resolution in the T first feature maps, T second feature maps, and T fourth feature maps are superimposed to obtain T fifth feature maps; perform upsampling and fusion processing on the T fifth feature maps to obtain the target depth map of the current image;

[0402] Among them, the third depth map can be the above-mentioned first depth map, or the one obtained by splicing the above-mentioned first depth map and the second depth map.

[0403] Optionally, the reference depth map is the depth map collected by the TOF camera.

[0404] In order to reduce the power consumption of the terminal device, the above-mentioned TOF camera collects depth maps at a frame rate lower than the preset frame rate, and the resolution of this depth map is lower than the preset resolution; the terminal device 100 projects the depth map collected by the TOF camera into the three-dimensional space according to the pose of the current image to obtain the fourth depth map; project the fourth depth map onto the reference image according to the pose of the reference image to obtain the reference depth map, and the reference image is the image adjacent to the current image in terms of acquisition time.

[0405] Optionally, the TOF camera can be the camera of the terminal device 100 or the camera of other terminal devices; after the depth map collected by the TOF camera of other terminal devices, the other terminal devices send the depth map collected by the TOF camera to the terminal device 100. In this way, the depth map collected by TOF can be introduced even if the terminal device 100 does not include a TOF camera, thereby improving the accuracy of the target depth map of the current image.

[0406] See Figure 11 , Figure 11 which is a schematic structural diagram of the terminal device provided by the embodiment of the present application. As Figure 11 shown, the terminal device 1100 includes:

[0407] An acquisition unit 1101, configured to acquire a current image and a virtual object image, and acquire a first depth map and a second depth map of the current image according to the current image, where the second depth map is acquired from a server;

[0408] An estimation unit 1102, configured to perform feature extraction according to the current image, the first depth map and the second depth map of the current image, and obtain a target depth map of the current image according to the result of the feature extraction.

[0409] In a feasible embodiment, the acquisition unit 1101 is further configured to acquire a virtual object image;

[0410] The terminal device 1100 further includes:

[0411] An overlay display unit 1103, configured to overlay and display the virtual object image and the current image according to the target depth map of the current image.

[0412] In a feasible embodiment, in terms of acquiring the first depth map of the current image according to the current image, the acquisition unit 1101 is specifically configured to:

[0413] Perform feature extraction on the current image to obtain the first 2D feature points of the current image; match the first 2D feature points of the current image with the pre-stored 2D feature points to obtain the second 2D feature points in the current image; according to the correspondence between the second 2D feature points in the current image and the pre-stored 2D feature points and 3D points, obtain the 3D points corresponding to the second 2D feature points in the current image; where the first depth map of the current image includes the 3D points corresponding to the second 2D feature points in the current image.

[0414] In a feasible embodiment, in terms of acquiring the first depth map of the current image according to the current image, the acquisition unit 1101 is specifically configured to:

[0415] Extract features from the current image to obtain the first 2D feature points of the current image; match the first 2D feature points of the current image with the 2D feature points of the local map to obtain the third 2D feature points in the local map; the local map is obtained by the server based on the current image; according to the second 2D feature points in the local map and the correspondence between the 2D feature points and 3D points in the local map, obtain the 3D points corresponding to the third 2D feature points in the local map; wherein, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

[0416] In a feasible embodiment, in terms of matching the first 2D feature points of the current image with the 2D feature points of the local map to obtain the third 2D feature points in the local map, the obtaining unit 1101 is specifically configured to:

[0417] Obtain a target map from the local map according to the first pose, and the position of the target map in the local map is associated with the position indicated by the angle information in the first pose; the first pose is the pose obtained by the terminal device based on the current image converted into the pose in the world coordinate system, match the 2D feature points in the target map with the first 2D feature points of the current image to obtain the third 2D feature points of the target map, and the third 2D feature points of the local map include the third 2D feature points of the target map.

[0418] In a feasible embodiment, the estimating unit 1102 is specifically configured to:

[0419] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform feature extraction on the third depth map to obtain T second feature maps; the resolutions of each of the T first feature maps are different, and the resolutions of each of the T second feature maps are different; T is an integer greater than 1; superimpose the first feature maps and the second feature maps with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps; perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image; wherein, the third depth map is the above-mentioned first depth map or is obtained by splicing the first depth map and the second depth map.

[0420] In a feasible embodiment, the estimating unit 1102 is specifically configured to:

[0421] Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; perform multi-scale feature extraction on the reference depth map to obtain T fourth feature maps. The resolutions of each of the T first feature maps are different from each other, the resolutions of each of the T second feature maps are different from each other, and the resolutions of each of the T fourth feature maps are different from each other; the reference depth map is obtained based on the depth map collected by the TOF camera, and T is an integer greater than 1; stack the first feature maps, the second feature maps, and the fourth feature maps with the same resolution among the T first feature maps, the T second feature maps, and the T fourth feature maps to obtain T fifth feature maps; perform upsampling and fusion processing on the T fifth feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by splicing the first depth map and the second depth map, or the third depth map is the above-mentioned first depth map.

[0422] In a feasible embodiment, the reference depth map is obtained based on the image collected by the TOF camera, specifically including:

[0423] Project the depth map collected by the TOF camera into the three-dimensional space according to the pose of the current image to obtain a fourth depth map; back-project the fourth depth map onto the reference image according to the pose of the reference image to obtain the reference depth map; the reference image is an image adjacent to the current image in terms of acquisition time; wherein, the resolution of the depth map collected by the TOF camera is lower than the preset resolution, and the frame rate when the TOF camera collects this depth map is lower than the preset frame rate.

[0424] In a feasible embodiment, the above upsampling and fusion processing includes:

[0425] S1: Upsample the feature map P’ j to obtain a feature map P” j , and the resolution of this feature map P” j is the same as the resolution of the (j + 1)-th feature map P j+1 in the processing object. The width of the feature map P j+1 is (j + 1) times the width of the feature map with the smallest resolution in the processing object, where j is an integer greater than 0 and less than T; T is the number of feature maps in the processing object;

[0426] S2: Fuse the feature map P” j with the feature map P j+1 to obtain a third feature map P’ j+1 ,

[0427] S3: Let j = j + 1, and repeat S1 - S3 until j = T - 1;

[0428] Among them, when j = 1, the third feature map P' j is the feature map with the smallest resolution in the processing object. When j = T - 1, the third feature map P' j+1 is the result of upsampling and fusion processing.

[0429] In a feasible embodiment, the estimation unit 1102 is specifically configured to:

[0430] Input the current image and the third depth map into the depth estimation model of the current image for feature extraction, and obtain the target depth map of the current image according to the feature extraction result, where the depth estimation model is implemented based on a convolutional neural network.

[0431] In a feasible embodiment, the terminal device 1100 further includes:

[0432] A sending unit 1104, configured to send a depth estimation model acquisition request to the server, where the depth estimation model acquisition request carries the current image and the location of the terminal device;

[0433] A receiving unit 1105, configured to receive a response message corresponding to the depth estimation model acquisition request sent by the server, where the response message carries the depth estimation model of the current image, and the depth estimation model of the current image is obtained by the server according to the current image and the location of the terminal device in the world coordinate system.

[0434] In a feasible embodiment, the terminal device 1100 further includes:

[0435] A training unit 1106, configured to train an initial convolutional neural network model to obtain a depth estimation model;

[0436] Among them, the training unit 1106 is specifically configured to:

[0437] Input multiple image samples and their corresponding multiple depth map samples into the initial convolutional neural network for processing to obtain multiple predicted depth maps; calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; adjust the parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of the current image; where the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0438] In a feasible embodiment, the superimposed display unit 1103 is specifically configured to:

[0439] Edge optimization is performed on the target depth map of the current image to obtain an optimized depth map; the virtual object image and the current image are superimposed and displayed according to the optimized depth map.

[0440] In a feasible embodiment, the superimposing and displaying unit 1103 is specifically configured to:

[0441] Segment the optimized depth map to obtain a foreground depth map and a background depth map of the current image, where the background depth map is the depth map containing the background area in the optimized depth map, and the foreground depth map is the depth map containing the foreground area in the optimized depth map. The optimized depth map is obtained by performing edge optimization on the target depth map of the current image; fuse the L background depth maps according to the L poses corresponding to the L background depth maps respectively to obtain a fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; perform back-projection on the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map; splice the fused background depth map and the foreground depth map of the current image to obtain an updated depth map; superimpose and display the virtual object image and the current image according to the updated depth map.

[0442] It should be noted that the above units (acquisition unit 1101, estimation unit 1102, superimposing and displaying unit 1103, sending unit 1104, receiving unit 1105, and training unit 1106) are used to execute the relevant steps of the above method. For example, the acquisition unit 1101, the estimation unit 1102, the sending unit 1104, and the receiving unit 1105 are used to execute the relevant content of steps S301 and S302, and the superimposing and displaying unit 1103 is used to execute the relevant content of step S303.

[0443] In this embodiment, the terminal device 1100 is presented in the form of units. Here, the "unit" may refer to an application-specific integrated circuit (ASIC), a processor and a memory that execute one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the above acquisition unit 1101, estimation unit 1102, superimposing and displaying unit 1103, and training unit 1106 can be implemented by Figure 13 the processor 1301 of the terminal device shown.

[0444] See Figure 12 , Figure 12 which is the structural schematic diagram of the server provided by the embodiment of the present application. As Figure 12 shown, the server 1200 includes:

[0445] A receiving unit 1201, configured to receive a depth estimation model acquisition request sent by a terminal device, where the depth estimation model acquisition request carries a current image collected by the terminal device and the location of the terminal device;

[0446] An obtaining unit 1202, configured to obtain a depth estimation model of the current image from a plurality of depth estimation models stored in the server according to the location of the current image;

[0447] A sending unit 1203, configured to send a response message in response to the depth estimation model acquisition request to the terminal device, where the response message carries the depth estimation model of the current image.

[0448] In a feasible embodiment, the obtaining unit 1202 is specifically configured to:

[0449] Obtain multiple first images according to the location of the terminal device, where the multiple first images are images within a preset range centered on the location of the terminal device in a base map, obtain a target image from the multiple first images, where the target image is the image with the highest similarity to the current image among the multiple first images; determine the depth estimation model corresponding to the target image as the depth estimation model of the current image.

[0450] In a feasible embodiment, the server 1200 further includes:

[0451] A training unit 1204, configured to respectively train depth estimation modules for each of the multiple first images among the multiple first images,

[0452] Wherein, for each of the multiple first images among the multiple first images, the following steps are performed for training to obtain the depth estimation model of each first image:

[0453] Input multiple image samples and their corresponding multiple depth map samples into an initial convolutional neural network for processing to obtain multiple predicted depth maps; calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; adjust the parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of each first image;

[0454] Wherein, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

[0455] In a feasible embodiment, the obtaining unit 1202 is further configured to obtain an initial depth map of the current image according to the current image and a pre-stored image; obtain a fifth depth map according to the current image and 3D points corresponding to a local map;

[0456] The server 1200 further includes:

[0457] An optimization unit 1205, configured to optimize the initial depth map and the fifth depth map according to the pose of the current image, so as to obtain a second depth map.

[0458] In a feasible embodiment, in terms of obtaining the initial depth map of the current image according to the current image and the pre-stored image, the obtaining unit 1202 is specifically configured to:

[0459] Match the first 2D feature points of the current image with the 2D feature points of the pre-stored image to obtain the sixth 2D feature points of the current image; remove the noise points in the sixth 2D feature points of the current image to obtain the seventh 2D feature points of the current image; perform triangulation calculation on each 2D feature point in the seventh 2D feature points of the current image to obtain the initial 3D points in space of the seventh 2D feature points of the current image; the initial depth map of the current image includes the initial 3D points in space of the seventh 2D feature points of the current image.

[0460] In a feasible embodiment, in terms of obtaining the fifth depth map according to the current image and the 3D points corresponding to the local map, the obtaining unit 1202 is specifically configured to:

[0461] Obtain M maps from multiple base maps, where the similarity between each of the M maps and the current image is greater than a first preset threshold; M is an integer greater than 0; match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; obtain the 3D points corresponding to each fourth 2D feature point in the multiple feature point matching pairs according to the correspondence between each fifth 2D feature point in the M maps and the 3D points; obtain the fifth depth map according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, where the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs.

[0462] It should be noted that the above units (transmission unit 1301, acquisition unit 1302, reception unit 1303, training unit 1304, and optimization unit 1305) are used to execute the relevant steps of the above method. For example, the transmission unit 1301 is used to execute the relevant content of step S501, the acquisition unit 1302, training unit 1304, and optimization unit 1305 are used to execute the relevant content of step S502, and the reception unit 1303 is used to execute the relevant content of step S503.

[0463] In this embodiment, the server 1300 is presented in the form of units. Here, the "unit" may refer to an application-specific integrated circuit (ASIC), a processor and memory that execute one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the above acquisition unit 1302, training unit 1304, and optimization unit 1305 can be implemented by Figure 14 the processor 1401 of the server shown.

[0464] As Figure 13 shown, the terminal device 1300 can Figure 13 be implemented in the structure shown in. The terminal device 1300 includes at least one processor 1301, at least one memory 1302, at least one communication interface 1303, and at least one display 1304. The processor 1301, the memory 1302, the display 1304, and the communication interface 1303 are connected through the communication bus and communicate with each other.

[0465] The processor 1301 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0466] The communication interface 1303 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0467] The memory 1302 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or can also be an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

[0468] Among them, the memory 1302 is used to store the application program code for executing the above solution, and is controlled by the processor 1301 for execution. The processor 1301 is used to execute the application program code stored in the memory 1302.

[0469] The code stored in the memory 1302 can execute a method for processing an image provided above. For example: obtaining a current image and a virtual object image, and obtaining a first depth map and a second depth map of the current image according to the current image, where the second depth map is obtained from a server; performing feature extraction according to the current image, the first depth map and the second depth map of the current image, and obtaining a target depth map of the current image according to the result of the feature extraction; superimposing and displaying the virtual object image and the current image according to the target depth map of the current image.

[0470] Such as Figure 14 shown, the server 1400 can Figure 14 be implemented in the structure shown in, and the server 1400 includes at least one processor 1401, at least one memory 1402, and at least one communication interface 1403. The processor 1401, the memory 1402, the display 1404, and the communication interface 1403 are connected through the communication bus and complete communication with each other.

[0471] The processor 1401 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above solution program.

[0472] A communication interface 1403 for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0473] The memory 1402 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

[0474] Among them, the memory 1402 is used to store the application program code for executing the above solution, and is controlled by the processor 1401 to execute. The processor 1401 is used to execute the application program code stored in the memory 1402.

[0475] The code stored in the memory 1402 can execute a method for processing an image provided above. For example: receiving a depth estimation model acquisition request sent by a terminal device, where the depth estimation model acquisition request carries the current image collected by the terminal device and the location of the terminal device; obtaining the depth estimation model of the current image from multiple depth estimation models stored in the server according to the location of the current image; sending a response message in response to the depth estimation model acquisition request to the terminal device, and the response message carries the depth estimation model of the current image.

[0476] The embodiments of the present application also provide a computer storage medium. Among them, the computer storage medium can store a program, and when the program is executed, it includes some or all of the steps of any one of the image processing methods described in the above method embodiments.

[0477] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0478] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0479] In the several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0480] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0481] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0482] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs.

[0483] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory, and the memory can include: flash drives, read-only memories (abbreviation: ROM, English: Read-Only Memory), random access memories (abbreviation: RAM, English: Random Access Memory), magnetic disks, or optical discs, etc.

[0484] As mentioned above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of this application.

Claims

1. A method for processing an image, characterized in that, The method includes: Obtain a current image, and obtain a first depth map and a second depth map based on the current image, where the second depth map is obtained by updating a local map sent by a server, and the local map is a part of a base map corresponding to the pose information of the current image; Extract features based on the current image, the first depth map, and the second depth map, and obtain a target depth map of the current image according to the result of feature extraction; Obtain a virtual object image; Overlay and display the virtual object image and the current image according to the target depth map of the current image.

2. The method according to claim 1, characterized in that The obtaining the first depth map according to the current image includes: Extract features from the current image to obtain first 2D feature points of the current image; Match the first 2D feature points of the current image with pre-stored 2D feature points to obtain second 2D feature points in the current image; According to the second 2D feature points in the current image and the corresponding relationship between the pre-stored 2D feature points and 3D points, obtain 3D points corresponding to the second 2D feature points in the current image; [[ID=,9]]Wherein, the first depth map includes the 3D points corresponding to the second 2D feature points in the current image.

3. The method according to claim 1, wherein The obtaining the first depth map of the current image according to the current image includes: Extract features from the current image to obtain first 2D feature points of the current image; Match the first 2D feature points of the current image with 2D feature points of a local map obtained from the server to obtain third 2D feature points in the local map; According to the third 2D feature points in the local map and the corresponding relationship between the 2D feature points and 3D points in the local map, obtain 3D points corresponding to the third 2D feature points in the local map; Wherein, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

4. The method according to claim 3, characterized in that, The matching the first 2D feature points of the current image with 2D feature points of a local map to obtain third 2D feature points in the local map includes: Obtain a target map from the local map according to a first pose, where the position of the target map in the local map is associated with the position indicated by the angle information in the first pose; the first pose is the pose obtained by converting the pose obtained by the terminal device according to the current image into a pose in the world coordinate system; Match the 2D feature points in the target map with the first 2D feature points of the current image to obtain third 2D feature points of the target map, and the third 2D feature points of the local map include the third 2D feature points of the target map.

5. The method according to any one of claims 1-4, characterized in that, The extracting features based on the current image, the first depth map, and the second depth map, and obtaining a target depth map of the current image according to the result of feature extraction includes: Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; the resolutions of each of the T first feature maps are different from each other, and the resolutions of each of the T second feature maps are different from each other; T is an integer greater than 1; Overlay the first feature map and the second feature map with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps; Perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by stitching the first depth map and the second depth map.

6. The method according to any one of claims 1-4, characterized in that The feature extraction based on the current image, the first depth map and the second depth map, and obtaining the target depth map of the current image according to the result of the feature extraction includes: Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; perform multi-scale feature extraction on the reference depth map to obtain T fourth feature maps, the resolutions of each of the T first feature maps are different from each other, the resolutions of each of the T second feature maps are different from each other, and the resolutions of each of the T fourth feature maps are different from each other; the reference depth map is obtained according to the depth map collected by a time-of-flight (TOF) camera, and T is an integer greater than 1; Overlay the first feature map, the second feature map and the fourth feature map with the same resolution among the T first feature maps, the T second feature maps and the T fourth feature maps to obtain T fifth feature maps; Perform upsampling and fusion processing on the T fifth feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by stitching the first depth map and the second depth map.

7. The method according to claim 6, wherein The reference depth map is obtained according to the image collected by a time-of-flight (TOF) camera, specifically including: Project the depth map collected by the TOF camera into three-dimensional space according to the pose of the current image to obtain a fourth depth map; Back-project the fourth depth map onto the reference image according to the pose of the reference image to obtain the reference depth map; the reference image is an image adjacent to the current image in terms of acquisition time; Wherein, the resolution of the depth map collected by the TOF camera is lower than a preset resolution, and the frame rate when the TOF camera collects this depth map is lower than a preset frame rate.

8. The method according to any one of claims 5 to 7, characterized in that, The upsampling and fusion processing includes: Upsample the feature map P’ j to obtain the feature map P” j . The resolution of this feature map P” j is the same as that of the (j + 1)-th feature map P j+1 in the processing object; the width of the (j + 1)-th feature map is (j + 1) times the width of the feature map with the smallest resolution in the processing object, where j is greater than or equal to 1 and less than or equal to T - 1; Fuse the feature map P” j with the feature map P j+1 to obtain the feature map P’ j+1 , Let j = j + 1, and repeat the above steps until j = T - 1; T is the number of feature maps in the processing object; Among them, when j = 1, the feature map P' j is the feature map with the smallest resolution in the processing object. When j = T - 1, the feature map P' j+1 is the result of the upsampling and fusion processing.

9. The method according to any one of claims 5-8, characterized in that, The feature extraction based on the current image, the first depth map and the second depth map, and obtaining the target depth map of the current image according to the result of the feature extraction includes: Input the current image and the third depth map into the depth estimation model of the current image for feature extraction, and obtain the target depth map of the current image according to the result of the feature extraction; Wherein, the depth estimation model is implemented based on a convolutional neural network.

10. The method according to claim 9, wherein The method further includes: Send a depth estimation model acquisition request to the server, where the depth estimation model acquisition request carries the current image and the location of the terminal device; Receive a response message in response to the depth estimation model acquisition request sent by the server, where the response message carries the depth estimation model of the current image, and the depth estimation model of the current image is obtained by the server according to the current image and the location of the terminal device in the world coordinate system.

11. The method according to claim 9, characterized in that The method further includes: Train an initial convolutional neural network model to obtain the depth estimation model; Wherein, the training of the initial convolutional neural network to obtain the depth estimation model includes: Input multiple image samples and their corresponding multiple depth map samples into the initial convolutional neural network for processing to obtain multiple predicted depth maps; Calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; Adjust the parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of the current image; Wherein, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

12. The method according to any one of claims 1 to 11, characterized in that, The superimposing and displaying the virtual object and the current image according to the target depth map of the current image includes: Segment the optimized depth map to obtain the foreground depth map and the background depth map of the current image, where the background depth map is the depth map containing the background area in the optimized depth map, and the foreground depth map is the depth map containing the foreground area in the optimized depth map; the optimized depth map is obtained by performing edge optimization on the target depth map of the current image; Fuse the L background depth maps according to the L poses respectively corresponding to the L background depth maps to obtain a fused three-dimensional scene; the L background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the L poses include the poses of the pre-stored images and the current image; L is an integer greater than 1; Back-project the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map; Stitch the fused background depth map and the foreground depth map of the current image to obtain an updated depth map; Superimpose and display the virtual object and the current image according to the updated depth map.

13. A method for processing an image, characterized in that, Includes: Receive a depth estimation model request message sent by a terminal device, where the request message carries the current image collected by the terminal device and the location of the terminal device; Obtain the depth estimation model of the current image from multiple depth estimation models stored in the server according to the current image and the location of the terminal device; Send a response message in response to the depth estimation model request message to the terminal device, where the response message carries the depth estimation model of the current image.

14. The method according to claim 13, wherein The obtaining the depth estimation model of the current image from multiple depth estimation models stored in the server according to the current image and the location of the terminal device includes: Obtain multiple first images according to the location of the terminal device, where the multiple first images are images within a preset range centered on the location of the terminal device in the base map; Obtain a target image from the multiple first images, where the target image is the image with the highest similarity between the multiple first images and the current image; Determine the depth estimation model corresponding to the target image as the depth estimation model of the current image.

15. The method according to claim 14, wherein The method further includes: For the multiple first images, respectively train depth estimation modules for each of the multiple first images, where, for each of the multiple first images, train according to the following steps to obtain the depth estimation model of each first image: Input multiple image samples and their corresponding multiple depth map samples into an initial convolutional neural network for processing to obtain multiple predicted depth maps; Calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; Adjust the parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of each frame of the first image; where, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

16. The method according to any one of claims 13 to 15, characterized in that, The method further includes: Obtain an initial depth map of the current image according to the current image and the pre-stored image; Obtain a fifth depth map according to the current image and the 3D points corresponding to the local map; Optimize the initial depth map and the fifth depth map according to the pose of the current image to obtain a second depth map.

17. The method according to claim 16, wherein The obtaining the third depth map according to the current image and the 3D points corresponding to the local map includes Obtain M maps from multiple frames of the base map, where the similarity between each of the M maps and the current image is greater than a first preset threshold; M is an integer greater than 0; Match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain multiple feature point matching pairs; each feature point matching pair in the multiple feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; Obtain the 3D points corresponding to each fourth 2D feature point in the multiple feature point matching pairs according to the correspondence between each fifth 2D feature point and the 3D points in the M maps. Obtain the fifth depth map according to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs, where the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the multiple feature point matching pairs.

18. A terminal device, characterized in that, Comprising: An acquisition unit, configured to acquire a current image and a virtual object image, and obtain a first depth map and a second depth map of the current image according to the current image, where the second depth map is obtained by updating a local map sent by a server, and the local map is a part of a base map corresponding to the pose information of the current image; An estimation unit, configured to perform feature extraction according to the current image, the first depth map, and the second depth map, and obtain a target depth map of the current image according to the feature extraction result; The acquisition unit is further configured to acquire a virtual object image; The terminal device further includes An overlay display unit, configured to overlay and display the virtual object image and the current image according to the target depth map of the current image.

19. The terminal device according to claim 18, characterized in that, In the aspect of obtaining the first depth map of the current image according to the current image, the acquisition unit specifically is configured to: Perform feature extraction on the current image to obtain first 2D feature points of the current image; Match the first 2D feature points of the current image with pre-stored 2D feature points to obtain second 2D feature points in the current image, Obtain the 3D points corresponding to the second 2D feature points in the current image according to the correspondence between the first 2D feature points in the current image and the pre-stored 2D feature points and the 3D points; Wherein, the first depth map of the current image includes the 3D points corresponding to the second 2D feature points in the current image.

20. The terminal device according to claim 18, characterized in that, In the aspect of obtaining the first depth map of the current image according to the current image, the acquisition unit specifically is configured to: Perform feature extraction on the current image to obtain first 2D feature points of the current image; Match the first 2D feature points of the current image with the 2D feature points of the local map obtained from the server to obtain third 2D feature points in the local map; Obtain the 3D points corresponding to the third 2D feature points in the local map according to the correspondence between the third 2D feature points in the local map and the 2D feature points in the local map and the 3D points; Wherein, the first depth map of the current image includes the 3D points corresponding to the third 2D feature points in the local map.

21. The terminal device according to claim 20, wherein In the aspect of matching the first 2D feature points of the current image with the 2D feature points of the local map to obtain third 2D feature points in the local map, the acquisition unit specifically is configured to: Obtain a target map from the local map according to the first pose, where the position of the target map in the local map is associated with the position indicated by the angle information in the first pose; the first pose is the pose obtained by the terminal device according to the current image converted into the pose in the world coordinate system. Match the 2D feature points in the target map with the first 2D feature points of the current image to obtain the third 2D feature points of the target map, where the third 2D feature points of the local map include the third 2D feature points of the target map.

22. The terminal device according to any one of claims 18-21, characterized in that, The estimation unit is specifically used for: Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; the resolutions of each of the T first feature maps are different, and the resolutions of each of the T second feature maps are different; T is an integer greater than 1. Overlay the first feature maps and the second feature maps with the same resolution among the T first feature maps and the T second feature maps to obtain T third feature maps. Perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by stitching the first depth map and the second depth map.

23. The terminal device according to any one of claims 18-21, characterized in that, The estimation unit is specifically used for: Perform multi-scale feature extraction on the current image to obtain T first feature maps, and perform multi-scale feature extraction on the third depth map to obtain T second feature maps; perform feature extraction on the reference depth map to obtain T fourth feature maps, where the resolutions of each of the T first feature maps are different, the resolutions of each of the T second feature maps are different, and the resolutions of each of the T fourth feature maps are different. The reference depth map is obtained according to the depth map collected by the time-of-flight (TOF) camera, and T is an integer greater than 1. Overlay the first feature maps, the second feature maps, and the fourth feature maps with the same resolution among the T first feature maps, the T second feature maps, and the T fourth feature maps to obtain T fifth feature maps. Perform upsampling and fusion processing on the T third feature maps to obtain the target depth map of the current image; wherein, the third depth map is obtained by stitching the first depth map and the second depth map.

24. The terminal device according to claim 23, characterized in that, The reference depth map is obtained according to the image collected by the time-of-flight (TOF) camera, and specifically includes: Project the depth map collected by the TOF camera into three-dimensional space according to the pose of the current image to obtain a fourth depth map. Back-project the fourth depth map onto the reference image according to the pose of the reference image to obtain the reference depth map; the reference image is an image adjacent to the current image in terms of the acquisition time. Wherein, the resolution of the depth map collected by the TOF camera is lower than the preset resolution, and the frame rate when the TOF camera collects this depth map is lower than the preset frame rate.

25. The terminal device according to any one of claims 22-24, characterized in that, The upsampling and fusion processing includes: Upsample the feature map P’ j to obtain the feature map P” j . The resolution of this feature map P” j is the same as that of the (j + 1)-th feature map P j+1 in the processing object; the width of the (j + 1)-th feature map is (j + 1) times the width of the feature map with the smallest resolution in the processing object, where j is greater than or equal to 1 and less than or equal to T - 1; Fuse the feature map P” j with the feature map P j+1 to obtain the feature map P’ j+1 , Let \(j = j + 1\), and repeat the above steps until \(j=T - 1\); where \(T\) is the number of feature maps in the object to be processed. Among them, when j = 1, the feature map P' j is the feature map with the smallest resolution in the processing object. When j = T - 1, the feature map P' j+1 is the result of the upsampling and fusion processing.

26. The terminal device according to any one of claims 22-25, characterized in that, The estimation unit is specifically configured to: Input the current image and the third depth map into the depth estimation model of the current image for feature extraction, and obtain the target depth map of the current image according to the result of the feature extraction. Wherein, the depth estimation model is implemented based on a convolutional neural network.

27. The terminal device according to claim 26, wherein The terminal device further includes: A sending unit, configured to send a depth estimation model acquisition request to the server, where the depth estimation model acquisition request carries the current image and the location of the terminal device. A receiving unit, configured to receive a response message corresponding to the depth estimation model acquisition request sent by the server, where the response message carries the depth estimation model of the current image, and the depth estimation model of the current image is obtained by the server according to the current image and the location of the terminal device in the world coordinate system.

28. The terminal device according to claim 26, characterized in that, The terminal device further includes: A training unit, configured to train an initial convolutional neural network model to obtain the depth estimation model. Wherein, the training module is specifically configured to: Input multiple image samples and their corresponding multiple depth map samples into the initial convolutional neural network for processing to obtain multiple predicted depth maps. Calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function. Adjust the parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of the current image. Wherein, the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

29. The terminal device according to any one of claims 18-27, characterized in that, The superimposed display unit is specifically configured to: Segment the optimized depth map to obtain the foreground depth map and the background depth map of the current image, where the background depth map is the depth map containing the background area in the optimized depth map of the current image, and the foreground depth map is the depth map containing the foreground area in the optimized depth map of the current image. The optimized depth map is obtained by performing edge optimization on the target depth map of the current image. Fuse the \(L\) background depth maps according to the \(L\) poses respectively corresponding to the \(L\) depth maps to obtain a fused three-dimensional scene; the \(L\) background depth maps include the background depth maps of the pre-stored images and the background depth map of the current image, and the \(L\) poses include the poses of the pre-stored images and the current image; \(L\) is an integer greater than 1. Back-project the fused three-dimensional scene according to the pose of the current image to obtain a fused background depth map. Stitch the fused background depth map and the foreground depth map of the current image to obtain an updated depth map. Superimpose and display the virtual object and the current image according to the updated depth map.

30. A server, characterized in that, Including: A receiving unit, configured to receive a depth estimation model request message sent by a terminal device, where the request message carries a current image collected by the terminal device and the location of the terminal device; An obtaining unit, configured to obtain a depth estimation model of the current image from multiple depth estimation models stored in a server according to the current image and the location of the terminal device; A sending unit, configured to send a response message in response to the depth estimation model request message to the terminal device, where the response message carries the depth estimation model of the current image.

31. The server according to claim 30, wherein The obtaining unit is specifically configured to: Obtain multiple first images according to the location of the terminal device, where the multiple first images are images within a preset range centered on the location of the terminal device in a base map; Obtain a target image from the multiple first images, where the target image is the image with the highest similarity between the multiple first images and the current image; Determine the depth estimation model corresponding to the target image as the depth estimation model of the current image.

32. The server according to claim 31, wherein The server further includes: A training unit, configured to respectively train depth estimation modules for each of the multiple first images among the multiple first images, where, for each of the multiple first images among the multiple first images, the following steps are performed for training to obtain the depth estimation model of each first image: Input multiple image samples and their corresponding multiple depth map samples into an initial convolutional neural network for processing to obtain multiple predicted depth maps; Calculate a loss value according to the multiple predicted depth maps, the true depth maps corresponding to the multiple image samples, and a loss function; Adjust parameters in the initial convolutional neural network according to the loss value to obtain the depth estimation model of each frame of the first image; where the loss function is determined based on the error between the predicted depth map and the true depth map, the error between the gradient of the predicted depth map and the gradient of the true depth map, and the error between the normal vector of the predicted depth map and the normal vector of the true depth map.

33. The server according to any one of claims 30-32, wherein the obtaining unit is further configured to obtain an initial depth map of the current image according to the current image and a pre-stored image, and obtain a fifth depth map according to the current image and 3D corresponding to a local map; The server further includes: An optimization unit, configured to optimize the initial depth map and the fifth depth map according to the pose of the current image to obtain a second depth map.

34. The server according to claim 33, wherein In terms of obtaining a third depth map according to the current image and 3D points corresponding to the local map, the obtaining unit is specifically configured to: Obtain M maps from multiple base maps, where the similarity between each of the M maps and the current image is greater than a first preset threshold; M is an integer greater than 0; Match the 2D feature points of the M maps with the first 2D feature points of the current image to obtain a plurality of feature point matching pairs; each feature point matching pair in the plurality of feature point matching pairs includes a fourth 2D feature point and a fifth 2D feature point, the fourth 2D feature point and the fifth 2D feature point are mutually matching feature points, the fourth 2D feature point is the first 2D feature point of the current image, and the fifth 2D feature point is the 2D feature point in the M maps; According to the correspondence between each fifth 2D feature point in the M maps and the 3D points, obtain the 3D points corresponding to each fourth 2D feature point in the plurality of feature point matching pairs; According to the 3D points corresponding to the local map and the 3D points corresponding to the fourth 2D feature points in the plurality of feature point matching pairs, obtain the fifth depth map, and the fifth depth map includes the 3D points corresponding to the local map that match the 3D points corresponding to the fourth 2D feature points in the plurality of feature point matching pairs.

35. A terminal device, comprising a memory, and one or more processors; wherein, The memory stores one or more programs; characterized in that when the one or more processors execute the one or more programs, the terminal device implements the method according to any one of claims 1 to 12.

36. A server, comprising a memory, one or more processors; wherein, The memory stores one or more programs; characterized in that when the one or more processors execute the one or more programs, the electronic device implements the method according to any one of claims 13 to 17.

37. A computer storage medium, characterized in that, Includes computer instructions, when the computer instructions run on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 17.

38. A computer program product, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 17.

Images