Image processing method, device, system and computer readable storage medium

Abstract

The application discloses an image processing method, device, system and computer readable storage medium, and belongs to the technical field of images. The method comprises the following steps: acquiring a first video frame in a video frame sequence as a target image; sending the target image or image features of the target image to a server; receiving a sparse depth map and pose information corresponding to the target image sent by the server; and determining a dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, a second video frame located before the target image in the video frame sequence, and a sparse depth map and pose information corresponding to the second video frame. In this way, the terminal can quickly obtain the dense depth map corresponding to the target image with the assistance of the server, without using a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements, reducing the operation amount of the terminal, improving the determination efficiency of the dense depth map, and being highly applicable.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, in particular to an image processing method, device, system and computer readable storage medium. BACKGROUND

[0002] A depth image is a three-dimensional representation of an object, which can directly reflect the geometric shape of the visible surface of the scene. According to the sparsity of the depth information, the depth image includes sparse depth image and dense depth image. In application, the dense depth image is usually used.

[0003] In one case, the dense depth image can be obtained by a depth camera with good indicators such as high resolution, that is, the target to be photographed is photographed by the depth camera, so as to obtain a dense depth image with high resolution. Or, the dense depth image is obtained by a binocular camera, that is, the target to be photographed is photographed by the binocular camera, and then the disparity of the binocular camera is depth estimated to obtain the dense depth image.

[0004] However, due to the fact that the cost, resolution and power consumption of the depth camera or binocular camera cannot meet the commercial requirements at the same time, the applicability of obtaining the dense depth image by the depth camera or binocular camera is poor. Therefore, how to obtain the dense depth image has become a research hotspot. SUMMARY

[0005] The present application provides an image processing method, device, system and computer readable storage medium, which can solve the problem of poor applicability of obtaining a dense depth image by a depth camera or binocular camera in related technologies. The technical solution is as follows:

[0006] In a first aspect, an image processing method is provided, which is applied in a terminal, and the method comprises:

[0007] obtaining a target image to be estimated, sending the target image or image features of the target image to a server, the target image being a first video frame in a video frame sequence, and the video frame sequence being collected by a moving RGB camera; then receiving sparse depth image and pose information corresponding to the target image sent by the server, and determining a dense depth image corresponding to the target image according to the target image, the sparse depth image and the pose information corresponding to the target image, a second video frame located before the target image in the video frame sequence, and sparse depth image and pose information corresponding to the second video frame.

[0008] Among them, the target image can be any video frame in the video frame sequence, such as any key frame in the video frame sequence.

[0009] As an example, the terminal is integrated with an RGB camera, and the terminal can perform mobile shooting on a target environment through the RGB camera, and take a first video frame in a video frame sequence obtained by mobile shooting as a target image.

[0010] For example, the terminal can obtain a video frame sequence by recording a target environment through the RGB camera after receiving a video recording instruction. The video recording instruction can be triggered by a user, and the user can trigger the instruction through a specified operation. For example, the specified operation can include, but is not limited to, a click operation, a sliding operation, a language operation, or a gesture operation.

[0011] As an example, the terminal can directly send the target image to the server after obtaining the target image.

[0012] As another example, the terminal can first extract feature points of the target image to obtain image features of the target image after obtaining the target image. The image features of the target image are a set of feature points of the target image. Then, the target features of the target image are sent to the server.

[0013] The image features of the target image include a set of feature points in the target image, and generally include sparse feature points of the target image. The feature points are generally feature points with key features in the target image.

[0014] In an embodiment, the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame can be input into a target model for processing, and a dense depth map corresponding to the target image is output.

[0015] The target model can be pre-trained. The target model is used to determine a dense depth map corresponding to any video frame according to the any video frame, the sparse depth map and the pose information corresponding to the any video frame, other video frames located before the any video frame, and the sparse depth map and the pose information corresponding to the other video frames.

[0016] In an embodiment, a sample data set can be obtained, and a network model to be trained is trained according to the sample data set to obtain a target model. The sample data set includes a plurality of groups of sample data, and each group of sample data in the plurality of groups of sample data includes a video frame sample sequence, a sparse depth map and a pose information corresponding to each video frame sample in the video frame sample sequence, and a real dense depth map corresponding to each video frame sample.

[0017] The sparse depth map and the pose information corresponding to each video frame sample can be obtained in the manner described above for obtaining the sparse depth map and the pose information corresponding to the target image, or can be obtained in other manners, which is not limited in the embodiments of the present application.

[0018] As an example, each video frame sample can be a key frame sample in the sequence of video frame samples.

[0019] As an example, the real dense depth map corresponding to each video frame sample can be acquired by a depth camera with good indicators, i.e., the depth camera can obtain dense depth information, for example, the depth camera can use a VGA depth camera.

[0020] As an example, the terminal stores a pre-trained target model. After receiving the sparse depth map and the pose information corresponding to the target image sent by the server, the terminal can call the target model, and then input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into the target model for processing. After that, the target image can output the dense depth map corresponding to the target image.

[0021] The dense depth map of the target image is determined by the target model, which can further improve the determination efficiency of the dense depth map.

[0022] In a possible implementation, an end-to-end network model can be constructed as the network model to be trained, and a loss function can be constructed, which is used to describe the degree of difference between the true value and the estimated value. In the embodiment, the true value is the real dense depth map, and the estimated value is the dense depth map estimated by the network model. In the training process, the model parameters of the network model can be updated according to the loss function and the sample data set. When the training satisfies the end condition, the network model with updated model parameters is used as the trained target model.

[0023] As an example, the loss function can be constructed according to the geometric error between the true value and the estimated value, the similarity error of the video frame sample, and the depth image smoothness. The similarity error of the video frame sample is used to describe the back projection error of the pose information. The depth image smoothness can be understood as a constraint condition, which can make the final obtained dense depth map smooth and there is no jump between each pixel point.

[0024] As another example, the target model can also be stored in the cloud. In this case, the terminal can input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into the target model uploaded to the cloud, and instruct the cloud to determine the dense depth map according to the uploaded data. After that, the cloud sends the determined dense depth map to the terminal. In this way, the operation amount of the terminal can be further reduced.

[0025] In an embodiment, the number of the second video frames is multiple, and the terminal can determine the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, each of the multiple second video frames, and the sparse depth map and the pose information corresponding to each of the multiple second video frames.

[0026] The sparse depth map and the pose information corresponding to each of the multiple second video frames can be obtained in the same manner as the sparse depth map and the pose information corresponding to the target image.

[0027] As an example, the terminal can input the target image, the sparse depth map and the pose information corresponding to the target image, each of the multiple second video frames, and the sparse depth map and the pose information corresponding to each of the multiple second video frames into the target model for processing, so as to determine the dense depth map corresponding to the target image through the target model.

[0028] That is, in the case where the number of the second video frames is multiple, the terminal can still determine the dense depth map corresponding to the target image through the target model.

[0029] In an embodiment, before sending the target image or the image feature of the target image to the server, the terminal can first obtain environment data and send the environment data to the server. The environment data is used to determine the RGB images of different perspectives in the target environment.

[0030] The environment data is the environment data of the target environment, and the target environment refers to the environment in which the target to be processed is located. The environment data is used to indicate the environmental features of the target environment, and the environmental images of the target environment can be determined according to the environment data. The environmental images of the target environment include the RGB images of different perspectives in the target environment, i.e., the RGB images of different perspectives of the target.

[0031] For example, the environment data can include one or more of a video frame sequence obtained by moving shooting of the target environment, a panoramic image of the target environment, and laser scanning information of the target environment. For example, the video frame sequence can be collected by a moving RGB camera. The panoramic image of the target environment can be obtained by panoramic shooting of the target environment through a fisheye camera or multiple RGB cameras. The laser scanning information of the target environment can be obtained by laser scanning of the target environment through a high-precision laser scanning system.

[0032] In the embodiments of the present application, the terminal can send a target image to be estimated or image features of the target image to the server, return the sparse depth map and the pose information corresponding to the target image to the terminal according to the first visual map, and then determine the dense depth map corresponding to the target image according to the sparse depth map and the pose information corresponding to the target image returned by the server. In this way, the terminal can quickly obtain the dense depth map corresponding to the target image with the assistance of the server. In this process, the terminal does not need to use a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements. Moreover, the terminal can directly obtain the sparse depth map and the pose information corresponding to the target image from the server without calculating them by itself, thereby reducing the calculation amount of the terminal, improving the determination efficiency of the dense depth map, and having high applicability.

[0033] In addition, the terminal can pre-collect environment data and send the environment data to the server, so that the server constructs the first visual map according to the environment data. The first data map includes a plurality of RGB images of different views in the environment, and the dense depth map and the pose information corresponding to each RGB image.

[0034] In a second aspect, an image processing apparatus is provided, which has functions to implement the behaviors of the image processing method in the first aspect. The image processing apparatus includes at least one module for implementing the image processing method provided in the first aspect.

[0035] In a third aspect, an image processing apparatus is provided, which includes a processor and a memory in its structure. The memory is configured to store a program supporting the image processing apparatus to execute the image processing method provided in the first aspect, and store data involved in the image processing method in the first aspect. The processor is configured to execute the program stored in the memory. The image processing apparatus can further include a communication bus for establishing a connection between the processor and the memory.

[0036] In a fourth aspect, a computer readable storage medium is provided, which stores instructions. When the instructions are run on a computer, the computer is caused to execute the image processing method in the first aspect.

[0037] In a fifth aspect, a computer program product is provided, which includes instructions. When the instructions are run on a computer, the computer is caused to execute the image processing method in the first aspect.

[0038] In a sixth aspect, an image processing method is provided, which is applied to a server. The method includes:

[0039] The target image or the image feature of the target image sent by the terminal is received. Then, according to the image feature of the target image, an RGB image corresponding to the image feature of the target image and having a difference less than a difference threshold is obtained from a stored first visual map, to obtain a plurality of candidate images, the first visual map including a plurality of RGB images and a dense depth map and pose information corresponding to each RGB image. Subsequently, according to the image feature of the target image and the dense depth map and pose information corresponding to the plurality of candidate images, the sparse depth map and the pose information corresponding to the target image are determined, and the sparse depth map and the pose information corresponding to the target image are sent to the terminal.

[0040] After receiving the target image or the image feature of the target image sent by the terminal, the server can determine the image feature of each RGB image included in the plurality of RGB images of the first visual map, and then compare the image feature of the target image with the image features of the plurality of RGB images in sequence, and take the RGB image corresponding to the image feature of the target image and having a difference less than a difference threshold as a candidate image.

[0041] It should be noted that if the terminal sends the image feature of the target image to the server, the terminal can directly compare the image feature of the target image with the image features of the plurality of RGB images in sequence to obtain the plurality of candidate images from the first visual map. If the terminal sends the target image to the server, the server needs to first extract the feature points of the target image to obtain the image feature of the target image, and then compare the image feature of the target image with the image features of the plurality of RGB images in sequence to obtain the plurality of candidate images from the first visual map.

[0042] Since the image feature of the target image is a set of feature points extracted from the target image, it usually only includes sparse feature points in the target image, and therefore the server can only obtain the sparse depth map and the pose information corresponding to the target image according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images.

[0043] In one embodiment, the sparse depth map corresponding to the target image can be determined according to the image feature of the target image and the dense depth map corresponding to the plurality of candidate images. The pose information corresponding to the target image can be determined according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images.

[0044] In an embodiment, determining the sparse depth map corresponding to the target image according to the image features of the target image and the dense depth maps corresponding to the plurality of candidate images comprises: determining target image features of a first candidate image of the plurality of candidate images, the target image features of the first candidate image being the image features of the first candidate image that match the image features of the target image, the first candidate image being any one of the plurality of candidate images; determining depth information corresponding to the target image features of the first candidate image from the dense depth map corresponding to the first candidate image; combining the depth information corresponding to the target image features of each candidate image of the plurality of candidate images according to the image features of the target image to obtain the depth information corresponding to the image features of the target image; and determining the sparse depth map corresponding to the target image according to the depth information corresponding to the image features of the target image.

[0045] By way of example, and not limitation, the server can perform pose estimation on the target image according to the image features of the target image and the dense depth maps and pose information corresponding to the plurality of candidate images using the PNP algorithm to obtain the pose information corresponding to the target image.

[0046] For example, performing pose estimation on the target image according to the image features of the target image and the dense depth maps and pose information corresponding to the plurality of candidate images using the PNP algorithm to obtain the pose information corresponding to the target image comprises: determining a plurality of co-visible feature points corresponding to the image features of the target image and the image features of the plurality of candidate images according to the image features of the target image and the pose information corresponding to the plurality of candidate images to obtain the plurality of co-visible feature points; determining image coordinates of the plurality of co-visible feature points; obtaining spatial coordinates of the plurality of co-visible feature points according to the dense depth maps corresponding to the plurality of candidate images; and determining the pose information corresponding to the target image according to the image coordinates and the spatial coordinates of the plurality of co-visible feature points.

[0047] The plurality of co-visible feature points are feature points having a co-visible relationship. For example, pixel coordinates of the plurality of co-visible feature points in corresponding images can be obtained, and then the pixel coordinates of the plurality of co-visible feature points in the corresponding images can be converted into image coordinates. Image coordinates are also referred to as 2D coordinates. The spatial coordinates of the co-visible feature points are the coordinates of the co-visible feature points in a world coordinate system. Spatial coordinates are also referred to as 3D coordinates.

[0048] For example, depth information corresponding to the plurality of co-visible feature points can be obtained according to the dense depth maps corresponding to the plurality of candidate images, the coordinates of the plurality of co-visible feature points in the world coordinate system can be determined according to the depth information corresponding to the plurality of co-visible feature points to obtain the spatial coordinates of the plurality of co-visible points, and the estimated pose information can be determined as the pose information corresponding to the target image according to the image coordinates and the spatial coordinates of the plurality of co-visible feature points using the PNP algorithm for pose estimation.

[0049] In an embodiment, the server can pre-receive the environment data sent by the terminal, and then generate a first visual map according to the environment data, the first visual map comprising a plurality of RGB images and a dense depth map and pose information corresponding to each RGB image, the plurality of RGB images being RGB images of different perspectives in the target environment.

[0050] As an example, the server can first generate a second visual map comprising a plurality of RGB images and a sparse depth map and pose information corresponding to each RGB image according to the environment data. Then, determine a dense depth map corresponding to each RGB image in the plurality of RGB images according to the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images, and construct the first visual map according to the plurality of RGB images and the dense depth map and the pose information corresponding to each RGB image.

[0051] For example, after the server receives the environment data sent by the terminal, the server can first determine the RGB images of different perspectives in the target environment according to the environment data to obtain a plurality of RGB images. Then, determine a sparse depth map and pose information corresponding to each RGB image in the plurality of RGB images. Construct a second visual map according to the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images.

[0052] As an example, the server can determine the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images using the SFM algorithm. The SFM algorithm is an offline algorithm for three-dimensional reconstruction based on various collected unordered pictures.

[0053] For example, determining the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images using the SFM algorithm comprises: extracting feature points of each RGB image in the plurality of RGB images to obtain image features of each RGB image, the image features of each RGB image being a set of feature points of each RGB image; performing feature matching on the image features of each pair of images in the plurality of RGB images to generate a first feature matching relationship, the first feature matching relationship comprising a matching point pair corresponding to any two images in the plurality of RGB images; performing three-dimensional reconstruction according to the first feature matching relationship, determining a sparse depth map corresponding to each RGB image in the plurality of RGB images according to the three-dimensional reconstruction result, and determining pose information corresponding to each RGB image in the plurality of RGB images according to the first feature matching relationship; and constructing a second visual map according to the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images.

[0054] After obtaining the sparse depth map and the pose information corresponding to each of the plurality of RGB images, a dense depth map corresponding to each of the plurality of RGB images can be further determined according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images.

[0055] By way of example and not limitation, the dense depth map corresponding to each of the plurality of RGB images can be determined according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images by using a MVS algorithm.

[0056] The MVS algorithm is a method for generating a dense 3D point cloud from a sparse 3D point cloud. The three-dimensional reconstruction result obtained by the SFM algorithm is a sparse 3D point cloud. The MVS algorithm is then used to process the sparse 3D point cloud to obtain a dense 3D point cloud. The dense depth map corresponding to each of the plurality of RGB images can be determined according to the dense 3D point cloud.

[0057] For example, the dense depth map corresponding to each of the plurality of RGB images can be determined according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images by using a MVS algorithm, which includes: performing image stereo rectification on any two images of the plurality of RGB images according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images to align the any two images to the same plane. Feature points of each of the plurality of RGB images are extracted after rectification to obtain image features of each of the plurality of RGB images. The image features of the two images of the plurality of RGB images are matched to obtain a second feature matching relationship of the plurality of RGB images. The dense depth map corresponding to each of the plurality of RGB images is determined according to the three-dimensional reconstruction result obtained by three-dimensional reconstruction according to the second feature matching relationship.

[0058] In the embodiments of the present application, the server can receive the target image or the image features of the target image sent by the terminal, then acquire, according to the image features of the target image, an RGB image corresponding to the image features of the target image and having a difference less than a difference threshold as a plurality of candidate images from the stored first visual map, determine the sparse depth map and the pose information corresponding to the target image according to the image features of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images, and send the sparse depth map and the pose information corresponding to the target image to the terminal. In this way, the terminal can quickly acquire the dense depth map corresponding to the target image with the assistance of the server, and in this process, the terminal does not need to use a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements, and the terminal can directly acquire the sparse depth map and the pose information corresponding to the target image from the server without calculating them by itself, thereby reducing the operation amount of the terminal, improving the determination efficiency of the dense depth map, and having high applicability.

[0059] In addition, the server can pre-receive the environment data sent by the terminal, construct a first visual map according to the environment data, and the first data map includes a plurality of RGB images of different views in the environment and the dense depth map and the pose information corresponding to each RGB image, so that the server can return the dense depth map and the pose information corresponding to the target image to the terminal according to the constructed first visual map subsequently.

[0060] In a seventh aspect, an image processing apparatus is provided, which has functions to realize the behaviors of the image processing method in the sixth aspect described above. The image processing apparatus includes at least one module for realizing the image processing method provided in the sixth aspect described above.

[0061] In an eighth aspect, an image processing apparatus is provided, which has a structure including a processor and a memory. The memory is configured to store a program supporting the image processing apparatus to execute the image processing method provided in the sixth aspect described above, and store data involved in the image processing method described in the sixth aspect. The processor is configured to execute the program stored in the memory. The image processing apparatus can further include a communication bus configured to establish a connection between the processor and the memory.

[0062] In a ninth aspect, a computer readable storage medium is provided, which has instructions stored therein, and when the instructions are run on a computer, the computer is caused to execute the image processing method described in the sixth aspect.

[0063] In a tenth aspect, a computer program product is provided, which includes instructions, and when the instructions are run on a computer, the computer is caused to execute the image processing method described in the sixth aspect.

[0064] In a eleventh aspect, an image processing system is provided, the system comprising a terminal and a server;

[0065] The terminal is configured to obtain a target image to be estimated, the target image being a first video frame in a sequence of video frames, the sequence of video frames being captured by an RGB camera in motion;

[0066] The terminal is further configured to send the target image or image features of the target image to the server;

[0067] The server is configured to receive the target image or image features of the target image sent by the terminal;

[0068] The server is further configured to obtain a plurality of candidate images from a stored first visual map according to the image features of the target image, the first visual map comprising a plurality of RGB images and corresponding dense depth maps and pose information of each RGB image, the plurality of candidate images being RGB images whose corresponding image features have a difference with the image features of the target image less than a difference threshold;

[0069] The server is further configured to determine a sparse depth map and pose information corresponding to the target image according to the image features of the target image and the dense depth maps and pose information of the plurality of candidate images;

[0070] The server is further configured to send the sparse depth map and pose information corresponding to the target image to the terminal;

[0071] The terminal is further configured to receive the sparse depth map and pose information corresponding to the target image sent by the server;

[0072] The terminal is further configured to determine a dense depth map corresponding to the target image according to the target image, the sparse depth map and pose information corresponding to the target image, a second video frame before the target image in the sequence of video frames, and a sparse depth map and pose information corresponding to the second video frame.

[0073] In an embodiment, the server is further configured to: for a first candidate image in the plurality of candidate images, determine depth information corresponding to target image features of the first candidate image from a dense depth map corresponding to the first candidate image, the target image features of the first candidate image being image features of the first candidate image that match the image features of the target image, the first candidate image being any one of the plurality of candidate images; combine the depth information corresponding to the target image features of each candidate image in the plurality of candidate images according to the image features of the target image to obtain depth information corresponding to the image features of the target image; and determine the sparse depth map corresponding to the target image according to the depth information corresponding to the image features of the target image.

[0074] In an embodiment, the server is further configured to: determine, according to the image feature of the target image and the pose information corresponding to the plurality of candidate images, a plurality of co-view feature points corresponding to the image feature of the target image, to obtain the plurality of co-view feature points; obtain spatial coordinates of the plurality of co-view feature points according to the dense depth map corresponding to the plurality of candidate images; and determine the pose information corresponding to the target image according to the image coordinates and the spatial coordinates of the plurality of co-view feature points.

[0075] In an embodiment, the terminal is further configured to: input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into a target model for processing, and output the dense depth map corresponding to the target image; wherein the target model is configured to determine the dense depth map corresponding to any video frame according to the any video frame, the sparse depth map and the pose information corresponding to the any video frame, other video frames located before the any video frame, and the sparse depth map and the pose information corresponding to the other video frames.

[0076] In an embodiment, the number of the second video frames is a plurality; and the terminal is further configured to: determine the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, each of the plurality of second video frames, and the sparse depth map and the pose information corresponding to each of the plurality of second video frames.

[0077] In an embodiment, the terminal is further configured to: obtain environment data, and send the environment data to the server, wherein the environment data is used to determine the RGB images of different perspectives in the target environment.

[0078] The server is further configured to receive the environment data sent by the terminal, and generate a first visual map according to the environment data, wherein the first visual map comprises a plurality of RGB images and the dense depth map and the pose information corresponding to each of the RGB images, and the plurality of RGB images are the RGB images of different perspectives in the target environment.

[0079] In an embodiment, the server is further configured to:

[0080] generate a second visual map according to the environment data, wherein the second visual map comprises a plurality of RGB images and the sparse depth map and the pose information corresponding to each of the RGB images; determine the dense depth map corresponding to each of the plurality of RGB images according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images; and construct the first visual map according to the plurality of RGB images and the dense depth map and the pose information corresponding to each of the RGB images.

[0081] In an embodiment, the server is further configured to: determine, according to the environment data, RGB images of different perspectives in the target environment to obtain a plurality of RGB images; extract feature points of each of the plurality of RGB images to obtain image features of each of the plurality of RGB images, the image features of each of the plurality of RGB images being a set of feature points of each of the plurality of RGB images; perform feature matching on the image features of any two of the plurality of RGB images to generate a first feature matching relationship, the first feature matching relationship including matching point pairs corresponding to any two of the plurality of RGB images; perform three-dimensional reconstruction according to the first feature matching relationship, determine a sparse depth map corresponding to each of the plurality of RGB images according to a result of the three-dimensional reconstruction, and determine pose information corresponding to each of the plurality of RGB images according to the first feature matching relationship; and construct a second visual map according to the sparse depth map corresponding to each of the plurality of RGB images and the pose information corresponding to each of the plurality of RGB images.

[0082] In an embodiment, the server is further configured to: perform image stereo rectification on any two of the plurality of RGB images according to the plurality of RGB images and the sparse depth map and the pose information corresponding to each of the plurality of RGB images, to align any two of the plurality of RGB images to the same plane; extract feature points of each of the plurality of rectified RGB images to obtain image features of each of the plurality of rectified RGB images, the image features of each of the plurality of rectified RGB images being a set of feature points of each of the plurality of rectified RGB images; perform stereo matching on the image features of any two of the plurality of rectified RGB images to obtain a second feature matching relationship, the second feature matching relationship including matching point pairs corresponding to any two of the plurality of rectified RGB images; perform three-dimensional reconstruction according to the second feature matching relationship, and determine a dense depth map corresponding to each of the plurality of rectified RGB images according to a result of the three-dimensional reconstruction.

[0083] In an embodiment, the environment data includes one or more of a sequence of video frames obtained by moving a camera in the target environment, a panoramic map of the target environment, and laser scanning information of the target environment. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 is a schematic diagram of an implementation environment related to an embodiment of the present application;

[0085] Figure 2 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0086] Figure 3 is a software structure block diagram of an electronic device of an embodiment of the present application;

[0087] Figure 4 is a flowchart of an image processing method provided by an embodiment of the present application;

[0088] Figure 5 is a schematic diagram of a user interface of a mobile phone provided by an embodiment of the present application;

[0089] Figure 6 is a schematic diagram of a user holding a mobile phone to take a mobile photo provided by an embodiment of the present application;

[0090] Figure 7 is a schematic diagram of a capture position of adjacent video frames provided by an embodiment of the present application;

[0091] Figure 8 is a schematic diagram of a matching point pair provided by an embodiment of the present application;

[0092] Figure 9 is a flowchart of another image processing method provided by an embodiment of the present application;

[0093] Figure 10 is a flowchart of a model training method provided by an embodiment of the present application;

[0094] Figure 11 is a structural schematic diagram of an image processing apparatus provided by an embodiment of the present application;

[0095] Figure 12 is a structural schematic diagram of another image processing apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0096] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0097] It should be understood that the "multiple" mentioned in the present application refers to two or more than two. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application only represents the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0098] Before the image processing method provided by the embodiments of the present application is described in detail, the names and implementation environment related to the embodiments of the present application are briefly introduced.

[0099] First, the names related to the embodiments of the present application are briefly introduced.

[0100] Depth information: represents the distance between a detection sensor (such as a depth camera) and a target, expressed in pixel values, with a value range of 0 to 255.

[0101] Sparse depth map: refers to a depth image with sparse depth information. For example, a depth image captured by a time of flight (TOF) camera can be referred to as a sparse depth map.

[0102] Dense depth map: refers to a depth image with dense depth information.

[0103] Pose information of an image: includes position information and attitude information of a photographing device (such as a camera) when capturing the image.

[0104] Image features of an image: the image features of an image include a set of feature points in the image. The feature points are generally feature points with key features in the image.

[0105] Feature point: composed of a key point and a descriptor. The key point is used to describe the position of the feature point in the video frame, and the descriptor is used to describe the relationship between the key point and the pixels around the key point. The feature point should satisfy the invariance of illumination, scale and rotation. Currently commonly used feature points include traditional geometric feature points and / or artificial intelligence (AI) feature points. As an example, the traditional geometric feature points can include but are not limited to scale invariant feature transform (SIFT), speed up robust feature (SURF), Harris Conner, oriented fast and rotated brief (ORB). The AI feature points can include but are not limited to Super Point, learned invariant feature transform (LIFT).

[0106] Next, an implementation environment related to the embodiments of the present application is introduced.

[0107] Please refer to Figure 1 , Figure 1 is a schematic diagram of an implementation environment related to the embodiments of the present application. As Figure 1 shown, the implementation environment includes a terminal 10 and a server 20, and the terminal 10 and the server 20 are connected through a wired network or a wireless network.

[0108] The terminal 10 can acquire the RGB image to be estimated, and has certain image processing functions. In addition, the terminal 10 can also acquire environmental data. The environmental data is used to determine the RGB (red green blue) images of different perspectives in the target environment. For example, the environmental data can include one or more of the following: a video frame sequence obtained by moving shooting in the target environment, a panoramic map of the target environment, and high-precision laser information of the target environment.

[0109] As an example, the terminal 10 has an image acquisition function and can acquire RGB images. In addition, the terminal 10 also has an environmental data acquisition function and can acquire environmental data of the target environment. It should be understood that the terminal 10 can also acquire environmental data of the target environment from other devices.

[0110] As an example, the terminal 10 can be an RGB camera or a mobile terminal integrated with an RGB camera. The mobile terminal can be a mobile phone, a tablet computer, a wearable device, or the like.

[0111] The server 20 has an image processing function. The server 20 is used to provide auxiliary functions for the terminal 10 to determine the dense depth map corresponding to the RGB image. The server 20 can be a separate server or a server cluster.

[0112] As an example, the terminal 10 is used to acquire environmental data of the target environment and send the environmental data to the server 20. The server 20 is used to receive the environmental data and generate a first visual map according to the environmental data. The first visual map includes a plurality of RGB images and the dense depth map and pose information corresponding to each RGB image, and the plurality of RGB images are RGB images of different perspectives in the target environment.

[0113] In addition, the terminal 10 is also used to acquire a target image to be estimated, the target image being an RGB image, and send the target image or image features of the target image to the server 20. The server 20 is used to receive the target image or image features of the target image sent by the terminal 10, acquire a plurality of candidate images from the stored first visual map according to the image features of the target image, the plurality of candidate images being RGB images whose corresponding image features have a difference less than a difference threshold from the image features of the target image, determine the sparse depth map and pose information corresponding to the target image according to the image features of the target image and the dense depth map and pose information corresponding to the plurality of candidate images, and send the sparse depth map and pose information corresponding to the target image to the terminal 10. The terminal 10 is used to receive the sparse depth map and pose information corresponding to the target image sent by the server 20, and determine the dense depth map corresponding to the target image according to the sparse depth map and pose information corresponding to the target image.

[0114] That is, the terminal 10 can pre-acquire the environment data of the target environment, send the environment data of the target environment to the server 20, and the server 20 constructs the first visual map of the target environment according to the environment data, and the first visual map includes a plurality of RGB images of different perspectives in the target environment and the dense depth map and pose information corresponding to each RGB image.

[0115] After that, when the terminal 10 needs to estimate the dense depth map corresponding to the target image, the terminal 10 can first send the target image or the image features of the target image to the server 20, and the server 20 returns the sparse depth map and pose information corresponding to the target image to the terminal 10 according to the image features of the target image and the first visual map. The terminal 10 determines the dense depth map corresponding to the target image according to the sparse depth map and pose information corresponding to the target image returned by the server 20.

[0116] In this way, the terminal 10 can quickly obtain the dense depth map corresponding to the target image with the assistance of the server 20, and in this process, the terminal 10 does not need to use a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements, and the terminal 10 can directly obtain the sparse depth map and pose information corresponding to the target image from the server 10 without the need to calculate it by itself, thereby reducing the calculation amount of the terminal 10, improving the determination efficiency of the dense depth map, and having high applicability.

[0117] Please refer to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device 100 provided by an embodiment of the present application. The electronic device 100 can be the terminal 10 or the server 20 in the above Figure 1 .

[0118] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyro sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0119] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0120] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0121] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0122] The electronic device 100 can implement a photographing function by the ISP, the camera 193, a video codec, a GPU, the display screen 194, and an application processor, and the like. For example, the electronic device 100 can implement a mobile photographing function to obtain a video frame sequence including RGB images of different perspectives.

[0123] The ISP is configured to process data fed back by the camera 193. For example, when taking a photo, a shutter is opened, light is transmitted to a camera photosensitive element through a lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing, and converts the electrical signal into an image visible to the naked eye. The ISP can also optimize algorithms for noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0124] The camera 193 is configured to capture still images or videos. The camera 193 can be an RGB camera. An object generates an optical image through a lens and projects the optical image onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB format or the like. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0125] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0126] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0127] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, the electronic device 100 can realize intelligent cognition applications such as image recognition, face recognition, voice recognition, text understanding, etc.

[0128] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, etc. Files are saved in the external memory card.

[0129] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), etc.

[0130] The electronic device 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0131] The audio module 170 is configured to convert digital audio information into analog audio signals and to convert analog audio input into digital audio signals. The audio module 170 can also be configured to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110 or some of the functions of the audio module 170 can be disposed in the processor 110. The speaker 170A, also referred to as a "loudspeaker", is configured to convert electrical audio signals into sound signals. The electronic device 100 can listen to music or engage in a hands-free call through the speaker 170A. The receiver 170B, also referred to as an "earpiece", is configured to convert electrical audio signals into sound signals. The microphone 170C, also referred to as a "microphone", "transducer", is configured to convert sound signals into electrical signals. The earphone interface 170D is configured to connect a wired earphone. The earphone interface 170D can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0132] The pressure sensor 180A is configured to sense a pressure signal and to convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed in the display screen 194. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view a short message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0133] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and lets the lens offset the shaking of the electronic device 100 by reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0134] The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100, and can be used for landscape / portrait switching, pedometer applications, and the like.

[0135] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locking, fingerprint photography, fingerprint call answering, and the like.

[0136] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement temperature processing strategies. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In another embodiment, when the temperature is lower than another threshold value, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by low temperature. In other embodiments, when the temperature is lower than yet another threshold value, the electronic device 100 performs voltage boosting on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0137] The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 together form a touch screen, also referred to as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0138] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support a Nano SIM card, a Micro SIM card, a SIM card, or the like.

[0139] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Embodiments of the present application take an Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0140] Figure 3 FIG. 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0141] The layered architecture divides software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Android runtime and system library, and a kernel layer.

[0142] The application layer can include a series of application packages.

[0143] As shown in FIG. 1, the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and the like. Figure 3

[0144] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0145] As shown in FIG. 1, the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like. Figure 3

[0146] The window manager is configured to manage window programs. The window manager can obtain the size of a display screen, determine whether there is a status bar, lock a screen, and take a screenshot, and the like.

[0147] The content provider is configured to store and obtain data, and make the data accessible to applications. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, and the like.

[0148] ​​The view system includes visual controls, such as controls that display text, controls that display pictures, and the like. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon can include a view that displays text and a view that displays a picture.

[0149] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call state (including the connection, hang-up, and the like).

[0150] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

[0151] The notification manager enables the application to display notification information in the status bar, which can be used to convey a type of message that can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of the download, the message reminder, and the like. The notification manager can also be a notification that appears in the top status bar of the system in the form of a chart or a scrolling text, for example, a notification of an application running in the background, and can also be a notification that appears on the screen in the form of a dialogue window. For example, the text information is prompted in the status bar, a prompt sound is emitted, the electronic device is vibrated, the indicator light flashes, and the like.

[0152] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0153] The core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0154] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to perform the management of the object life cycle, the stack management, the thread management, the security and exception management, and the garbage collection, and the like.

[0155] The system library can include a plurality of functional modules. For example: the surface manager, the media library, the three-dimensional graphics processing library (for example: OpenGL ES), the 2D graphics engine (for example: SGL), and the like.

[0156] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for a plurality of applications.

[0157] The media library supports multiple commonly used audio, video format playback and recording, and static image files, etc. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0158] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0159] The 2D graphics engine is a drawing engine for 2D drawing.

[0160] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, camera drivers, audio drivers, and sensor drivers.

[0161] After introducing the implementation environment related to the embodiments of the present application, the method provided by the embodiments of the present application will be described in detail in combination with the drawings.

[0162] Please refer to Figure 4 , Figure 4 is a flowchart of an image processing method provided by the embodiments of the present application. As an example but not limitation, the method can be applied to the terminal and the server in the above Figure 1 , and the method can include the following steps:

[0163] Step 401: The terminal acquires environment data, which is used to determine RGB images of different perspectives in a target environment.

[0164] The environment data is environment data of a target environment, and the target environment refers to an environment in which a target to be processed is located. The environment data is used to indicate environmental characteristics of the target environment, and the environmental image of the target environment can be determined according to the environment data. The environmental image of the target environment includes RGB images of different perspectives in the target environment, i.e., RGB images of the target of different perspectives.

[0165] For example, the environment data can include one or more of a video frame sequence obtained by moving shooting of the target environment, a panoramic image of the target environment, and laser scanning information of the target environment. For example, the video frame sequence can be collected by a moving RGB camera. The panoramic image of the target environment can be obtained by panoramic shooting of the target environment by a fisheye camera or multiple RGB cameras. The laser scanning information of the target environment can be obtained by laser scanning of the target environment by a high-precision laser scanning system.

[0166] As an example, the terminal is integrated with an RGB camera, and the terminal can move the target environment by the RGB camera to shoot, and take the video frame sequence obtained by the moving shooting as the environment data of the target environment.

[0167] For example, after receiving a video recording command, the terminal can record the target environment using an RGB camera to obtain a video frame sequence. This video recording command can be triggered by a user through a specified operation. For example, the specified operation may include, but is not limited to, click operations, swipe operations, voice operations, or gesture operations.

[0168] In one embodiment, taking a mobile phone as an example, the phone has a camera app installed. When a user wants to take a picture, they can trigger the camera app, for example, by clicking the app icon. The phone responds to the user's trigger operation by launching the camera app. Please refer to [link / reference]. Figure 5 In Figure (a), the camera app provides a video recording option 51. The user can trigger the video recording option 51, at which point the phone detects a video recording command. In response to the video recording command, the phone starts the video recording function, as shown below. Figure 5 As shown in Figure (b), the phone activates its RGB camera to record video. As an example, during video recording, the user can move the phone while holding it, thus keeping the phone in motion. For example... Figure 6 As shown, the user can hold the phone and walk around the target. During this process, the phone uses the video recorded by the RGB camera as a sequence of video frames.

[0169] Since the terminal moves while capturing the video frame sequence, it's easy to understand that adjacent video frames in the sequence are captured from two different locations. For example, please refer to... Figure 7 The first video frame in the video frame sequence was captured at point A, and the adjacent video frame was captured at point B. This can be understood as having one camera at point A (assuming a first camera) and another camera at point B (assuming a second camera). Logically, the first video frame can be considered to have been captured by the first camera, and the adjacent video frame by the second camera. In other words, the first video frame and the adjacent video frame can be understood as two video frames captured by a dual-camera system (including both the first and second cameras), or two video frames captured from two different angles of the same target. Therefore, the video frames in the video frame sequence are RGB images of the target environment from different perspectives.

[0170] Step 402: The terminal sends the environmental data to the server.

[0171] Step 403: The server receives environmental data sent by the terminal and generates a second visual map based on the environmental data. The second visual map includes multiple RGB images and a sparse depth map and pose information corresponding to each RGB image. The multiple RGB images are RGB images from different perspectives in the target environment.

[0172] In an embodiment of the present application, after the server receives the environment data, the server can generate a first visual map according to the environment data, the first visual map comprising a plurality of RGB images and a dense depth map and pose information corresponding to each RGB image. Before generating the first visual map, the server can first generate a second visual map according to the environment data, so as to generate the first visual map according to the second visual map.

[0173] For example, after the server receives the environment data sent by the terminal, the server can first determine the RGB images of different perspectives in the target environment according to the environment data, to obtain a plurality of RGB images. Then, the server determines a sparse depth map and pose information corresponding to each RGB image in the plurality of RGB images. According to the sparse depth map and pose information corresponding to each RGB image in the plurality of RGB images, the server constructs a second visual map.

[0174] As an example, the server can use a structured from motion (SFM) algorithm to determine the sparse depth map and pose information corresponding to each RGB image in the plurality of RGB images. It should be understood that the server can also use other algorithms to determine the sparse depth map and pose information corresponding to each RGB image in the plurality of RGB images, and the embodiments of the present application do not limit this.

[0175] The SFM algorithm is an offline algorithm for three-dimensional reconstruction based on various collected unordered pictures. As an example, using the SFM algorithm to determine the sparse depth map and pose information corresponding to each RGB image in the plurality of RGB images includes the following steps:

[0176] 1) Extract feature points of each RGB image in the plurality of RGB images to obtain image features of each image.

[0177] The feature points extracted from each RGB image refer to invariant feature points in the image, i.e., feature points that are independent of the shooting angle. The image features of each image refer to a set of feature points extracted from the image. The plurality of RGB images are a series of images taken around the same target. In fact, there are some feature points in the images taken of the same target, which are independent of the shooting angle. These feature points can be extracted first.

[0178] For example, a scale-invariant feature transform (SIFT) algorithm can be used to extract the feature points of each RGB image, i.e., to extract SIFT feature points of each image.

[0179] 2) performing feature matching on the image features of each pair of images in the plurality of RGB images to generate a first feature matching relationship of the plurality of RGB images.

[0180] The feature matching on the first image features of each pair of images is mainly to compare the feature points of each pair of images. The first feature matching relationship includes the matching point pairs corresponding to any two images in the plurality of RGB images, and the matching point pairs are 2D point pairs. In addition, since the feature matching is performed on the image features of each pair of images in the plurality of RGB images, the matching comparison is blind, and there is a point of mismatching points. Therefore, the feature matching relationship of the plurality of RGB images usually includes sparse matching point pairs.

[0181] The matching point pairs corresponding to any two images include the feature points corresponding to a certain part (or a certain point) of the target in the first image and the feature points corresponding to the second image, or it can be understood that the two feature points included in the matching point pairs are used to describe the characteristics of a certain part of the target at different angles. For example, please refer to Figure 8 , Figure 8 is a schematic diagram of part of the matching point pairs in the two images according to an example embodiment. In order to facilitate understanding, the two feature points in each matching point pair are connected by the same dashed line.

[0182] As an example but not limitation, the matching of the two feature points can be performed by calculating the Euler distance between any two corresponding feature points in each pair of images. If the Euler distance between the two feature points is greater than or equal to a preset threshold, it is determined that the two feature points are matched, and further determined that the two feature points are the matching point pairs corresponding to the two images.

[0183] 3) performing three-dimensional reconstruction according to the first feature matching relationship, and determining the sparse depth map corresponding to each RGB image in the plurality of RGB images according to the three-dimensional reconstruction result.

[0184] Since the matching point pairs in the first feature matching relationship are sparse matching point pairs, the three-dimensional reconstruction performed according to the first feature matching relationship is usually called sparse reconstruction. According to the first feature matching relationship, sparse reconstruction can be performed to reconstruct a sparse 3D point cloud, which includes the 3D points corresponding to the matching point pairs in the plurality of RGB images. According to the 3D points corresponding to the matching point pairs in the plurality of RGB images included in the 3D point cloud, the sparse depth map corresponding to each RGB image in the plurality of RGB images can be determined.

[0185] In one embodiment, the camera extrinsic parameters can be determined according to the first feature matching relationship and the camera intrinsic parameters. Then, the conversion relationship between the camera coordinate system and the world coordinate system can be determined according to the camera extrinsic parameters and the camera intrinsic parameters. After that, the coordinates of the 3D points corresponding to the matching point pairs in the multiple RGB images in the world coordinate system can be determined according to the coordinates of the matching point pairs (2D point pairs) in the multiple RGB images in the camera coordinate system and the conversion relationship between the camera coordinate system and the world coordinate system. The sparse depth map corresponding to each RGB image in the multiple RGB images can be determined according to the coordinates of the 3D points corresponding to the matching point pairs in the multiple RGB images in the world coordinate system.

[0186] As an example, x, K, R and T can be determined according to the feature matching relationship of the multiple RGB images. Wherein, x refers to the homogeneous representation of the coordinates of the matching point pairs in the multiple RGB images in the camera coordinate system. K is the camera intrinsic parameters. R and T are the camera extrinsic parameters, R is a rotation matrix, and T is a displacement vector.

[0187] For example, a projection equation of the 3D points projected onto the image plane of the camera is constructed in advance. According to the projection equation, the projection error between the projection points and the real points on the image is determined. Then, the target optimization equation of the projection error is optimized to determine the camera extrinsic parameters. In the target optimization equation of the projection error, the coordinates of the matching point pairs in the multiple RGB images in the camera coordinate system and the camera intrinsic parameters are known numbers, and the camera extrinsic parameters are unknown numbers. By optimizing the target optimization equation of the projection error, the camera extrinsic parameters that can minimize the projection error can be solved.

[0188] 4) The pose information corresponding to each RGB image in the multiple RGB images is determined according to the first feature matching relationship.

[0189] As an example but not limitation, the pose information corresponding to each RGB image in the multiple RGB images can be determined according to the first feature matching relationship by pose tracking and the like.

[0190] Step 404: The server determines the dense depth map corresponding to each RGB image in the multiple RGB images according to the sparse depth map corresponding to each RGB image in the multiple RGB images and the pose information.

[0191] After obtaining the sparse depth map corresponding to each RGB image in the multiple RGB images and the pose information, the dense depth map corresponding to each RGB image in the multiple RGB images can be further determined according to the sparse depth map corresponding to each RGB image in the multiple RGB images and the pose information.

[0192] As an example but not limitation, the dense depth map corresponding to each of the plurality of RGB images can be determined by using a multi view system (MVS) algorithm according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images.

[0193] The MVS algorithm is a method of generating a dense 3D point cloud from a sparse 3D point cloud. In the above step 403, the three-dimensional reconstruction result obtained by the three-dimensional reconstruction by the SFM algorithm is a sparse 3D point cloud, and then the MVS algorithm is used to process the sparse 3D point cloud to obtain a dense 3D point cloud. According to the dense 3D point cloud, the dense depth map corresponding to each of the plurality of RGB images can be determined.

[0194] As an example, the dense depth map corresponding to each of the plurality of RGB images can be determined by using the MVS algorithm according to the sparse depth map and the pose information corresponding to each of the plurality of RGB images, which can include the following steps:

[0195] 1) According to the sparse depth map and the pose information corresponding to each of the plurality of RGB images, performing image stereo rectification on any two images of the plurality of RGB images to align the any two images to the same plane.

[0196] For example, the camera parameters obtained in the above step 403 during the generation of the sparse depth map and the pose information corresponding to each of the plurality of RGB images can be used to perform image stereo rectification on any two images of the plurality of RGB images to align the any two images to the same plane. The camera parameters can include camera intrinsic parameters and camera extrinsic parameters.

[0197] 2) Extracting feature points of each of the plurality of RGB images to obtain image features of each of the plurality of RGB images.

[0198] The image features of each of the plurality of RGB images are a set of feature points of each of the plurality of RGB images.

[0199] It should be noted that the extraction of the feature points of each of the plurality of RGB images is the same as the extraction of the feature points of each of the plurality of RGB images in the above step 403, and the specific extraction process can refer to the related description of the above step 403, which will not be repeated here.

[0200] 3) Performing stereo matching on the image features of the plurality of RGB images to obtain a second feature matching relationship of the plurality of RGB images.

[0201] The second feature matching relationship includes matching point pairs corresponding to any two images in the plurality of corrected RGB images.

[0202] As an example but not limitation, a polar line constraint algorithm can be used to perform stereo matching on the image features of any two images in the plurality of corrected RGB images.

[0203] The stereo matching on the image features of any two images in the plurality of corrected RGB images, that is, the stereo matching on any two aligned images. Since any two aligned images can be aligned to the same plane, when performing stereo matching on any two aligned images, each feature point in any two aligned images can be matched, thereby obtaining relatively dense matching point pairs. Therefore, the second feature matching relationship can include dense matching point pairs.

[0204] 4) performing three-dimensional reconstruction according to the second feature matching relationship, and determining a dense depth map corresponding to each RGB image in the plurality of RGB images according to a three-dimensional reconstruction result.

[0205] Since the second feature matching relationship can include dense matching point pairs, the three-dimensional reconstruction performed according to the second feature matching relationship is generally referred to as dense reconstruction. According to the second feature matching relationship, dense reconstruction can be performed to obtain a dense 3D point cloud, which includes 3D point clouds corresponding to the matching point pairs in the plurality of corrected RGB images. According to the 3D point clouds corresponding to the matching point pairs in the plurality of corrected RGB images included in the dense 3D point cloud, a dense depth map corresponding to each RGB image in the plurality of corrected RGB images can be determined, and the dense depth map corresponding to the plurality of corrected RGB images is taken as a dense depth map corresponding to each RGB image in the plurality of corrected RGB images.

[0206] It should be noted that the implementation of the three-dimensional reconstruction according to the second feature matching relationship is the same as the implementation of the three-dimensional reconstruction according to the first feature matching relationship in step 403 described above. For specific implementation process, reference can be made to the related description in step 403 described above, and the embodiments of the present application will not be repeated here.

[0207] Step 405: The server constructs a first visual map according to the plurality of RGB images and the dense depth map and pose information corresponding to each RGB image.

[0208] For example, the server can construct a first visual map list in advance, and then store the plurality of RGB images and the dense depth map and pose information corresponding to each RGB image in the first visual map list.

[0209] In the embodiments of the present application, the terminal can collect environment data and send the environment data to the server, and the server can construct a first visual map according to the environment data, the first data map including a plurality of RGB images of different views in the environment, and a dense depth map and pose information corresponding to each RGB image. In the application, the terminal can send a target image to be estimated to the server, and the server can return a sparse depth map and pose information corresponding to the target image to the terminal according to the second visual map, and the terminal can determine a dense depth map corresponding to the target image according to the sparse depth map and pose information corresponding to the target image returned by the server. In this process, the terminal does not need to use a depth camera or a binocular camera to obtain a dense depth map, thereby reducing the hardware requirements, and the terminal can directly obtain the sparse depth map and pose information corresponding to the target image from the server without the need for the terminal itself to calculate, thereby reducing the computational load of the terminal, improving the determination efficiency of the dense depth map, and having high applicability.

[0210] It should be understood that the serial numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of the processes should be determined according to their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0211] After the server constructs the first visual map, the terminal can quickly obtain the dense depth map of the target image to be estimated with the assistance of the server. Next, the process of the terminal obtaining the dense depth map of the target image with the assistance of the server will be described in detail.

[0212] Please refer to Figure 9 , Figure 9 is a flowchart of another image processing method provided by the embodiments of the present application, which is an example and is not limited, and the method can be applied to the terminal and the server in the above Figure 1 , the method can include the following steps:

[0213] Step 901: The terminal obtains a target image to be estimated.

[0214] The target image is an RGB image. The target image to be estimated is an RGB image whose dense depth map is to be estimated, and can be any RGB image obtained by photographing a target.

[0215] As an example, the target image can be any video frame in a video frame sequence, such as any key frame in a video frame sequence. The video frame sequence can be collected by a mobile RGB camera. For details of the collection method, please refer to the related description of step 401 in the above Figure 4 embodiments, which will not be described here again.

[0216] For example, the terminal can extract key frames in the video frame sequence, and then take the extracted key frames as target images of the dense depth map to be estimated.

[0217] In step 902, the terminal sends the target image or the image features of the target image to the server.

[0218] As an example, after obtaining the target image, the terminal can directly send the target image to the server.

[0219] As another example, after obtaining the target image, the terminal can first extract feature points of the target image to obtain image features of the target image, and then send the target features of the target image to the server.

[0220] It should be noted that the implementation of the terminal extracting the feature points of the target image is the same as the implementation of the server extracting the feature points of each RGB image in step 403 of the above Figure 4 embodiment, and the specific implementation process can refer to the related description in the above step 403, which will not be described here again in the embodiments of the present application.

[0221] It should also be noted that the image features of the target image usually only include sparse feature points in the target image.

[0222] In step 903, the server receives the target image or the image features of the target image sent by the terminal, and according to the image features of the target image, obtains a plurality of candidate images from the stored first visual map, the plurality of candidate images being RGB images corresponding to image features having a difference with the image features of the target image less than a difference threshold.

[0223] That is, after receiving the target image or the image features of the target image sent by the terminal, the server can determine the image features of each RGB image included in the plurality of RGB images of the first visual map, and then compare the image features of the target image with the image features of the plurality of RGB images in turn, and take the RGB images corresponding to the image features having a difference with the image features of the target image less than the difference threshold as the candidate images.

[0224] Among them, the server can extract feature points of each RGB image in the plurality of RGB images to obtain image features of each RGB image. The specific process can refer to the related description in step 403 of the above Figure 4 embodiment, which will not be described here again in the embodiments of the present application.

[0225] It should be noted that if the terminal sends the image feature of the target image to the server, the terminal can directly compare the image feature of the target image with the image features of the plurality of RGB images in turn to obtain the plurality of candidate images from the first visual map. If the terminal sends the target image to the server, the server needs to first extract the feature points of the target image to obtain the image feature of the target image, and then compare the image feature of the target image with the image features of the plurality of RGB images in turn to obtain the plurality of candidate images from the first visual map.

[0226] Step 904: The server determines the sparse depth map and the pose information corresponding to the target image according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images.

[0227] It should be noted that since the image feature of the target image is a set of feature points extracted from the target image, it usually only includes sparse feature points in the target image, and therefore the server can only obtain the sparse depth map and the pose information corresponding to the target image according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images.

[0228] Among them, determining the sparse depth map and the pose information corresponding to the target image according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images means determining the sparse depth map and the pose information corresponding to the target image respectively according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images. That is, the sparse depth map corresponding to the target image is determined according to the image feature of the target image and the dense depth map corresponding to the plurality of candidate images. The pose information corresponding to the target image is determined according to the image feature of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images.

[0229] As an example, determining the sparse depth map corresponding to the target image according to the image feature of the target image and the dense depth map corresponding to the plurality of candidate images can include the following steps:

[0230] 1) For a first candidate image in the plurality of candidate images, determine the target image feature of the first candidate image, which refers to the part of the image feature of the first candidate image that matches the part of the image feature of the target image, and the first candidate image is any one of the plurality of candidate images.

[0231] As an example, feature matching can be performed on the image feature of the target image and the image feature of the first candidate image to obtain a matching point pair corresponding to the target image and the first candidate image, and then the feature point corresponding to the matching point in the image feature of the first candidate image is determined as the target image feature of the first candidate image.

[0232] 2) determining, from the dense depth map corresponding to the first candidate image, depth information corresponding to the target image feature of the first candidate image.

[0233] The dense depth map corresponding to the first candidate image generally includes depth information corresponding to dense feature points in the first candidate image, and thus the depth information corresponding to sparse feature points included in the target image feature of the first candidate image can be obtained from the depth information corresponding to dense feature points in the first candidate image.

[0234] In the embodiments of the present application, the steps 1) to 2) can be performed on each of the plurality of candidate images to obtain depth information corresponding to the target image feature of each candidate image.

[0235] 3) combining the depth information corresponding to the target image feature of each candidate image in the plurality of candidate images according to the image feature of the target image to obtain depth information corresponding to the image feature of the target image.

[0236] For example, the depth information corresponding to the target image feature of each candidate image in the plurality of candidate images can be combined according to the positions of the feature points in the corresponding matching target image feature, so that the depth information corresponding to each feature point in the image feature of the target image can be obtained.

[0237] 4) determining a sparse depth map corresponding to the target image according to the depth information corresponding to the image feature of the target image.

[0238] That is, the sparse depth map constructed according to the depth information corresponding to the image feature of the target image can be used as the sparse depth map corresponding to the target image. Since the image feature of the target image generally only includes sparse feature points in the target image, the depth map constructed according to the depth information corresponding to the image feature of the target image is a sparse depth map.

[0239] By way of example but not limitation, the server can perform pose estimation using a perspective-n-point (PNP) algorithm according to the image feature of the target image and the dense depth maps and pose information corresponding to the plurality of candidate images to obtain the pose information corresponding to the target image. The PNP algorithm is a method for solving the motion estimation of 3D-2D point pairs, which can estimate the pose of a camera when the coordinates of n 3D points and their two-dimensional projection positions are known.

[0240] For example, the server can perform pose estimation using the PNP algorithm according to the image feature of the target image and the dense depth maps and pose information corresponding to the plurality of candidate images to obtain the pose information corresponding to the target image, which can include the following steps:

[0241] 1) determining, according to the image feature of the target image and the pose information corresponding to the plurality of candidate images, a plurality of co-visible feature points corresponding to the image feature of the target image, to obtain a plurality of co-visible feature points.

[0242] The plurality of co-visible feature points refer to feature points having a co-visible relationship.

[0243] 2) determining the image coordinates of the plurality of co-visible feature points.

[0244] For example, the pixel coordinates of the plurality of co-visible feature points in the corresponding image can be obtained, and then the pixel coordinates of the plurality of co-visible feature points in the corresponding image are converted into image coordinates. The image coordinates are also referred to as 2D coordinates.

[0245] 3) obtaining the spatial coordinates of the plurality of co-visible feature points according to the dense depth map corresponding to the plurality of candidate images.

[0246] The spatial coordinates of the co-visible feature points refer to the coordinates of the co-visible feature points in the world coordinate system. The spatial coordinates are also referred to as 3D coordinates.

[0247] For example, the depth information corresponding to the plurality of co-visible feature points can be obtained according to the dense depth map corresponding to the plurality of candidate images, the coordinates of the plurality of co-visible feature points in the world coordinate system are determined according to the depth information corresponding to the plurality of co-visible feature points, and the spatial coordinates of the plurality of co-visible points are obtained.

[0248] 4) determining the pose information corresponding to the target image according to the image coordinates and the spatial coordinates of the plurality of co-visible feature points.

[0249] For example, the PNP algorithm can be used for pose estimation according to the image coordinates and the spatial coordinates of the plurality of co-visible feature points, and the estimated pose information is determined as the pose information corresponding to the target image.

[0250] Step 905: The server sends the sparse depth map and the pose information corresponding to the target image to the terminal.

[0251] Step 906: The terminal receives the sparse depth map and the pose information corresponding to the target image sent by the server, and determines the dense depth map corresponding to the target image according to the sparse depth map and the pose information corresponding to the target image.

[0252] In an embodiment, the terminal can determine the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, and the adjacent image of the target image, the sparse depth map and the pose information corresponding to the adjacent image. The adjacent image of the target image is an RGB image collected at a visual angle different from that of the target image.

[0253] As an example, the target image is a first video frame in a video frame sequence, and the video frame sequence is captured by a RGB camera in motion. Accordingly, determining the dense depth map corresponding to the target image according to the sparse depth map and the pose information corresponding to the target image can include: determining the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, a second video frame located before the target image in the video frame sequence, and the sparse depth map and the pose information corresponding to the second video frame.

[0254] The second video frame is a neighboring video frame of the target image, and the number of the second video frames can be one or more. The sparse depth map and the pose information corresponding to the second video frame can be obtained in the same manner as the sparse depth map and the pose information corresponding to the target image.

[0255] As an example, the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame can be input into the target model for processing, and the dense depth map corresponding to the target image is output.

[0256] The target model is configured to determine the dense depth map corresponding to any video frame according to the video frame, the sparse depth map and the pose information corresponding to the video frame, other video frames located before the video frame, and the sparse depth map and the pose information corresponding to the other video frames.

[0257] As an example but not limitation, the sparse depth map and the pose information corresponding to the any video frame can be obtained in the same manner as the sparse depth map and the pose information corresponding to the target image.

[0258] It should be noted that the target model can be obtained by pre-training, and the training process can be referred to the embodiments shown in the following Figure 10 The embodiments of the present application will not be described here in detail.

[0259] As an example, the terminal stores a pre-trained target model. After receiving the sparse depth map and the pose information corresponding to the target image sent by the server, the terminal can call the target model, and then input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into the target model for processing. Then, the target image can output the dense depth map corresponding to the target image.

[0260] The dense depth map of the target image is determined by the target model, which can further improve the determination efficiency of the dense depth map.

[0261] As another example, the target model can also be stored in the cloud, in which case the terminal can input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame to the target model uploaded to the cloud, and instruct the cloud to determine the dense depth map according to the uploaded data. Then, the cloud sends the determined dense depth map to the terminal. In this way, the computational load of the terminal can be further reduced.

[0262] In one embodiment, the number of second video frames is multiple, and the terminal can determine the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, each of the multiple second video frames, and the sparse depth map and the pose information corresponding to each of the second video frames.

[0263] The sparse depth map and the pose information corresponding to each of the second video frames can be obtained in the same way as the sparse depth map and the pose information corresponding to the target image.

[0264] As an example, the terminal can input the target image, the sparse depth map and the pose information corresponding to the target image, each of the multiple second video frames, and the sparse depth map and the pose information corresponding to each of the second video frames into the target model for processing to determine the dense depth map corresponding to the target image through the target model.

[0265] That is, when the number of second video frames is multiple, the terminal can still determine the dense depth map corresponding to the target image through the target model.

[0266] It is worth mentioning that when the number of second video frames is multiple, it means that the depth information of the target can be determined from more angles, i.e., rich depth information can be determined, so that a depth image with dense and accurate depth information can be determined.

[0267] In the embodiments of the present application, the terminal can send the target image to be estimated or the image features of the target image to the server, and the server returns the sparse depth map and the pose information corresponding to the target image to the terminal according to the first visual map, and then the terminal determines the dense depth map corresponding to the target image according to the sparse depth map and the pose information corresponding to the target image returned by the server. In this way, the terminal can quickly obtain the dense depth map corresponding to the target image with the assistance of the server. In this process, the terminal does not need to use a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements. Moreover, the terminal can directly obtain the sparse depth map and the pose information corresponding to the target image from the server, without the need for the terminal to calculate them by itself, thereby reducing the computational load of the terminal, improving the determination efficiency of the dense depth map, and having high applicability.

[0268] Please refer to 10, 10 is a flow chart of a model training method provided by an embodiment of the present application. The method can be applied in an electronic device, which can be a terminal or a server. The embodiment of the present application takes the method applied in a terminal as an example for illustration. The terminal can be the terminal 10 in the above Figure 1 , or other terminals other than the terminal 10. That is, the terminal performing model training and the terminal determining the dense depth map in the above Figure 9 embodiment can be the same terminal or different terminals. The method can include the following steps.

[0269] Step 1001: The terminal obtains a sample data set. The sample data set includes multiple groups of sample data. Each group of sample data includes a video frame sample sequence, a sparse depth map and pose information corresponding to each video frame sample in the video frame sample sequence, and a real dense depth map corresponding to each video frame sample.

[0270] As an example of the present application, the video frame sample sequence is collected by a mobile RGB camera.

[0271] In an embodiment, the sparse depth map and the pose information corresponding to each video frame sample can be obtained in the manner of obtaining the sparse depth map and the pose information corresponding to the target image, or can be obtained in other manners. The embodiments of the present application do not limit this.

[0272] In an embodiment, each video frame sample can be a key frame sample in the video frame sample sequence.

[0273] In an embodiment, the real dense depth map corresponding to each video frame sample can be collected by a depth camera with excellent indicators, that is, the depth camera can obtain dense depth information. For example, the depth camera can be a Video Graphics Array (VGA) depth camera.

[0274] Step 1002: The terminal trains a network model to be trained according to the sample data set to obtain a target model.

[0275] In implementation, an end-to-end network model is constructed as the network model to be trained. In an embodiment, the network model can be a U-net network model, a Visual Geometry Group (VGG) network model, or a Residual Network (RESNET)-50 network model, etc.

[0276] The loss function is constructed to describe the degree of difference between the real value and the estimated value. The real value in this embodiment is the real dense depth map, and the estimated value is the dense depth map estimated by the network model.

[0277] As an example of the present application, the loss function can be constructed according to the geometric error between the real value and the estimated value, the similarity error of the video frame sample, and the depth image smoothness. The similarity error of the video frame sample is used to describe the back-projection error of the pose information. The depth image smoothness can be understood as a constraint condition, which can make the finally obtained dense depth map smooth and there is no jump between the pixels.

[0278] Exemplarily, the geometric error can be determined by formula (1):

[0279]

[0280] wherein, l d represents the geometric error, i represents the ith second sample data, d i represents the real value, represents the estimated value.

[0281] In addition, the similarity error of the video frame sample can be determined by formula (2):

[0282]

[0283] wherein, is determined by formula (3) as follows:

[0284]

[0285] The meaning of is to project a video frame sample (such as a key frame sample) I t to another video frame sample I S , and the projection video frame obtained after I t . Wherein, K is an intrinsic matrix, T is the pose information of the video frame sample I t , and D is the depth information of the video frame sample I .

[0286] In formula (2), l p represents the similarity error, a is an empirical value, which can be 0.85, for example, and SSIM(*) is the structural similarity error, which is a function for calculating the structural similarity error.

[0287] In addition, the depth image smoothness can be determined by formula (4) as follows:

[0288]

[0289] wherein, l s represents the depth image smoothness, n represents the number of samples, F(*) is a vector differential operator used to calculate the gradient, represents the gradient in the x direction, represents the gradient in the y direction,

[0290] Then, the loss function is obtained by weighted sum of the geometric error, the similarity error of the video frame samples, and the depth image smoothness.

[0291] In the training process, the model parameters of the network model can be updated according to the loss function and the sample data set. When the training satisfies the end condition, the network model with updated model parameters is taken as the trained target model.

[0292] wherein, the end condition can be set according to actual requirements. Illustratively, the end condition can be that the number of training times reaches a number threshold. The number threshold can be set according to actual requirements.

[0293] In the embodiments of the present application, the network model is trained based on the sample data set to obtain the target model. In application, based on the target image, the sparse depth map and the pose information corresponding to the target image, and the adjacent video frame, the sparse depth map and the pose information corresponding to the adjacent video frame located before the target image, the target model is processed to determine the dense depth map corresponding to the target image. In this way, the efficiency of determining the dense depth map can be improved.

[0294] It should be understood that the serial numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0295] Figure 11 is a structural schematic diagram of an image processing device provided by the embodiments of the present application. The device can be realized by software, hardware or a combination of both to become part or all of a computer device, which can be a terminal 10 as shown in Figure 1 . Referring to Figure 11 , the device includes a first acquisition module 1101, a sending module 1102, a receiving module 1103 and a determination module 1104.

[0296] The first acquisition module 1101 is configured to perform step 901 in the above Figure 9 embodiments.

[0297] The sending module 1102 is configured to perform step 902 in the above Figure 9 embodiments.

[0298] The receiving module 1103 and the determining module 4404 are configured to perform the steps 906 in the above embodiments. Figure 9

[0299] Optionally, the determining module 4404 is configured to input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into a target model for processing, and output the dense depth map corresponding to the target image; wherein the target model is configured to determine the dense depth map corresponding to any video frame according to any video frame, the sparse depth map and the pose information corresponding to the any video frame, other video frames located before the any video frame, and the sparse depth map and the pose information corresponding to the other video frames.

[0300] Optionally, the apparatus further includes a second obtaining module and a training module.

[0301] The second obtaining module is configured to perform the step 1001 in the above embodiments. Figure 10

[0302] The training module is configured to perform the step 1002 in the above embodiments. Figure 10

[0303] Optionally, the apparatus further includes a third obtaining module.

[0304] The third obtaining module is configured to perform the step 401 in the above embodiments. Figure 4

[0305] The sending module is further configured to perform the step 402 in the above embodiments. Figure 4

[0306] In the embodiments of the present application, the apparatus can send the target image to be estimated or the image features of the target image to a server, and the server returns the sparse depth map and the pose information corresponding to the target image to the terminal according to the first visual map, and then determines the dense depth map corresponding to the target image according to the sparse depth map and the pose information corresponding to the target image returned by the server. In this way, the apparatus can quickly obtain the dense depth map corresponding to the target image with the assistance of the server. In this process, the apparatus does not need to use a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements, reducing the computational load of the apparatus, improving the determination efficiency of the dense depth map, and having high applicability.

[0307] Figure 12 FIG. 4 is a structural schematic diagram of another image processing apparatus provided in the embodiments of the present application. The apparatus can be realized by software, hardware, or a combination of both, as part or all of a computer device, which can be the server 20 shown in FIG. 2. Figure 1 Figure 12 ​​​​​​The apparatus comprises a receiving module 1201, an obtaining module 1202, a determining module 1203, and a sending module 1204.

[0308] The receiving module 1201 and the obtaining module 1202 are configured to perform the step 903 in the method of any of the above embodiments. Figure 9

[0309] The determining module 1203 is configured to perform the step 904 in the method of any of the above embodiments. Figure 9

[0310] The sending module 1204 is configured to perform the step 905 in the method of any of the above embodiments. Figure 9

[0311] Optionally, the determining module 1203 comprises:

[0312] a first determining unit, configured to determine the sparse depth map corresponding to the target image according to the image feature of the target image and the dense depth maps corresponding to the plurality of candidate images;

[0313] a second determining unit, configured to determine the pose information corresponding to the target image according to the image feature of the target image, the dense depth maps corresponding to the plurality of candidate images, and the pose information.

[0314] Optionally, the first determining unit is configured to:

[0315] for a first candidate image in the plurality of candidate images, determine the depth information corresponding to the target image feature of the first candidate image from the dense depth map corresponding to the first candidate image, the target image feature of the first candidate image being part of the image feature of the first candidate image that matches the part of the image feature of the target image, and the first candidate image being any one of the plurality of candidate images;

[0316] combine the depth information corresponding to the target image feature of each candidate image in the plurality of candidate images according to the image feature of the target image, to obtain the depth information corresponding to the image feature of the target image;

[0317] determine the sparse depth map corresponding to the target image according to the depth information corresponding to the image feature of the target image.

[0318] Optionally, the second determining unit is configured to:

[0319] determine the co-visible feature points of the image features of the plurality of candidate images and the image feature of the target image according to the image feature of the target image and the pose information corresponding to the plurality of candidate images, to obtain a plurality of co-visible feature points;

[0320] ​​​According to the dense depth maps corresponding to the plurality of candidate images, spatial coordinates of the plurality of co-visible feature points are obtained;

[0321] According to the image coordinates and the spatial coordinates of the plurality of co-visible feature points, pose information corresponding to the target image is determined.

[0322] Optionally, the apparatus further includes a generating module;

[0323] The receiving module 1201 is further configured to receive environment data sent by the terminal;

[0324] The generating module is configured to generate, according to the environment data, a first visual map, the first visual map including a plurality of RGB images and dense depth maps and pose information corresponding to each RGB image, the plurality of RGB images being RGB images of different perspectives in a target environment.

[0325] Optionally, the generating module is configured to perform steps 403-405 in the above Figure 4 embodiment.

[0326] Optionally, the generating module is configured to:

[0327] According to the environment data, determine the RGB images of different perspectives in the target environment, to obtain the plurality of RGB images;

[0328] Extract feature points of each RGB image in the plurality of RGB images, to obtain image features of each RGB image, the image features of each RGB image being a set of feature points of each RGB image;

[0329] Perform feature matching on the image features of every two images in the plurality of RGB images, to generate a first feature matching relationship, the first feature matching relationship including matching point pairs corresponding to any two images in the plurality of RGB images;

[0330] According to the first feature matching relationship, perform three-dimensional reconstruction, determine sparse depth maps corresponding to each RGB image in the plurality of RGB images according to a three-dimensional reconstruction result, and determine pose information corresponding to each RGB image in the plurality of RGB images according to the feature matching relationship;

[0331] According to the sparse depth maps and the pose information corresponding to each RGB image in the plurality of RGB images, construct a second visual map.

[0332] Optionally, the generating module is configured to:

[0333] According to the plurality of RGB images and the sparse depth maps and the pose information corresponding to each RGB image, perform image stereo rectification on any two images in the plurality of RGB images, to align the any two images to the same plane;

[0334] extract feature points of each of the plurality of corrected RGB images to obtain image features of each of the plurality of corrected RGB images, the image features of each of the plurality of corrected RGB images being a set of feature points of each of the plurality of corrected RGB images;

[0335] perform stereo matching on the image features of any two images of the plurality of corrected RGB images to obtain a second feature matching relationship, the second feature matching relationship including a matching point pair corresponding to any two images of the plurality of corrected RGB images;

[0336] perform three-dimensional reconstruction according to the second feature matching relationship, and determine a dense depth map corresponding to each of the plurality of RGB images according to a result of the three-dimensional reconstruction.

[0337] In the embodiments, the server can receive a target image or image features of the target image sent by the terminal, and then acquire, according to the image features of the target image, an RGB image with a difference between corresponding image features and the image features of the target image being less than a difference threshold from the stored first visual map as a plurality of candidate images, determine a sparse depth map and pose information corresponding to the target image according to the image features of the target image and the dense depth map and the pose information corresponding to the plurality of candidate images, and send the sparse depth map and the pose information corresponding to the target image to the terminal. In this way, the terminal can quickly acquire the dense depth map corresponding to the target image with the assistance of the server. In this process, the terminal does not need to use a depth camera or a binocular camera to obtain the dense depth map, thereby reducing the hardware requirements. Moreover, the terminal can directly acquire the sparse depth map and the pose information corresponding to the target image from the server, instead of calculating them by itself, thereby reducing the computation amount of the terminal, improving the determination efficiency of the dense depth map, and having high applicability.

[0338] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual applications, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0339] In an embodiment of the present application, all or part of the flow of the above-mentioned method can be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunications signal.

[0340] Finally, it should be noted that the above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, Applied to a terminal, the method comprises: obtaining environment data, the environment data being used to determine RGB images of different perspectives in a target environment, the environment data comprising one or more of a video frame sequence obtained by moving shooting in the target environment, a panoramic map of the target environment, and laser scanning information of the target environment; obtaining a target image to be estimated, the target image being a first video frame in a video frame sequence, the video frame sequence being collected by a moving RGB camera; sending the environment data to a server, and sending the target image or image features of the target image to the server; receiving a sparse depth map and pose information corresponding to the target image sent by the server; determining a dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, a second video frame located before the target image in the video frame sequence, and a sparse depth map and pose information corresponding to the second video frame, the number of the second video frames being one or more.

2. The method of claim 1, wherein, The determining of the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame located before the target image in the video frame sequence, and the sparse depth map and the pose information corresponding to the second video frame comprises: inputting the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into a target model for processing, and outputting the dense depth map corresponding to the target image; wherein the target model is used to determine a dense depth map corresponding to any video frame according to the any video frame, a sparse depth map and pose information corresponding to the any video frame, other video frames located before the any video frame, and sparse depth maps and pose information corresponding to the other video frames.

3. The method of claim 2, wherein, The method further comprises: obtaining a sample data set, the sample data set comprising a plurality of groups of sample data, each group of sample data in the plurality of groups of sample data comprising a video frame sample sequence, a sparse depth map and pose information corresponding to each video frame sample in the video frame sample sequence, and a real dense depth map corresponding to the each video frame sample; training a network model to be trained according to the sample data set to obtain the target model.

4. The method according to any of claims 1 to 3, characterized in that, The number of the second video frames is a plurality, and the determining of the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame located before the target image in the video frame sequence, and the sparse depth map and the pose information corresponding to the second video frame comprises: determining the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, each second video frame in the plurality of second video frames, and a sparse depth map and pose information corresponding to the each second video frame.

5. An image processing system, characterized by The system comprises a terminal and a server; The terminal is configured to acquire environment data, the environment data being used to determine RGB images of different perspectives in a target environment, and the environment data comprising one or more of a video frame sequence obtained by moving photographing of the target environment, a panoramic map of the target environment, and laser scanning information of the target environment; The terminal is further configured to acquire a target image to be estimated, the target image being a first video frame in a video frame sequence, and the video frame sequence being collected by a moving photographing RGB camera; The terminal is further configured to send the environment data to a server and send the target image or image features of the target image to the server; The server is configured to receive the target image or image features of the target image sent by the terminal and the environment data sent by the terminal; The server is further configured to generate a first visual map according to the environment data, acquire a plurality of candidate images from the first visual map according to the image features of the target image, the first visual map comprising a plurality of RGB images, and a dense depth map and pose information corresponding to each RGB image, and the plurality of candidate images being RGB images with a difference between corresponding image features and the image features of the target image being less than a difference threshold; The server is further configured to determine a sparse depth map and pose information corresponding to the target image according to the image features of the target image and the dense depth map and pose information corresponding to the plurality of candidate images; The server is further configured to send the sparse depth map and pose information corresponding to the target image to the terminal; The terminal is further configured to receive the sparse depth map and pose information corresponding to the target image sent by the server; The terminal is further configured to determine a dense depth map corresponding to the target image according to the target image, the sparse depth map and pose information corresponding to the target image, a second video frame before the target image in the video frame sequence, and a sparse depth map and pose information corresponding to the second video frame, the number of the second video frames being one or more.

6. The system of claim 5, wherein, The server is further configured to: determine, for a first candidate image in the plurality of candidate images, depth information corresponding to target image features of the first candidate image from a dense depth map corresponding to the first candidate image, the target image features of the first candidate image being part of image features of the first candidate image that are mutually matched with part of the image features of the target image, the first candidate image being any one of the plurality of candidate images; combine, according to the image features of the target image, the depth information corresponding to the target image features of each candidate image in the plurality of candidate images, to obtain depth information corresponding to the image features of the target image; determine, according to the depth information corresponding to the image features of the target image, a sparse depth map corresponding to the target image.

7. The system of claim 5 or 6, wherein, The server is further configured to: determine, according to the image features of the target image and the pose information corresponding to the plurality of candidate images, co-view feature points corresponding to the image features of the target image and the image features of the plurality of candidate images, to obtain a plurality of co-view feature points; obtain spatial coordinates of the plurality of co-view feature points according to the dense depth maps corresponding to the plurality of candidate images; determine the pose information corresponding to the target image according to the image coordinates and the spatial coordinates of the plurality of co-view feature points.

8. The system according to any of claims 5-7, characterized in that, The terminal is further configured to: input the target image, the sparse depth map and the pose information corresponding to the target image, the second video frame, and the sparse depth map and the pose information corresponding to the second video frame into a target model for processing, and output a dense depth map corresponding to the target image; wherein the target model is configured to determine a dense depth map corresponding to any video frame according to the any video frame, the sparse depth map and the pose information corresponding to the any video frame, other video frames located before the any video frame, and the sparse depth map and the pose information corresponding to the other video frames.

9. The system according to any of claims 5-8, characterized in that, The number of the second video frames is a plurality. The terminal is further configured to determine the dense depth map corresponding to the target image according to the target image, the sparse depth map and the pose information corresponding to the target image, each second video frame in the plurality of second video frames, and the sparse depth map and the pose information corresponding to each second video frame.

10. The system according to any of claims 5-9, characterized in that, The server is further configured to: generate a second visual map according to the environment data, the second visual map including a plurality of RGB images and a sparse depth map and pose information corresponding to each RGB image; determine a dense depth map corresponding to each RGB image in the plurality of RGB images according to the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images; construct the first visual map according to the plurality of RGB images and the dense depth map and the pose information corresponding to each RGB image.

11. The system of claim 10, wherein, The server is further configured to: determine RGB images of different perspectives in the target environment according to the environment data, to obtain the plurality of RGB images; extract feature points of each RGB image in the plurality of RGB images to obtain image features of each RGB image, the image features of each RGB image being a set of feature points of each RGB image; perform feature matching on the image features of each pair of images in the plurality of RGB images to generate a first feature matching relationship, the first feature matching relationship including a matching point pair corresponding to any two images in the plurality of RGB images; perform three-dimensional reconstruction according to the first feature matching relationship, determine a sparse depth map corresponding to each RGB image in the plurality of RGB images according to a three-dimensional reconstruction result, and determine pose information corresponding to each RGB image in the plurality of RGB images according to the first feature matching relationship; construct the second visual map according to the sparse depth map and the pose information corresponding to each RGB image in the plurality of RGB images.

12. The system of claim 10 or 11, wherein, The server is further configured to: According to the plurality of RGB images and the sparse depth map and pose information corresponding to each RGB image, image stereo rectification is performed on any two images in the plurality of RGB images to align the any two images to the same plane; Feature points of each RGB image in the plurality of corrected RGB images are extracted to obtain image features of each RGB image in the plurality of corrected RGB images, and the image features of each corrected RGB image are a set of feature points of each corrected RGB image; Stereo matching is performed on the image features of the two images in the plurality of corrected RGB images to obtain a second feature matching relationship, and the second feature matching relationship includes a matching point pair corresponding to any two images in the plurality of corrected RGB images; According to the second feature matching relationship, three-dimensional reconstruction is performed, and a dense depth map corresponding to each RGB image in the plurality of RGB images is determined according to a three-dimensional reconstruction result.

13. A terminal, characterized by comprising: The terminal comprises a memory and a processor; The memory is configured to store a program supporting the terminal to execute the method of any one of claims 1-4, and store data involved in the method of any one of claims 1-4; and the processor is configured to execute the program stored in the memory.

14. A computer-readable storage medium having stored therein instructions, the computer-readable storage medium comprising: When it runs on a computer, it makes the computer execute the method of any one of claims 1-4.

Images