Depth map generation method, device, equipment and storage medium based on plane prior

Abstract

The present application discloses a method, device, equipment and storage medium for generating a depth map based on plane prior, which relates to the field of image processing technology and is used to improve the accuracy of image depth estimation. The method comprises: acquiring multiple scene images of a target scene shot from different perspectives, and performing plane detection on a reference perspective image in the multiple scene images to obtain plane information in the reference perspective image; wherein the plane information is used to indicate pixel points belonging to the same plane; based on the pixel feature correlation between the multiple scene images, determining a reference pixel point set whose pixel feature correlation meets a set correlation condition from each pixel point included in the reference perspective image; based on the plane information and the reference pixel point set, obtaining a plane prior condition of the reference perspective image; and taking the plane prior condition as a constraint and based on the pixel feature correlation between the multiple scene images, obtaining a depth map of the reference perspective image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to the field of image processing technology, and provides a method, device, equipment and storage medium for generating a depth map based on plane prior. Background Art

[0002] Depth estimation is the process of estimating the distance of each pixel in an image relative to the shooting source using a single image or multiple images from different perspectives. Depth estimation is a key step in the task of 3D scene reconstruction. It is a very important part in the field of computer vision and has very important applications in scenarios such as Virtual Reality (VR) or Augmented Reality (AR).

[0003] Multi-view stereo matching (Patch match Multi-View Stereo, Patch match MVS) technology is a three-dimensional vision technology that uses a stereo matching algorithm to restore the stereo structure from color images of multiple perspectives with known internal and external parameters of the camera. It has achieved relatively good results for depth estimation.

[0004] However, in weak texture areas, since the pixel differences in the area are small, the pixels in these areas cannot be reliably distinguished, making it impossible to accurately estimate the depth value. Therefore, the depth estimation effect of related technologies in weak texture areas is poor. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for generating a depth map based on plane prior, which are used to improve the accuracy of image depth estimation.

[0006] On the one hand, a method for generating a depth map based on a plane prior is provided, the method comprising:

[0007] Acquire multiple scene images of a target scene shot from different perspectives, and perform plane detection on a reference perspective image among the multiple scene images to obtain plane information in the reference perspective image; wherein the plane information is used to indicate pixel points belonging to the same plane:

[0008] Based on the pixel feature correlations between the multiple scene images, determining a reference pixel point set whose pixel feature correlations satisfy a set correlation condition from the pixels included in the reference view image;

[0009] Based on the plane information and the reference pixel point set, obtaining a plane priori condition of the reference viewing angle image;

[0010] Taking the plane prior condition as a constraint and based on the pixel feature correlation between the multiple scene images, a depth map of the reference perspective image is obtained.

[0011] On the one hand, a three-dimensional modeling method based on the above method comprises:

[0012] Acquire multiple scene images of the target scene shot based on different viewing angles, and use the above method to obtain depth maps corresponding to each of the multiple scene images;

[0013] Based on the obtained depth maps, a three-dimensional stereogram corresponding to the target scene is obtained.

[0014] On the one hand, an automatic driving method based on the above method comprises:

[0015] Acquire multiple scene images of the target scene shot based on different viewing angles, and use the above method to obtain depth maps corresponding to each of the multiple scene images;

[0016] Based on the obtained depth maps, obstacle distances around the target vehicle are determined, and autonomous driving is performed based on the obtained obstacle distances.

[0017] On the one hand, a device for generating a depth map based on a plane prior is provided, the device comprising:

[0018] A plane detection unit, used to acquire multiple scene images of a target scene shot from different perspectives, and perform plane detection on a reference perspective image among the multiple scene images to obtain plane information in the reference perspective image; wherein the plane information is used to indicate pixel points belonging to the same plane;

[0019] A pixel screening unit, configured to determine, based on the pixel feature correlations between the plurality of scene images, a reference pixel point set whose pixel feature correlations satisfy a set correlation condition from among the pixels included in the reference view image;

[0020] A plane priori generating unit, configured to obtain a plane priori condition of the reference viewing angle image based on the plane information and the reference pixel point set;

[0021] A depth estimation unit is used to obtain a depth map of the reference view image based on the pixel feature correlation between the multiple scene images, with the plane prior condition as a constraint.

[0022] Optionally, the plane priori generating unit is specifically used to:

[0023] Based on the reference pixel point set, a triangulated mesh method is used to perform plane fitting processing to obtain a triangulated mesh structure, wherein each triangular plane in the triangulated mesh structure includes three reference pixel points in the reference pixel point set;

[0024] Based on the plane information, merging the triangular planes corresponding to the reference pixel points located on the same plane in the triangular mesh structure to obtain a merged triangular mesh structure;

[0025] Based on the merged triangular mesh structure, the plane prior condition is obtained.

[0026] Optionally, the plane priori generating unit is specifically used to:

[0027] Based on the plane information, determine a plurality of reference pixel point groups belonging to the same plane in the reference pixel point set, each reference pixel point group including at least one reference pixel point;

[0028] Performing plane fitting processing on the obtained multiple reference pixel point groups respectively to obtain a plane combination structure composed of a corresponding multiple fitting planes; wherein each reference pixel point group corresponds to a fitting plane among the multiple fitting planes;

[0029] Based on the plane combination structure, the plane priori condition is obtained.

[0030] Optionally, the plane detection unit is specifically used to:

[0031] Performing image semantic segmentation on the reference view image to obtain plane masks corresponding to respective plane regions in the reference view image;

[0032] Performing depth estimation processing on the reference perspective image to obtain an estimated depth map of the reference perspective image;

[0033] Based on the estimated depth map, the plane mask is updated to obtain the plane information of the reference view image.

[0034] Optionally, the pixel screening unit is specifically used to:

[0035] For each pixel point, perform the following steps respectively:

[0036] For a pixel point, determine a stereo matching area of ​​the pixel point in the reference view image, each stereo matching area includes an image area of ​​a set range with the pixel point as a reference point;

[0037] Respectively determine the mapping area of ​​the stereo matching area in other scene images and the pixel feature correlation between the stereo matching area; wherein the other scene images are scene images other than the reference perspective image among the multiple scene images;

[0038] From the individual pixel points, a reference pixel point set whose pixel feature association degree satisfies a set association condition is determined.

[0039] Optionally, the pixel screening unit is further used for:

[0040] Initializing parameters of each pixel point respectively to obtain initial disparity plane parameters corresponding to each pixel point;

[0041] Based on the pixel feature correlation between the reference view image and the corresponding area in the other scene image, performing multiple iterative update processes on the initial disparity plane parameters corresponding to each pixel point, to obtain the estimated disparity plane parameters corresponding to each pixel point;

[0042] Based on the obtained estimated disparity plane parameters, the stereo matching regions corresponding to the respective pixel points and the mapping regions in the other scene images are determined.

[0043] Optionally, the pixel screening unit is specifically used to:

[0044] Acquire a plurality of sampling pixel points from pixel points in a set area around the one pixel point;

[0045] Determining an iterative update loss value of the pixel point based on a correlation degree between a stereo matching area corresponding to the pixel point and pixel features in each corresponding mapping area;

[0046] Determine the iterative update loss values ​​corresponding to the multiple sampling pixels respectively based on the correlation between the stereo matching areas corresponding to the multiple sampling pixels and the pixel features in the corresponding mapping areas;

[0047] If, among the multiple sampling pixel points, there is a target sampling pixel point whose iterative update loss value is less than the iterative update loss value corresponding to the one pixel point, the current disparity plane parameter corresponding to the target sampling pixel point is used to update the current disparity plane parameter of the one pixel point.

[0048] Optionally, the pixel screening unit is specifically used to:

[0049] Based on the current disparity plane parameters corresponding to the one pixel point and the homography transformation matrix between various perspectives, determining the stereo matching area corresponding to the one pixel point and the mapping area in the other scene image;

[0050] Based on the pixel feature values ​​in the stereo matching area and each mapping area, correspondingly determining the pixel feature correlation between the stereo matching area and each mapping area;

[0051] Based on the obtained correlation degree of each pixel feature, an iterative update loss value corresponding to the one pixel point is determined, wherein the iterative update loss value is negatively correlated with the pixel feature correlation degree.

[0052] Optionally, the depth estimation unit is specifically used to:

[0053] Based on the plane prior conditions and the pixel feature correlation, the current disparity plane parameters corresponding to each pixel point are updated multiple times to obtain the target disparity plane parameters corresponding to each pixel point;

[0054] Based on the obtained target disparity plane parameters, the depth values ​​corresponding to the respective pixel points are determined to obtain the depth map.

[0055] Optionally, the depth estimation unit is specifically used to:

[0056] Starting from the specified pixel point of the reference view image, the following steps are performed one by one for each pixel point in a traversal manner:

[0057] For a pixel point, obtaining a plurality of sampling pixel points from pixel points in a set area around the pixel point;

[0058] Based on the plane where each pixel point in the plane prior condition is located and the correlation degree of pixel features in the corresponding stereo matching area and each mapping area, respectively determine the iterative update loss value corresponding to the one pixel point and the multiple sampling pixel points;

[0059] If, among the multiple sampling pixel points, there is a target sampling pixel point whose corresponding iterative update loss value is less than the iterative update loss value corresponding to the one pixel point, the current disparity plane parameter of the one pixel point is updated according to the current disparity plane parameter corresponding to the target sampling pixel point.

[0060] Optionally, the pixel screening unit and the depth estimation unit are further specifically used for:

[0061] According to the set number of processing channels N, the reference view image is divided into N channel images; wherein any two adjacent pixels included in each channel image are separated by N-1 pixels from their original positions in the reference view image;

[0062] The multiple sampling pixel points are obtained from the pixel points in the surrounding set area included in the channel image where the one pixel point is located.

[0063] On the one hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0064] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0065] In one aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above methods.

[0066] In the embodiment of the present application, when performing depth estimation for the target scene, the plane information in the reference perspective image is obtained by performing plane detection on the reference perspective image, that is, it is possible to obtain which pixels in the reference perspective image belong to the same plane, and also select a reference pixel set whose pixel feature correlation satisfies the set correlation condition from the reference perspective image through the pixel feature correlation between scene images of different perspectives of the target scene, and then generate a plane priori condition of the reference perspective image based on the obtained plane information and the reference pixel set, and then use the plane prior condition as a constraint to generate a depth map corresponding to the reference perspective image. It can be seen that the embodiment of the present application makes full use of the plane information in the image by introducing plane detection, improves the accuracy of the plane prior condition, and then the depth value of the weak texture area in the depth map solved in combination with the plane prior condition is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0068] Figure 1 A schematic diagram of patch mapping in images of different viewing angles provided in an embodiment of the present application;

[0069] Figure 2a to Figure 2d A depth map obtained by performing depth estimation for related technologies;

[0070] Figure 3 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0071] Figure 4 A schematic diagram of a depth estimation method based on plane prior provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of a process flow of a plane detection process provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a process for determining a reference pixel set from each pixel provided in an embodiment of the present application;

[0074] Figure 7a to Figure 7c Schematic diagrams of several traversal methods are provided for embodiments of the present application;

[0075] Figure 8 A schematic diagram of a flow chart of an iterative update process of a pixel point Pi provided in an embodiment of the present application;

[0076] Figure 9a and Figure 9b A schematic diagram of a sampling pixel point set is provided for an embodiment of the present application;

[0077] Fig.10 A schematic diagram of a process for obtaining a plane prior based on plane information and a reference pixel point set provided in an embodiment of the present application;

[0078] Fig.11 A schematic diagram of a flow chart for obtaining a plane priori condition provided in an embodiment of the present application;

[0079] Fig.12 A schematic diagram of a plane prior condition provided in an embodiment of the present application;

[0080] Fig.13 Another schematic diagram of a process for obtaining a plane priori condition provided in an embodiment of the present application;

[0081] Fig.14 A schematic diagram of a process for obtaining a depth map of a reference viewing angle image provided in an embodiment of the present application;

[0082] Figure 15a to Figure 15b A depth map obtained by using the method of an embodiment of the present application;

[0083] Fig.16 A schematic diagram of a structure of a depth map generation device based on plane prior provided in an embodiment of the present application;

[0084] Fig.17 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0086] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0087] Target scene: A scene is composed of one or more scene elements. A scene element can be a person, a game character, an object, and the environment in which it is located. For example, the game character or environment in the game can be used as scene elements to constitute a game scene. Alternatively, during driving, the vehicle in front and the road test environment can be used as scene elements to constitute a driving scene.

[0088] Stereo matching area: also called patch, refers to a pixel defined on the reference view image, corresponding to a tiny plane in three-dimensional space, which can express depth and normal vector. Figure 1 As shown, for the pixel point Pi in the reference view, its patch in the reference view image is Figure 1 As shown in Qi, the size and shape of Qi can be defined by oneself, for example, Qi can be a plane of size N*N with Pi as the center.

[0089] Pixel feature correlation: also known as normalized cross correlation (NCC), is used to describe the similarity between two patches. The larger the NCC value, the more similar the two patches are. Otherwise, the smaller the NCC value, the less similar the two patches are. In multiple scene images from different perspectives, although the perspectives are different, the scene elements presented are the same, so the pixel features in the presented scene images have certain similarities. When performing depth estimation, the disparity plane of each pixel can be solved based on this characteristic of multi-view images.

[0090] Here, we take the pixel feature as an example. Photometric consistency means that no matter from which perspective a certain point of the target object is observed, the photometric intensity it receives in space is the same. Therefore, the point to be reconstructed must have the same color in the image of each perspective. That is, the core of 3D reconstruction is to restore points with photometric consistency in space. See Figure 1 As shown in , a patch of a certain size is taken with pixel Pi as the center. When the patch is mapped to the scene image of other perspectives, as shown in Figure 1 It is mapped to the patch centered at pi1 in C1, namely qi1. Similarly, it is mapped to qi2 and qi3 in C2 and C3 respectively. Then the similarity of the patch colors can be calculated to determine whether Pi is the same point as pi1, pi2 and pi3 respectively.

[0091] Homography matrix: mainly refers to the plane homography matrix, through which the projection mapping from one plane to another can be realized. The homography matrix can be used to solve two problems. One is to describe the perspective transformation of a plane and its corresponding image in the real world, and the other is to transform the image from one view to another through perspective transformation.

[0092] Disparity plane parameters: also known as plane assumptions. When there are N scene images of different perspectives for the target scene, if we assume that we want to solve the depth map of the i-th scene image, we set the i-th scene image as the reference perspective image (ReferenceImage), and the other images are Source Images. For a pixel coordinate (x0, y0) of the reference perspective image, randomly assign it a plane parameter (a, b, c, d). Then the pixel belongs to the plane ax+by+cz+d=0 in space, then (a, b, c, d) is the plane assumption parameter of the pixel (x0, y0), where d is the depth corresponding to the pixel (x0, y0). According to the plane assumption parameters, the patch with (x0, y0) as the center and N*N as the size can be mapped to images of other perspectives according to the homography transformation. See the mapping results. Figure 1 shown.

[0093] Plane prior conditions: Generally speaking, the process of solving the depth map is to determine the parallax plane parameters of each pixel. There may be some weak texture areas in an image, and the depth cannot be accurately perceived by edge characteristics, such as the above Figure 2aThe plane area of ​​the table in the middle should theoretically have a smooth transition in the depth map, but the actual algorithm is unable to accurately perceive the depth of this area. However, from a practical point of view, since the area belongs to the same plane, plane constraints can be added when solving the depth map, so that these weak texture areas can add plane constraints when selecting the disparity plane parameters, forcing the disparity plane parameters between pixels in the area to meet certain constraints. This plane constraint is the plane prior condition. Generally speaking, strict geometric consistency verification can retain the depth values ​​of some reliable pixels. In addition, plane detection can more accurately detect the plane area in the image, so that based on these reliable pixels and combined with the plane detection results, a triangular mesh can be constructed on the image plane. The triangular mesh constructed in this way can be used as a plane prior condition for depth estimation.

[0094] Triangulation: In geometry, triangulation refers to the subdivision of planar objects into triangles, and by extension, the subdivision of high-dimensional geometric objects into simplexes. For a given set of points, there are many ways to triangulate. Delaunay triangulation is a common triangulation method, which satisfies the following two properties:

[0095] (1) Empty circle characteristic: The Delaunay triangulation is unique, that is, any four points cannot be in a circle, and there will be no other points within the circumscribed circle of any triangle in the Delaunay triangulation.

[0096] (2) Maximizing the minimum angle property: Among the possible triangulations formed by the scattered point set, the minimum angle of the triangle formed by the Delaunay triangulation is the largest. In this sense, the Delaunay triangulation is the "closest to regularization" triangulation. Specifically, it means that when the diagonals of two adjacent triangles form a convex quadrilateral, after they are exchanged, the minimum angle of the six internal angles will no longer increase.

[0097] The embodiments of the present application relate to artificial intelligence and machine learning technologies, and are mainly designed based on computer vision technology in artificial intelligence.

[0098] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0099] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0100] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0101] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0102] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Artificial neural networks (ANN) abstract the human brain neural network from the perspective of information processing, establish a simple model, and form different networks according to different connection methods. A neural network is a computational model composed of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of the artificial neural network. The output of the network varies depending on the network connection method, weight value and activation function. The network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0103] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0104] The solution provided in the embodiment of the present application involves artificial intelligence computer vision technology and machine learning and other technologies. The embodiment of the present application uses a machine learning plane detection model to perform plane detection on the reference view image to obtain plane information in the reference view image, and then combines the plane information to generate a more accurate plane prior condition, and uses the plane prior condition as a constraint to perform depth estimation of the reference view image.

[0105] The following is a brief introduction to the design concept of the embodiments of the present application.

[0106] Multi-view stereo matching technology is a 3D vision technology that uses a stereo matching algorithm to restore the stereo structure from color images of multiple perspectives with known internal and external parameters of the camera. It has achieved relatively good results for depth estimation. However, in weak texture areas, since the pixel differences in the area are small, the pixels in these areas cannot be reliably distinguished, making it impossible to accurately estimate the depth value. Therefore, the depth estimation effect of related technologies in weak texture areas is poor.

[0107] See also Figure 2a to Figure 2dAs shown in the figure, the depth map obtained by using the relevant technology for depth estimation can be clearly seen. Figure 2a The middle round table area is the plane corresponding to the round table. Theoretically, the depth map at the round table position should be a smooth transition, but see Figure 2b As shown in the figure, the estimated depth of the round table area circled in the corresponding box does not match the actual depth. Figure 2c The middle right door section should be flat. Theoretically, the depth of this section should be smoothly transitioned. However, see Figure 2d As shown, the estimated depth of the part circled in the corresponding box obviously does not match the actual depth, which shows that the depth estimation result obtained by the related art is not accurate.

[0108] Considering that Figure 2a The position of the round table belongs to the same plane, so the pixel points in this area are slightly different. By applying a plane constraint to it, the depth values ​​in this area are forced to have a certain constraint relationship. In order to better integrate the plane information, plane detection can be performed on the image to obtain more accurate plane information.

[0109] In view of this, the embodiment of the present application provides a depth estimation method based on plane prior, in which, when performing depth estimation for a target scene, plane information in the reference perspective image is obtained by performing plane detection on the reference perspective image, that is, it is possible to obtain which pixels in the reference perspective image belong to the same plane, and also select a reference pixel set whose pixel feature correlation satisfies the set correlation condition from the reference perspective image by the pixel feature correlation between scene images of different perspectives of the target scene, and then generate a plane prior condition of the reference perspective image based on the obtained plane information and the reference pixel set, and then use the plane prior condition as a constraint to generate a depth map corresponding to the reference perspective image. It can be seen that the embodiment of the present application makes full use of the plane information in the image by introducing plane detection, improves the accuracy of the plane prior condition, and then the depth value of the weak texture area in the depth map solved in combination with the plane prior condition is more accurate.

[0110] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0111] The solution provided in the embodiment of the present application can be applied to most application scenarios involving depth estimation, such as VR, AR, 3D games, 3D film and television works, short videos or autonomous driving. Figure 3As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, an image acquisition device 301, a depth estimation device 302 and a depth map processing device 303 may be included.

[0112] The image acquisition device 301 may be, for example, a camera device for acquiring scene images of a target scene. In a possible implementation, in order to capture scene images from different viewing angles, the image acquisition device 301 may be a binocular camera device or a multi-camera device.

[0113] The depth estimation device 302 is used to implement the depth estimation function, and may include one or more processors 3021, a memory 3022, and an I / O interface 3023 that interacts with the image acquisition device 301. Among them, the memory 3022 of the depth estimation device 302 may also store program instructions of the depth estimation method based on plane prior provided in the embodiment of the present application, and these program instructions can be used to implement the steps of the depth estimation method based on plane prior provided in the embodiment of the present application when executed by the processor 3021, so as to implement the depth estimation process and obtain the corresponding depth map.

[0114] In one implementation, the depth map processing device 303 may be, for example, a display device, which may directly output the depth map obtained by the depth estimation device 302 .

[0115] In one embodiment, the above-mentioned scene can be an autonomous driving scene, and the image acquisition device 301 can be set on the vehicle. By taking images of the scenes around the vehicle and transmitting the images to the depth estimation device 302, after the depth estimation device 302 obtains the depth map, the depth map processing device 303 determines the distance of the surrounding obstacles based on the depth map, and then performs assisted driving.

[0116] In actual application, the depth estimation device 302 may be a computer device, which is installed on the vehicle together with the image acquisition device 301; or, the depth estimation device 302 may be a background server. After the image acquisition device 301 acquires the scene image, it may be transmitted to the depth estimation device 302 via the network to generate a depth map.

[0117] In one implementation, the above-mentioned scene may be an application scene such as VR, AR, 3D games, 3D film and television works, and the image acquisition device 301 may be used to acquire scene images of real scenes, or, if there are multiple scene images of different perspectives in the scene, the image acquisition device 301 may not be provided. Furthermore, the depth estimation device 302 may estimate the depth of each object in the image based on the scene image to obtain a corresponding depth map, and the depth map processing device 303 may restore the 3D structure corresponding to each object based on the depth map.

[0118] The above-mentioned devices can be directly or indirectly connected to each other through one or more networks. The network can be a wired network or a wireless network, for example, the wireless network can be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course, it can also be other possible networks, which are not limited in the embodiments of the present invention.

[0119] It should be noted that the above-mentioned devices can be deployed separately or in combination according to the actual application scenario. For example, one or more depth estimation devices 302 can be deployed, or the depth estimation device 302 and the depth map processing device 303 can also be implemented by the same device.

[0120] Of course, the method provided in the embodiment of the present application is not limited to Figure 3 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 3 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0121] The method flow provided in each embodiment of the present application can be Figure 3 The depth estimation device 302 in FIG. Figure 4 , which is a flow chart of a depth estimation method based on plane prior provided in an embodiment of the present application.

[0122] Step 401: Acquire multiple scene images of a target scene shot from different viewing angles.

[0123] In an embodiment of the present application, in order to more accurately generate a depth map of a target scene, a multi-view stereo matching method is adopted to generate a depth map of a reference view based on scene images of a plurality of different viewpoints of the target scene.

[0124] In one embodiment, such as in an autonomous driving scenario, multiple scene images around the vehicle, such as multiple scene images in front of the vehicle, can be captured by a multi-camera device installed on the vehicle. Generally speaking, multi-camera devices have a certain spatial distance in their layout, thereby generating parallax to obtain scene images from multiple different angles.

[0125] In another embodiment, in scenarios involving 3D modeling to generate corresponding 3D stereograms, such as AR / VR, 3D games, 3D film and television works, etc., a mobile camera device can be used to capture multiple frames of scene images from different perspectives for real scenes or constructed physical model scenes.

[0126] Of course, other possible methods may also be used to acquire scene images, and the embodiments of the present application do not limit this.

[0127] Step 402: Perform plane detection on a reference view image in a plurality of scene images to obtain plane information in the reference view image; wherein the plane information is used to indicate pixel points belonging to the same plane.

[0128] In an embodiment of the present application, in order to improve the accuracy of the depth map finally generated, plane detection is performed on the reference perspective image to extract plane information in the image, and then the plane information is fully utilized to generate a plane prior, so that a relatively good depth estimation effect can still be achieved in the weak texture area in the reference perspective image.

[0129] Specifically, plane estimation refers to detecting pixels in a scene image that belong to the same plane, and then outputting which pixels in the scene image belong to the same plane.

[0130] In one embodiment, when performing plane detection on a reference perspective image, plane detection can be performed only based on the reference perspective image to obtain plane information in the reference perspective image. This method has relatively low computational consumption and can output plane information faster, thereby improving the efficiency of the entire depth estimation process.

[0131] In another embodiment, considering that the scene images of other perspectives still contain the image feature information in the reference perspective image, the scene images of other perspectives can also be referred to to comprehensively obtain the plane information in the reference perspective image. For example, the scene images of other perspectives can be subjected to plane detection, thereby combining the plane detection results of each scene image, or the image features of the scene images of other perspectives can be fused into the corresponding image features of the reference perspective image in the image feature extraction stage. The corresponding image features can be corresponding pixel features or corresponding regional features. This method of fusing the image information of multiple perspective images can improve the accuracy of the plane information, and correspondingly improve the accuracy of the depth estimation results.

[0132] In practical applications, a plane detection method may be used. For example, the plane detection method may be a 3D Hough transform point cloud plane detection algorithm or a regional plane detection method based on convolutional neural network features (plane Regions with CNN features, plane RCNN). Of course, other possible plane detection methods may also be used, and the embodiments of the present application do not limit this.

[0133] Below, plane detection of the reference view image using a plane detection model based on one of the plane detection methods is introduced as an example.

[0134] In actual applications, the plane detection model needs to be pre-trained using machine learning methods until the convergence conditions are reached and can be applied to the actual plane detection process.

[0135] In one embodiment, in the machine learning stage, a supervised learning-based method can be used for model training. Specifically, multiple training samples can be collected, each of which can include a training image and a training label of the training image. The training label can be used to indicate whether each image area in the training image is a plane, and then the trained plane detection model can accurately detect which areas in an image are planes and which areas are non-planes. Of course, the training labels can be appropriately adjusted based on actual training needs, and the embodiments of the present application do not limit this.

[0136] In one implementation, during the machine learning stage, a self-supervised learning method may be used for model training. Specifically, multiple training samples may be acquired, each of which may include a training image. During the training process, whether each region in the image belongs to the same plane may be identified based on the correlation between the images in the training image.

[0137] After the plane detection model is obtained through machine learning training, the plane detection model can be applied to the plane detection model. Figure 5 As shown, it is a flow chart of the plane detection process.

[0138] S4021: Perform image semantic segmentation on the reference view image to obtain plane masks corresponding to each plane area in the reference view image.

[0139] In an embodiment of the present application, the plane detection model includes a semantic segmentation part, which is used to perform image semantic segmentation on the reference perspective image to obtain plane masks corresponding to each plane area in the reference perspective image.

[0140] Among them, the semantic segmentation part can adopt a mask-CNN network structure, for example, and of course other possible network structures can also be adopted, and the embodiments of the present application do not limit this.

[0141] S4022: Perform depth estimation processing on the reference view image to obtain an estimated depth map of the reference view image.

[0142] Specifically, in order to estimate the depth map of the entire image, a decoder can be added after the feature pyramid networks (FPN) of Mask R-CNN. Each layer of this depth map decoder uses a convolution kernel of size 3x3 and stride 1 and a convolution kernel of size 4x4 and stride 2. Finally, the bilinear upsampling method is used to obtain a depth map of the same size as the input image.

[0143] S4023: Based on the estimated depth map, the plane mask is updated to obtain the plane information of the reference view image.

[0144] In specific implementation, the plane mask obtained by the above semantic segmentation is only a rough plane estimation, and there are large errors in its edge parts. Therefore, the plane mask obtained by the above process can be optimized and updated in combination with the estimated depth map to obtain the reference perspective image plane information more accurately.

[0145] Step 403: Based on the pixel feature correlations between the multiple scene images, a reference pixel point set whose pixel feature correlations satisfy a set correlation condition is determined from the pixels included in the reference view image.

[0146] In the embodiment of the present application, since multiple scene images are all obtained by shooting the target scene, the same object exists in these multiple scene images, which makes there a certain correlation between them. Therefore, the corresponding pixel points in these multiple scene images can be located based on this correlation. The similarity between the corresponding pixel points in different scene images is high, that is, the pixel points that are more likely to belong to the same pixel points can be used as reference pixel points.

[0147] Specifically, a multi-view stereo matching method can be used to determine the optimal plane parameters for each pixel, that is, to obtain the plane with the minimum cost for the pixel, so as to determine a reference pixel set whose pixel feature association degree meets the set association conditions from each pixel according to the optimal plane parameters. Alternatively, the similarity between the pixel features in each scene image can be calculated to find the similarity pixel with the maximum similarity for each pixel, and determine whether the pixel is a reference pixel according to the similarity between the pixel and the similar pixel.

[0148] Next, we take multi-view stereo matching as an example. The basic idea of ​​multi-view stereo matching is to randomly initialize a plane hypothesis parameter for each pixel, and then replace the current plane hypothesis parameter with its surrounding plane hypothesis parameters to calculate the cost. The plane hypothesis parameter with the minimum cost is used as the new plane hypothesis parameter of the pixel. Generally speaking, the cost can be calculated using photometric consistency. It mainly includes four basic steps, namely random initialization, propagation, view selection and refinement.

[0149] See also Figure 6 , which is a schematic diagram of a process of determining a reference pixel point set from individual pixel points.

[0150] Step 4031: Initialize parameters of each pixel point respectively to obtain initial disparity plane parameters corresponding to each pixel point.

[0151] Specifically, a random initialization method may be used to randomly assign a disparity plane to each pixel point. By randomly assigning values, it is expected that at least one pixel point in each plane region can be randomly assigned a correct disparity plane parameter.

[0152] Of course, any other possible initialization method may also be used, such as assigning values ​​according to pixel values ​​in a reference viewing angle image, and the embodiments of the present application are not limited to this.

[0153] Step 4032: for a pixel point, determine a stereo matching region of each pixel point in the reference view image, each stereo matching region includes an image region within a set range with each pixel point as a reference point.

[0154] In the embodiment of the present application, the stereo matching area corresponding to each pixel point, that is, patch refers to an image area of ​​size N*N centered on the pixel point, and the value of N can be set according to actual needs and is not limited to this.

[0155] Step 4033: Based on the pixel feature correlation between the reference view image and the corresponding areas in other scene images, the initial disparity plane parameters corresponding to each pixel point are respectively updated multiple times to obtain the estimated disparity plane parameters corresponding to each pixel point.

[0156] In the embodiment of the present application, after the initialization process is completed, the parameters can be propagated according to the current disparity plane parameters of each pixel point, that is, the initial disparity plane parameters corresponding to each pixel point are respectively subjected to multiple iterative update processes, and in the iterative update process, a better disparity plane parameter is selected, and finally the estimated disparity plane parameters corresponding to each pixel point are obtained. Among them, an iterative update process can be performed in a specified pixel point order to complete the iterative update process of all pixel points.

[0157] In one implementation, starting from the first pixel in the upper left corner of the reference view image, the disparity plane parameters may be iteratively updated for each pixel one by one. Figure 7a As shown, it is a schematic diagram of a traversal method. When performing iterative update, it can start from the first pixel in the upper left corner, and traverse each pixel from left to right one by one to iteratively update the disparity plane parameters until the traversal of the first row of pixels is completed, and then traverse the next row of pixels until the iterative update of the disparity plane parameters of all pixels is completed.

[0158] Of course, the above-mentioned traversal direction is only one possible direction. In specific implementation, other traversal directions can also be set as long as the iterative update of the parallax plane parameters of all pixels can be completed. The embodiment of the present application does not limit this.

[0159] In one embodiment, in order to facilitate the graphics processing unit (GPU) to perform multi-channel parallel processing on the image, so as to fully utilize the processing resources of the GPU and improve the efficiency of image processing, the reference view image can also be divided into N channel images according to the set number of processing channels N, and any two adjacent pixels included in each channel image are separated by N-1 pixels from their original positions in the reference view image.

[0160] See also Figure 7b and Figure 7c As shown in FIG. 1 , it is a schematic diagram of another traversal method, where the number of channels N is 2 as an example. Figure 7b or Figure 7c As shown, all pixels included in the reference perspective image can be divided into two channels, and the vector pixels of each channel are spaced one pixel apart in the original image. Figure 7b or Figure 7c The dots of different colors in represent the pixels of different channels. Then on each channel, Figure 7b or Figure 7c As shown by the middle arrow, each pixel point can be traversed row by row starting from the first pixel point in the upper left corner to perform an iterative update process for each pixel point.

[0161] In the following, since the iterative update of each pixel is similar, the iterative update process of one pixel Pi is taken as an example for description. Figure 8 , which is a flowchart of the iterative update process of pixel point Pi.

[0162] Step 40331: Obtain multiple sampling pixel points from the pixel points in a set area around the pixel point Pi.

[0163] In the embodiment of the present application, when different traversal methods are adopted, the sampling methods are also correspondingly different to obtain a sampling point set S corresponding to the pixel point Pi, where S = (s1, s2, ..., sn).

[0164] In one embodiment, when using Figure 7a When the traversal method shown in FIG. 1 is used, pixel sampling can be performed directly from the pixel points in the set area around the pixel point in the reference view image to obtain multiple sampled pixel points. Figure 9a FIG. 1 is a schematic diagram of a sampling pixel point set, wherein a sampling area of ​​a reference pixel point set can be preset. Figure 9a The central black dot represents the pixel Pi, and the surrounding white dots are the pixels corresponding to the set sampling area.

[0165] In one embodiment, when using Figure 7b or Figure 7c When the traversal method shown in FIG. 1 is used, pixel sampling can be performed from the pixel points in the surrounding set area of ​​the channel image where the pixel point Pi is located to obtain multiple sampled pixel points. Figure 9b As shown, it is a schematic diagram of another set of sampling pixel points, where: Figure 9b The central black dot represents the pixel Pi, and the surrounding white dots are the pixels corresponding to the set sampling area. It can be seen that there is a pixel interval between the sampled pixels.

[0166] Step 40332: Determine the iterative update loss value of the pixel point Pi based on the correlation between the stereo matching area corresponding to the pixel point Pi and the pixel features in each corresponding mapping area.

[0167] In an embodiment of the present application, the patch can be transformed into an image of any perspective through the homography transformation matrix between various perspectives. Based on this characteristic, the patch corresponding to the pixel point Pi can be determined based on the current disparity plane parameters corresponding to the pixel point Pi and the homography transformation matrix between various perspectives, and the mapping patch in other scene images.

[0168] For details, please refer to Figure 1 As shown in , a patch of a certain size is taken with pixel Pi as the center. When the patch is mapped to the scene image of other perspectives, as shown in Figure 1 The patch centered on pi1 in C1 is mapped to qi1. Similarly, it is mapped to qi2 and qi3 in C2 and C3 respectively.

[0169] Theoretically speaking, if the current disparity plane parameters corresponding to Pi are accurate enough, then the mapped pi1~pi3 should be the same pixel points of Pi in the scene images of other perspectives, and the corresponding patches are also the same areas, so their similarity should be high enough. Therefore, the similarity between patches can be calculated to determine whether Pi is the same point as pi1, pi2 and pi3 respectively.

[0170] Specifically, based on the stereo matching area corresponding to the pixel point Pi and the pixel feature values ​​in each mapping area, the pixel feature correlation between the stereo matching area and each mapping area can be determined accordingly, and the pixel feature correlation is used to characterize the similarity between two patches.

[0171] In one embodiment, the pixel feature may be a photometric feature. Photometric consistency means that no matter from which perspective a certain point of the target object is observed, the spatial photometric intensity is the same. Therefore, the point to be reconstructed must have the same color in the image of each perspective. Thus, the similarity of the patch color can be calculated to determine whether Pi is the same point as pi1, pi2, and pi3. Therefore, the pixel feature correlation between two patches can be calculated in the following way:

[0172]

[0173] Among them, NCC represents the pixel feature correlation between two patches, f and t are two different patches, μ f and μ t is the average of the pixel features in patch f and patch t, ​​σ f and σ t are the standard deviations of the pixel features corresponding to patch f and patch t, ​​respectively. Patch f is the patch in the reference view image, and patch t is the mapping patch in the scene image of other views.

[0174] Through the above process, the correlation degrees of each pixel feature between the patch in the reference perspective image and the mapped patch in the scene image of other perspectives can be calculated respectively, and then the iterative update loss value corresponding to the pixel point Pi can be determined based on the obtained correlation degrees of each pixel feature, wherein the iterative update loss value is negatively correlated with the pixel feature correlation degree.

[0175] Specifically, a certain weight value may be assigned to each perspective, and then the pixel feature correlation degree corresponding to the perspective with the largest weight value is selected from the above-mentioned pixel feature correlation degrees according to the weight value of each perspective, and the iterative update loss value corresponding to the pixel point Pi is determined according to the pixel feature correlation degree. Alternatively, a weighted summation may be performed according to the weight value to determine the iterative update loss value corresponding to the pixel point Pi according to the pixel feature correlation degree obtained by the weighted summation. It should be noted that the weight values ​​of each perspective corresponding to different pixels may be different.

[0176] In one implementation, the iterative update loss value cost may be calculated in the following manner:

[0177] m(pi,l)=1-NCC

[0178] Wherein, l is the current disparity plane parameter corresponding to pixel pi, function m is the cost function, that is, the cost of pixel p when the current disparity plane is l, and NCC in the above formula is the above-mentioned pixel feature correlation degree finally obtained. Of course, other methods can also be used to calculate the cost, and the embodiment of the present application does not limit this.

[0179] Step 40333: Based on the correlation between the stereo matching areas corresponding to the multiple sampling pixels and the pixel features in the corresponding mapping areas, determine the iterative update loss values ​​corresponding to the multiple sampling pixels respectively.

[0180] Similar to the calculation process of the iterative update loss value of the above pixel point Pi, the iterative update loss values ​​corresponding to other sampling pixel points can be calculated by the same logic. Therefore, the acquisition process of other sampling pixel points will not be repeated here.

[0181] Step 40334: If there is a target sampling pixel point among the multiple sampling pixel points whose iterative update loss value is less than the iterative update loss value corresponding to the pixel point Pi, the current disparity plane parameters corresponding to the target sampling pixel point are used to update the current disparity plane parameters of the pixel point Pi.

[0182] In the embodiment of the present application, the propagation of parameters is to update the current disparity plane parameters of the pixel itself or the neighboring pixel with the disparity plane parameters of the pixel itself. The goal is to find the disparity plane parameters with the minimum cost for each pixel, that is, the objective function f pi for:

[0183]

[0184] Wherein, F is a disparity plane parameter set, for example, it may be a set consisting of the sampling point set S and the current disparity plane parameters corresponding to the pixel point Pi.

[0185] Specifically, for each sampling pixel point in the sampling pixel point set, determine whether the following results are true:

[0186] m(pi,l)>m(si,li)

[0187] Among them, si is one of the sampling pixel points in the sampling pixel point set, and li is the current disparity plane parameter corresponding to si. If the above formula is established, it means that li is more suitable for the current pixel point pi. The current disparity plane parameter of the pixel point Pi is updated with the current disparity plane parameter corresponding to the target sampling pixel point, that is, the current disparity plane parameter of pi is updated to li.

[0188] Step 4034: Based on the obtained estimated disparity plane parameters corresponding to each pixel point, determine the stereo matching area corresponding to each pixel point, and the mapping area in the other scene image.

[0189] In an embodiment of the present application, the above-mentioned iterative update process can be performed multiple times until the convergence condition is met. The convergence condition can be that the number of iterative updates reaches a preset number, or that the update rate of the disparity plane parameters meets certain conditions. For example, in one iterative update process, if the disparity plane parameters of 90% of the pixels no longer change, it can be considered that the convergence condition is met, and the current disparity plane parameters finally obtained are the estimated disparity plane parameters.

[0190] Furthermore, based on the estimated disparity plane parameters corresponding to each pixel point and the homography transformation matrix between each perspective, the patch of each pixel point in the reference perspective image can be mapped to the scene image of other perspectives to obtain the mapping patch in the other scene image.

[0191] Step 4035: respectively determine the mapping area of ​​the stereo matching area in other scene images and the pixel feature correlation between the stereo matching area; wherein the other scene images are scene images other than the reference view image among the multiple scene images.

[0192] Among them, the calculation method of the pixel feature correlation degree corresponding to each pixel point can be the same as that in the above-mentioned iterative update process, so please refer to the description of the above-mentioned iterative update process, and no further details will be given here.

[0193] Step 4036: From each pixel point, determine a reference pixel point set whose pixel feature correlation degree satisfies a set correlation condition.

[0194] Specifically, for each pixel point pi, if the pixel feature correlation m(pi, ​​l) corresponding to the pixel point is greater than the set correlation threshold, it is considered that the patch of the pixel point pi is very similar to a patch of other perspectives, and the accuracy of the estimated disparity plane parameters corresponding to the pixel point pi is very high. The estimated disparity plane parameters of such pixel points have high reliability and can be used as reference pixel points to obtain a reference pixel point set. Usually, there are not many such pixel points, so they can also be called sparse reliable points.

[0195] Step 404: Based on the plane information and the reference pixel point set, obtain the plane prior condition of the reference viewing angle image.

[0196] In an embodiment of the present application, the reference perspective image can be plane-divided based on a set of reference pixel points, and the plane information can be combined to determine which pixels in the reference perspective image belong to the same plane, thereby making the plane prior conditions obtained in the reference perspective image plane-dividing process more accurate.

[0197] See also Fig.10 As shown in FIG. 1 , it is a schematic diagram of the process of obtaining plane prior based on plane information and a reference pixel set, wherein the reference pixel points are extracted from the reference view image to obtain the following Fig.10 The reference pixel set shown, Fig.10 The depth map of each pixel point finally obtained based on the above iterative update process is shown in FIG. In addition, the plane detection of the reference view image can be performed through planeRCNN to obtain the following: Fig.10 The plane information shown in FIG. 1 divides the image areas into planes, and then the depth map can be optimized by combining the plane information to obtain the following: Fig.10 From the plane prior shown, it can be clearly felt that after the optimized plane prior, the depth in the plane area is better filtered, which is more consistent with the actual situation in the image and has higher accuracy.

[0198] In one embodiment, see Fig.11 , which is a schematic diagram of a process for obtaining a plane prior condition of a reference viewing angle image.

[0199] Step 4041a: Based on the reference pixel point set, a triangulation method is used to perform plane fitting processing to obtain a triangulated mesh structure, where each triangular plane in the triangulated mesh structure includes three reference pixel points in the reference pixel point set.

[0200] For details, see Fig.12As shown, each pixel in the reference pixel set can be used as a vertex and the Delaunay triangulation method can be used to obtain the triangular mesh structure shown on the right, where each triangular plane includes three reference pixel points in the reference pixel set.

[0201] Step 4042a: Based on the plane information, the triangular planes corresponding to the reference pixel points located on the same plane in the triangular mesh structure are merged to obtain a merged triangular mesh structure.

[0202] See also Fig.12 As shown, every three reference pixels form a triangular plane, and different triangular planes belong to different disparity planes. In the actual reference perspective image, there may be multiple reference pixels belonging to the same plane, which makes the above triangular mesh structure fitting result not in line with the actual situation. Therefore, in order to alleviate the problem of inaccurate final depth estimation caused by this situation, the triangular mesh structure is corrected by introducing the plane detection result.

[0203] Specifically, through the plane information obtained by plane detection, it can be known which pixel points belong to the same plane. Similarly, it can also be known which of the above-mentioned reference pixel points belong to the same plane. Therefore, if there are reference pixel points that should belong to the same plane, but belong to multiple triangular planes, then these triangular planes should be merged. For example, re-plane fitting can be performed based on these triangular pixel points belonging to the same plane to update the triangular mesh structure, so as to obtain a merged triangular mesh structure.

[0204] As shown in FIG12 , based on the plane information, it is determined that the five reference pixels in the dotted box should belong to the same plane. However, the triangulated mesh structure obtained by the above triangulation can clearly show that these reference pixels are not placed in the same plane. Therefore, these triangulated planes can be merged to obtain Fig.12 The triangular mesh structure after merging.

[0205] Step 4043a: Based on the merged triangular mesh structure, obtain a plane prior condition.

[0206] Specifically, based on the above merged triangular mesh structure, corresponding plane parameters are obtained as plane prior conditions.

[0207] In another embodiment, see Fig.13 As shown, it is a schematic diagram of another process of obtaining the plane prior condition of the reference viewing angle image.

[0208] Step 4041b: Based on the plane information, determine multiple reference pixel point groups belonging to the same plane in the reference pixel point set, each reference pixel point group includes at least one reference pixel point.

[0209] Specifically, the plane information specifies which pixels belong to the same plane. Then, based on the plane information, it can be determined which reference pixels belong to the same plane, and multiple reference pixels belonging to the same plane can be merged into the same reference pixel group to obtain multiple reference pixel groups, each reference pixel group including one or more reference pixel groups.

[0210] Step 4042b: Perform plane fitting processing on the obtained multiple reference pixel point groups respectively to obtain a plane combination structure composed of a corresponding multiple fitting planes; wherein each reference pixel point group corresponds to a fitting plane among the multiple fitting planes.

[0211] For each reference pixel point group, a plane fitting process may be performed based on the reference pixel points included therein to obtain a fitting plane corresponding to the reference pixel point group, thereby obtaining a plane combination structure composed of multiple fitting planes.

[0212] Step 4043b: Based on the plane combination structure, obtain the plane prior condition.

[0213] Specifically, based on the above merged triangular mesh structure, corresponding plane parameters are obtained as plane prior conditions.

[0214] Step 405: Taking the plane prior condition as a constraint, based on the pixel feature correlation between multiple scene images, a depth map of the reference view image is obtained.

[0215] In the embodiment of the present application, the process of step 405 is similar to the process of step 403 described above, except that a plane prior condition is added as a constraint during the iterative update process.

[0216] See also Fig.14 , which is a schematic diagram of the process of obtaining a depth map of a reference viewing angle image.

[0217] Step 4051: Initialize parameters of each pixel point respectively to obtain initial disparity plane parameters corresponding to each pixel point.

[0218] Specifically, when initializing parameters, a random initialization method may be used, or initialization may be performed based on the plane information indicated by the above-mentioned plane priori condition.

[0219] Step 4052: Based on the plane priori condition and the pixel feature correlation, the current disparity plane parameters corresponding to each pixel point are updated multiple times through iterations to obtain the target disparity plane parameters corresponding to each pixel point.

[0220] Similarly, according to the introduction of step 403 above, starting from the designated pixel of the reference view image, the iterative update process can be performed on each pixel one by one in a traversal manner, so that the disparity plane parameters of each pixel are sampled and propagated, and the disparity plane parameters of each pixel are updated. The difference from step 403 above is that the cost at this time is a combination of photometric consistency and plane prior. The traversal method can also refer to the introduction of step 403 above, and will not be repeated here.

[0221] Here again, taking the pixel point Pi as an example, a plurality of sampling pixel points may be obtained from the pixel points in a set area around the pixel point Pi.

[0222] Furthermore, based on the plane where each pixel point in the plane prior condition is located, and the correlation between the corresponding stereo matching area and the pixel features in each mapping area, the iterative update loss values ​​corresponding to the pixel point Pi and multiple sampling pixel points can be determined respectively.

[0223] Specifically, for each pixel point, the stereo matching area of ​​the pixel point in the reference perspective image can be determined, and based on the current disparity plane parameters and the homography transformation matrix between each perspective, the stereo matching area is mapped to the scene image of other perspectives to determine the mapping area in the scene image of other perspectives.

[0224] In the first round of iterative updating process, the current disparity plane parameters are the initial disparity plane parameters, and in the subsequent iterative updating process, they are the disparity plane parameters obtained after the last iterative updating.

[0225] Through the above process, the correlation degrees of each pixel feature between the patch in the reference perspective image and the mapped patch in the scene image of other perspectives can be calculated respectively, and then the iterative update loss value corresponding to the pixel point Pi can be determined based on the obtained correlation degrees of each pixel feature and the plane where the pixel point Pi is located in the plane prior condition, where the iterative update loss value is negatively correlated with the pixel feature correlation degree.

[0226] In one implementation, the iterative update loss value cost may be calculated in the following manner:

[0227]

[0228] Among them, m(pi, ​​l) represents the cost part obtained based on the correlation of pixel features, α and γ are both constants, and d i and n i The current disparity plane parameter used to characterize the pixel Pi, d i Indicates the depth value of the pixel Pi in the current parallax plane parameter, n iIndicates the normal vector of the pixel point Pi corresponding to the current parallax plane, that is, the vector expression corresponding to (a, b, c) in the current parallax plane parameters; d p and n p The plane prior condition used to characterize the pixel Pi, d p Represents the depth value of the pixel Pi in the plane prior condition, n p Indicates the normal vector of the plane prior condition corresponding to the pixel point Pi, that is, the vector expression corresponding to (a, b, c) in the plane prior condition, λ d Represents the corresponding coefficient of the depth difference between the current disparity plane parameter and the plane prior condition, λ n The corresponding coefficient representing the difference between the current parallax plane parameters and the normal vector of the plane prior condition.

[0229] Similar to the calculation process of the iterative update loss value of the above pixel point Pi, the iterative update loss values ​​corresponding to other sampling pixel points can be calculated by the same logic. Therefore, the acquisition process of other sampling pixel points will not be repeated here.

[0230] After obtaining each iterative update loss value, if among multiple sampling pixel points, there is a target sampling pixel point whose corresponding iterative update loss value is less than the iterative update loss value corresponding to a pixel point, the current disparity plane parameters of the pixel point Pi are updated according to the current disparity plane parameters corresponding to the target sampling pixel point.

[0231] In the embodiment of the present application, the above-mentioned iterative updating process may be performed multiple times until a convergence condition is satisfied, so that the current disparity plane parameters finally obtained are the target disparity plane parameters.

[0232] Step 4053: Based on the obtained target disparity plane parameters, the depth values ​​corresponding to the respective pixel points are determined to obtain a depth map corresponding to the reference viewing angle image.

[0233] Specifically, the obtained target disparity plane parameters corresponding to each pixel point include plane parameters (a, b, c, d), that is, the pixel point belongs to the plane ax+by+cz+d=0 in space, where d is the depth value corresponding to the pixel point.

[0234] It should be noted that the execution order between the above steps is not strictly in accordance with the above order, and can be adjusted according to actual conditions in practical applications, that is, it can be executed in the above order, or the above multiple steps can be executed simultaneously. The embodiments of the present application do not limit this.

[0235] See also Fig.15a and Fig.15b As shown, the depth map obtained by the method of the embodiment of the present application is Fig.15a For Figure 2aThe image shown is the depth map obtained from the reference view image. Fig.15b For Figure 2c The image shown is the depth map obtained from the reference view image. It can be clearly seen that Fig.15a The round table area in Figure 2b For the depth, there is a very obvious transition effect, which is more consistent with the actual depth. Fig.15b In the selected area, compared with Figure 2d In terms of depth, the depth is more consistent with the actual depth situation. It can be seen that the technical solution of the embodiment of the present application can effectively increase the depth estimation effect in the weak texture area.

[0236] In the embodiment of the present application, the depth map corresponding to the reference perspective image can be obtained through the above process. Similarly, the above process can also be used to obtain the depth map of any scene image from other perspectives, so that the obtained depth map can be used in downstream applications.

[0237] In one implementation, three-dimensional modeling may be performed on the target scene based on each obtained depth map to obtain a three-dimensional stereogram corresponding to the target scene.

[0238] In one embodiment, the obstacle distances around the target vehicle can also be determined based on the obtained depth maps, and autonomous driving can be performed based on the obtained obstacle distances.

[0239] In summary, in the embodiments of the present application, by introducing plane detection and fully mining the plane information in the image, the plane prior depth map can be used to recover the information of the extremely weak texture area, and the quality of depth estimation in the weak texture area can be better improved. This method can be applied to AR / VR, 3D games, 3D film and television works, short videos, autonomous driving, free viewpoints and other application scenarios, and can effectively assist in restoring better 3D structures, generate more beautiful results, improve product experience, and improve product user experience.

[0240] See also Fig.16 Based on the same inventive concept, the embodiment of the present application further provides a depth map generating device 160 based on plane prior, the device comprising:

[0241] The plane detection unit 1601 is used to obtain multiple scene images of the target scene shot based on different viewing angles, and perform plane detection on the reference viewing angle image in the multiple scene images to obtain plane information in the reference viewing angle image; wherein the plane information is used to indicate pixel points belonging to the same plane;

[0242] The pixel screening unit 1602 is used to determine, from the pixels included in the reference view image, a reference pixel point set whose pixel feature correlation degree meets a set correlation condition based on the pixel feature correlation degree between the multiple scene images;

[0243] A plane priori generating unit 1603, configured to obtain a plane priori condition of a reference viewing angle image based on plane information and a reference pixel point set;

[0244] The depth estimation unit 1604 is used to obtain a depth map of the reference view image based on the pixel feature correlation between multiple scene images with the plane prior condition as a constraint.

[0245] Optionally, the plane priori generating unit 1603 is specifically used for:

[0246] Based on the reference pixel point set, a triangulated mesh method is used to perform plane fitting processing to obtain a triangulated mesh structure, wherein each triangular plane in the triangulated mesh structure includes three reference pixel points in the reference pixel point set;

[0247] Based on the plane information, the triangular planes corresponding to the reference pixel points located on the same plane in the triangular mesh structure are merged to obtain a merged triangular mesh structure;

[0248] Based on the merged triangular mesh structure, the plane prior condition is obtained.

[0249] Optionally, the plane priori generating unit 1603 is specifically used for:

[0250] Based on the plane information, determine multiple reference pixel point groups belonging to the same plane in the reference pixel point set, each reference pixel point group includes at least one reference pixel point;

[0251] Performing plane fitting processing on the obtained multiple reference pixel point groups respectively to obtain a plane combination structure composed of corresponding multiple fitting planes; wherein each reference pixel point group corresponds to a fitting plane among the multiple fitting planes;

[0252] Based on the plane combination structure, the plane prior condition is obtained.

[0253] Optionally, the plane detection unit 1601 is specifically used for:

[0254] Performing image semantic segmentation on the reference view image to obtain plane masks corresponding to each plane area in the reference view image;

[0255] Performing depth estimation processing on the reference perspective image to obtain an estimated depth map of the reference perspective image;

[0256] Based on the estimated depth map, the plane mask is updated to obtain the plane information of the reference view image.

[0257] Optionally, the pixel screening unit 1602 is specifically configured to:

[0258] For each pixel, perform the following steps respectively:

[0259] For a pixel point, determine a stereo matching area of ​​the pixel point in the reference view image, each stereo matching area includes an image area of ​​a set range with the pixel point as a reference point;

[0260] Determine respectively the mapping area of ​​the stereo matching area in other scene images and the pixel feature correlation between the stereo matching area; wherein the other scene images are scene images other than the reference view image among the multiple scene images;

[0261] From each pixel point, a reference pixel point set whose pixel feature correlation degree meets the set correlation condition is determined.

[0262] Optionally, the pixel screening unit 1602 is further configured to:

[0263] Initialize the parameters of each pixel point respectively to obtain the initial disparity plane parameters corresponding to each pixel point;

[0264] Based on the pixel feature correlation of the reference view image and the corresponding area of ​​the other scene images, the initial disparity plane parameters corresponding to each pixel point are updated multiple times to obtain the estimated disparity plane parameters corresponding to each pixel point;

[0265] Based on the obtained estimated disparity plane parameters, the stereo matching area corresponding to each pixel point is determined, and the mapping area in other scene images is determined.

[0266] Optionally, the pixel screening unit 1602 is specifically configured to:

[0267] Obtain multiple sampling pixel points from the pixel points in a set area around a pixel point;

[0268] Determine an iterative update loss value of a pixel based on the correlation between the stereo matching area corresponding to the pixel and the pixel features in each corresponding mapping area;

[0269] Based on the correlation between the stereo matching areas corresponding to the multiple sampling pixels and the pixel features in the corresponding mapping areas, respectively determine the iterative update loss values ​​corresponding to the multiple sampling pixels;

[0270] If, among the multiple sampling pixels, there is a target sampling pixel whose iterative update loss value is less than the iterative update loss value corresponding to a pixel, the current disparity plane parameter of the pixel is updated with the current disparity plane parameter corresponding to the target sampling pixel.

[0271] Optionally, the pixel screening unit 1602 is specifically configured to:

[0272] Based on the current disparity plane parameters corresponding to a pixel point and the homography transformation matrix between various perspectives, determine the stereo matching area corresponding to a pixel point and the mapping area in other scene images;

[0273] Based on the pixel feature values ​​in the stereo matching region and each mapping region, correspondingly determining the pixel feature correlation between the stereo matching region and each mapping region;

[0274] Based on the obtained correlation degree of each pixel feature, an iterative update loss value corresponding to a pixel point is determined, wherein the iterative update loss value is negatively correlated with the pixel feature correlation degree.

[0275] Optionally, the depth estimation unit 1604 is specifically configured to:

[0276] Based on the plane prior conditions and the pixel feature correlation, the current disparity plane parameters corresponding to each pixel point are updated multiple times to obtain the target disparity plane parameters corresponding to each pixel point;

[0277] Based on the obtained target disparity plane parameters, the depth value corresponding to each pixel point is determined to obtain a depth map.

[0278] Optionally, the depth estimation unit 1604 is specifically configured to:

[0279] Starting from the specified pixel point of the reference view image, the following steps are performed for each pixel point one by one in a traversal manner:

[0280] For a pixel point, multiple sampling pixel points are obtained from the pixel points in a set area around the pixel point;

[0281] Based on the plane where each pixel point in the plane prior condition is located, and the correlation between the corresponding stereo matching area and the pixel features in each mapping area, the iterative update loss value corresponding to a pixel point and multiple sampling pixels is determined respectively;

[0282] If, among multiple sampling pixels, there is a target sampling pixel whose corresponding iterative update loss value is less than the iterative update loss value corresponding to a pixel, the current disparity plane parameter of the pixel is updated according to the current disparity plane parameter corresponding to the target sampling pixel.

[0283] Optionally, the pixel screening unit and the depth estimation unit are further specifically used for:

[0284] According to the set number of processing channels N, the reference view image is divided into N channel images; wherein any two adjacent pixels included in each channel image are separated by N-1 pixels from their original positions in the reference view image;

[0285] A plurality of sampling pixel points are obtained from the pixel points in a surrounding set area included in the channel image where the pixel point is located.

[0286] Through the above-mentioned device, by introducing plane detection, the plane information in the image can be fully mined, and the depth map can be solved as a plane priori, so as to restore the information of the extremely weak texture area, better improve the quality of depth estimation in the weak texture area, and effectively assist in restoring a better 3D structure, generate more beautiful results, improve the product experience, and improve the user experience of the product.

[0287] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0288] See also Fig.17 Based on the same technical concept, the present application embodiment also provides a computer device 170, which can be Figure 3 In the depth estimation device shown, the computer device 170 may include a memory 1701 and a processor 1702 .

[0289] The memory 1701 is used to store computer programs executed by the processor 1702. The memory 1701 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the computer device, etc. The processor 1702 may be a central processing unit (CPU), or a digital processing unit, etc. The specific connection medium between the memory 1701 and the processor 1702 is not limited in the embodiments of the present application. The embodiments of the present application are Fig.17 In the embodiment, the memory 1701 and the processor 1702 are connected via a bus 1703. The bus 1703 is connected to the processor 1702 via a bus 1703. Fig.17 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 1703 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.17 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0290] The memory 1701 may be a volatile memory, such as a random-access memory (RAM); the memory 1701 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1701 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1701 may be a combination of the above memories.

[0291] The processor 1702 is used to execute the method executed by the device in each embodiment of the present application when calling the computer program stored in the memory 1701.

[0292] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to the various exemplary implementations of the present application described above in this specification. For example, the computer device can execute the method executed by the device in each embodiment of the present application.

[0293] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0294] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0295] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating a depth map based on a plane prior, characterized in that: The method comprises: Acquire multiple scene images of a target scene shot from different perspectives, and perform plane detection on a reference perspective image among the multiple scene images to obtain plane information in the reference perspective image; wherein the plane information is used to indicate pixel points belonging to the same plane; Based on the pixel feature correlations between the multiple scene images, determining a reference pixel point set whose pixel feature correlations satisfy a set correlation condition from the pixels included in the reference view image; Determine, based on the plane information, a plurality of reference pixel point groups in the reference pixel point set that belong to the same plane, wherein at least one reference pixel point included in each reference pixel point group is located in the same plane; Optimizing the plane determined by the reference pixel point set according to the multiple reference pixel point groups to obtain a plane priori condition of the reference viewing angle image; Taking the plane prior condition as a constraint and based on the pixel feature correlation between the multiple scene images, a depth map of the reference perspective image is obtained.

2. The method according to claim 1, characterized in that The determining, based on the plane information, a plurality of reference pixel point groups belonging to the same plane in the reference pixel point set comprises: Based on the reference pixel point set, a triangulated mesh method is used to perform plane fitting processing to obtain a triangulated mesh structure, wherein each triangular plane in the triangulated mesh structure includes three reference pixel points in the reference pixel point set; Based on the plane information, determining the triangular planes corresponding to the reference pixels located on the same plane in the triangular mesh structure, wherein the reference pixels included in the triangular planes corresponding to the reference pixels on the same plane are included in one reference pixel group; The step of optimizing the plane determined by the reference pixel set according to the multiple reference pixel groups to obtain a plane priori condition of the reference viewing angle image includes: Merging the triangular planes corresponding to the reference pixel points located on the same plane in the triangular mesh structure to obtain a merged triangular mesh structure; Based on the merged triangular mesh structure, the plane prior condition is obtained.

3. The method according to claim 1, characterized in that The step of optimizing the plane determined by the reference pixel set according to the multiple reference pixel groups to obtain a plane priori condition of the reference viewing angle image includes: Performing plane fitting processing on the multiple reference pixel point groups respectively to obtain a plane combination structure composed of a corresponding multiple fitting planes; wherein each reference pixel point group corresponds to a fitting plane among the multiple fitting planes; Based on the plane combination structure, the plane priori condition is obtained.

4. The method according to claim 1, characterized in that Performing plane detection on a reference viewing angle image corresponding to a target viewing angle using a plane detection method to obtain plane information in the reference viewing angle image includes: Performing image semantic segmentation on the reference view image to obtain plane masks corresponding to respective plane regions in the reference view image; Performing depth estimation processing on the reference perspective image to obtain an estimated depth map of the reference perspective image; Based on the estimated depth map, the plane mask is updated to obtain the plane information of the reference view image.

5. The method according to any one of claims 1 to 4, characterized in that: Based on the pixel feature correlations between the multiple scene images, a reference pixel point set whose pixel feature correlations satisfy a set correlation condition is determined from the pixels included in the reference view image, including: For each pixel point, perform the following steps respectively: For a pixel point, determine a stereo matching area of ​​the pixel point in the reference view image, each stereo matching area includes an image area of ​​a set range with the pixel point as a reference point; Respectively determine the mapping area of ​​the stereo matching area in other scene images and the pixel feature correlation between the stereo matching area; wherein the other scene images are scene images other than the reference perspective image among the multiple scene images; From the individual pixel points, a reference pixel point set whose pixel feature association degree satisfies a set association condition is determined.

6. The method according to claim 5, characterized in that Before respectively determining the pixel feature correlation between the mapping area of ​​the stereo matching area in other scene images and the stereo matching area, the method further includes: Initializing parameters of each pixel point respectively to obtain initial disparity plane parameters corresponding to each pixel point; Based on the pixel feature correlation between the reference view image and the corresponding area in the other scene image, performing multiple iterative update processes on the initial disparity plane parameters corresponding to each pixel point, to obtain the estimated disparity plane parameters corresponding to each pixel point; Based on the obtained estimated disparity plane parameters, the stereo matching regions corresponding to the respective pixel points and the mapping regions in the other scene images are determined.

7. The method according to claim 6, characterized in that For the pixel point, an iterative update process includes the following steps: Acquire a plurality of sampling pixel points from pixel points in a set area around the one pixel point; Determining an iterative update loss value of the pixel point based on a correlation degree between a stereo matching area corresponding to the pixel point and pixel features in each corresponding mapping area; Determine the iterative update loss values ​​corresponding to the multiple sampling pixels respectively based on the correlation between the stereo matching areas corresponding to the multiple sampling pixels and the pixel features in the corresponding mapping areas; If, among the multiple sampling pixel points, there is a target sampling pixel point whose iterative update loss value is less than the iterative update loss value corresponding to the one pixel point, the current disparity plane parameter corresponding to the target sampling pixel point is used to update the current disparity plane parameter of the one pixel point.

8. The method according to claim 7, characterized in that Determining an iterative update loss value of the pixel point based on a correlation degree between a stereo matching area corresponding to the pixel point and pixel features in corresponding mapping areas includes: Based on the current disparity plane parameters corresponding to the one pixel point and the homography transformation matrix between various perspectives, determining the stereo matching area corresponding to the one pixel point and the mapping area in the other scene image; Based on the pixel feature values ​​in the stereo matching area and each mapping area, correspondingly determining the pixel feature correlation between the stereo matching area and each mapping area; Based on the obtained correlation degree of each pixel feature, an iterative update loss value corresponding to the one pixel point is determined, wherein the iterative update loss value is negatively correlated with the pixel feature correlation degree.

9. The method according to any one of claims 1 to 4, characterized in that: Taking the plane prior condition as a constraint and based on the pixel feature correlation between the multiple scene images, determining the depth map of the reference perspective image includes: Based on the plane prior conditions and the pixel feature correlation, the current disparity plane parameters corresponding to each pixel point are updated multiple times to obtain the target disparity plane parameters corresponding to each pixel point; Based on the obtained target disparity plane parameters, the depth values ​​corresponding to the respective pixel points are determined to obtain the depth map.

10. The method according to claim 9, characterized in that An iterative update process includes: Starting from the designated pixel point of the reference view image, the following steps are performed one by one for each pixel point in a traversal manner: For a pixel point, obtaining a plurality of sampling pixel points from pixel points in a set area around the pixel point; Based on the plane where each pixel point in the plane prior condition is located and the correlation degree of pixel features in the corresponding stereo matching area and each mapping area, respectively determine the iterative update loss value corresponding to the one pixel point and the multiple sampling pixel points; If, among the multiple sampling pixel points, there is a target sampling pixel point whose corresponding iterative update loss value is less than the iterative update loss value corresponding to the one pixel point, the current disparity plane parameter of the one pixel point is updated according to the current disparity plane parameter corresponding to the target sampling pixel point.

11. The method according to claim 10, characterized in that Acquiring a plurality of sampling pixel points from pixel points in a set area around the one pixel point includes: According to the set number of processing channels N, the reference view image is divided into N channel images; wherein any two adjacent pixels included in each channel image are separated by N-1 pixels from their original positions in the reference view image; The multiple sampling pixel points are obtained from the pixel points in the surrounding set area included in the channel image where the one pixel point is located.

12. A depth map generation device based on plane prior, characterized in that: The device comprises: A plane detection unit, used to acquire a plurality of scene images of a target scene shot from different perspectives, and perform plane detection on a reference perspective image among the plurality of scene images to obtain plane information in the reference perspective image; wherein the plane information is used to indicate pixel points belonging to the same plane; A pixel screening unit, configured to determine, based on the pixel feature correlations between the plurality of scene images, a reference pixel point set whose pixel feature correlations satisfy a set correlation condition from among the pixels included in the reference view image; a plane priori generating unit, configured to determine, based on the plane information, a plurality of reference pixel point groups belonging to the same plane in the reference pixel point set, wherein at least one reference pixel point included in each reference pixel point group is located in the same plane; and, optimizing the plane determined by the reference pixel point set according to the plurality of reference pixel point groups, to obtain a plane priori condition of the reference view image; A depth estimation unit is used to obtain a depth map of the reference view image based on the pixel feature correlation between the multiple scene images, with the plane prior condition as a constraint.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images