Image processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method and device, a storage medium and a program product. According to the embodiment of the invention, the method comprises the steps: obtaining a target image with an initial pose, converting the target image into point cloud data according to the depth information of the target image, and carrying out the plane segmentation based on a normal vector of the point cloud data, so as to recognize at least one target plane; if the number of the target planes reaches or exceeds a set number threshold value, extracting at least one group of plane combinations formed by three mutually vertical target planes from the planes, and determining a correction pose of the target image through a normal vector matrix of the combinations; if the number of the target planes is insufficient, extracting a reference plane from the point cloud data, and determining a correction pose according to a normal vector; and finally, performing vertical correction on the target image according to the correction pose, thereby improving the accuracy and robustness of target image correction, simplifying the operation process, reducing manual intervention, and improving the three-dimensional reconstruction quality or the VR display effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and program product. Background Art

[0002] Compared with ordinary single-view images, panoramas can provide a wider field of view, allowing people to restore scenes to a certain extent, so panoramas are becoming more and more popular. In particular, with the continuous development of virtual reality (VR) technology, panoramas are widely used in house rentals, hotels, home furnishings, various exhibitions and other fields for three-dimensional reconstruction or VR display.

[0003] When using a panorama, vertical correction of the panorama can improve the quality of 3D reconstruction or VR display effects to a certain extent. For example, in the field of 3D reconstruction, the degrees of freedom of a panorama after vertical correction are simplified from 3 constraints to 2, which makes the pose calculation results more robust and efficient, thereby improving the accuracy of 3D reconstruction. In the field of VR display, the vertical direction of a vertically corrected panorama is consistent with the direction of gravity, which is more in line with the user's observation habits and can improve the VR display effect.

[0004] In the prior art, the panorama is either corrected vertically using the straight line segments in the panorama, or the horizontal direction is manually selected and then other software is used to correct the panorama vertically, or the posture of the inertial measurement unit (IMU) when the panorama is collected is used to correct the panorama vertically. These correction methods have poor accuracy and robustness, or the implementation process is complicated, time-consuming and labor-intensive. Summary of the invention

[0005] Multiple aspects of the present application provide an image processing method, device, storage medium and program product to improve the accuracy and robustness of image vertical correction, simplify the operation process, reduce manual intervention, and thereby improve the quality of three-dimensional reconstruction or VR display effect.

[0006] An embodiment of the present application provides an image processing method, including: acquiring a target image, the target image having an initial posture; converting the target image into point cloud data according to depth information corresponding to the target image; performing plane segmentation on the point cloud data based on the normal vector of the point cloud data to obtain at least one target plane; if the number of the at least one target plane is greater than or equal to a set number threshold, extracting at least one group of plane combinations from the at least one target plane, the plane combination including three mutually perpendicular target planes; determining a correction posture corresponding to the target image according to a normal vector matrix of the at least one group of plane combinations; if the number of the at least one target plane is less than the set number threshold, extracting a reference plane from the point cloud data, and determining a correction posture corresponding to the target image according to the normal vector of the reference plane; and performing vertical correction on the target image according to the correction posture.

[0007] The present application also provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the processor is enabled to implement the steps in the image processing method provided in the embodiment of the present application.

[0008] The present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps of the image processing method provided in the embodiment of the present application.

[0009] The present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the image processing method provided in the embodiment of the present application.

[0010] In an embodiment of the present application, a target image with an initial posture is acquired and converted into point cloud data according to its depth information, and plane segmentation is performed based on the normal vector of the point cloud data to identify at least one target plane; if the number of target planes reaches or exceeds a set number threshold, at least one group of plane combinations consisting of three mutually perpendicular target planes are extracted from these planes, and the corrected posture of the target image is determined by the normal vector matrix of these combinations; if the number of target planes is insufficient, a reference plane is extracted from the point cloud data, and the corrected posture is determined according to the normal vector; finally, the target image is vertically corrected according to the corrected posture, thereby improving the accuracy and robustness of the target image correction, simplifying the operation process, reducing manual intervention, and improving the quality of three-dimensional reconstruction or the effect of VR display. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A flowchart of an image processing method provided for an exemplary embodiment of the present application;

[0013] Figure 2 A flowchart of an image processing method provided for another exemplary embodiment of the present application;

[0014] Figure 3 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0016] In view of the technical problems in the prior art that the vertical correction of panoramic images is low in accuracy, poor in robustness, and has a complex implementation process and is time-consuming and labor-intensive, in an embodiment of the present application, a target image with an initial posture is obtained and converted into point cloud data based on its depth information, and plane segmentation is performed based on the normal vector of the point cloud data to identify at least one target plane; if the number of target planes reaches or exceeds a set number threshold, at least one group of plane combinations consisting of three mutually perpendicular target planes is extracted from these planes, and the corrected posture of the target image is determined by the normal vector matrix of these combinations; if the number of target planes is insufficient, a reference plane is extracted from the point cloud data, and the corrected posture is determined according to the normal vector; finally, the target image is vertically corrected according to the corrected posture, thereby improving the accuracy and robustness of the target image correction, simplifying the operation process, reducing manual intervention, and improving the quality of three-dimensional reconstruction or the effect of VR display.

[0017] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0018] Figure 1 A flowchart of an image processing method is provided for applying for an exemplary embodiment. Figure 1 As shown, the method includes:

[0019] 11. Choose a suitable shooting position and initial posture;

[0020] 12. Use a shooting device to shoot the target object at the shooting position according to the initial posture to obtain a target image;

[0021] 13. Perform vertical correction on the target image;

[0022] 14. Output the vertically corrected target image.

[0023] In the embodiment of the present application, steps 11 and 12 focus on describing the image capture process, which can be completed using a capture device; steps 13 and 14 end with describing the image correction process, which can be completed by an image correction device. It is noted here that the capture device and the image correction device can be implemented as the same device, such as a mobile phone or tablet with a camera, or a camera device with an image correction function, etc., so that the image can be captured while performing posture correction on the image; of course, the capture device and the image correction device can also be two different devices, and the capture device and the image correction device are communicatively connected, and the capture device sends the captured target image to the image correction device, and the image correction device performs posture correction on the target image.

[0024] In this embodiment, the implementation form of the shooting device is not limited. Any device that can shoot images can be used as the shooting device in the embodiment of this application. For example, the shooting device can be a handheld mobile phone with a camera function, a fixed panoramic camera, a drone with a rotating mechanism, or a professional panoramic scanning device, such as a laser radar.

[0025] Regardless of whether the image correction device has the function of shooting images, in the embodiments of the present application, the image correction device can be implemented as an electronic device, and the electronic device can be a mobile phone, a laptop computer, a tablet computer, a desktop computer, or other devices with a screen. A graphical user interface can be displayed on the screen of the electronic device, and the graphical user interface can be an application interface provided by an application program that needs to perform image correction, or a web interface, etc.; further, a target image can be displayed on the graphical user interface, and a virtual three-dimensional scene of the target image can also be displayed, etc., depending on the application scenario.

[0026] In this embodiment, the type of target image is not limited. For example, the target image can be a 360-degree panoramic image obtained by direct photography, a 360-degree panoramic image stitched together from multiple overlapping images, a wide-angle image, a fisheye image with obvious distortion effect, a conventional perspective image, etc.

[0027] In an embodiment of the present application, the target image has an initial posture, which refers to the posture of the target image when the shooting device captures the target image. No matter what shooting device is used to shoot the target image, the target image will have an initial posture. The initial posture can be used as a reference in the subsequent image processing correction process.

[0028] In addition, in the embodiment of the present application, the shooting position when shooting the target image is not limited, nor is the choice of shooting position limited, which can be determined according to the target object and shooting requirements. In addition, the shooting angle is also not limited, and it can be placed horizontally, placed vertically, or tilted at any angle such as 30 degrees for shooting. It is explained here that the target object in the embodiment of the present application refers to a three-dimensional space object, such as a target house, office, shopping mall, supermarket, etc. The target image refers to an image containing a three-dimensional space object. For example, in order to shoot an effect that can fully display the appearance of the house and its surrounding environment, a position farther from the front of the house can be selected as a suitable shooting position, and a horizontal position can be selected for the shooting angle, so that the lens can capture the appearance of the entire house and the surrounding natural environment or landscape, such as trees, gardens or other architectural elements. If you want to highlight the layering and three-dimensional sense of the house, you can shoot from a 45-degree angle on the side of the house, using a slightly upward angle to increase the spatial sense of the photo; in order to shoot a visual effect that can reflect the spacious and bright interior of the mall, rich in goods and reasonable layout, you can choose to shoot from the entrance of the mall inward as a suitable shooting position, so that you can show the design style, lighting effects and product display of the mall from an open angle.

[0029] During the capture of the target image, the target image generated after the capture may be tilted in the vertical direction due to the possible deviation or tilt of the initial posture of the shooting device or the complexity of the shooting environment. This tilt not only affects the user's visual experience, but also introduces errors in 3D reconstruction or VR display, reducing the accuracy of reconstruction and the realism of the display. Therefore, it is very necessary to perform vertical correction on the target image. Through vertical correction, the vertical direction of the image can be ensured to be consistent with the direction of gravity, thereby improving the quality of 3D reconstruction or the effect of VR display.

[0030] Therefore, in the embodiment of the present application, when the target image is captured, the target image is further vertically corrected, and the vertically corrected target image is output, so as to facilitate three-dimensional reconstruction, VR display, etc. based on the vertically corrected target image. For the process of vertically correcting the target image, please refer to Figure 2 The illustrated embodiment will not be described in detail here.

[0031] Figure 2A flowchart of an image processing method provided for another exemplary embodiment of the present application is shown in the figure. As shown in the figure, the method includes:

[0032] 21. Obtain a target image with an initial pose;

[0033] 22. Convert the depth information corresponding to the target image into point cloud data;

[0034] 23. Based on the normal vector of the point cloud data, perform plane segmentation on the point cloud data to obtain at least one target plane;

[0035] 24. If the number of target planes is greater than or equal to the set number threshold, extract at least one set of plane combinations from the target planes, each plane combination includes three mutually perpendicular target planes, and determine the corrected pose of the target image according to the normal vector matrices of these plane combinations;

[0036] 25. If the number of target planes is less than the set number threshold, extract the reference plane from the point cloud data, and determine the corrected pose of the target image according to the normal vector of the reference screen;

[0037] 26. According to the determined correction posture, vertical correction is performed on the target image.

[0038] In this embodiment, the method for acquiring the target image with the initial posture is not limited. For example, the target image captured in real time by the image capturing device can be acquired from the image capturing device, or the target image stored in the preset target image library can be acquired from the preset target image library, or the target image captured in real time or pre-stored can be acquired from other electronic devices connected to and communicating with the current electronic device, etc.

[0039] In this embodiment, the target image is converted into point cloud data according to the depth information corresponding to the target image. Depth information refers to the distance information from each pixel to the shooting device when shooting the target image; point cloud information is a data set composed of a series of discrete points, each point contains its position information in space, and is used to represent objects or scenes in three-dimensional space.

[0040] In the embodiments of the present application, the method for obtaining the depth information corresponding to the target image is not limited. For example, the target image and its depth information can be directly obtained by using an RGB-D camera (such as Microsoft Kinect, Intel RealSense) or a camera plus a laser radar (LiDAR). These devices can provide high-precision depth data. Alternatively, the target image can also be obtained by taking pictures using some shooting devices that cannot obtain depth information. In this case, a deep learning model can be used to extract depth information from the target image. The model can be pre-trained with a large number of sample images with real depth labels to predict the depth information of the image without a depth sensor. In addition, by introducing geometric constraint information such as planes and vertical planes during the model training process, the model is more accurate in depth prediction.

[0041] An implementation method for extracting depth information from a target image based on a deep learning model includes: inputting the target image into a convolutional layer in a depth estimation model for feature extraction to obtain a first feature map of the target image; inputting the first feature map into an encoder in the depth estimation model to encode the first feature map to obtain a second feature map; inputting the second feature map into a decoder in the depth estimation model to decode the second feature map to obtain a depth map corresponding to the target image, wherein the depth map includes depth information; wherein the depth estimation model is obtained by model training based on a sample image and a real depth map of the sample image. In the process of depth estimation, the target image is first subjected to feature extraction through a series of convolutional layers to capture key visual features in the image, such as edges and textures, which is the basis for generating a depth map. Subsequently, the extracted first feature map enters the encoder, which gradually compresses the spatial dimensions of the feature map and increases the number of channels through a multi-layer neural network to obtain a deeper and richer feature representation. This step helps to reduce the amount of computation and memory consumption while retaining the key information of the original image. The encoded second feature map is then sent to the decoder, whose task is to restore the compressed feature map to a spatial size close to that of the input image and generate the final depth map. During this period, upsampling and other technologies will be used to ensure the high resolution and accuracy of the depth map. In order to improve the prediction accuracy of the depth map, the model needs to be trained based on a large number of data sets with real depth annotations. The internal parameters are adjusted to minimize the difference between the predicted value and the true value. The loss functions include mean squared error (MSE), mean absolute error (MAE) and structural similarity index (Structural Similarity Index). These methods work together to optimize the model performance, making the three-dimensional spatial information recovered from the two-dimensional image more accurate, thereby obtaining the corresponding depth information in the target image.

[0042] According to the depth information corresponding to the target image obtained above, the target image is converted into point cloud data. Each pixel has a corresponding depth information, that is, a depth value, and the corresponding pixel point can be converted into a coordinate point in three-dimensional space according to the depth information.

[0043] After the target image is converted into point cloud data, the point cloud data is plane segmented based on the normal vector of the point cloud data to obtain at least one target plane. This step includes: calculating the normal vector of each point in the point cloud data, and then using these normal vectors to segment the target plane in the point cloud data. Each target plane contains a group of points with the same or similar normal vectors. These target planes can be used for subsequent vertical correction image processing tasks.

[0044] In this embodiment, when the target plane is obtained, the number of the target planes can be counted; if the number of the target planes is greater than or equal to the set number threshold, at least one set of plane combinations is extracted from the target planes, each plane combination includes three mutually perpendicular target planes, and the correction posture of the target image is determined according to the normal vector matrix of these plane combinations. In this embodiment, the value of the set number threshold is not limited, and it can be an integer greater than or equal to 3. Among them, if the number of sampled target planes is greater than or equal to the set number threshold (for example, 3 or more), at least one set of plane combinations is extracted from these target planes, and the number of plane combinations is not limited, and can be one or more groups. The more the number of plane combinations, the more accurate the correction posture determined. The correction posture of the target image can be calculated through the normal vector matrix of these plane combinations. Specifically, a normal vector matrix is ​​constructed using the normal vectors of the three planes in each plane combination, and the correction posture corresponding to the target image is determined by mathematically processing the matrix (such as singular value decomposition), thereby achieving vertical correction, making the image more visually beautiful and natural, avoiding visual discomfort caused by tilt, and improving the user's viewing experience.

[0045] Based on the above content, several optional implementation methods are listed below.

[0046] In Example A1, the target plane quantity threshold is set to 3. The target image is a typical indoor room, including target plane walls, ground and desk. The target planes are four planes, three walls and the ground. If the number is greater than the set quantity threshold of 3, two of the walls and the ground are selected to form a set of mutually perpendicular plane combinations. By calculating the normal vector matrices of these three planes and performing singular value decomposition, the corrected posture corresponding to the target image is determined, so that the walls, ground and desk in the image remain visually upright.

[0047] In Example A2, the target plane quantity threshold is set to 4. The target image is an office space, including multiple walls, floors, and desks. The target planes are three walls, the floor, and the ceiling. If the number is greater than the set number threshold of 4, two walls and the floor are selected to form a set of mutually perpendicular plane combinations. By calculating the normal vector matrices of these three planes and performing singular value decomposition, the corrected posture corresponding to the target image is determined, so that the walls, floors, and desks in the image remain visually upright.

[0048] In Example A3, the target plane quantity threshold is set to 5. The target image is an industrial plant, which includes multiple walls, floors, and workbenches. The target planes are three walls, the floor, and the workbench. The number is determined to be equal to the set number threshold of 5, and two of the walls and the floor are selected to form a set of mutually perpendicular plane combinations. By calculating the normal vector matrix of these three planes and performing singular value decomposition, the corrected posture corresponding to the target image is determined, so that the walls, the floor, and the workbench in the image remain visually upright.

[0049] In this embodiment, if the number of target planes is less than a set number threshold, a reference plane is extracted from the point cloud data, and the corrected pose of the target image is determined based on the normal vector of the reference screen. This means that if the number of target planes identified in the point cloud data is not enough to form a combination of multiple mutually perpendicular planes, a significant reference plane is extracted from the point cloud data. Among them, the reference plane refers to a plane that is significant and has a large area in the scene, which is not limited in this embodiment. For example, it can be the ground, wall or ceiling. The main function of the reference plane is to provide a stable reference frame to help determine the position and posture of other objects. In this way, even when the number of target planes is less than the set number threshold, the pose of the target image can be corrected by extracting the normal vector of the reference plane to ensure that the vertical direction of the image is consistent with the direction of gravity, thereby improving the user experience of applications such as AR, VR and MR.

[0050] Based on the above content, several optional implementation methods are listed below.

[0051] In Example B1, the target plane quantity threshold is set to 3. The target image is a small outdoor square, including the ground and an irregular slate plane. The target plane has only one ground plane and one slate plane. If the number is less than the set quantity threshold of 3, the ground is extracted from the point cloud data as the reference plane. By calculating the normal vector of the ground, the corrected posture of the target image is determined so that the ground and the slate in the image remain visually upright.

[0052] In Example B2, the target plane quantity threshold is set to 3. The target image is an outdoor garden, including the ground and an irregular flower bed plane. The target plane has only one ground plane and one flower bed plane. If the number is less than the set quantity threshold of 3, the flower bed plane is extracted from the point cloud data as the reference plane. By calculating the normal vector of the flower bed plane, the corrected posture of the target image is determined so that the ground and flower bed in the image remain visually upright. This ensures that even if the number of target planes is insufficient, the vertical elements in the image can still be correctly aligned, thereby improving the user's viewing experience.

[0053] In this embodiment, the target image is vertically corrected according to the determined correction posture, and there is no limitation on how to output the corrected target image posture according to the correction posture of the target image. For example, a correction mapping can be calculated based on data information such as the correction posture. This mapping is a mapping from coordinates in the correction posture to coordinates of the corrected image. Then, the correction posture is sampled according to this mapping to generate a corrected image. Alternatively, the initial image and the correction posture and other data information can be directly accepted. Then, the pixel position of the corrected image is calculated based on the initial image and the correction posture and other data information, and the corresponding pixel values ​​in the initial image are mapped to the corrected image. This process involves complex geometric transformations and interpolation, and the corrected image can be directly output.

[0054] In an optional embodiment, the target image is converted into point cloud data according to the depth information corresponding to the target image, including: according to the conversion relationship between the camera coordinate system and the spherical coordinate system, the pixel coordinates of each pixel in the target image are converted into longitude information and dimension information; according to the longitude information and dimension information of each pixel point, the three-dimensional coordinates of each pixel point in the spherical coordinate system are calculated; according to the depth information of each pixel point in the target image and the three-dimensional coordinates of each pixel point in the spherical coordinate system, the three-dimensional coordinates of each pixel point in the world coordinate system are determined to obtain point cloud data. Specifically, this process involves converting from the camera coordinate system to the spherical coordinate system, that is, converting from the depth map pixel coordinates to the longitude and latitude in the spherical coordinate system, and then further converting the depth information of the pixel point into the three-dimensional coordinates in the world coordinate system, that is, converting the pixel points in the depth map into points in three-dimensional space, thereby constructing complete point cloud data. In the conversion from depth map pixel coordinates to spherical coordinates, each pixel in the depth map represents a distance value in a specific direction. To convert these pixel points into positions in the spherical coordinate system, the following formulas (1) and (2) can be used to calculate the longitude and latitude of the pixel points in the spherical coordinate system respectively:

[0055]

[0056] In formula (1), lon represents the longitude of the pixel in the spherical coordinate system; the value of 0.5f is used for normalization of pixel coordinates from the camera coordinate system to the spherical coordinate system; depth_pixel.x is the X component of the position coordinates of the pixel in the depth map; depth_cols is the width of the depth map, that is, the number of columns, which defines the size of the depth map; M_PI is π in mathematics, and the role of *2π is to convert the relative position into radians, representing the complete longitude range on the sphere.

[0057]

[0058] In formula (2), lat represents the latitude of the pixel in the spherical coordinate system; the value of 0.5f is used for normalization processing to convert the pixel coordinates from the camera coordinate system to the spherical coordinate system; depth_pixel.y is the Y component of the position coordinates of the pixel in the depth map; depth_rows is the height of the depth map, that is, the number of rows, which defines the size of the depth map; M_PI is π in mathematics, and the role of *π is to convert the relative position into a radian value, indicating the maximum value from the southern hemisphere to the northern hemisphere on the sphere.

[0059] In the process of converting from the spherical coordinate system to the world coordinate system, the depth value is very important because it represents the actual distance from the camera center to the pixel point in the depth map. The position of the point in the world coordinate system is determined based on the angle of longitude and latitude in the spherical coordinate system and the depth information of the pixel point. The following formulas (3), (4) and (5) can be used to calculate the x, y and z coordinates of the point converted to the world coordinate system respectively:

[0060] x = cos (lat) * sin(lon) * d (3)

[0061] In formula (3), x represents the x coordinate of the point in the world coordinate system; cos(lat) represents the cosine value of the latitude of the point in the spherical coordinate system; sin(lon) represents the sine value of the longitude of the point in the spherical coordinate system; and d represents the actual depth value.

[0062] y = sin(lat) * d (4)

[0063] In formula (4), y represents the y coordinate of the point in the world coordinate system; sin(lat) represents the sine value of the latitude of the point based on the spherical coordinate system; and d represents the actual depth value.

[0064] z = cos(lat) * cos(lon) * d (5)

[0065] Wherein, in formula (5), z represents the z coordinate of the point in the world coordinate system; cos(lat) represents the cosine value of the latitude of the point in the spherical coordinate system; cos(lon) represents the cosine value of the longitude of the point in the spherical coordinate system; and d represents the actual depth value.

[0066] The actual depth value d in formulas (3), (4) and (5) can be calculated using formula (6):

[0067] d = point3d * scale (6)

[0068] In formula (6), d represents the actual depth value; point3d represents the original depth value in the depth map; and scale represents the scaling factor.

[0069] In an optional embodiment, for each point in the point cloud data, the normal vector of each point is estimated based on the information of the local neighborhood of each point. The normal vector of the point cloud data is obtained by determining the local neighborhood of each point in the point cloud data, calculating the covariance matrix of the points in the local neighborhood, solving the eigenvalues ​​and eigenvectors of the covariance matrix, and selecting the eigenvector corresponding to the minimum eigenvalue as the normal vector. Specifically, for each point P in the point cloud data i , use K nearest neighbor or fixed radius search method to select points in its local neighborhood, assuming that the points in the local neighborhood are P i1 , P i2 , ..., P iN Next, use formula (7) to calculate the center point P of the local neighborhood C :

[0070]

[0071] In formula (7), N represents the number of points in the local neighborhood; P ij Represents the coordinates of the jth point in the local neighborhood.

[0072] Then, the deviation P of each point relative to the center point is calculated using formula (8): ij -P C , and construct the covariance matrix C:

[0073]

[0074] In formula (8), P ij -P C represents the deviation vector of the jth point relative to the center point; (P ij -P C ) T Represents the transpose of the bias vector.

[0075] Next, the eigenvalue decomposition method is used to solve the eigenvalues ​​λ1, λ2, λ3 and the corresponding eigenvectors v1, v2, v3 of the covariance matrix C. The eigenvector corresponding to the minimum eigenvalue usually represents the normal vector in the local neighborhood. Assuming that the minimum eigenvalue is λ3 and the corresponding eigenvector is v3, then v3 is the current point p i For example, suppose there is a cloud point dataset containing P 1 =(1,2,3),P 2 =(1.1, 2.1, 3.1), P 3 = (0.9, 1.9, 2.9) three points, choose P 1 As the current point, its local neighborhood includes P 2 and P 3 , according to P 1 , P 2 and P 3 Calculate the center point P of the local neighborhood C =(P 1 +P 2 +P 3 ) / 3, where the deviation P of each point relative to the central location is calculated 1 -P C =(1,2,3)-(1,2,3)=(0,0,0),P 2 -P C = (1.1, 2.1, 3.1)-(1, 2, 3) = (0.1, 0.1, 0.1), P 3 -P C =(0.9, 1.9, 2.9)-(1, 2, 3)=(-0.1, -0.1, -0.1), calculate the covariance matrix

[0076] Use the eigenvalue decomposition method to solve the eigenvalues ​​and eigenvectors of the covariance matrix C. Assuming that the eigenvalues ​​are λ1, λ2, λ3, the corresponding eigenvectors are v1, v2, v3, and assuming that the minimum eigenvalue is λ3 and the corresponding eigenvector is v3, then v3 is the current point P 1 The normal vector of .

[0077] Based on the above method, the normal vector of each point cloud data is calculated, and the point cloud data is fitted and segmented using a random extraction algorithm to obtain multiple initial planes; the angle between the normal vectors of the two initial planes is calculated, and the two initial planes whose angle is less than the first set angle threshold are merged to obtain at least one target plane. Among them, based on the calculated normal vector of each point cloud data, the point cloud data is fitted and segmented using a random extraction algorithm RANSAC to obtain multiple initial planes. The random selection mechanism helps to reduce redundant planes and improve the accuracy and efficiency of plane segmentation. Specifically, the RANSAC algorithm defines a plane model by randomly selecting three points, calculates the distance from all points to the plane, and regards points whose distance is less than the set threshold as internal points. This process is repeated multiple times, and the plane model with the most internal points is selected as the initial plane. The angle between the normal vectors of the two initial planes is calculated. If the angle is less than the first set angle threshold, the two initial planes are merged to finally obtain at least one target plane.

[0078] In the above embodiment, the first set angle threshold is a key parameter used to determine whether two initial planes should be merged. The setting of this threshold directly affects the number and quality of the final target plane. If the normal vector angle of the two initial planes is less than the first set angle threshold, it means that the two planes are very close in direction and can be considered to belong to the same plane. Therefore, merging them can reduce redundant planes and improve the accuracy of plane segmentation. Among them, this embodiment does not limit the setting of the threshold. For example, it can be 5° to 10°, or it can be 15° to 20°. In short, the setting of the first set angle threshold needs to comprehensively consider factors such as application scenarios, plane features and data noise, and determine the most appropriate value through experimental verification. For example, in an indoor environment, planes such as walls, floors and ceilings are usually more regular and have a smaller angle. In this case, a smaller angle threshold, such as 5°, can be set to ensure that the merged planes do belong to the same physical plane; in an outdoor environment, the terrain and buildings may be more complex, and the angles between planes vary more. At this time, a larger angle threshold, such as 15°, can be set to accommodate more plane combinations.

[0079] In an optional embodiment, according to the normal vector of each point cloud data, a random extraction algorithm is used to fit and segment the point cloud data to obtain multiple initial planes, including: step 1, randomly sampling a group of sample points from the point cloud data as points on an initial plane; step 2, using the normal vector of the sample point to calculate the reference normal vector of the initial plane; step 3, calculating the angle between the reference normal vector and the normal vectors of other points in the point cloud data, and selecting points with an angle less than a second set angle threshold from other points as points of the initial plane; repeating steps 1-3 until all points in the point cloud data are divided into an initial plane. Among them, the reference normal vector refers to a vector representing the direction of the plane calculated from a group of sample points. Specifically, a plane can be fitted through randomly selected sample points, and the normal vector of the plane is the reference normal vector. The reference normal vector is used to describe the direction of the initial plane and help screen out other points belonging to the same plane. The second set angle threshold is a key parameter for judging whether other points in the point cloud data belong to the same initial plane. The angle between the normal vectors of other points in the point cloud data and the reference normal vector is calculated. If the angle is less than the second set angle threshold, these points are considered to belong to the same initial plane. By setting a reasonable angle threshold, the probability of misclassifying points that do not belong to the same plane into the same plane can be reduced, thereby improving the accuracy of plane segmentation.

[0080] In the above embodiment, the purpose of repeatedly executing steps 1-3 is to gradually divide all points in the point cloud data into different initial planes to ensure that each point is correctly classified. Specifically, in each iteration, a new set of sample points is randomly sampled from the remaining unclassified point cloud data, a new reference normal vector is calculated, and points that meet the conditions are selected to be added to the current initial plane. This process is repeated until all points are assigned to a certain initial plane. In this way, the point cloud data can be effectively plane segmented to improve the accuracy and completeness of the segmentation. For example, there is a set of point cloud data containing 1,000 points, representing the internal structure of a room. In order to divide these point cloud data into multiple initial planes, a random extraction algorithm is used for fitting and segmentation. The number of sample points taken is 3, and the second set angle threshold is set to 10°. In the first iteration: 3 points are randomly selected from the 1,000 points. Assume that these points are located on the ground, and the reference normal vectors of these 3 points are calculated. Assume that the normal vector obtained is n 1 , calculate the reference normal vector n 1 The angle between the normal vectors of the remaining 997 points and the points with an angle less than 10° are selected. Assuming that 500 points are found, these 500 points are marked as ground points and removed from the point cloud data. Second iteration: randomly select 3 points from the remaining 497 points, assuming that these points are located on a wall, calculate the reference normal vectors of these 3 points, and assume that the obtained normal vector is n 2 , calculate the reference normal vector n2 The angle between the normal vectors of the remaining 494 points and the points with an angle less than 10° are selected. Assuming that 300 points are found, these 300 points are marked as wall points and removed from the point cloud data. The third iteration: randomly select 3 points from the remaining 197 points, assume that these points are located on the ceiling, calculate the reference normal vectors of these 3 points, and assume that the normal vector obtained is n 3 , calculate the reference normal vector n 3 The angle between the normal vectors of the remaining 194 points is selected, and the points with an angle less than 10° are selected. Assuming that 194 points are found, these 194 points are marked as ceiling points and removed from the point cloud data. Through the above steps, the 1000 point cloud data is successfully segmented into three initial planes: ground (500 points), wall (300 points) and ceiling (194 points). This process ensures that each point is correctly classified and improves the accuracy and completeness of plane segmentation.

[0081] In an optional embodiment, extracting at least one group of plane combinations from at least one target plane includes: randomly sampling at least one target plane at least once, randomly sampling two target planes each time, until randomly sampling a first target plane and a second target plane whose angle is close to 90°; according to the first target plane and the second target plane, determining a third target plane perpendicular to both the first target plane and the second target plane from other target planes except the first target plane and the second target plane; taking the first target plane, the second target plane and the third target plane as a first group of plane combinations, and the three target planes in the first group of plane combinations meet the right-hand coordinate system principle; according to the first group of plane combinations, selecting other plane combinations from the remaining target planes, and the three target planes in the other plane combinations are respectively parallel to the three target planes in the first group of plane combinations and the distances are less than the corresponding distance thresholds. Wherein, the target plane is a plane obtained by segmenting point cloud data, and loc-ransac (a variant of RANSAC) is used to iteratively solve, and two planes are randomly extracted. If the angle is close to the threshold, the next step is entered, otherwise the extraction is continued. That is, random sampling is to randomly select some planes from the target plane for further processing. The angle is the angle between the normal vectors of the two target planes. For example, the vector n 1 and n 2 The cosine value of the angle θ between them is equal to their dot product divided by the product of their moduli. The angle θ can be calculated by the inverse cosine function. Among them, the right-hand coordinate system principle is satisfied when the normal vectors of the three planes satisfy the right-hand rule, that is, if the normal vector of the first plane is n 1 , the normal vector of the second plane point is n 2 , the normal vector of the third plane is n 3 , then n 1 ×n 2Should be n 3 The same direction; close and approximate here means that the angle is within a certain range, for example, an angle close to 90° can be defined as an angle within the range of 90°±x. Among them, x is a small angle value. For example, when x is 5°, an angle close to 90° can be defined as an angle between 85° and 95°. The third target plane perpendicular to both the first target plane and the second target plane can be determined by calculating the cross product of the normal vector points of the first target plane and the second target plane to obtain a new vector, which is perpendicular to the normal vectors of the first target plane and the second target plane. From the remaining target planes, the plane whose angle between the normal vector and the vector of the new point is less than a threshold is selected as the third target plane.

[0082] In the above embodiment, the normal vectors of the three planes obtained satisfy the right-hand rule, that is, the normal vectors of the first target plane and the second target plane point are in the same direction as the new vector, which can be confirmed by calculating the product of the normal vectors of the first target plane and the second target plane point to verify whether their directions are consistent with the new vector. Among them, accurate angle calculation and normal vector verification are used to ensure that the geometric relationship of the extracted plane combination is correct and the accuracy of plane segmentation is improved; random sampling and multiple iterations improve the robustness of the algorithm and can stably extract key planes in complex point cloud data. The extracted key plane combination can be directly used for subsequent three-dimensional reconstruction, which improves the efficiency and reliability of the overall correction. For example, two planes P are randomly selected from the target plane. 1 and P 2 , calculate P 1 and P 2 The normal vector n 1 and n 2 If the angle θ between them is within the range of 90°±x (e.g., 85° to 95°), P is retained. 1 and P 2 As a candidate plane, calculate n 1 and n 2 The cross product of 3 , from the remaining target planes, select the normal vector and n 3 The plane with an angle less than a threshold (e.g., 5°) is taken as the third target plane P 3 , calculate n 1 ×n 2 , if the direction of the resulting vector is the same as n 3 The direction of P is the same, then the right-hand coordinate system principle is satisfied. 1 , P 2 and P 3 As the first set of plane combinations, according to the first set of plane combinations P 1 , P 2 and P 3, select other plane combinations from the remaining target planes, the three target planes in the other plane combinations are parallel to the three target planes in the first set of plane combinations and the distances are less than their respective corresponding distance thresholds.

[0083] In an optional embodiment, according to the first target plane and the second target plane, a third target plane perpendicular to both the first target plane and the second target plane is determined from other target planes except the first target plane and the second target plane, including: according to the first target plane and the second target plane, a candidate target plane perpendicular to both the first target plane and the second target plane is determined from other target planes; from other target planes, a target plane parallel or approximately parallel to the candidate target plane is selected, and the candidate target plane and the target planes parallel or approximately parallel to the candidate target plane form a plane set; according to the number of points contained in each target plane in the plane set and the angle between each other, the score of each target plane in the plane set is calculated; and the target plane with the highest score is selected from the plane set as the third target plane. Among them, the loc-ransac iterative solution is used to randomly select two planes, and once two nearly perpendicular planes are found, the normal vector of the new plane perpendicular to both planes is calculated by the normal vectors of the two planes. The normal vector of the calculated new plane is used to find a plane set similar to it, and it can be scored using formula (9):

[0084] score = (1-pow(1-angle, 0.25))×point.size (9)

[0085] Among them, in formula (9), score represents the scoring result; angle is the angle between the normal vector of the target plane and the normal vector of the new plane; point.size represents the number of points contained in the target plane; pow represents power operation, and pow is a power operation function. In formula (9), pow(1-angle, 0.25) represents (1-angle) to the power of 0.25; 1 is a unit value, and 1-pow(1-angle, 0.25) represents the difference between the angle angle and 1; 0.25 is an empirical value used to adjust the degree of the angle on the score. The choice of empirical values ​​is usually based on the effects of experiments and actual applications. In this scenario, 0.25 is selected to make the influence of the angle smoother and more gradual, and to avoid a slight deviation in the angle causing a large change in the score. This value can better balance the accuracy of the angle and the stability of the score. Among them, the more points there are, the more stable the plane is and the higher the score is. When the angle between the two planes is close to 0 (almost parallel), 1-angle is close to 1, and pow(1-angle, 0.25) is also close to 1, so 1-pow(1-angle, 0.25) is close to 0, and the score will be very low; when the angle between the two planes is close to 90 degrees (almost perpendicular), 1-angle is close to 0, pow(1-angle, 0.25) is also close to 0, so 1-pow(1-angle, 0.25) is close to 1, and the score will be very high; at the same time, if there are many points on a plane, even if the angle difference between it and the main plane is not large, its score will increase due to the number of points. After iteration, the plane with the highest score is selected as the third target plane, and the order of the plane set obtained above is adjusted to meet the right-hand coordinate system principle. Through these steps, the third target plane perpendicular to the first target plane and the second target plane can be determined from the target plane, thereby forming a plane combination that meets the right-hand coordinate system principle and improving the accuracy and efficiency of plane segmentation.

[0086] In an optional embodiment, the correction posture corresponding to the target image is determined according to the normal vector matrix of at least one group of plane combinations, including: constructing the normal vector matrix corresponding to each group of plane combinations according to the normal vectors of the three target planes in each group of plane combinations; using the number of points contained in the three target planes in each group of plane combinations as weights to obtain the weight vector corresponding to each group of plane combinations; performing weighted summation on the normal vector matrix corresponding to each group of plane combinations according to the weight vector corresponding to each group of plane combinations to obtain the target normal vector matrix; performing singular value decomposition on the target normal vector matrix to obtain the correction posture corresponding to the target image. For example, there are three normal vectors of target planes in a group of plane combinations, and the three target planes are perpendicular to each other, and their normal vectors are n respectively. 0 、n 1 、n 2, we can construct a matrix A, where each row corresponds to a normal vector of a plane, and each row is multiplied by the corresponding number of point clouds as a weight. The form of matrix A is where ω 0 ,ω 1 ,ω 2 are the number of points contained in the three target planes respectively; the number of points contained in the three target planes in each set of plane combinations is used as the weight to obtain the weight vector ω corresponding to each set of plane combinations. The weight vector ω can be expressed as According to the weight vector corresponding to each set of plane combinations, the normal vector matrix corresponding to each set of plane combinations is weighted summed to obtain the target normal vector matrix A target , if there are two sets of plane combinations A 1 and A 2 , and the corresponding weight vectors are ω 1 and ω 2 , then the target normal vector matrix A target It can be expressed as For the target normal vector matrix A target Perform singular value decomposition (SVD) to obtain the corrected pose corresponding to the target image, where the form of singular value decomposition is A target =UΣV T , where U and V are orthogonal matrices, Σ is a diagonal matrix, and the corrected pose can be obtained through U and V. Through the above steps, the corrected pose corresponding to the target image can be determined from at least one set of plane combinations, thereby improving the accuracy and reliability of plane segmentation and pose estimation.

[0087] In an optional embodiment, the target image is an image corresponding to the target physical space, and a reference plane is extracted from the point cloud data, including: using a sliding window with a set width to slide in the point cloud data along the y-axis direction representing the height to determine the maximum height value and / or the minimum height value, the maximum height value corresponds to the ceiling of the target physical space, and the minimum height value corresponds to the ground of the target physical space; continuing to use the sliding window to slide along the x-axis and z-axis directions in the point cloud data with the maximum height value and / or the minimum height value to obtain the points that fall into the sliding window each time at the maximum height value and / or the minimum height value; fitting the points that fall into the sliding window each time at the maximum height value and / or the minimum height value to obtain the ceiling and / or the ground as the reference plane. Among them, for extracting the reference plane from the point cloud data, first, obtain the y-axis coordinate values ​​of all points in the corona data, divide the y-axis coordinate values ​​into windows with a set width in height, such as 10 cm, count the number of points in each window, and select windows with more concentrated points. The maximum height value of these windows corresponds to the ceiling of the target physical space, and the minimum height value corresponds to the ground of the target physical space; then, select point cloud data from the windows corresponding to the determined maximum height value and / or minimum height value, use a sliding window to slide along the x-axis and z-axis directions, set the window width, such as 10 cm, count the number of points in each sliding window, and record the windows with more points; finally, select the points in the window with more points, perform plane fitting on these points, and obtain the equations of the ceiling and the ground. Fitting methods including least squares method, RANSAC, etc. can be used. The normal vector of the plane is calculated by fitting the plane equation obtained. For example, if the plane equation is ax+by+cz+d=0, then the normal vector is n=(a, b, c). Through the above steps, the ground plane and / or ceiling plane can be extracted from the point cloud data and used as reference planes for the vertical correction pose of the target image.

[0088] In an optional embodiment, when the corrected posture corresponding to the target image cannot be determined according to the normal vector of the reference plane, the posture of the IMU when the target image is collected is obtained as the corrected posture corresponding to the target image. According to the above method, the ground plane or ceiling plane is extracted from the point cloud data, and the normal vectors of these planes are calculated. If the extracted normal vector is not accurate enough or cannot be determined (for example, the point cloud data is of poor quality, has a lot of noise, the plane is not obvious, etc.), the corrected posture of the target image cannot be determined according to the normal vector of the reference plane. In this case, the posture information of the IMU when the target image is collected is obtained, including the roll angle (roll), pitch angle (pitch) and yaw angle (yaw). IMU (inertial measurement unit) is a sensor that can provide posture information of the device, including rotation angle and acceleration, etc. The posture information recorded by the IMU is usually stored in the form of Euler angles (roll, pitch, yaw), which describe the rotation state of the device in three-dimensional space. Specifically, the roll angle (roll) represents the rotation angle of the device around the x-axis, the pitch angle (pitch) represents the rotation angle of the device around the y-axis, and the yaw angle (yaw) represents the rotation angle of the device around the z-axis. By reading the roll angle, pitch angle, and yaw angle recorded by the IMU, a rotation matrix or quaternion can be constructed to correct the target image so that it is aligned with the vertical direction of the physical space. Through the above steps, when the corrected pose corresponding to the target image cannot be determined based on the normal vector of the reference plane, the pose information of the IMU can be used as a substitute to ensure the vertical corrected pose of the target image.

[0089] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 11, 12, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0090] Figure 3 The following is a schematic diagram of an electronic device structure provided by an exemplary embodiment of the present application. Figure 3 As shown, the device includes: a processor 35 and a memory 34.

[0091] The memory 34 is used to store computer programs and can be configured to store various other data to support operations on the image processing device. Examples of such data include instructions for any application or method operating on the electronic device, image data, user interface elements, image files, three-dimensional models, etc.

[0092] The memory 34 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0093] The processor 35 is coupled to the memory 34 and is used to execute the computer program in the memory 34, so as to: obtain a target image, the target image having an initial posture; convert the target image into point cloud data according to the depth information corresponding to the target image; perform plane segmentation on the point cloud data based on the normal vector of the point cloud data to obtain at least one target plane; if the number of at least one target plane is greater than or equal to a set number threshold, extract at least one group of plane combinations from the at least one target plane, the plane combination includes three target planes perpendicular to each other; determine the correction posture corresponding to the target image according to the normal vector matrix of at least one group of plane combinations; if the number of at least one target plane is less than the set number threshold, extract a reference plane from the point cloud data, and determine the correction posture corresponding to the target image according to the normal vector of the reference plane; and perform vertical correction on the target image according to the correction posture.

[0094] In an optional embodiment, before the processor 35 converts the target image into point cloud data according to the depth information corresponding to the target image, it also includes: inputting the target image into the convolution layer in the depth estimation model for feature extraction to obtain a first feature map of the target image; inputting the first feature map into the encoder in the depth estimation model to encode the first feature map to obtain a second feature map; inputting the second feature map into the decoder in the depth estimation model to decode the second feature map to obtain a depth map corresponding to the target image, wherein the depth map includes depth information; wherein the depth estimation model is obtained by model training based on the sample image and the real depth map of the sample image.

[0095] In an optional embodiment, the processor 35 converts the target image into point cloud data based on the depth information corresponding to the target image, including: converting the pixel coordinates of each pixel in the target image into longitude information and latitude information based on the conversion relationship between the camera coordinate system and the spherical coordinate system; calculating the three-dimensional coordinates of each pixel in the spherical coordinate system based on the longitude information and latitude information of each pixel; determining the three-dimensional coordinates of each pixel in the world coordinate system based on the depth information of each pixel in the target image and the three-dimensional coordinates of each pixel in the spherical coordinate system to obtain point cloud data.

[0096] In an optional embodiment, the processor 35 performs plane segmentation on the point cloud data based on the normal vector of the point cloud data to obtain at least one target plane, including: estimating the normal vector of each point in the point cloud data according to the information of the local neighborhood of each point; fitting and segmenting the point cloud data according to the normal vector of each point cloud data using a random sampling algorithm to obtain multiple initial planes; calculating the angle between the normal vectors of each pair of initial planes, and merging two initial planes whose angle is less than a first set angle threshold to obtain at least one target plane.

[0097] In an optional embodiment, the processor 35 fits and segments the point cloud data using a random extraction algorithm based on the normal vector of each point cloud data to obtain multiple initial planes, including: step 1, randomly sampling a group of sample points from the point cloud data as points on an initial plane; step 2, using the normal vectors of the sample points to calculate the reference normal vector of the initial plane; step 3, calculating the angle between the reference normal vector and the normal vectors of other points in the point cloud data, and selecting points from other points whose angles are less than a second set angle threshold as points of the initial plane; repeating steps 1-3 until all points in the point cloud data are divided into an initial plane.

[0098] In an optional embodiment, the processor 35 extracts at least one group of plane combinations from at least one target plane, including: randomly sampling at least one target plane at least once, randomly sampling two target planes each time, until randomly sampling a first target plane and a second target plane whose angle is close to 90°; based on the first target plane and the second target plane, determining a third target plane perpendicular to both the first target plane and the second target plane from other target planes except the first target plane and the second target plane; taking the first target plane, the second target plane and the third target plane as a first group of plane combinations, the three target planes in the first group of plane combinations satisfying the right-hand coordinate system principle; based on the first group of plane combinations, selecting other plane combinations from the remaining target planes, the three target planes in the other plane combinations being parallel to the three target planes in the first group of plane combinations and the distances being less than their respective corresponding distance thresholds. In an optional embodiment, the processor 35 determines a third target plane perpendicular to both the first target plane and the second target plane from other target planes except the first target plane and the second target plane based on the first target plane and the second target plane, including: determining a candidate target plane perpendicular to both the first target plane and the second target plane from other target planes based on the first target plane and the second target plane; selecting a target plane that is parallel or approximately parallel to the candidate target plane from other target planes, and the candidate target plane and the target planes that are parallel or approximately parallel to the candidate target plane form a plane set; calculating the score of each target plane in the plane set based on the number of points contained in each target plane in the plane set and the angles between them; and selecting the target plane with the highest score from the plane set as the third target plane.

[0099] In an optional embodiment, the processor 35 determines the corrected posture corresponding to the target image based on the normal vector matrix of at least one group of plane combinations, including: constructing a normal vector matrix corresponding to each group of plane combinations based on the normal vectors of the three target planes in each group of plane combinations; using the number of points contained in the three target planes in each group of plane combinations as weights to obtain a weight vector corresponding to each group of plane combinations; performing weighted summation of the normal vector matrices corresponding to each group of plane combinations based on the weight vectors corresponding to each group of plane combinations to obtain a target normal vector matrix; and performing singular value decomposition on the target normal vector matrix to obtain the corrected posture corresponding to the target image.

[0100] In an optional embodiment, the target image is an image corresponding to the target physical space, and the processor 35 extracts a reference plane from the point cloud data, including: using a sliding window with a set width to slide in the point cloud data along the y-axis direction representing the height to determine the maximum height value and / or the minimum height value, the maximum height value corresponds to the ceiling of the target physical space, and the minimum height value corresponds to the ground of the target physical space; continue to use the sliding window to slide along the x-axis and z-axis directions in the point cloud data with the maximum height value and / or the minimum height value to obtain the points that fall into the sliding window each time at the maximum height value and / or the minimum height value; fit the points that fall into the sliding window each time at the maximum height value and / or the minimum height value to obtain the ceiling and / or the ground as the reference plane.

[0101] In an optional embodiment, when the processor 35 is unable to determine the corrected posture corresponding to the target image based on the normal vector of the reference plane, the processor 35 obtains the posture of the IMU when the target image is acquired as the corrected posture corresponding to the target image.

[0102] Further, if Figure 3 As shown, the computing platform also includes: a communication component 36, a display 37, a power component 38, an audio component 39 and other components. Figure 3 Only some components are shown schematically, which does not mean that the computing platform only includes Figure 3 In addition, Figure 3 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the working node. The working node of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server-side device such as a conventional server, a cloud server, or a server array. If the working node of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 3 If the working node of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 3 Components within the dashed box.

[0103] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the computing platform in the above method embodiment.

[0104] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0105] Above Figure 3 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0106] Above Figure 3 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0107] Above Figure 3 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0108] Above Figure 3The audio component in can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a target image, wherein the target image has an initial pose; Converting the target image into point cloud data according to the depth information corresponding to the target image; Based on the normal vector of the point cloud data, plane segmentation is performed on the point cloud data to obtain at least one target plane; If the number of the at least one target plane is greater than or equal to a set number threshold, extract at least one plane combination from the at least one target plane, the plane combination including three target planes perpendicular to each other; determine a corrected posture corresponding to the target image according to a normal vector matrix of the at least one plane combination; If the number of the at least one target plane is less than a set number threshold, extracting a reference plane from the point cloud data, and determining a corrected pose corresponding to the target image according to a normal vector of the reference plane; The target image is vertically corrected according to the corrected posture.

2. The method according to claim 1, characterized in that Before converting the target image into point cloud data according to the depth information corresponding to the target image, the method further includes: Inputting the target image into a convolutional layer in a depth estimation model for feature extraction to obtain a first feature map of the target image; Inputting the first feature map into an encoder in the depth estimation model to encode the first feature map to obtain a second feature map; Inputting the second feature map into a decoder in the depth estimation model to decode the second feature map to obtain a depth map corresponding to the target image, wherein the depth map includes depth information; The depth estimation model is obtained by performing model training based on the sample image and the real depth map of the sample image.

3. The method according to claim 1, characterized in that According to the depth information corresponding to the target image, the target image is converted into point cloud data, including: According to the conversion relationship between the camera coordinate system and the spherical coordinate system, the pixel coordinates of each pixel in the target image are converted into longitude information and latitude information; Calculate the three-dimensional coordinates of each pixel point in the spherical coordinate system according to the longitude information and latitude information of each pixel point; According to the depth information of each pixel point in the target image and the three-dimensional coordinates of each pixel point in the spherical coordinate system, the three-dimensional coordinates of each pixel point in the world coordinate system are determined to obtain the point cloud data.

4. The method according to claim 1, characterized in that: Based on the normal vector of the point cloud data, plane segmentation is performed on the point cloud data to obtain at least one target plane, including: For each point in the point cloud data, estimating a normal vector of each point based on information of a local neighborhood of each point; According to the normal vector of each point cloud data, the point cloud data is fitted and segmented by using a random extraction algorithm to obtain a plurality of initial planes; The angle between the normal vectors of the two initial planes is calculated, and two initial planes whose angle is smaller than a first set angle threshold are merged to obtain at least one target plane.

5. The method according to claim 4, characterized in that According to the normal vector of each point cloud data, the point cloud data is fitted and segmented by using a random extraction algorithm to obtain multiple initial planes, including: Step 1: randomly sampling a group of sample points from the point cloud data as points on an initial plane; Step 2: Calculate the reference normal vector of the initial plane using the normal vector of the sample point; Step 3, calculating the angle between the reference normal vector and the normal vectors of other points in the point cloud data, and selecting a point whose angle is less than a second set angle threshold from the other points as a point of the initial plane; Repeat steps 1-3 until all points in the point cloud data are divided into an initial plane.

6. The method according to any one of claims 1 to 5, characterized in that: Extracting at least one set of plane combinations from the at least one target plane comprises: Randomly sampling the at least one target plane at least once, randomly sampling two target planes each time, until a first target plane and a second target plane with an angle close to 90° are randomly sampled; According to the first target plane and the second target plane, determining a third target plane perpendicular to both the first target plane and the second target plane from other target planes except the first target plane and the second target plane; The first target plane, the second target plane and the third target plane are used as a first set of plane combinations, wherein the three target planes in the first set of plane combinations satisfy the right-hand coordinate system principle; According to the first group of plane combinations, other plane combinations are selected from the remaining target planes, and three target planes in the other plane combinations are respectively parallel to the three target planes in the first group of plane combinations and the distances therebetween are less than their respective corresponding distance thresholds.

7. The method according to claim 6, characterized in that According to the first target plane and the second target plane, determining a third target plane perpendicular to both the first target plane and the second target plane from other target planes except the first target plane and the second target plane, comprising: According to the first target plane and the second target plane, determining a candidate target plane perpendicular to both the first target plane and the second target plane from the other target planes; Selecting a target plane that is parallel or approximately parallel to the candidate target plane from the other target planes, wherein the candidate target plane and the target planes that are parallel or approximately parallel to the candidate target plane form a plane set; Calculate the score of each target plane in the plane set according to the number of points contained in each target plane in the plane set and the angles between the points; A target plane with the highest score is selected from the plane set as the third target plane.

8. The method according to any one of claims 1 to 7, characterized in that: Determining a corrected posture corresponding to the target image according to a normal vector matrix of the at least one set of plane combinations includes: Construct a normal vector matrix corresponding to each group of plane combinations according to the normal vectors of the three target planes in each group of plane combinations; The number of points contained in the three target planes in each plane combination is used as the weight to obtain the weight vector corresponding to each plane combination; According to the weight vector corresponding to each group of plane combinations, the normal vector matrix corresponding to each group of plane combinations is weighted summed to obtain the target normal vector matrix; Perform singular value decomposition on the target normal vector matrix to obtain a corrected pose corresponding to the target image.

9. The method according to any one of claims 1 to 7, characterized in that: The target image is an image corresponding to the target physical space, and extracting a reference plane from the point cloud data includes: Using a sliding window with a set width, sliding in the point cloud data along the y-axis direction representing the height, to determine a maximum height value and / or a minimum height value, the maximum height value corresponding to the ceiling of the target physical space, and the minimum height value corresponding to the ground of the target physical space; Continue to use the sliding window to slide along the x-axis and z-axis directions in the point cloud data with the maximum height value and / or the minimum height value to obtain the points that fall into the sliding window at the maximum height value and / or the minimum height value each time; Fitting is performed on the points falling into the sliding window at the maximum height value and / or the minimum height value each time to obtain the ceiling and / or the ground as the reference plane.

10. The method according to any one of claims 1 to 7, characterized in that: Also includes: When the corrected posture corresponding to the target image cannot be determined according to the normal vector of the reference plane, the posture of the IMU when the target image is acquired is obtained as the corrected posture corresponding to the target image.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of the method according to any one of claims 1 to 10.

Images