Camera Calibration Method, Device, and Storage Medium

Abstract

The present disclosure relates to a camera calibration method, device, and storage medium. By projecting a fisheye image of a room onto a panoramic view and performing gravity correction on the projected panoramic image to obtain a target panoramic image, it can ensure that all three-dimensional points on the ground are on the same plane. Based on the target panoramic image, the contour points of the ground included in the target panoramic image are determined, and the contour points are projected into three-dimensional space. The projected points of the contour points are clustered in three-dimensional space, and the contour of the ground is determined based on the clustering result, which can reduce the influence of contour point calculation errors on the ground contour estimation and improve the accuracy of ground contour determination. Furthermore, aligning the determined ground contour with the house type vector of the room can improve the accuracy of the alignment result, and thus improve the accuracy of camera calibration.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional fusion technology, and in particular, to a camera calibration method, device, and storage medium. Background Art

[0002] Related technologies can perform three-dimensional fusion on a fisheye image captured by a fisheye camera set in a room and a three-dimensional (i.e., 3D) box of a building information model (BIM) of the room to obtain a global viewing experience from a third-person perspective. To achieve the three-dimensional fusion of the fisheye image and the 3D box of the BIM, the house type vector of the room and the external parameters of the camera are necessary parameters that need to be known. Therefore, how to accurately perform camera calibration and obtain the external parameters of the camera is a technical problem that needs to be solved. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a camera calibration method, device, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a camera calibration method, the method comprising:

[0005] Project a fisheye image of a room onto a panoramic view, and perform gravity correction on the projected panoramic view to obtain a target panoramic view;

[0006] Based on the target panoramic view, determine the contour points of the ground included in the target panoramic view, and project the contour points into three-dimensional space;

[0007] Cluster the projection points of the contour points in the three-dimensional space, and determine the contour of the ground based on the clustering result;

[0008] Align the contour with the house type vector of the room, and determine the external parameters of the camera based on the alignment result.

[0009] Optionally, the clustering the projection points of the contour points in the three-dimensional space and determining the contour of the ground based on the clustering result includes:

[0010] Cluster the projection points in the directions of a first coordinate axis and a second coordinate axis respectively to obtain a plurality of point clusters in the directions of the first coordinate axis and the second coordinate axis, where the first coordinate axis and the second coordinate axis are the coordinate axes of a ground coordinate system;

[0011] Generate a plurality of first straight lines based on the plurality of point clusters in the direction of the first coordinate axis;

[0012] Generate multiple second straight lines based on multiple point clusters in the direction of the second coordinate axis;

[0013] Based on the multiple first straight lines and the multiple second straight lines, determine the contour of the ground in the three-dimensional space.

[0014] Optionally, the determining the contour of the ground in the three-dimensional space based on the multiple first straight lines and the multiple second straight lines includes:

[0015] Respectively determine multiple target straight lines with a distance greater than a preset distance from each other from the multiple first straight lines and the multiple second straight lines;

[0016] Determine the contour enclosed by the multiple target straight lines as the contour of the ground in the three-dimensional space.

[0017] Optionally, the aligning process of the contour and the house type vector of the room includes:

[0018] Extract the same number of points from the contour and the house type vector according to the same extraction method;

[0019] Perform an alignment process on the points on the contour and the house type vector.

[0020] Optionally, the aligning process of the points on the contour and the house type vector includes:

[0021] Perform an alignment process on the points on the contour and the house type vector based on the Plücker analysis method.

[0022] Optionally, the method of the first aspect further includes:

[0023] Perform semantic segmentation processing on the fisheye image to obtain a semantic segmentation result.

[0024] Optionally, the semantic segmentation result includes the position of the ground in the fisheye image;

[0025] The projecting the fisheye image of the room onto the panoramic view and performing gravity correction on the projected panoramic view to obtain a target panoramic view includes:

[0026] Project the fisheye image and the position of the ground in the fisheye image onto the panoramic view, and perform gravity correction on the projected panoramic view to obtain a target panoramic view and the position of the ground in the target panoramic view;

[0027] The determining the contour points of the ground included in the target panoramic view based on the target panoramic view includes:

[0028] Determine the contour points of the ground in the target panoramic view based on the position of the ground in the target panoramic view.

[0029] Optionally, the semantic segmentation result includes: the position of the door and / or window in the fisheye view;

[0030] The determining the external parameters of the camera based on the alignment processing result includes:

[0031] Based on the position of the door and / or window in the house type vector and the projection position of the door and / or window on the contour of the ground, adjust the alignment processing result so that the positions of the door and / or window in the house type vector and the contour are consistent;

[0032] Determine the external parameters of the camera based on the adjusted alignment processing result.

[0033] In a second aspect, an embodiment of the present disclosure provides a camera calibration device, and the device includes:

[0034] A first processing module, configured to project a fisheye view of a room onto a panoramic view perspective and perform gravity correction on the projected panoramic view to obtain a target panoramic view;

[0035] A second processing module, configured to determine the contour points of the ground included in the target panoramic view based on the target panoramic view and project the contour points into three-dimensional space;

[0036] A first determination module, configured to cluster the projected points of the contour points in the three-dimensional space and determine the contour of the ground based on the clustering result;

[0037] A second determination module, configured to perform alignment processing on the contour and the house type vector of the room and determine the external parameters of the camera based on the alignment processing result.

[0038] Optionally, the first determination module is configured to:

[0039] Cluster the projected points in the directions of the first coordinate axis and the second coordinate axis respectively to obtain a plurality of point clusters in the directions of the first coordinate axis and the second coordinate axis, where the first coordinate axis and the second coordinate axis are the coordinate axes of the ground coordinate system;

[0040] Generate a plurality of first straight lines based on the plurality of point clusters in the direction of the first coordinate axis;

[0041] Generate a plurality of second straight lines based on the plurality of point clusters in the direction of the second coordinate axis;

[0042] Determine the contour of the ground in the three-dimensional space based on the plurality of first straight lines and the plurality of second straight lines.

[0043] Optionally, the first determination module is configured to:

[0044] Determine a plurality of target straight lines with a distance greater than a preset distance from each other from the plurality of first straight lines and the plurality of second straight lines respectively;

[0045] Determine the contour surrounded by the plurality of target straight lines as the contour of the ground in the three-dimensional space.

[0046] Optionally, the second determination module is configured to:

[0047] Extract the same number of points from the contour and the house type vector according to the same extraction method;

[0048] Perform alignment processing on the points on the contour and the house type vector.

[0049] Optionally, the second determination module is configured to:

[0050] Perform alignment processing on the points on the contour and the house type vector based on the Plücker analysis method.

[0051] Optionally, the camera calibration device provided in the second aspect further includes:

[0052] A semantic segmentation module, configured to perform semantic segmentation processing on the fisheye image to obtain a semantic segmentation result.

[0053] Optionally, the semantic segmentation result includes the position of the ground in the fisheye image;

[0054] The first processing module is configured to: project the fisheye image and the position of the ground in the fisheye image onto the panoramic view perspective, and perform gravity correction on the projected panoramic view to obtain a target panoramic view, and the position of the ground in the target panoramic view;

[0055] The second processing module is configured to: determine the contour points of the ground in the target panoramic view based on the position of the ground in the target panoramic view.

[0056] Optionally, the semantic segmentation result includes: the position of the door and / or window in the fisheye image;

[0057] The second determination module is configured to:

[0058] Based on the positions of the door and / or window in the house type vector and the projected positions of the door and / or window on the contour of the ground, adjust the alignment processing result to make the positions of the door and / or window on the house type vector and the contour consistent;

[0059] Based on the adjusted alignment processing result, determine the external parameters of the camera.

[0060] In a third aspect, an embodiment of the present disclosure provides a computer device, which includes:

[0061] A memory;

[0062] A processor; and

[0063] A computer program;

[0064] Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the method described in the first aspect.

[0065] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method described in the first aspect.

[0066] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer program instructions, and when the computer program instructions are executed by a processor, the method described in the first aspect can be implemented.

[0067] The camera calibration method, device and storage medium provided by the embodiments of the present disclosure project the fisheye image of the room onto the panoramic view, and perform gravity correction on the projected panoramic view to obtain the target panoramic view, which can ensure that all three-dimensional points on the ground are on the same plane. By determining the contour points of the ground included in the target panoramic view based on the target panoramic view, projecting the contour points into the three-dimensional space, clustering the projected points of the contour points in the three-dimensional space, and determining the contour of the ground based on the clustering result, the influence of the contour point calculation error on the ground contour estimation can be reduced, and the accuracy of the ground contour determination can be improved. Furthermore, aligning the determined ground contour with the house type vector of the room can improve the accuracy of the alignment result, thereby improving the accuracy of camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0070] Figure 1 is a flowchart of a camera calibration method provided by an embodiment of the present disclosure;

[0071] Figure 2 It is a schematic diagram of a method for determining ground contour points provided by an embodiment of the present disclosure;

[0072] Figure 3 It is a flowchart of a method for determining the contour of the ground provided by an embodiment of the present disclosure;

[0073] Figure 4a It is a top view of the projection of ground contour points in three-dimensional space provided by an embodiment of the present disclosure;

[0074] Figure 4b It is for Figure 4a The schematic diagram of the clustering result obtained after clustering the projection points in;

[0075] Figure 4c It is a schematic diagram of generating a straight line based on a point cluster;

[0076] Figure 5 It is a flowchart of a method for aligning the contour of the ground and the house type vector of the room provided by an embodiment of the present disclosure;

[0077] Figure 6 It is a schematic diagram of a method for extracting points on the ground contour provided by an embodiment of the present disclosure;

[0078] Figure 7 It is a schematic structural diagram of a camera calibration device provided by an embodiment of the present disclosure;

[0079] Figure 8 It is a schematic structural diagram of an embodiment of a computer device provided by an embodiment of the present disclosure. Detailed implementation manners

[0080] In order to be able to more clearly understand the above objects, features, and advantages of the present disclosure, the solution of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0081] In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0082] Referring to the background art, in the related art, the fisheye image of the room can be three-dimensionally fused with the 3D box of the bim to obtain a third-person perspective global viewing experience. And the difficulty of this solution lies in:

[0083] In view of the above problems in the related art, the embodiments of the present disclosure provide a camera calibration method, device, and storage medium to accurately estimate the contour of the ground in three-dimensional space, solve the problem of greedy matching, and improve the accuracy of camera calibration.

[0084] The solutions of the embodiments of the present disclosure will be described below in conjunction with exemplary embodiments.

[0085] Exemplarily, Figure 1 is a flowchart of a camera calibration method provided by an embodiment of the present disclosure. This method can be exemplarily executed by a computer device, which can be any device with image processing and computing capabilities, such as terminal devices like mobile phones and computers, computers and servers equipped with image processing systems, etc., but is not limited to the devices listed here. As Figure 1 shown, in some embodiments, the camera calibration method provided by the embodiments of the present disclosure may include steps 101 - 104.

[0086] Step 101: Project the fisheye image of the room onto the panoramic view, and perform gravity correction on the projected panoramic image to obtain a target panoramic image.

[0087] The fisheye image referred to in the embodiments of the present disclosure may be captured by a fisheye camera set at a higher or lower position in the room. For ease of understanding, in the embodiments of the present disclosure, the fisheye image may be exemplarily understood as a fisheye image with a bird's-eye view captured by a fisheye camera set at a higher position (such as the roof) in the room. The fisheye image may exemplarily include images of the ground, walls, doors, windows, etc. The embodiments of the present disclosure take the case where the fisheye camera is set at a higher position in the room as an example to illustrate the camera calibration method. The camera calibration method when the fisheye camera is set at a lower position (such as the center of the ground) in the room is similar to the case where the fisheye camera is set at a higher position in the room, and the embodiments of the present disclosure will not repeat the description.

[0088] In the embodiments of the present disclosure, any method in the related art can be used to project the fisheye image onto the panoramic view to obtain a panoramic image. For example, in one example, the fisheye image can be mapped to the camera coordinate system according to the mapping relationship between the image coordinate system and the camera coordinate system of the fisheye image, and then based on the mapping relationship between the camera coordinate system and the world coordinate system, the fisheye image is mapped from the camera coordinate system to the world coordinate system. Furthermore, according to the mapping relationship between the world coordinate system and the two-dimensional coordinate system of the panoramic view, the fisheye image is mapped from the world coordinate system to the two-dimensional coordinate system of the panoramic view, thereby obtaining the panoramic image corresponding to the fisheye image. Of course, this is only an example for illustration and not the only limitation on the method of projecting the fisheye image onto the panoramic view.

[0089] In the embodiments of the present disclosure, the method for gravity correction of the panoramic view is similar to the gravity correction method provided by the related art, which will not be elaborated here. By performing gravity correction on the panoramic view, it can be ensured that the points on the ground are in the same plane in the three-dimensional space, so that when the ground contour in the panoramic view is projected into the three-dimensional space, contour information that conforms to the physical reality can be obtained.

[0090] Step 102: Based on the target panoramic view, determine the contour points of the ground included in the target panoramic view, and project the contour points into the three-dimensional space.

[0091] In the embodiments of the present disclosure, there are various methods for determining the contour points of the ground included in the target panoramic view based on the target panoramic view. For the sake of easy understanding, two examples are used for illustrative purposes below. It should be noted that the following two methods for determining the ground contour points are only two exemplary methods rather than all methods. In fact, any method in the related art that can determine the ground contour points based on the target panoramic view can be included within the protection scope of the embodiments of the present disclosure.

[0092] Exemplarily, in some exemplary embodiments, the target panoramic view can be input into a preset ground contour point extraction model, and through the ground contour point extraction model, the contour points of the ground in the target panoramic view can be extracted from the target panoramic view. Among them, the ground contour point extraction model can adopt the model training method provided by the related art and be trained with the panoramic view marked with the ground contour points. The specific training method can refer to the related art and will not be elaborated here.

[0093] Exemplarily, Figure 2 is a schematic diagram of a method for determining the ground contour points provided by the embodiments of the present disclosure. As Figure 2 shown, in some other exemplary embodiments, the fisheye view of the room can also be subjected to semantic segmentation processing through a preset semantic segmentation model to obtain the positions of the ground, walls, doors, and windows included in the fisheye view, such as Figure 2 the semantic segmentation result shown in 2a. Then, the fisheye view and the positions of the ground, walls, doors, and windows are projected into the panoramic view perspective, and the projected panoramic view is subjected to gravity correction to obtain the target panoramic view and the position of the ground in the target panoramic view, such as Figure 2 the target panoramic view and the position of the ground included in the target panoramic view shown in 2b. Further, based on the position of the ground in the target panoramic view, the contour points of the ground are extracted from the ground edge. For example, Figure 2 the contour points shown in 2c.

[0094] Optionally, in some exemplary embodiments, after obtaining the contour points of the ground in the target panoramic image, the contour points of the ground can be projected into a three-dimensional space based on the Manhattan assumption (i.e., the ground, walls, and ceiling of an indoor scene are usually aligned in three mutually perpendicular principal directions), and the projection points of the contour points of the ground in the three-dimensional space are obtained.

[0095] Step 103: Cluster the projection points of the contour points in the three-dimensional space, and determine the contour of the ground based on the clustering result.

[0096] Exemplarily, Figure 3 is a flowchart of a method for determining the contour of the ground provided by an embodiment of the present disclosure. As Figure 3 shown, in some embodiments, the contour of the ground of a room in the three-dimensional space can be determined through the following steps 301-step 304.

[0097] Step 301: Cluster the projection points in the directions of the first coordinate axis and the second coordinate axis respectively to obtain a plurality of point clusters in the directions of the first coordinate axis and the second coordinate axis, where the first coordinate axis and the second coordinate axis are the coordinate axes of the ground coordinate system.

[0098] Step 302: Generate a plurality of first straight lines based on the plurality of point clusters in the direction of the first coordinate axis.

[0099] Step 303: Generate a plurality of second straight lines based on the plurality of point clusters in the direction of the second coordinate axis.

[0100] Among them, the execution order between step 302 and step 303 can be arbitrary.

[0101] Step 304: Determine the contour of the ground in the three-dimensional space based on the plurality of first straight lines and the plurality of second straight lines.

[0102] For example, Figure 4a is a top view of the projection of the contour points of the ground in the three-dimensional space provided by an embodiment of the present disclosure. Figure 4b is for Figure 4a the schematic diagram of the clustering result obtained after clustering the projection points in, where the projection points in a dashed box are a point cluster. Figure 4c is a schematic diagram of generating a straight line based on the point cluster. As Figure 4a and Figure 4b shown, the horizontal axis of the ground coordinate system is x (which can be exemplarily understood as the first coordinate axis), and the vertical axis is y (which can be exemplarily understood as the second coordinate axis). The projection points in Figure 4a can be clustered in the x-axis direction and the y-axis direction respectively to obtain Figure 4bThe clustering results shown. Further, based on the points in the same point cluster, a straight line is fitted to obtain multiple first straight lines in the x-axis direction and multiple second straight lines in the y-axis direction, resulting in the straight lines shown as Figure 4c The straight lines shown, so that the contour enclosed by these straight lines is the contour of the ground in three-dimensional space. By clustering the projection points of the contour points of the ground in three-dimensional space, fitting a straight line according to the clustering results, and determining the contour of the ground in three-dimensional space based on the fitted straight line, the influence of the calculation error of the ground contour points on the ground contour estimation can be reduced, and the accuracy of the ground contour estimation can be improved.

[0103] It can be understood that the above examples are only for illustrative purposes and not the only limitations. For example, in some embodiments, multiple first straight lines and multiple second straight lines can also be screened according to a preset distance, the straight lines among the multiple first straight lines with a distance less than or equal to the preset distance between each other are removed, and the remaining first straight lines are used as target straight lines. Similarly, the straight lines among the multiple second straight lines with a distance less than or equal to the preset distance between each other are removed, and the remaining second straight lines are used as target straight lines. Thus, the contour enclosed by the target straight lines is determined as the contour of the ground in three-dimensional space. For example, among the multiple first straight lines, the distance between straight line a and straight line b is less than the preset distance, and the distance between straight line a and straight line c is greater than the preset distance, then straight line b is deleted, and straight line a and c are used as target straight lines. That is, for any straight line among the multiple first straight lines, as long as the distance between this straight line and any other straight line among the multiple first straight lines is less than or equal to the preset distance, then this straight line is removed.

[0104] Step 104: Align the contour of the ground and the house type vector of the room, and determine the external parameters of the camera based on the alignment result.

[0105] In the embodiments of the present disclosure, the contour of the ground and the house type vector of the room can be aligned in three-dimensional space to obtain the scale relationship, rotation angle, and translation relationship between the contour of the ground and the house type vector. Among them, for the alignment method of the contour of the ground and the house type vector of the room, for example, the alignment methods provided by related technologies (such as point pair matching methods) can be used, but it is not limited to the alignment methods provided by related technologies.

[0106] Further, after obtaining the scale relationship, rotation angle, and translation relationship between the ground contour and the house type vector, the alignment result (mainly the rotation angle) can be adjusted according to the positions of the doors and / or windows in the house type vector and the projected positions of the doors and / or windows on the ground contour, so that the positions of the adjusted doors and / or windows on the house type vector are consistent with the projected positions of the doors and / or windows on the ground contour. Thus, based on the adjustment result, the mapping relationship between the points on the fisheye image and the points in the three-dimensional space can be determined, and the external parameters of the camera can be obtained. The specific method for determining the external parameters of the camera based on the alignment result can refer to the related technology and will not be elaborated here.

[0107] Adjusting the alignment result through the positions of the doors and / or windows can make the alignment relationship between the ground contour and the house type vector more in line with the physical reality and improve the accuracy of alignment.

[0108] In some embodiments, after obtaining the external parameters of the camera, the fisheye image captured by the camera can be fused into the 3D box of the BIM based on the external parameters of the camera to improve the fusion effect.

[0109] In the embodiments of the present disclosure, by projecting the fisheye image of the room onto the panoramic view and performing gravity correction on the projected panoramic view to obtain the target panoramic view, it can be ensured that all the three-dimensional points on the ground are on the same plane. By determining the contour points of the ground included in the target panoramic view based on the target panoramic view and projecting the contour points into the three-dimensional space, clustering the projected points of the contour points in the three-dimensional space, and determining the contour of the ground based on the clustering result, the influence of the contour point calculation error on the ground contour estimation can be reduced, and the accuracy of the ground contour determination can be improved. Furthermore, aligning the determined ground contour with the house type vector of the room can improve the accuracy of the alignment result, and thus improve the accuracy of camera calibration.

[0110] Figure 5 is a flowchart of a method for aligning the ground contour and the house type vector of a room provided by the embodiments of the present disclosure. As Figure 5 shown, in some embodiments, the ground contour and the house type vector of the room can be aligned by the method of step 501-step 502.

[0111] Step 501: Extract the same number of points from the ground contour and the house type vector according to the same extraction method.

[0112] Exemplarily, Figure 6 is a schematic diagram of a method for extracting points on the ground contour provided by the embodiments of the present disclosure. In Figure 6 the point o is the geometric center of the ground contour, and the coordinate axes x and y are the coordinate axes of the ground coordinate system. As Figure 6As shown, in some embodiments, rays can be drawn starting from the geometric center o at every preset angle Q of rotation, and the intersection points between the rays and the contour of the ground can be obtained. By analogy, multiple points can be obtained from the contour of the ground. Similarly, starting from the geometric center of the house type vector, the same number of points can be obtained from the house type vector using the same extraction method (such as the same rotation direction and the same angles between the first ray drawn from the geometric center and the x-axis and y-axis).

[0113] It should be noted that Figure 6 This is only an example illustration and not the only illustration of the extraction method referred to in the embodiments of the present disclosure.

[0114] Step 502: Align the points on the contour of the ground and the house type vector of the room.

[0115] In 3D space, as long as two contours have the same points, the Plücker analysis method can be used to align the two contours. Therefore, in an exemplary embodiment, after obtaining the same number of points from the contour of the ground and the house type vector of the room based on the method of step 501, the points on the contour of the ground and the house type vector of the room can be aligned by the Plücker analysis method to obtain the scale relationship, rotation angle, and translation relationship between the contour of the ground and the house type vector.

[0116] In the embodiments of the present disclosure, by extracting the same number of points from the contour of the ground and the house type vector in the same extraction method, the unification of the dimensions of the contour of the ground and the house type vector is achieved. Then, the Plücker analysis method is used to align these points, which can avoid the problem of greedy matching, reduce the error of contour matching, improve the accuracy of contour alignment, and further improve the accuracy of camera calibration based on the alignment result.

[0117] Figure 7 It is a schematic structural diagram of a camera calibration device provided by the embodiments of the present disclosure. This camera calibration device can be exemplarily understood as the computer device or a partial functional module in the computer device in the above method embodiments, such as Figure 7 As shown, in some embodiments, the camera calibration device 70 provided by the embodiments of the present disclosure may include:

[0118] A first processing module 71, configured to project the fisheye image of the room onto the panoramic view and perform gravity correction on the projected panoramic view to obtain a target panoramic view;

[0119] A second processing module 72, configured to determine the contour points of the ground included in the target panoramic view based on the target panoramic view and project the contour points into the three-dimensional space;

[0120] The first determination module 73 is configured to cluster the projection points of the contour points in the three-dimensional space, and determine the contour of the ground based on the clustering result;

[0121] The second determination module 74 is configured to align the contour and the house type vector of the room, and determine the external parameters of the camera based on the alignment result.

[0122] Optionally, the first determination module 73 is configured to:

[0123] Cluster the projection points in the directions of the first coordinate axis and the second coordinate axis respectively, to obtain multiple point clusters in the directions of the first coordinate axis and the second coordinate axis, where the first coordinate axis and the second coordinate axis are the coordinate axes of the ground coordinate system;

[0124] Generate multiple first straight lines based on the multiple point clusters in the direction of the first coordinate axis;

[0125] Generate multiple second straight lines based on the multiple point clusters in the direction of the second coordinate axis;

[0126] Determine the contour of the ground in the three-dimensional space based on the multiple first straight lines and the multiple second straight lines.

[0127] Optionally, the first determination module 73 is configured to:

[0128] Determine multiple target straight lines with a distance greater than a preset distance from each other respectively from the multiple first straight lines and the multiple second straight lines;

[0129] Determine the contour surrounded by the multiple target straight lines as the contour of the ground in the three-dimensional space.

[0130] Optionally, the second determination module 74 is configured to:

[0131] Extract the same number of points from the contour and the house type vector according to the same extraction method;

[0132] Align the points on the contour and the house type vector.

[0133] Optionally, the second determination module 74 is configured to:

[0134] Align the points on the contour and the house type vector based on the Plücker analysis method.

[0135] Optionally, the camera calibration device provided in the second aspect further includes:

[0136] A semantic segmentation module, configured to perform semantic segmentation processing on the fisheye image to obtain a semantic segmentation result.

[0137] Optionally, the semantic segmentation result includes the position of the ground in the fisheye image;

[0138] A first processing module 71, configured to project the fisheye image and the position of the ground in the fisheye image onto a panoramic view perspective, and perform gravity correction on the projected panoramic image to obtain a target panoramic image and the position of the ground in the target panoramic image;

[0139] A second processing module 72, configured to determine the contour points of the ground in the target panoramic image based on the position of the ground in the target panoramic image.

[0140] Optionally, the semantic segmentation result includes the position of a door and / or a window in the fisheye image;

[0141] A second determination module 74, configured to:

[0142] Based on the positions of the door and / or window in the house type vector and the projected positions of the door and / or window on the contour of the ground, adjust the alignment processing result so that the positions of the door and / or window in the house type vector and the contour are consistent;

[0143] Based on the adjusted alignment processing result, determine the external parameters of the camera.

[0144] The device provided in the embodiments of the present disclosure can execute the methods in any of the above method embodiments, and its execution manner and beneficial effects are similar and will not be elaborated here.

[0145] It should also be noted that the division of modules in the camera calibration device in the embodiments of the present disclosure is illustrative, merely a logical function division, and there may be other division methods in actual implementation. In addition, in each embodiment of the present application, each functional module may be integrated into one processing module, may exist separately physically for each module, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0146] If the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a processor-readable storage medium. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in the embodiments of the present disclosure.

[0147] Figure 8 The following is a schematic structural diagram of an embodiment of a computer device provided by an embodiment of the present disclosure. As Figure 8 shown, the computer device includes a memory 121 and a processor 122.

[0148] The memory 121 is used to store programs. In addition to the above programs, the memory 121 can also be configured to store various other data to support operations on the computer device. Examples of such data include instructions for any application or method for operating on the computer device, contact member data, phone book member data, messages, pictures, videos, etc.

[0149] The memory 121 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0150] The processor 122 is coupled to the memory 121 and executes the programs stored in the memory 121 for:

[0151] Projecting the fisheye image of the room onto a panoramic view, and performing gravity correction on the projected panoramic view to obtain a target panoramic view;

[0152] Based on the target panoramic view, determining the contour points of the ground included in the target panoramic view, and projecting the contour points into three-dimensional space;

[0153] Clustering the projection points of the contour points in the three-dimensional space, and determining the contour of the ground based on the clustering result;

[0154] Performing alignment processing on the contour and the house type vector of the room, and determining the external parameters of the camera based on the result of the alignment processing.

[0155] Optionally, the clustering the projection points of the contour points in the three-dimensional space and determining the contour of the ground based on the clustering result includes:

[0156] Clustering the projection points in the directions of the first coordinate axis and the second coordinate axis respectively to obtain a plurality of point clusters in the directions of the first coordinate axis and the second coordinate axis, where the first coordinate axis and the second coordinate axis are the coordinate axes of the ground coordinate system;

[0157] Generating a plurality of first straight lines based on the plurality of point clusters in the direction of the first coordinate axis;

[0158] Generate multiple second straight lines based on multiple point clusters in the direction of the second coordinate axis;

[0159] Based on the multiple first straight lines and the multiple second straight lines, determine the contour of the ground in the three-dimensional space.

[0160] Optionally, the determining the contour of the ground in the three-dimensional space based on the multiple first straight lines and the multiple second straight lines includes:

[0161] Respectively determine multiple target straight lines with a distance greater than a preset distance from each other from the multiple first straight lines and the multiple second straight lines;

[0162] Determine the contour enclosed by the multiple target straight lines as the contour of the ground in the three-dimensional space.

[0163] Optionally, the aligning process of the contour and the house type vector of the room includes:

[0164] Extract the same number of points from the contour and the house type vector according to the same extraction method;

[0165] Perform an alignment process on the points on the contour and the house type vector.

[0166] Optionally, the aligning process of the points on the contour and the house type vector includes:

[0167] Perform an alignment process on the points on the contour and the house type vector based on the Plücker analysis method.

[0168] Optionally, the method of the first aspect further includes:

[0169] Perform semantic segmentation processing on the fisheye image to obtain a semantic segmentation result.

[0170] Optionally, the semantic segmentation result includes the position of the ground in the fisheye image;

[0171] The projecting the fisheye image of the room to the panoramic view and performing gravity correction on the projected panoramic view to obtain a target panoramic view includes:

[0172] Project the fisheye image and the position of the ground in the fisheye image to the panoramic view, and perform gravity correction on the projected panoramic view to obtain a target panoramic view and the position of the ground in the target panoramic view;

[0173] The determining the contour points of the ground included in the target panoramic view based on the target panoramic view includes:

[0174] Determine the contour points of the ground in the target panoramic view based on the position of the ground in the target panoramic view.

[0175] Optionally, the semantic segmentation result includes: the positions of doors and / or windows in the fisheye view;

[0176] The determining the external parameters of the camera based on the alignment processing result includes:

[0177] Based on the positions of the doors and / or windows in the house type vector and the projection positions of the doors and / or windows on the contour of the ground, adjust the alignment processing result so that the positions of the doors and / or windows in the house type vector and on the contour are consistent;

[0178] Determine the external parameters of the camera based on the adjusted alignment processing result.

[0179] Furthermore, as Figure 8 shown, the computer device may further include: other components such as a communication component 123, a power supply component 124, an audio component 125, a display 126, etc. Figure 8 Only some components are schematically shown in Figure 8 shown, which does not mean that the computer device only includes

[0180] The communication component 123 is configured to facilitate wired or wireless communication between the computer device and other devices. The computer device can access a wireless network based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 123 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 123 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0181] The power supply component 124 provides power for various components of the computer device. The power supply component 124 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the computer device.

[0182] The audio component 125 is configured to output and / or input audio signals. For example, the audio component 125 includes a microphone (MIC), which is configured to receive external audio signals when the computer device is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 121 or transmitted via the communication component 123. In some embodiments, the audio component 125 further includes a speaker for outputting audio signals.

[0183] The display 126 includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations.

[0184] In addition, embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method described in any of the above method embodiments.

[0185] In embodiments of the present disclosure, the above computer-readable storage medium may be any available medium or data storage device accessible by a processor, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid state drives (SSD)).

[0186] Those skilled in the art should understand that embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0187] Embodiments of the present disclosure provide a computer program product, including computer program instructions, and when the computer program instructions are executed by a processor, the method described in any of the above method embodiments can be implemented.

[0188] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0189] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but rather will conform to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera calibration method, characterized in that: The method comprises: Project the fisheye image of the room to the panoramic view, and perform gravity correction on the projected panoramic image to obtain the target panoramic image; Based on the target panoramic image, determining contour points of the ground contained in the target panoramic image, and projecting the contour points into a three-dimensional space; Clustering the projection points of the contour points in the three-dimensional space, and determining the contour of the ground based on the clustering result; The outline and the room type vector are aligned, and the external parameters of the camera are determined based on the alignment result.

2. The method according to claim 1, characterized in that The step of clustering the projection points of the contour points in the three-dimensional space and determining the contour of the ground based on the clustering result includes: Clustering the projection points in the directions of a first coordinate axis and a second coordinate axis respectively to obtain a plurality of point clusters in the directions of the first coordinate axis and the second coordinate axis, wherein the first coordinate axis and the second coordinate axis are coordinate axes of a ground coordinate system; Generating a plurality of first straight lines based on a plurality of point clusters in the direction of the first coordinate axis; generating a plurality of second straight lines based on a plurality of point clusters in the direction of the second coordinate axis; Based on the plurality of first straight lines and the plurality of second straight lines, a contour of the ground in the three-dimensional space is determined.

3. The method according to claim 2, characterized in that The step of determining the contour of the ground in the three-dimensional space based on the plurality of first straight lines and the plurality of second straight lines comprises: Determining, from the plurality of first straight lines and the plurality of second straight lines, a plurality of target straight lines whose distances from each other are greater than a preset distance; The outline enclosed by the multiple target straight lines is determined as the outline of the ground in the three-dimensional space.

4. The method according to claim 1, characterized in that: The aligning process of the outline and the room type vector comprises: Extracting the same number of points from the contour and the apartment type vector in the same extraction manner; Align the outline and the points on the apartment type vector.

5. The method according to claim 4, characterized in that The aligning of the points on the contour and the apartment type vector comprises: The points on the outline and the apartment type vector are aligned based on the Prucker analysis method.

6. The method according to claim 1, characterized in that The method further comprises: Performing semantic segmentation processing on the fisheye image to obtain a semantic segmentation result, wherein the semantic segmentation result includes a position of the ground in the fisheye image; The fisheye image of the room is projected to a panoramic view, and gravity corrected for the projected panoramic image to obtain a target panoramic image, including: Projecting the fisheye image and the position of the ground in the fisheye image to a panoramic view, and performing gravity correction on the projected panoramic image to obtain a target panoramic image and the position of the ground in the target panoramic image; The step of determining the contour points of the ground contained in the target panoramic image based on the target panoramic image includes: Based on the position of the ground in the target panoramic image, contour points of the ground in the target panoramic image are determined.

7. The method according to claim 6, characterized in that The semantic segmentation result includes: the position of the door and / or window in the fisheye image; The step of determining the external parameters of the camera based on the alignment processing result includes: Based on the positions of the door and / or window in the apartment vector and the projection positions of the door and / or window on the outline of the ground, adjusting the alignment processing result so that the positions of the door and / or window on the apartment vector and the outline are consistent; Based on the adjusted alignment processing result, the extrinsic parameters of the camera are determined.

8. A computer device, characterized in that: The computer device comprises: Memory; Processor; and Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising computer program instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 7.

Images