Image acquisition equipment external parameter calibration method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image acquisition equipment external parameter calibration method, electronic equipment and a storage medium, and the method comprises the steps: determining a building information model corresponding to a target room based on a grid house type diagram corresponding to the target room; wherein the building information model comprises a plurality of 3D key points in the target room; key point extraction is carried out on a real plane image collected by an image collection device and corresponding to the target room, and at least one 2D key point group is obtained; wherein each 2D key point group comprises a plurality of 2D key points corresponding to the target room; determining n groups of matching point pairs corresponding to the target room; wherein n is greater than or equal to 4, and each group of matching point pairs comprises a 3D key point and a corresponding 2D key point; and based on the n groups of matching point pairs, external parameter information of image acquisition equipment for acquiring the real plane image in the target room is determined.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional fusion technology, and in particular to a calibration method for external parameters of an image acquisition device, an electronic device, and a storage medium. Background Art

[0002] 3D fusion technology has multiple meanings in decoration scenarios. First, the video content presents the status of the construction site in a very intuitive way, whether it is online inspection or quality inspection, which greatly improves the efficiency of construction site management; for users, being able to see the decoration status of the house in real time can make users more confident in the decoration work; second, it is a promotional experience. When fisheye cameras are installed on the decoration site, more areas can be observed through the larger wide-angle range of the fisheye camera; and the 3D fusion technology successfully compensates for the poor viewing experience caused by the distortion of the wide-angle fisheye camera image.

[0003] 3D fusion technology maps points on a 2D image to a 3D world coordinate system when the camera's internal and external parameters are known, presenting the 2D image in 3D to restore the real physical scene. The internal parameters are generally determined by the camera hardware, so the camera's external parameters need to be determined. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a calibration method for external parameters of an image acquisition device, an electronic device and a storage medium.

[0005] According to one aspect of an embodiment of the present disclosure, a method for calibrating external parameters of an image acquisition device is provided, comprising:

[0006] Determine a building information model corresponding to the target room based on a grid floor plan corresponding to the target room; wherein the building information model includes a plurality of 3D key points in the target room;

[0007] Extract key points from the real plane image corresponding to the target room and acquired by the image acquisition device to obtain at least one group of 2D key points; wherein each group of 2D key points includes a plurality of 2D key points corresponding to the target room;

[0008] Determine n groups of matching point pairs corresponding to the target room; wherein n is greater than or equal to 4, and each group of matching point pairs includes a 3D key point and a corresponding 2D key point;

[0009] Based on the n groups of matching point pairs, external parameter information of an image acquisition device that acquires the real plane image in the target room is determined.

[0010] Optionally, determining the building information model corresponding to the target room based on the grid floor plan corresponding to the target room includes:

[0011] The grid floor plan is converted by a preset conversion model to obtain a vector map corresponding to the target room; wherein the vector map includes plane information of at least one structure included in the target room;

[0012] The building information model corresponding to the target room is determined based on the vector diagram and preset height information corresponding to different structures in the target room.

[0013] Optionally, extracting key points from the real plane image corresponding to the target room and acquired by the image acquisition device to obtain at least one group of 2D key points includes:

[0014] For a real plane image corresponding to the target room, line segment detection is performed on the real plane image to obtain a plurality of first line segments included in the target room; wherein the first line segments represent intersection lines between walls, or represent intersection lines between a wall and a door;

[0015] A 2D key point group corresponding to the room is determined based on an endpoint of each first line segment of the plurality of first line segments close to a center point of the real plane image.

[0016] Optionally, performing line segment detection on the real plane image to obtain a plurality of first line segments included in the target room includes:

[0017] Perform line segment detection on the real plane image using a segmentation model and / or a line segment detection model to obtain the plurality of first line segments.

[0018] Optionally, determining n groups of matching point pairs corresponding to the target room includes:

[0019] Based on the internal reference information of the image acquisition device corresponding to the target room and multiple assumed external reference information, projecting the building information model corresponding to the room into the 2D image to obtain multiple simulated plane images;

[0020] Based on the real plane image, determine the simulated plane image having the highest similarity to the real plane image from among the multiple simulated plane images as the target plane image;

[0021] Based on the multiple 3D key points corresponding to the target plane image and the multiple 2D key points in the real plane image, n groups of matching point pairs corresponding to the room are obtained.

[0022] Optionally, obtaining n groups of matching point pairs corresponding to the room based on the multiple 3D key points corresponding to the target plane image and the multiple 2D key points in the real plane image includes:

[0023] Performing line segment detection on the target plane image to obtain a plurality of second line segments included in the target plane image;

[0024] The target plane image is rotated multiple times at preset angle intervals to obtain multiple third line segment groups; wherein each third line segment group includes multiple third line segments in the target plane image corresponding to a rotation angle;

[0025] Based on the plurality of first line segments, determining a plurality of third line segments matching the plurality of first line segments from the plurality of groups of third line segments;

[0026] Based on the endpoint of each third line segment in the plurality of the third line segments close to the center point of the target plane image, a plurality of 3D key points matching the 2D key point group are determined to obtain n groups of matching point pairs corresponding to the room.

[0027] Optionally, the determining, based on the plurality of first line segments, from the plurality of third line segment groups, a plurality of third line segments matching the plurality of first line segments comprises:

[0028] Determine a plurality of cost matrices based on each first line segment of the plurality of first line segments and each third line segment group of the plurality of third line segment groups; wherein each cost matrix corresponds to a group of the third line segment groups, and each row in the cost matrix represents a plurality of cost values ​​between one of the first line segments and a plurality of the third line segments;

[0029] Based on the matrix cost values ​​corresponding to each of the cost matrices, determining the cost matrix with the smallest matrix cost value as the target cost matrix;

[0030] Based on the target cost matrix, a plurality of the third line segments matching the plurality of the first line segments are determined.

[0031] Optionally, determining the external parameter information of an image acquisition device for acquiring the real plane image in the target room based on the n groups of matching point pairs includes:

[0032] Reorganize the n groups of matching point pairs into multiple groups of group matching point pairs; wherein each group of group matching point pairs includes 4 groups of matching point pairs;

[0033] Based on the multiple groups of group matching point pairs, determining 4 groups of target matching point pairs;

[0034] Based on the four groups of target matching point pairs, external parameter information of an image acquisition device that acquires the real plane image in the target room is determined.

[0035] Optionally, the determining four groups of target matching point pairs based on the multiple groups of group matching point pairs includes:

[0036] Re-projection errors and position information of the image acquisition device are calculated based on the multiple groups of group matching point pairs to obtain multiple groups of re-projection errors and position information; wherein each group of group matching points corresponds to a group of re-projection errors and position information;

[0037] Based on the multiple groups of reprojection errors and position information, determining at least one group of the grouped matching point pairs from the multiple groups of grouped matching point pairs as a target matching point pair;

[0038] Four groups of target matching point pairs are obtained from the at least one group of target matching point pairs.

[0039] Optionally, obtaining four groups of target matching point pairs from the at least one group of target matching point pairs comprises:

[0040] Determining the frequency of occurrence of each of the target matching point pairs in the at least one set of target matching point pairs;

[0041] Based on the frequency of occurrence, the four groups of target matching point pairs with the highest frequency of occurrence are determined.

[0042] Optionally, before extracting key points from the real plane image corresponding to each room in the at least one room to obtain at least one group of 2D key points, the method further includes:

[0043] At least one of the real plane images is obtained by performing image acquisition using an image acquisition device preset in each of the at least one room.

[0044] According to another aspect of an embodiment of the present disclosure, a device for calibrating external parameters of an image acquisition device is provided, comprising:

[0045] An information model module, used to determine a building information model corresponding to the target room based on a grid floor plan corresponding to the target room; wherein the building information model includes a plurality of 3D key points in the target room;

[0046] An image key point extraction module is used to extract key points from a real plane image corresponding to the target room and acquired by an image acquisition device to obtain at least one group of 2D key point groups; wherein each group of 2D key point groups includes a plurality of 2D key points corresponding to the target room;

[0047] A key point matching module, used to determine n groups of matching point pairs corresponding to the target room; wherein n is greater than or equal to 4, and each group of matching point pairs includes a 3D key point and a corresponding 2D key point;

[0048] An external parameter determination module is used to determine the external parameter information of the image acquisition device that acquires the real plane image in the target room based on the n groups of matching point pairs.

[0049] Optionally, the information model module is specifically used to convert the grid floor plan through a preset conversion model to obtain a vector diagram corresponding to the target room; wherein the vector diagram includes plane information of at least one structure included in the target room; based on the vector diagram and preset height information corresponding to different structures in the target room, determine the building information model corresponding to the target room.

[0050] Optionally, the image key point extraction module includes:

[0051] A line segment detection unit, configured to perform line segment detection on the real plane image corresponding to the target room to obtain a plurality of first line segments included in the target room; wherein the first line segments represent intersection lines between walls, or represent intersection lines between a wall and a door;

[0052] A key point determination unit is used to determine a 2D key point group corresponding to the room based on an endpoint of each first line segment in the plurality of first line segments close to a center point of the real plane image.

[0053] Optionally, the line segment detection unit is specifically configured to perform line segment detection on the real plane image using a segmentation model and / or a line segment detection model to obtain the plurality of first line segments.

[0054] Optionally, the key point matching module includes:

[0055] An image simulation unit, configured to project a building information model corresponding to the target room into a 2D image based on the internal reference information of the image acquisition device corresponding to the target room and a plurality of assumed external reference information, so as to obtain a plurality of simulated plane images;

[0056] an image screening unit, configured to determine, based on the real plane image, from among the multiple simulated plane images, the simulated plane image having the highest similarity to the real plane image as a target plane image;

[0057] A point pair matching unit is used to obtain n groups of matching point pairs corresponding to the room based on multiple 3D key points corresponding to the target plane image and multiple 2D key points in the real plane image.

[0058] Optionally, the point pair matching unit is specifically used to perform line segment detection on the target plane image to obtain multiple second line segments included in the target plane image; rotate the target plane image multiple times at preset angle intervals to obtain multiple groups of third line segment groups; wherein each group of the third line segment groups includes multiple third line segments in the target plane image corresponding to a rotation angle; based on the multiple first line segments, determine multiple third line segments matching the multiple first line segments from the multiple groups of third line segment groups; based on the endpoints of each third line segment in the multiple third line segments close to the center point of the target plane image, determine multiple 3D key points matching the 2D key point group to obtain n groups of matching point pairs corresponding to the room.

[0059] Optionally, when the point pair matching unit determines multiple third line segments matching the multiple first line segments from the multiple groups of third line segment groups based on the multiple first line segments, it is used to determine multiple cost matrices based on each first line segment in the multiple first line segments and each group of third line segment groups in the multiple groups of third line segment groups; wherein each cost matrix corresponds to a group of third line segment groups, and each row in the cost matrix represents multiple cost values ​​between one first line segment and multiple third line segments; based on the matrix cost value corresponding to each cost matrix, the cost matrix with the smallest matrix cost value is determined as the target cost matrix; based on the target cost matrix, multiple third line segments matching the multiple first line segments are determined.

[0060] Optionally, the external parameter determination module is specifically used to reorganize the n groups of matching point pairs into multiple groups of group matching point pairs; wherein each group of group matching point pairs includes 4 groups of matching point pairs; based on the multiple groups of group matching point pairs, 4 groups of target matching point pairs are determined; based on the 4 groups of target matching point pairs, the external parameter information of the image acquisition device that captures the real plane image in the target room is determined.

[0061] Optionally, when the external parameter determination module determines four groups of target matching point pairs based on the multiple groups of group matching point pairs, it is used to calculate the reprojection errors and the position information of the image acquisition device based on the multiple groups of group matching point pairs, so as to obtain multiple groups of reprojection errors and position information; wherein each group of the group matching points corresponds to a group of reprojection errors and position information; based on the multiple groups of reprojection errors and position information, at least one group of the group matching point pairs is determined from the multiple groups of group matching point pairs as the target matching point pairs; and four groups of target matching point pairs are obtained from the at least one group of target matching point pairs.

[0062] Optionally, when the external parameter determination module obtains 4 groups of target matching point pairs from the at least one group of target matching point pairs, it is used to determine the frequency of occurrence of each target matching point pair in the at least one group of target matching point pairs; based on the frequency of occurrence, the 4 groups of target matching point pairs with the highest frequency of occurrence are determined.

[0063] Optionally, the device further comprises:

[0064] The image acquisition module is used to acquire at least one real plane image by performing image acquisition through an image acquisition device preset in each room of the at least one room.

[0065] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0066] A memory for storing a computer program product;

[0067] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the calibration method of the external parameters of the image acquisition device described in any of the above embodiments is implemented.

[0068] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for calibrating the external parameters of the image acquisition device described in any of the above embodiments is implemented.

[0069] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the method for calibrating external parameters of an image acquisition device as described in any of the above embodiments.

[0070] Based on the above-mentioned embodiment of the present disclosure, a method for calibrating the external parameters of an image acquisition device, an electronic device and a storage medium are provided. Based on the grid floor plan corresponding to the target room, a building information model corresponding to the target room is determined; wherein the building information model includes multiple 3D key points in the target room; key points are extracted from the real plane image corresponding to the target room and acquired by the image acquisition device, so as to obtain at least one group of 2D key point groups; wherein each group of 2D key point groups includes multiple 2D key points corresponding to the target room; n groups of matching point pairs corresponding to the target room are determined; wherein n is greater than or equal to 4, and each group of the matching point pairs includes a 3D key point and a corresponding 2D key point; based on the n groups of matching point pairs, the external parameter information of the image acquisition device that acquires the real plane image in the target room is determined; this embodiment automatically determines the building information model based on the grid floor plan corresponding to the room, avoids manual drawing of the building information model, improves the efficiency of external parameter calibration, and reduces labor costs.

[0071] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0073] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0074] Figure 1 It is a flowchart of a method for calibrating external parameters of an image acquisition device provided by an exemplary embodiment of the present disclosure;

[0075] Figure 2 This disclosure Figure 1 A schematic flow chart of step 102 in the embodiment shown;

[0076] Figure 3a is a schematic grid floor plan in an exemplary embodiment of the present disclosure;

[0077] Figure 3b is based on Figure 3a The vector diagram obtained by processing the raster floor plan shown;

[0078] Figure 3c is a simulated plan view of an optional example in an exemplary embodiment of the present disclosure;

[0079] Figure 3d is a simulated plan view of another optional example in an exemplary embodiment of the present disclosure;

[0080] Figure 3e is a simulated plan view of yet another optional example in an exemplary embodiment of the present disclosure;

[0081] Figure 3f is a schematic diagram of a schematic real plane image after segmentation processing in an exemplary embodiment of the present disclosure;

[0082] Figure 4 This disclosure Figure 1 A schematic flow chart of step 104 in the embodiment shown;

[0083] Figure 5 This disclosure Figure 1 A schematic flow chart of step 106 in the embodiment shown;

[0084] Figure 6a is a schematic diagram of line segment matching obtained by rotating a target plane image by 0° in an exemplary embodiment of the present disclosure;

[0085] Figure 6bis a schematic diagram of line segment matching obtained by rotating the target plane image by 100° in an exemplary embodiment of the present disclosure;

[0086] Figure 6c is a schematic diagram of line segment matching obtained by rotating the target plane image by 200° in an exemplary embodiment of the present disclosure;

[0087] Figure 6d is a schematic diagram of line segment matching obtained by rotating the target plane image by 300° in an exemplary embodiment of the present disclosure;

[0088] Figure 7 This disclosure Figure 1 A schematic flow chart of step 108 in the embodiment shown;

[0089] Figure 8 It is a structural schematic diagram of a device for calibrating external parameters of an image acquisition device provided by an exemplary embodiment of the present disclosure;

[0090] Fig. 9 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. DETAILED DESCRIPTION

[0091] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0092] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0093] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0094] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0095] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0096] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the previous and next associated objects are in an "or" relationship. The data referred to in this disclosure may include unstructured data such as text, images, and videos, and may also be structured data.

[0097] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0098] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0099] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0100] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0101] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0102] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0103] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0104] Exemplary Methods

[0105] Figure 1 FIG. 1 is a flow chart of a method for calibrating external parameters of an image acquisition device provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0106] Step 102: Determine a building information model corresponding to the target room based on the grid floor plan corresponding to the target room.

[0107] The building information model includes a plurality of 3D key points in the target room; each room corresponds to a plurality of 3D key points. The target room in this embodiment can be any room in a house with a known grid floor plan.

[0108] The Building Information Modeling (BIM) file contains all the information about the apartment, such as the names of rooms, the locations of key elements such as walls, doors, and windows; it also includes the three-dimensional coordinate information of key points determined by a series of algorithms based on the complete information of the apartment, such as the three-dimensional coordinate information of key points such as corner points and door points.

[0109] The spatial information in the building information model plays a vital role in the external parameter calibration. In order to overcome the limitations of the solution without a building information model, this embodiment proposes a solution to automatically obtain spatial information from a grid floor plan (such as an original floor plan CAD drawing) to complete three-dimensional fusion; the solution provided by this embodiment achieves both accuracy and low cost by automatically obtaining a building information model. Optionally, the grid floor plan of the house may include the most basic room plan information, and this embodiment may determine the grid floor plan of the target room based on the grid floor plan of the house.

[0110] Step 104: extract key points from the real plane image corresponding to the target room and captured by the image capture device to obtain at least one set of 2D key points.

[0111] Each group of 2D key points includes multiple 2D key points corresponding to the target room.

[0112] In one embodiment, the real plane image can be acquired by setting up an image acquisition device (e.g., a fisheye camera, etc.) in the target room. Optionally, any key point extraction technology can be used to obtain key points such as corner points and door points in the real plane image.

[0113] Step 106: Determine n groups of matching point pairs corresponding to the target room.

[0114] Among them, n is greater than or equal to 4, and each set of matching point pairs includes a 3D key point and a corresponding 2D key point.

[0115] Optionally, this embodiment determines n groups of matching point pairs for the target room, and improves the calibration accuracy of the external parameters of the image acquisition device by obtaining more matching point pairs.

[0116] Step 108: Determine the external parameter information of the image acquisition device that acquires the real plane image in the target room based on the n groups of matching point pairs.

[0117] In this embodiment, the external parameter information of the image acquisition device can be determined based on the more accurate part of the matching point pairs in the n groups of matching point pairs, thereby reducing the impact of erroneous matching point pairs on the external parameter calibration and improving the accuracy of the external parameter calibration.

[0118] The above-mentioned embodiment of the present disclosure provides a method for calibrating the external parameters of an image acquisition device, which determines a building information model corresponding to a target room based on a grid floor plan corresponding to the target room; wherein the building information model includes multiple 3D key points in the target room; extracts key points from the real plane image corresponding to the target room acquired by the image acquisition device, and obtains at least one group of 2D key point groups; wherein each group of 2D key point groups includes multiple 2D key points corresponding to the target room; determines n groups of matching point pairs corresponding to the target room; wherein n is greater than or equal to 4, and each group of matching point pairs includes a 3D key point and a corresponding 2D key point; based on the n groups of matching point pairs, determines the external parameter information of the image acquisition device that acquires the real plane image in the target room; this embodiment automatically determines the building information model based on the grid floor plan corresponding to the room, avoids manual drawing of the building information model, improves the efficiency of external parameter calibration, and reduces labor costs.

[0119] like Figure 2 As shown in the above Figure 1 Based on the illustrated embodiment, step 102 may include the following steps:

[0120] Step 1021, convert the grid floor plan through a preset conversion model to obtain a vector map corresponding to the target room.

[0121] The vector diagram includes the plan information of at least one structure included in the target room; optionally, the vector diagram of the target room can be determined by the vector diagram of the house corresponding to the target room, and the vector diagram of the house includes the compartment information of the house and the plan information of at least one structure included in each room.

[0122] Optionally, a raster to vector (R2V) model is used to convert the raster floor plan into a vector map, wherein the R2V model may be a deep neural network, which can be trained to realize room division, coordinate calibration, and calibration of multiple structures of the raster floor plan; wherein at least one structure of the room may include but is not limited to: walls, floors, doors, windows, and other structures. For example, in some optional examples, Figure 3a The grid floor plan shown in the figure can be converted to Figure 3b The vector diagram shown (the numbers in the figure only represent the results in the example and are for illustration only. Since some walls are close to each other, the overlapping numbers do not affect the implementation of the embodiment of the present disclosure) marks the properties of the target room (for example, living room, bedroom, bathroom, etc.) and the coordinate information of the doors, windows and walls included in the target room.

[0123] Step 1022: Determine a building information model corresponding to the target room based on the vector diagram and preset height information corresponding to different structures in the target room.

[0124] For external parameter calibration, absolute spatial position coordinates are not necessary, and the proportional relationship of the partition wall length has a greater impact on external parameter calibration. In fact, the line length in the grid floor plan already contains the proportional relationship of the wall in the real world, so in theory, accurate relative spatial position information can be obtained based on the grid floor plan. In this embodiment, the grid floor plan is converted into a vector diagram through the R2V model, and the plane coordinate information of structures such as walls and doors can be directly obtained based on the vector diagram; on the basis of determining the plane coordinate information, in order to determine the three-dimensional information of key points, the ground height can be set to 0, and the same or different preset height information can be assigned to different structures. For example, the same or different preset height information is assigned to walls, doors and windows respectively (the preset height information can be set according to empirical values, for example, set to 1 meter for windows, 2 meters for doors, 3 meters for walls, etc.). Since ground points are preferentially selected when performing external parameter calibration, the heights of walls, doors and windows can be preset. Even if there are differences from the actual situation, the impact on the completion of the image acquisition equipment calibrated in this application is not significant. The preset height can obtain three-dimensional space information close to the real scene; combined with the set height information and plane coordinate information, the building information model corresponding to the target room can be determined; wherein at least the three-dimensional information of key points such as wall corner points and door points is included.

[0125] like Figure 4 As shown in the above Figure 1 Based on the illustrated embodiment, step 104 may include the following steps:

[0126] Step 1041 : For the real plane image corresponding to the target room, line segment detection is performed on the real plane image to obtain a plurality of first line segments included in the target room.

[0127] The first line segment represents the intersection line between walls, or represents the intersection line between a wall and a door.

[0128] Optionally, a segmentation model and / or a line segment detection model may be used to perform line segment detection on the real plane image to obtain a plurality of first line segments.

[0129] The segmentation model can be an image segmentation model in a deep neural network. After training with a large number of room plane images and their corresponding annotated segmentation results, the obtained image segmentation model can segment the real plane image to obtain multiple line segments in the real plane image (for example, wall lines, door lines, window lines, etc.). In order to improve efficiency, only specific line segments (for example, vertically distributed wall lines, door lines, window lines, etc.) can be trained during the training process, so that the segmentation model only outputs specific line segments in the real plane image, thereby improving detection efficiency.

[0130] In addition, a unified linesegment detection algorithm across pinhole, fisheye, and spherical cameras (ULSD) can be used to determine at least one specific line segment in the real plane image; the unified line segment detection algorithm can also be fine-tuned based on a large amount of real estate data to make the fine-tuned unified line segment detection algorithm more suitable for the line segment detection of the real plane image of the house in this embodiment.

[0131] The above two methods can also be combined, that is, the image segmentation model and the unified line segment detection algorithm are respectively used to perform line segment detection on the real plane image, and the detection results obtained by the two methods are combined to determine multiple first line segments; for example, the image segmentation model detects 3 segmentation line segments, and the unified line segment detection algorithm obtains 4 detection line segments. The distances between the 4 detection line segments and the 3 segmentation line segments are calculated to determine the 3 matching detection line segments, and then the 3 first line segments are determined based on the position mean of each detection line segment in the 3 detection line segments and the corresponding segmentation line segment (the mean of each coordinate in the two antenna segments).

[0132] Step 1042: Determine a 2D key point group corresponding to the room based on an endpoint of each first line segment in the plurality of first line segments close to the center point of the real plane image.

[0133] In this embodiment, the image acquisition device for acquiring the real plane image can be a fisheye camera. The image acquired by the fisheye camera is usually circular in reality. Therefore, the center point of the real plane image is the center of the circle. At this time, each of the two ends of the first line segment has one end close to the center of the circle and the other end far away from the center of the circle. In this embodiment, the ground is used as a reference in the three-dimensional coordinates. Therefore, the endpoint close to the center of the circle in the first line segment is determined as a 2D key point, and a 2D key point group consisting of multiple 2D key points in the room is obtained.

[0134] like Figure 5 As shown in the above Figure 1 Based on the illustrated embodiment, step 106 may include the following steps:

[0135] Step 1061, based on the internal reference information of the image acquisition device corresponding to the target room and multiple assumed external reference information, the building information model corresponding to the room is projected into the 2D image to obtain multiple simulated plane images.

[0136] In this embodiment, when projecting the three-dimensional spatial information into the 2D image corresponding to the fisheye camera, it is necessary to combine the camera's intrinsic parameters (including distortion parameters) and extrinsic parameters, wherein the intrinsic parameters have been determined when the camera leaves the factory and are known data. Therefore, it is necessary to assume the extrinsic parameters. In order to improve the calibration accuracy, this embodiment needs to find the 2D and 3D point pairs as accurately as possible; optionally, a spatial grid with a preset range (for example, 200 (cm) x200 (cm) x100 (cm), where the three values ​​correspond to the x-axis, y-axis, and z-axis, respectively) with the geometric center as the origin is constructed; with the geometric center as the origin, within the spatial grid, a preset distance is moved on the x-axis, y-axis, and z-axis to obtain multiple assumed extrinsic parameter information (mainly the camera position); based on each assumed extrinsic parameter information, a simulated plane image is projected. For example, in an optional example, when the extrinsic parameter information is assumed to be at the center of the set, the projection is obtained as follows Figure 3c The simulated plane image shown in FIG. 1 is shown in FIG. 1 . When the external parameter information is assumed to be offset by 150 cm on the x-axis, the projection is as follows: Figure 3d The simulated plane image shown in FIG. 1 is shown in FIG. 1 . When the external reference information is assumed to be offset by 200 cm on the x-axis, the projection is as follows: Figure 3e Simulated flat image shown.

[0137] Step 1062 : Based on the real plane image, a simulated plane image having the highest similarity to the real plane image is determined from the multiple simulated plane images as the target plane image.

[0138] In this embodiment, the best matching camera position is determined by performing key point matching or line segment matching (for example, edge line segment matching) on ​​multiple simulated plane images and real plane images. Optionally, key point matching is performed on the real plane image processed by the segmentation model and multiple simulated plane images processed by the segmentation model, and the simulated plane image with the highest similarity to the real plane image is determined as the target plane image, wherein the similarity can be determined according to the number of matched feature points (the more the number, the higher the similarity) or the distance between the matched line segments (the smaller the distance, the higher the similarity); for example, in an optional example, Figure 3f As shown in the figure, it is the real plane image after segmentation processing. Based on the real plane image, it can be determined Figure 3e The simulated plane image shown is the target plane image.

[0139] Step 1063: Based on the multiple 3D key points corresponding to the target plane image and the multiple 2D key points in the real plane image, n groups of matching point pairs corresponding to the room are obtained.

[0140] In this embodiment, the installation position of the fisheye camera at the decoration site is relatively random, and its specific installation position cannot be determined. In order to match the image 2D points with the spatial 3D point pairs, it is necessary to assume the camera's posture, project the three-dimensional spatial information in the building information model to obtain a simulated 2D map, and then compare it with the real 2D scene. The assumed camera posture has an important impact on the final matching result; and in order to overcome the problem of uncertain camera position, this embodiment proposes a simulated camera posture space search strategy (with the geometric center as the origin, in the spatial grid, on the x-axis, y-axis, and z-axis Move at a preset distance), determine an optimal camera position for point pair matching, and improve the accuracy of external parameter calibration.

[0141] In some optional embodiments, step 1063 may include:

[0142] Performing line segment detection on the target plane image to obtain a plurality of second line segments included in the target plane image;

[0143] Optionally, the line segment detection means of the target plane image may be similar to the line segment detection means of the real plane image; or, a plurality of second line segments corresponding to the projection onto the plane image may be determined by a building information model (for example, wall lines, door lines, window lines, etc. included in the room may be determined based on the building information model, and a plurality of second line segments in the target plane image may be obtained based on the projection of these line segments).

[0144] The target plane image is rotated multiple times at preset angle intervals to obtain multiple third line segment groups; wherein each third line segment group includes multiple third line segments in the target plane image corresponding to a rotation angle;

[0145] Optionally, in a fisheye scene, the camera is installed at an arbitrary angle. In order to obtain the best match, this embodiment rotates the target plane image within a range of 360° to obtain a result that better matches the real plane image.

[0146] Based on the plurality of first line segments, determining a plurality of third line segments matching the plurality of first line segments from a plurality of third line segment groups;

[0147] Based on the endpoint of each third line segment in the plurality of third line segments close to the center point of the target plane image, a plurality of 3D key points matching the 2D key point group are determined to obtain n groups of matching point pairs corresponding to the room.

[0148] In this embodiment, the key point matching of the target plane image and the real plane image is achieved through the line segment matching strategy (matching each group of third line segments with each first line segment respectively). In the prior art, the key point matching of the two images is based on the principle of segmenting the key elements in the picture, such as doors and windows, and then calculating the overlapping area between the doors and windows in the simulated image and the doors and windows in the real image, and the match with the largest overlapping area is taken as the final match; however, the prior art is feasible on the premise that the room structure is accurately analyzed, and the training of the segmentation model consumes a lot of time and data annotation. The disclosed embodiment uses a line matching scheme, and after the line is matched, the corresponding matching point pairs can be directly determined based on the endpoints of the line, and the annotation cost and training cost are greatly reduced, which is concise and effective.

[0149] Optionally, based on the plurality of first line segments, determining a plurality of third line segments matching the plurality of first line segments from a plurality of third line segment groups comprises:

[0150] 1) Based on each first line segment in the plurality of first line segments and each third line segment group in the plurality of third line segment groups, a plurality of cost matrices are determined.

[0151] Each cost matrix corresponds to a group of third line segments, and each row in the cost matrix represents a plurality of cost values ​​between a first line segment and a plurality of third line segments.

[0152] In this embodiment, each element of each row in the cost matrix represents a cost value between a first line segment and a third line segment. The cost value cost can be determined based on the following formula (1):

[0153] cost=α*angle+β*length+γ*distance formula (1)

[0154] Wherein, cost represents the cost value; angle represents the angle between the first line segment and the third line segment; length represents the absolute value of the difference between the lengths of the first line segment and the third line segment; distance represents the distance between the end point of the first line segment close to the center of the circle and the end point of the third line segment close to the center of the circle; α, β, γ represent balance coefficients, which are determined through multiple experiments (in addition, since the effect of the line segment length on the cost value is small, the value of β is usually small). The cost value between each first line segment and each third line segment can be determined by the above formula (1), and the cost matrix can be obtained.

[0155] 2) Based on the matrix cost value corresponding to each cost matrix, determine the cost matrix with the minimum matrix cost value as the target cost matrix.

[0156] In this embodiment, each group of third line segments and multiple first line segments correspond to a cost matrix, that is, each rotation of the target plane image by one angle corresponds to a cost matrix, and each cost matrix corresponds to a matrix cost value, which can be determined based on the sum of the minimum value of each row in the cost matrix (the minimum cost value in a row corresponds to the third line segment that best matches the first line segment corresponding to the row); for example, if the cost matrix is In this embodiment, two first line segments and two third line segments are represented, wherein the cost values ​​of one first line segment and the two third line segments are 0.5 and 2 respectively; the cost values ​​of the other first line segment and the two third line segments are 1 and 0.3 respectively, and at this time, the matrix cost value of the cost matrix is ​​0.5+0.3=0.8. By comparing the matrix cost values ​​corresponding to the cost matrices corresponding to each group of third line segments, the cost matrix with the smallest matrix cost value is determined as the target cost matrix; optionally, the target cost matrix can be quickly determined by a greedy algorithm.

[0157] 3) Based on the target cost matrix, determine a plurality of third line segments that match the plurality of first line segments.

[0158] In this embodiment, after determining the target cost matrix, a set of third line segment groups can be determined, and the third line segment corresponding to each first line segment can be determined based on the corresponding matrix cost value in the target cost matrix. This embodiment uses a greedy algorithm to determine the optimal set of third line segment groups. For example, in some optional examples, the target plane image is rotated within a range of 360°. For example, the target plane image is rotated by 0° to obtain the following: Figure 6a The line segment matching diagram shown in FIG. 1 is shown in FIG. 2 ; the target plane image is rotated 100° to obtain the following Figure 6b The line segment matching diagram shown in FIG. 1 is shown in FIG. 2 ; the target plane image is rotated 200° to obtain the following Figure 6c The line segment matching diagram shown in FIG. 1 is shown in FIG. 2 ; the target plane image is rotated 300° to obtain the following Figure 6dThe line segment matching schematic diagram is shown; in the figure, the first line segment and the third line segment can be distinguished by different colors (different grayscales in the black and white figure). After the matrix cost value calculation in this example, it can be determined that the matrix cost value of the cost matrix corresponding to the rotation of 200° is the smallest, that is, the third line segment group when rotated 200° is more matched with the multiple first line segments. At this time, based on the third line segment corresponding to the cost value of each row of the calculated matrix cost value in the cost matrix, the third line segment matching each first line segment can be determined.

[0159] like Figure 7 As shown in the above Figure 1 Based on the illustrated embodiment, step 108 may include the following steps:

[0160] Step 1081, reorganize n groups of matching point pairs into multiple groups of grouped matching point pairs.

[0161] Each group of matching point pairs includes 4 groups of matching point pairs.

[0162] Since, when determining the external parameters of the image acquisition device based on the matching point pairs, it can be achieved with 4 groups of matching point pairs, therefore, in this embodiment, n groups of matching point pairs are reorganized (for example, by permutation and combination) into multiple groups of grouped matching point pairs; assuming that there are n groups of matching point pairs, n>4, 4 point pairs are randomly selected from all n matching point pairs for external parameter calibration, and there are a total of A combination of methods.

[0163] Step 1082, based on the multiple groups of group matching point pairs, determine 4 groups of target matching point pairs.

[0164] Step 1083, based on the four groups of target matching point pairs, determine the external parameter information of the image acquisition device that captures the real plane image in the target room.

[0165] In this embodiment, some of the matching point pairs obtained by the above matching strategy may be wrong. If these wrong matching point pairs are used in the external parameter calculation, the external parameters finally obtained will be unusable. Therefore, this embodiment uses the Monte Carlo method to filter the matching point pairs. The Monte Carlo method is also called the statistical simulation method or the statistical test method. It is a numerical simulation method that takes probability phenomena as the research object. It is a calculation method that obtains statistical values ​​according to the sampling survey method to infer unknown characteristic quantities.

[0166] In some optional embodiments, step 1082 may include:

[0167] Reprojection errors and position information of the image acquisition device are calculated based on the multiple groups of group matching point pairs to obtain multiple groups of reprojection errors and position information; wherein each group of group matching points corresponds to a group of reprojection errors and position information;

[0168] Based on the multiple groups of reprojection errors and position information, determining at least one group of matching point pairs from the multiple groups of matching point pairs as a target matching point pair;

[0169] Four sets of target matching point pairs are obtained from at least one set of target matching point pairs.

[0170] Optionally, the frequency of occurrence of each target matching point pair in at least one group of target matching point pairs is determined; based on the frequency of occurrence, four groups of target matching point pairs with the highest frequency of occurrence are determined.

[0171] In this embodiment, it can be assumed that a group of group matching point pairs is used for matching. After the extrinsic parameters are calculated based on the group matching point pairs, the four 3D points in the group matching point pairs under the spatial coordinates will be projected onto the real plane image through the internal and external parameters, and then the error is calculated with the actual position in the real plane image (the position of the four 2D points in the group matching point pairs), and the average is taken to determine the reprojection error; after the camera extrinsic parameters are determined based on the group matching point pairs, the camera extrinsic parameters represent the rotation and translation from the spatial coordinates to the camera coordinates, so the coordinates of the camera in space are the rotation matrix multiplied by the translation matrix. Therefore, the position information of a camera can be determined based on each group of group matching points. By identifying the reprojection error and position information corresponding to each group of grouped matching point pairs, it is determined whether the grouped matching point pairs meet the requirements. For example, when the reprojection error is less than an error threshold (for example, 20 pixels, etc.) and the position information is within a preset range, it is determined that the grouped matching point pairs corresponding to the reprojection error and the position information meet the requirements and are used as target matching point pairs. At this time, multiple groups of target matching point pairs may be obtained, each group of target matching point pairs includes 4 target matching point pairs, that is, the number is much larger than the required 4 point pairs. In order to further improve the accuracy of extrinsic parameter calibration, the 4 groups of target matching point pairs with the highest frequency among all target matching points can be counted, and the final camera extrinsic parameters are calculated based on the 4 groups of target matching point pairs with the highest frequency.

[0172] Optionally, before step 104, the following steps may also be included:

[0173] At least one real plane image is obtained by performing image acquisition through an image acquisition device preset in each room of at least one room.

[0174] Optionally, the image acquisition device in this embodiment may be a fisheye camera. This embodiment is to determine the camera extrinsic parameters of the fisheye camera in the target room.

[0175] Optionally, step 108 of determining the external parameter information of the image acquisition device based on n groups of matching point pairs can be implemented based on any existing technology. For example, a mapping relationship of 2D key point coordinates to 3D key point coordinates (i.e., projecting three-dimensional spatial information into a 2D image) is established. The relationship can be expressed as 2D key point coordinates equal to 3D key point coordinates multiplied by a camera matrix. By bringing n groups of matching point pairs into this relationship, the camera matrix can be determined. The camera matrix is ​​obtained by multiplying the intrinsic parameter matrix and the extrinsic parameter matrix. Under the premise of knowing the camera intrinsic parameter matrix, the external parameter information of the camera can be determined, and the external parameter information that needs to be solved in this embodiment can be determined.

[0176] Any calibration method for the external parameters of an image acquisition device provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any calibration method for the external parameters of an image acquisition device provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any calibration method for the external parameters of an image acquisition device mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0177] Exemplary Devices

[0178] Figure 8 FIG. 1 is a schematic diagram of a structure of an image acquisition device extrinsic parameter calibration device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the device provided in this embodiment includes:

[0179] The information model module 81 is used to determine the building information model corresponding to the target room based on the grid floor plan corresponding to the target room.

[0180] The building information model includes multiple 3D key points in the target room.

[0181] The image key point extraction module 82 is used to extract key points from the real plane image corresponding to the target room and acquired by the image acquisition device to obtain at least one set of 2D key points.

[0182] Wherein, each group of 2D key points includes multiple 2D key points corresponding to the target room;

[0183] The key point matching module 83 is used to determine n groups of matching point pairs corresponding to the target room.

[0184] Among them, n is greater than or equal to 4, and each set of matching point pairs includes a 3D key point and a corresponding 2D key point.

[0185] The external parameter determination module 84 is used to determine the external parameter information of the image acquisition device that captures the real plane image in the target room based on n groups of matching point pairs.

[0186] The above-mentioned embodiment of the present disclosure provides a device for calibrating external parameters of an image acquisition device. This embodiment automatically determines a building information model based on a grid floor plan corresponding to a room, thereby avoiding manual drawing of the building information model, improving the efficiency of external parameter calibration, and reducing labor costs.

[0187] In some optional embodiments, the information model module 81 is specifically used to convert the grid floor plan through a preset conversion model to obtain a vector diagram corresponding to the target room; wherein the vector diagram includes plane information of at least one structure included in the target room; based on the vector diagram and preset height information corresponding to different structures in the target room, determine the building information model corresponding to the target room.

[0188] In some optional embodiments, the image key point extraction module 82 includes:

[0189] A line segment detection unit is used to perform line segment detection on the real plane image corresponding to the target room to obtain a plurality of first line segments included in the target room; wherein the first line segment represents the intersection line between walls, or represents the intersection line between a wall and a door;

[0190] The key point determination unit is used to determine a 2D key point group corresponding to the room based on an endpoint of each first line segment in the plurality of first line segments close to a center point of the real plane image.

[0191] Optionally, the line segment detection unit is specifically configured to perform line segment detection on the real plane image using a segmentation model and / or a line segment detection model to obtain a plurality of first line segments.

[0192] In some optional embodiments, the key point matching module 83 includes:

[0193] An image simulation unit is used to project a building information model corresponding to the room into a 2D image based on the internal reference information of the image acquisition device corresponding to the target room and a plurality of assumed external reference information to obtain a plurality of simulated plane images;

[0194] An image screening unit, used for determining, based on the real plane image, from a plurality of simulated plane images, a simulated plane image having the highest similarity to the real plane image as a target plane image;

[0195] The point pair matching unit is used to obtain n groups of matching point pairs corresponding to the room based on multiple 3D key points corresponding to the target plane image and multiple 2D key points in the real plane image.

[0196] Optionally, the point pair matching unit is specifically used to perform line segment detection on the target plane image to obtain multiple second line segments included in the target plane image; rotate the target plane image multiple times at preset angle intervals to obtain multiple groups of third line segment groups; wherein each group of third line segment groups includes multiple third line segments in the target plane image corresponding to a rotation angle; based on the multiple first line segments, determine multiple third line segments matching the multiple first line segments from the multiple groups of third line segment groups; based on the endpoints of each third line segment in the multiple third line segments close to the center point of the target plane image, determine multiple 3D key points matching the 2D key point group to obtain n groups of matching point pairs corresponding to the room.

[0197] Optionally, when the point pair matching unit determines multiple third line segments that match the multiple first line segments from multiple groups of third line segment groups based on the multiple first line segments, it is used to determine multiple cost matrices based on each first line segment in the multiple first line segments and each group of third line segment groups in the multiple groups of third line segment groups; wherein each cost matrix corresponds to a group of third line segment groups, and each row in the cost matrix represents multiple cost values ​​between a first line segment and multiple third line segments; based on the matrix cost value corresponding to each cost matrix, determine the cost matrix with the smallest matrix cost value as the target cost matrix; based on the target cost matrix, determine multiple third line segments that match the multiple first line segments.

[0198] In some optional embodiments, the external parameter determination module 84 is specifically used to reorganize n groups of matching point pairs into multiple groups of grouped matching point pairs; each group of grouped matching point pairs includes 4 groups of matching point pairs; based on the multiple groups of grouped matching point pairs, 4 groups of target matching point pairs are determined; based on the 4 groups of target matching point pairs, the external parameter information of the image acquisition device that captures the real plane image in the target room is determined.

[0199] Optionally, when the external parameter determination module 84 determines four groups of target matching point pairs based on multiple groups of group matching point pairs, it is used to calculate the reprojection errors and the position information of the image acquisition device based on the multiple groups of group matching point pairs, so as to obtain multiple groups of reprojection errors and position information; wherein each group of group matching points corresponds to a group of reprojection errors and position information; based on the multiple groups of reprojection errors and position information, at least one group of group matching point pairs is determined from the multiple groups of group matching point pairs as the target matching point pairs; and four groups of target matching point pairs are obtained from the at least one group of target matching point pairs.

[0200] Optionally, when the external parameter determination module obtains 4 groups of target matching point pairs from at least one group of target matching point pairs, it is used to determine the frequency of occurrence of each target matching point pair in at least one group of target matching point pairs; based on the frequency of occurrence, the 4 groups of target matching point pairs with the highest occurrence frequency are determined.

[0201] In some optional embodiments, the device provided in this embodiment may further include:

[0202] The image acquisition module is used to acquire at least one real plane image by acquiring images through an image acquisition device preset in each room of at least one room.

[0203] Exemplary Electronic Devices

[0204] Below, reference Fig. 9 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0205] Fig. 9 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0206] like Fig. 9 As shown, the electronic device includes one or more processors and memory.

[0207] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0208] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the calibration method of the external parameters of the image acquisition device of each embodiment of the present disclosure described above and / or other desired functions.

[0209] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0210] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0211] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0212] Of course, to simplify, Fig. 9Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0213] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for calibrating the external parameters of the image acquisition device according to various embodiments of the present disclosure described in the above part of this specification.

[0214] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0215] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for calibrating the external parameters of the image acquisition device according to various embodiments of the present disclosure described in the above part of this specification.

[0216] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0217] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0218] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0219] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0220] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0221] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0222] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0223] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for calibrating external parameters of an image acquisition device, characterized in that: include: Determine a building information model corresponding to the target room based on a grid floor plan corresponding to the target room; wherein the building information model includes a plurality of 3D key points in the target room; Extract key points from the real plane image corresponding to the target room and acquired by the image acquisition device to obtain at least one group of 2D key points; wherein each group of 2D key points includes a plurality of 2D key points corresponding to the target room; Determine n groups of matching point pairs corresponding to the target room; wherein n is greater than or equal to 4, and each group of matching point pairs includes a 3D key point and a corresponding 2D key point; Based on the n groups of matching point pairs, external parameter information of an image acquisition device that acquires the real plane image in the target room is determined.

2. The method according to claim 1, characterized in that The determining, based on the grid floor plan corresponding to the target room, a building information model corresponding to the target room includes: The grid floor plan is converted by a preset conversion model to obtain a vector map corresponding to the target room; wherein the vector map includes plane information of at least one structure included in the target room; The building information model corresponding to the target room is determined based on the vector diagram and preset height information corresponding to different structures in the target room.

3. The method according to claim 1 or 2, characterized in that: The step of extracting key points from the real plane image corresponding to the target room and acquired by the image acquisition device to obtain at least one group of 2D key points includes: For a real plane image corresponding to the target room, line segment detection is performed on the real plane image to obtain a plurality of first line segments included in the target room; wherein the first line segments represent intersection lines between walls, or represent intersection lines between a wall and a door; A 2D key point group corresponding to the room is determined based on an endpoint of each first line segment of the plurality of first line segments close to a center point of the real plane image.

4. The method according to claim 3, characterized in that The determining of n groups of matching point pairs corresponding to the target room includes: Based on the internal reference information of the image acquisition device corresponding to the target room and multiple assumed external reference information, projecting the building information model corresponding to the room into the 2D image to obtain multiple simulated plane images; Based on the real plane image, determine the simulated plane image having the highest similarity to the real plane image from among the multiple simulated plane images as the target plane image; Based on the multiple 3D key points corresponding to the target plane image and the multiple 2D key points in the real plane image, n groups of matching point pairs corresponding to the room are obtained.

5. The method according to claim 4, characterized in that The obtaining n groups of matching point pairs corresponding to the room based on the multiple 3D key points corresponding to the target plane image and the multiple 2D key points in the real plane image comprises: Performing line segment detection on the target plane image to obtain a plurality of second line segments included in the target plane image; The target plane image is rotated multiple times at preset angle intervals to obtain multiple groups of third line segment groups; wherein each group of the third line segment groups includes multiple third line segments in the target plane image corresponding to a rotation angle; Based on the plurality of first line segments, determining a plurality of third line segments matching the plurality of first line segments from the plurality of groups of third line segments; Based on the endpoint of each third line segment in the plurality of the third line segments close to the center point of the target plane image, a plurality of 3D key points matching the 2D key point group are determined to obtain n groups of matching point pairs corresponding to the room.

6. The method according to claim 5, characterized in that The determining, based on the plurality of first line segments, a plurality of third line segments matching the plurality of first line segments from the plurality of third line segment groups comprises: Determine a plurality of cost matrices based on each first line segment of the plurality of first line segments and each third line segment group of the plurality of third line segment groups; wherein each cost matrix corresponds to a group of the third line segment groups, and each row in the cost matrix represents a plurality of cost values ​​between one of the first line segments and a plurality of the third line segments; Based on the matrix cost values ​​corresponding to each of the cost matrices, determining the cost matrix with the smallest matrix cost value as the target cost matrix; Based on the target cost matrix, a plurality of the third line segments matching the plurality of the first line segments are determined.

7. The method according to any one of claims 1 to 6, characterized in that: The determining, based on the n groups of matching point pairs, the external parameter information of the image acquisition device for acquiring the real plane image in the target room comprises: Reorganize the n groups of matching point pairs into multiple groups of group matching point pairs; wherein each group of group matching point pairs includes 4 groups of matching point pairs; Based on the multiple groups of group matching point pairs, determining 4 groups of target matching point pairs; Based on the four groups of target matching point pairs, external parameter information of an image acquisition device that acquires the real plane image in the target room is determined.

8. The method according to claim 7, characterized in that The step of determining four groups of target matching point pairs based on the multiple groups of group matching point pairs includes: Re-projection errors and position information of the image acquisition device are calculated based on the multiple groups of group matching point pairs to obtain multiple groups of re-projection errors and position information; wherein each group of group matching points corresponds to a group of re-projection errors and position information; Based on the multiple groups of reprojection errors and position information, at least one group of the grouped matching point pairs is determined from the multiple groups of grouped matching point pairs as a target matching point pair.

9. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method for calibrating the external parameters of the image acquisition device described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method for calibrating the external parameters of the image acquisition device described in any one of claims 1 to 8 is implemented.

Images