Camera calibration method and system

Abstract

Provided are a camera calibration method and system, relating to the technical field of vision. The camera calibration method comprises: on the basis of a line segment set associated with a terminal device, acquiring an orthogonal vanishing point pair corresponding to the model of the terminal device and a line segment group corresponding to the orthogonal vanishing point pair (S101), the line segment set associated with the terminal device comprising line segments identified on the basis of an image captured by the terminal device; and generating a camera internal parameter of the terminal device of the model on the basis of orthogonal vanishing point pairs corresponding to terminal devices of the same model and corresponding line segment groups (S102). The method eliminates the need for dedicated personnel to capture images for calibration, thereby significantly improving calibration efficiency, and facilitating the maintenance of a high level of model coverage of calibrated terminal devices.

Description

Camera calibration method and system

[0001] The present disclosure claims priority to the Chinese patent application No. 202410851332.4, filed on June 27, 2024, and entitled "Camera calibration method and system", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the field of vision technology, and in particular, to a camera calibration method and system. BACKGROUND

[0003] Augmented Reality (AR) is a technology that fuses virtual information with the real world. With the improvement of device performance, AR technology is widely used. For example, in AR navigation, the pose of a terminal device is calculated in real time based on images captured by a camera of the terminal device, and computer graphics technology is used to fuse and display navigation signs (virtual information) with images (real-world images) captured by the camera of the terminal device. The calculation of the pose of the terminal device is a necessary technical means to implement AR-related functions on the terminal device. To accurately calculate the pose of the terminal device, the camera intrinsic parameters of the terminal device need to be calibrated.

[0004] Currently, camera intrinsic parameter calibration requires a special person to use a terminal device to capture images of a specific pattern (such as a checkerboard) from different angles, and then determine the camera intrinsic parameters of the terminal device based on these images using a calibration algorithm. The present inventors have found that, due to the large number of terminal device models, the existing calibration method has low calibration efficiency and cannot achieve high coverage, thereby limiting the implementation of AR-related functions on terminal devices. SUMMARY

[0005] To overcome the problems in the related art, embodiments of the present disclosure provide a camera calibration method and system.

[0006] According to a first aspect of embodiments of the present disclosure, a camera calibration method is provided, the method comprising:

[0007] obtaining, based on a line segment set associated with a terminal device, an orthogonal vanishing point pair corresponding to a model of the terminal device and a line segment group corresponding to the orthogonal vanishing point pair, the line segment set associated with the terminal device comprising line segments identified based on images captured by the terminal device;

[0008] generating camera intrinsic parameters of the terminal device of the model based on the orthogonal vanishing point pair corresponding to the terminal device of the model and the line segment group corresponding to the terminal device of the model.

[0009] According to a second aspect of the embodiments of the present disclosure, a camera calibration method is provided, and the method comprises:

[0010] In response to a camera intrinsic parameter acquisition request sent by the terminal device, the camera intrinsic parameter corresponding to the model of the terminal device is queried from the pre-stored camera intrinsic parameter calibration data, and the camera intrinsic parameter is generated according to the method of the first aspect;

[0011] The camera intrinsic parameter is sent to the terminal device, so that the terminal device calculates the pose of the terminal device according to the camera intrinsic parameter.

[0012] According to a third aspect of the embodiments of the present disclosure, a camera calibration system is provided, comprising a terminal device and a server, and the terminal device is provided with a camera, wherein,

[0013] The terminal device is configured to capture an image and identify a line segment set of the image, and send the line segment set to the server;

[0014] The server is configured to acquire a normal vanishing point pair corresponding to the model of the terminal device and a line segment group corresponding to the normal vanishing point pair based on the line segment set associated with the terminal device, and generate a camera intrinsic parameter of the terminal device of the same model according to the normal vanishing point pair and the line segment group corresponding to the normal vanishing point pair corresponding to the terminal device of the same model;

[0015] The terminal device is further configured to send a camera intrinsic parameter acquisition request to the server, and the camera intrinsic parameter acquisition request carries the model of the terminal device;

[0016] The server is further configured to query the camera intrinsic parameter corresponding to the model of the terminal device from the pre-stored camera intrinsic parameter calibration data, and send the camera intrinsic parameter to the terminal device;

[0017] The terminal device is further configured to calculate the pose of the terminal device according to the camera intrinsic parameter.

[0018] According to a fourth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed on a computer, cause the computer to perform the method of the first aspect or the second aspect.

[0019] According to a fifth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, and the computer-readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the method of the first aspect or the second aspect.

[0020] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0021] In the embodiments of the present disclosure, for any type of terminal device, the server can obtain the orthogonal vanishing point pair corresponding to the type of the terminal device and the line segment group corresponding to the orthogonal vanishing point pair based on the line segment set associated with the terminal device, so as to calculate the camera intrinsic parameter of the terminal device of the corresponding type according to the obtained data. Since the line segment is identified based on the image captured by the terminal device, the camera parameter of the terminal device is calibrated. It can be understood that the image in which the line segment is identified does not need to be specially captured by a specific pattern, so the calibration efficiency can be obviously improved, and the coverage rate of the calibrated type can be maintained at a high level.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments of the present disclosure will be briefly introduced. It should be understood that other drawings can also be obtained by those of ordinary skill in the art without creative labor on the basis of these drawings.

[0024] FIG. 1 is a structural schematic diagram of an exemplary network architecture provided by the present disclosure;

[0025] FIG. 2 is a scene schematic diagram of an exemplary technical flow provided by the present disclosure;

[0026] FIG. 3A is an exemplary method flow schematic diagram one of a camera calibration method provided by the present disclosure;

[0027] FIG. 3B is an exemplary method flow schematic diagram two of a camera calibration method provided by the present disclosure;

[0028] FIG. 4A is an exemplary component schematic diagram one of a camera calibration device provided by the present disclosure;

[0029] FIG. 4B is an exemplary component schematic diagram two of a camera calibration device provided by the present disclosure;

[0030] FIG. 5 is an exemplary structural schematic diagram of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0031] The technical solutions of the embodiments of the present disclosure will be described below in combination with the drawings in the embodiments of the present disclosure.

[0032] The terms used in the following embodiments of the present disclosure are for the purpose of describing specific embodiments and are not intended to be limiting of the technical solutions of the present disclosure. As used in the specification and the appended claims of the present disclosure, the singular forms "a," "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise.

[0033] It should also be understood that, although the terms first, second, etc. can be employed in the following embodiments to describe a certain class of objects, the objects should not be limited by these terms. These terms are used to distinguish a specific implementation of the class of objects.

[0034] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0035] The related technologies involved in the embodiments of the present disclosure are described below.

[0036] AR technology is a technology of fusing virtual information generated by a terminal device, such as an object, an identifier, a video, a sound, etc., with a real scene. Specifically, the AR technology collects a real scene through a camera (also called a camera or a shooting component) of a terminal device, and fuses virtual information in the real scene, so that a user can interact with the real scene based on the virtual information, achieving the purpose of augmented reality. Typical application scenarios of AR technology include AR navigation, AR boarding point guidance, etc.

[0037] To ensure that the picture of the fused virtual information and real scene is correct (i.e., the perspective relationship is correct), it is necessary to ensure that the pose of the virtual information matches the real scene. The pose of the virtual information is related to the pose of the terminal device, and the accuracy of the pose of the terminal device is related to the camera intrinsics. The camera intrinsics can include but are not limited to focal length (fx, fy) and optical center (cx, cy) of the camera, and the optical center of the camera is also called the principal point of the camera.

[0038] Since different types of mobile terminals are equipped with different cameras, the intrinsic parameters (such as camera focal length and optical center) of different cameras are also different, and the camera intrinsic parameters are necessary conditions for pose calculation. If the intrinsic parameters of the camera equipped on a certain type of mobile terminal are unknown, or the known intrinsic parameters are inaccurate, pose calculation cannot be performed or deviation occurs in pose calculation, resulting in that the user using the type of mobile terminal cannot use the function implemented by the AR technology or the function effect is not as expected. Therefore, terminal devices implementing AR technology all need to calibrate camera intrinsic parameters, but terminal devices are not only various in types, but also have a large number of models of the same type, and new models of terminal devices are constantly being introduced and used. How to improve the calibration efficiency of camera intrinsic parameters and maintain a high level of coverage of terminal devices with calibrated intrinsic parameters is a problem to be solved by the technical personnel.

[0039] Therefore, the embodiments of the present disclosure provide a method for calibrating camera intrinsic parameters of a plurality of different types and different models of terminal devices by capturing images, obtaining a set of line segments based on the line segments recognized from the images captured by terminal devices of the same type and the same model, obtaining a set of orthogonal vanishing point pairs and line segments corresponding to the orthogonal vanishing point pairs as calibration data, and then determining the camera intrinsic parameters of the terminal devices of the same type and the same model according to the calibration data. In this way, it is not necessary to specially collect images for calibration, which can significantly improve the calibration efficiency and is conducive to maintaining a high level of coverage of the types and models of terminal devices with calibrated intrinsic parameters.

[0040] The technical scenario related to the embodiments of the present disclosure is described below.

[0041] The embodiments of the present disclosure can be applied to a camera calibration system. As shown in FIG. 1, the network architecture of the camera calibration system related to the embodiments of the present disclosure can include a server 1000 and a terminal device cluster. The terminal device cluster can include one or more models of terminal devices. As shown in FIG. 1, the plurality of terminal devices can include terminal device 2000a, terminal device 2000b, terminal device 2000c, and the like. The terminal device 2000a, the terminal device 2000b, the terminal device 2000c, and the like can be directly or indirectly connected to the server 1000 through wired or wireless communication, so that each terminal device can interact with the server 1000 through the network connection.

[0042] Any terminal device in the terminal device cluster can include any device with a camera component, such as a tablet computer, a smartphone, a smart watch, or smart glasses. The model of the terminal device is determined by the manufacturer of the corresponding device, and the model of the terminal device can be obtained from the device information of the terminal device when implementing the solutions of the present disclosure. In some implementations, any two terminal devices in FIG. 1 can be implemented as different types of electronic devices, for example, the terminal device 2000a is implemented as a smartphone, the terminal device 2000b is implemented as a tablet computer, and the terminal device 2000c is implemented as a portable computer. In other implementations, even if two terminal devices in FIG. 1 are implemented as the same type of electronic device, the models of the two terminal devices are different, for example, the terminal device 2000c can also be implemented as a smartphone, but the terminal device 2000a is, for example, a smartphone of model X1, and the terminal device 2000c is, for example, a smartphone of model X00.

[0043] It should be understood that any terminal device can be integrated with a camera component for collecting images of a real scene. Here, the camera component can be a camera on the terminal device for taking photos or videos. Optionally, a terminal device can be integrated with multiple cameras. The camera involved in the embodiments of the present disclosure can include multiple cameras on the corresponding terminal device. In some embodiments, any terminal device can be configured to support the implementation of virtual-real fusion functions, such as video production (virtual production), live streaming, virtual-real fusion in post-video special effects, AR navigation, virtual-real fusion in a metaverse scene, and the like. In the case where an instruction triggering the virtual-real fusion function of an application (APP) is received, the terminal device can obtain the camera intrinsic parameter corresponding to the model of the terminal device from the server 1000 (in the case where the server 1000 has calculated the camera intrinsic parameter), and generate a virtual model (for example, an AR model). In the process of calling the camera component to obtain images, the pose of the terminal device is determined in combination with the camera intrinsic parameter, so as to fuse the virtual model with the real scene according to the pose of the terminal device.

[0044] In other embodiments, in the case where the server 1000 does not contain the camera intrinsic parameter corresponding to the terminal device, the terminal device can call the camera component to collect images of a real scene, and extract a plurality of spatial line segments meeting a preset condition from the collected images, and send a line segment set composed of the plurality of spatial line segments to the server 1000, so as to enable the server 1000 to calculate the camera intrinsic parameter corresponding to the model of the terminal device according to the line segment set. For example, the image is an image taken on a road in the real world, and any line segment in the line segment set can be a straight line segment of an object recorded in the image, such as a lane line on the road surface, a boundary line of the road, a boundary line of the edge of a building (such as the side of a building), and the like.

[0045] The server 1000 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0046] The server 1000 can be configured to calculate (calibrate) the camera intrinsic parameters of the camera assembly in the terminal device of the corresponding type and model based on the line segment set from terminal devices of different types and different models, and store the corresponding camera intrinsic parameters according to the type and model. For the line segment set of any model of terminal device, the server 1000 can extract the orthogonal vanishing point pair and the line segment group corresponding to the orthogonal vanishing point pair, and then calculate the camera intrinsic parameters corresponding to the model according to the orthogonal vanishing point pair and the line segment group corresponding to the orthogonal vanishing point pair, and store the corresponding camera intrinsic parameters according to the type and model. When other terminal devices of the model send a camera intrinsic parameter acquisition request, the corresponding camera intrinsic parameters of the model can be queried in response to the request and sent to the terminal device.

[0047] In some embodiments, the server 1000 can deploy program instructions and data, algorithms, and modules required for processing line segment sets to obtain corresponding camera intrinsic parameters. The data may, for example, include the focal length of the corresponding camera, the focal length in the ultra-wide-angle, wide-angle, and telephoto states, respectively; the algorithms may, for example, include vanishing point extraction algorithms, SVD (Singular Value Decomposition) algorithms, LU (Lower-Upper) decomposition algorithms, and nonlinear optimization algorithms; and the modules may, for example, include storage modules and functional modules carrying the above algorithms, such as vanishing point extraction modules, vanishing point optimization modules, intrinsic parameter calibration modules, and the like. The program instructions can cause the server 1000 to call the relevant data, algorithms, or modules during execution to perform the relevant processing steps of camera intrinsic parameter calibration. Refer to the description of the corresponding embodiments of FIG. 2.

[0048] In this way, by deploying algorithms for calculating camera intrinsic parameters on the server side, for terminal devices of different types, the server can calibrate the camera intrinsic parameters of terminal devices of the corresponding model according to the line segment set associated with at least one terminal device of the same model, thereby facilitating the saving of the cost of manual calibration and maintaining a high level of coverage of the calibrated terminal device models.

[0049] It should be understood that the camera intrinsic parameters of the terminal device generally do not change with use. In some embodiments, for any model, if the server 1000 calculates the camera intrinsic parameters corresponding to the terminal device of the model that satisfy the preset condition, the server can further delete the line segment set and the vanishing point corresponding to the model of the terminal device for calibrating the camera intrinsic parameters, thereby saving the storage space occupancy of the server.

[0050] The camera intrinsic parameters that satisfy the preset condition can refer to camera intrinsic parameters that are relatively accurate according to empirical values. In actual implementation scenarios, camera intrinsic parameters calibrated according to a sufficient amount of data are generally considered to have relatively high accuracy. For example, camera intrinsic parameters calculated according to a preset number of vanishing points can be determined as camera intrinsic parameters that satisfy the preset condition.

[0051] It should be noted that the server 1000 and any terminal device in FIG. 1 are logical equivalents and are schematic. In actual implementation scenarios, the functions of the server 1000 can be implemented by multiple server entities or computer device entities. Any terminal device can include multiple terminal device entities of the same model.

[0052] In combination with the network architecture of the camera calibration system shown in FIG. 1, the technical process related to camera calibration in the embodiments of the present disclosure is described below.

[0053] As shown in FIG. 2, FIG. 2 is a schematic diagram of a scene of an exemplary technical process provided by the embodiments of the present disclosure. The server 10 in FIG. 2 can be implemented as the server 1000 in the embodiments corresponding to FIG. 1, and the terminal device 20a and the terminal device 20b shown in FIG. 2 can be implemented as any terminal device in the embodiments corresponding to FIG. 1. The server 10 can be deployed with a vanishing point extraction module, a vanishing point optimization module, a vanishing point storage module, an intrinsic parameter calibration module, and an intrinsic parameter storage module, and at least one of algorithms and data implementing related functions is deployed in different modules. The camera assembly is integrated in the terminal device 20a and the terminal device 20b. The intrinsic parameter storage module of the server 10 does not store the camera intrinsic parameters corresponding to the model of the terminal device 20a, for example.

[0054] Referring to FIG. 2, the terminal device 20a can start the AR function in a case where a corresponding user 1 triggers an AR function start instruction is received. In a case where it is determined that the server 10 does not store the camera intrinsic parameters corresponding to the model of the corresponding terminal device, the terminal device 20a can call the camera assembly to obtain an image, for example, to obtain an image by shooting. The terminal device 20a can identify a line segment set from the obtained image, and then send the identified line segment set to the server 10.

[0055] In a case where the server 10 receives the line segment set associated with the terminal device 20a, the server 10 can obtain the orthogonal vanishing point pair corresponding to the model of the terminal device and the line segment group corresponding to the orthogonal vanishing point pair, and generate the camera intrinsic parameter of the terminal device of the same model according to the orthogonal vanishing point pair corresponding to the terminal device of the same model and the line segment group corresponding to the terminal device of the same model. If the model of the terminal device 20b is the same as the model of the terminal device 20a, and the terminal device 20b receives the AR function opening instruction of the corresponding user after the terminal device 20a, the terminal device 20b can send a camera intrinsic parameter acquisition request to the server 10, and the request carries the model of the terminal device. The server 10 can query the camera intrinsic parameter corresponding to the model of the terminal device carried in the request from the pre-stored camera intrinsic parameter calibration data, and send the camera intrinsic parameter to the terminal device 20b. The terminal device 20b can calculate the pose of the terminal device according to the camera intrinsic parameter.

[0056] Wherein, after the server 10 receives the line segment set associated with the terminal device 20a, the server 10 can call the vanishing point extraction module to extract the orthogonal vanishing point pair corresponding to the line segment set and the line segment group corresponding to the orthogonal vanishing point pair based on the pre-deployed algorithm, and then call the vanishing point optimization module to optimize the coordinates of the two orthogonal vanishing points in the orthogonal vanishing point pair, and store the optimized orthogonal vanishing point pair in the vanishing point storage module. If the number of orthogonal vanishing point pairs corresponding to the model of the terminal device 20a in the vanishing point storage module reaches a preset number threshold, the server 10 can call the intrinsic parameter calibration module to calculate the preset number threshold of orthogonal vanishing point pairs and corresponding line segment groups according to the pre-deployed algorithm. The calculation result can be the camera intrinsic parameter corresponding to the model of the terminal device 20a.

[0057] In some embodiments, the terminal device 20a can also send the model of the terminal device 20a to the server 10 in the process of sending the line segment set to the server 10, so that after the intrinsic parameter calibration module of the server 10 calculates the camera intrinsic parameter corresponding to the model of the terminal device 20a, the server 10 can store the model of the terminal device 20a and the camera intrinsic parameter in the intrinsic parameter storage module.

[0058] It should be noted that the above process of terminal device 20a extracting line segment set from image, and the process of each module in server 10 processing corresponding data, can include multiple processing steps, and different processing steps can be related to different algorithms. For example, the process of vanishing point extraction module extracting an orthogonal vanishing point pair from line segment set can include two processing steps: extracting multiple vanishing points corresponding to line segment set, and identifying an orthogonal vanishing point pair from multiple vanishing points, and each of the two processing steps can correspond to an algorithm; for another example, the process of intrinsic parameter calibration module calculating camera intrinsic parameter from orthogonal vanishing point pair can include three processing steps: constructing parameter matrix, performing SVD decomposition on parameter matrix to obtain intrinsic parameter variable matrix to be processed, and performing LU decomposition on intrinsic parameter variable matrix and calculating camera intrinsic parameter, and each of the three processing steps can correspond to an algorithm. For specific implementation mode, refer to the description of the following embodiments, which will not be expanded here.

[0059] Optionally, in the case that the model of terminal device 20b is the same as the model of terminal device 20a, terminal device 20b can obtain camera intrinsic parameter corresponding to the model from server 10. In this embodiment, terminal device 20b can no longer obtain image and send line segment set to server 10. If the model of terminal device 20b is different from the model of terminal device 20a, in the case that it is determined that server 10 does not store camera intrinsic parameter corresponding to the model of terminal device 20b, terminal device 20b obtains line segment set and sends line segment set to server 10, so that server 10 calculates camera intrinsic parameter corresponding to the model of terminal device 20b. Wherein, the process of terminal device 20b obtaining line segment set, and the process of server 10 calculating camera intrinsic parameter corresponding to the model of terminal device 20b can refer to the corresponding processing process of terminal device 20a, which will not be described here.

[0060] It can be seen that the technical solution deploys the algorithm, data and related processing flow of calibrating camera intrinsic parameter to the server, and calibrates camera intrinsic parameter of terminal devices of different models by using crowd-sourced collection data (i.e. line segment set) corresponding to the model, without the need for special personnel to shoot specific patterns, which not only has high calibration efficiency, but also can obtain a large number of camera intrinsic parameters of different models at relatively low calibration cost.

[0061] The technical solution of the present disclosure and how the technical solution of the present disclosure solves the above technical problems will be described in detail in the following specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments. The embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0062] Referring to FIG. 3A, FIG. 3A is an exemplary method flow diagram of a camera calibration method provided by an embodiment of the present disclosure, which can be performed by a server, which can be as shown in the corresponding embodiments of FIG. 1 or FIG. 2. The camera calibration method can include the following steps:

[0063] In step S101, based on a line segment set associated with a terminal device, an orthogonal vanishing point pair corresponding to a model of the terminal device and a line segment group corresponding to the orthogonal vanishing point pair are obtained.

[0064] The line segment set associated with the terminal device includes line segments identified based on an image captured by the terminal device. The line segments can be acquired by the terminal device in response to a line segment collection instruction sent by the server, by invoking a camera mounted on the terminal device to capture at least one image and based on the image.

[0065] In some embodiments, upon receiving the line segment collection instruction from the server, the terminal device can invoke the camera to collect at least one image, filter at least one candidate image that meets a preset condition from the at least one image, and then extract a plurality of spatial line segments from the at least one candidate image to obtain the line segment set, and send the line segment set to the server. Therefore, the image needs to record an object that can identify a straight line, for example, the image can be an image captured on a road, which records lane lines of the road that can identify a straight line, edges of buildings, billboards, etc.

[0066] It should be noted that the user can hold the terminal device, and the terminal device can collect at least one image at a set collection time interval, but there can be a situation that the pose of the terminal device deviates by too large an angle from the normal position of the terminal device, resulting in that the collected image deviates by too large an angle from the real scene during this process, or the terminal device shakes too much, resulting in that the collected image has poor definition during this process, and so on. Therefore, the above candidate image that meets the preset condition can include at least one of the following:

[0067] The pose of the terminal device deviates by less than a preset angle when capturing the image, the brightness value of the image is greater than a preset brightness value, the definition of the image is greater than a preset value, and the collection time interval of the image and a previous candidate image is greater than a certain time length.

[0068] It should be understood that the above candidate image that meets the preset condition is only illustratively described, and does not limit the embodiments of the present disclosure. In actual implementation, the preset condition for filtering the candidate image can be more or less. The embodiments of the present disclosure do not limit this.

[0069] The pose offset angle of the terminal device may, for example, include an offset angle of a pitch angle and an offset angle of a roll angle, and correspondingly, the pose offset angle of the terminal device being less than a preset angle may include the offset angle of the pitch angle of the terminal device being less than the preset angle and the offset angle of the roll angle being less than the preset angle. The preset angle may be, for example, 20 degrees. In actual implementation, the terminal device may obtain parameters of a gyroscope sensor of the terminal device in the process of collecting each image, to determine the pose of the terminal device according to the parameters of the gyroscope sensor.

[0070] For example, after collecting an image, the terminal device may read a brightness value and a gradient value of the image. If the brightness value is greater than 128, it may be considered that the brightness value of the image is greater than a preset brightness value. If the gradient value of the image is greater than a preset gradient value, it may be considered that the definition of the image is greater than a preset value.

[0071] In some embodiments, the above-mentioned collection time interval may be flexibly set based on actual implementation scenarios. For example, if the model of the terminal device is a model that has been on the market for less than 3 days, the number of orthogonal vanishing points corresponding to the model stored in the server is relatively small, and the collection time interval may be determined as 3 seconds. For another example, if the model of the terminal device is a model that has been on the market for more than 10 days, the number of orthogonal vanishing points corresponding to the model stored in the server may be relatively large, and the collection time interval may be determined as 10 seconds.

[0072] Further, after obtaining at least one candidate image, the terminal device may extract a plurality of spatial line segments from the at least one candidate image based on a pre-deployed line segment extraction algorithm, and may filter out spatial line segments with a length less than a preset length, and send the remaining spatial line segments as a line segment set to the server. For example, the line segment extraction algorithm includes, but is not limited to, Edge Drawing Lines (EDLines), Line Segment Detector (LSD), and Hough transform.

[0073] In summary, by using the technical solutions of the embodiments of the present disclosure, for any type and model of terminal device, the terminal device only needs to obtain an image and extract a line segment set and send it to the server, without the need for special personnel to shoot a specific pattern for line segment extraction. The server will process and calculate based on the line segment set, so as to calculate the camera intrinsic parameters of the terminal device corresponding to the corresponding model, which can significantly improve the calibration efficiency, and is also conducive to maintaining the coverage rate of the calibrated models at a high level.

[0074] Vanishing Point, also known as vanishing point, in two parallel lines, two lines will disappear in the distance of a point. For example, looking at the distance along the two parallel rails, the two parallel rails intersect at a certain point far away, and the point is the vanishing point.

[0075] Orthogonal Vanishing Point, when the line connecting two vanishing points and the camera optical center has an orthogonal relationship, it can be called an orthogonal vanishing point.

[0076] Among them, for two vanishing points, if any line segment in the line segment set corresponding to one vanishing point is perpendicular to any line segment in the line segment set corresponding to the other vanishing point, then the two vanishing points are mutually orthogonal vanishing points, and the two vanishing points are an orthogonal vanishing point pair. The line segment set corresponding to any orthogonal vanishing point pair forms the line segment group of the orthogonal vanishing point pair.

[0077] Since the coordinates of the vanishing points are determined according to the images collected by the camera, the geometric features of the vanishing point coordinates are related to the camera intrinsic parameters. Based on this, the camera intrinsic parameters can usually be derived from the coordinates of the vanishing points. Further, calibrating the camera intrinsic parameters usually maps the image coordinate system (two-dimensional coordinate system) to the camera coordinate system (three-dimensional coordinate system). Correspondingly, in order to provide more accurate constraints and spatial information to improve the accuracy and reliability of the calibration, the orthogonal vanishing point pair is usually used as the basic data. Therefore, in the embodiments of the present disclosure, after obtaining the vanishing point set, the initial orthogonal vanishing point pair can be extracted from the vanishing point set.

[0078] It should be noted that whether any two vanishing points are orthogonal can be determined by the included angle of the light rays when the two vanishing points are back projected to the camera optical center, and the included angle should be calculated based on the camera focal length. However, the camera focal length and the optical center coordinates of the camera have not been calibrated. In actual implementation scenarios, the camera can support multiple field of view modes for shooting, such as ultra-wide angle, wide angle, and long focal length. The three modes are related to the camera focal length. In addition, the optical center coordinates of the camera are close to the coordinates of the center pixel of the image collected by the camera. Therefore, in order to support the extraction of the orthogonal vanishing point pair, the camera focal length corresponding to at least one field of view mode can be used as a reference intrinsic parameter, and the coordinates of the center pixel of the image can be determined as the reference optical center coordinates. For example, the at least one reference intrinsic parameter includes reference intrinsic parameter 1, reference intrinsic parameter 2, and reference intrinsic parameter 3. The reference intrinsic parameter 1 is, for example, the reference intrinsic parameter corresponding to the focal length when the camera is in the ultra-wide angle field of view mode. The reference intrinsic parameter 2 is, for example, the reference intrinsic parameter corresponding to the focal length when the camera is in the wide angle field of view mode. The reference intrinsic parameter 3 is, for example, the reference intrinsic parameter corresponding to the focal length when the camera is in the long focal length field of view mode.

[0079] For example, the focal length in the ultra-wide angle field of view mode is, for example:

[0080] The focal length in the case of the wide field of view mode is, for example, f2 = min(width, height);

[0081] The focal length in the case of the long field of view mode is, for example, f3 = max(width, height);

[0082] The reference optical center coordinates can be, for example,

[0083] where width can represent the width of the image size, and height can represent the height of the image size.

[0084] The reference intrinsic matrix can satisfy:

[0085] Further, for any two vanishing points vp i and vp j in the vanishing point set, the server can calculate the included angle θ n of the two vanishing points mapping to the reference optical center coordinates of the camera according to each reference intrinsic in the at least one reference intrinsic, respectively.

[0086] wherein n = 1, 2, 3.

[0087] If the included angle θ corresponding to any reference intrinsic satisfies a preset orthogonal condition, the two vanishing points are determined as an initial orthogonal vanishing point pair.

[0088] For example, the preset orthogonal condition can be, for example, that the included angle θ satisfies is less than a preset angle threshold. The preset angle threshold can be flexibly set according to an empirical value. In one optional example, the preset angle threshold is, for example, 20 degrees.

[0089] In this way, the server can extract the vanishing point set from the line segment set, and then determine the reference focal length and the reference optical center coordinates that can be referenced based on the known data of the camera (such as the image size, the parameters of the field of view mode corresponding to the camera, and the like), thereby supporting the extraction of the initial orthogonal vanishing point pair based on the reference focal length and the reference optical center coordinates, and thereby providing data support for the calculation of the camera intrinsic.

[0090] In combination with the embodiment shown in FIG. 2, the above algorithms for extracting a set of vanishing points from a set of line segments, determining at least one reference intrinsic parameter and a reference optical center coordinate, and determining an initial orthogonal vanishing point pair from the set of vanishing points based on the at least one reference intrinsic parameter and the reference optical center coordinate, as well as related data and program instructions, can be deployed in a vanishing point extraction module, and the server can invoke the vanishing point extraction module to perform the above related steps.

[0091] Although the initial orthogonal vanishing point pair has been extracted, the reference intrinsic parameter and the reference optical center coordinate used to extract the initial orthogonal vanishing point pair are relatively close data determined based on the geometric transformation relationship between the camera and the image, resulting in some distortion in the coordinates of the vanishing points in each initial orthogonal vanishing point pair. In order to improve the accuracy of the calibrated camera intrinsic parameter, the embodiments of the present disclosure can correct the coordinates of each initial orthogonal vanishing point pair.

[0092] For example, for any initial orthogonal vanishing point pair vp1 and vp2, the server can determine the initial orthogonal vanishing point pair as a vanishing point pair to be corrected, calculate a loss value corresponding to the coordinates of the two vanishing points to be corrected in the vanishing point pair to be corrected, if the loss value does not reach a convergence condition, adjust the coordinates of the two vanishing points in the vanishing point pair to be corrected to obtain a new vanishing point pair to be corrected, and repeatedly perform the step of calculating the loss value corresponding to the coordinates of the two vanishing points in the vanishing point pair to be corrected; if the loss value reaches the convergence condition, the vanishing point pair to be corrected is determined as the orthogonal vanishing point pair.

[0093] The loss value Loss(f, vp1, vp2) is the sum of a distance loss Loss vp and an orthogonal loss Loss orth For example, it can satisfy: Loss(f, vp1, vp2) = Loss vp + Losso rth

[0094] The distance loss Loss vp refers to the loss between the distance of any vanishing point to each line segment group {L1, L2} corresponding to the respective vanishing point and the preset distance condition, for example, it can satisfy:

[0095] The orthogonal loss refers to the loss between the distance of the two vanishing points relative to the reference optical center coordinate and the to-be-calibrated focal length of the camera, that is, the orthogonal loss can represent the correlation between the distance of the two vanishing points relative to the reference optical center coordinate and the to-be-calibrated focal length of the camera. For example, it can satisfy:

[0096] Where O refers to the reference optical center, and f refers to the to-be-calibrated focal length.

[0097] In some embodiments, in the case that the loss value does not reach the convergence condition, the coordinates of the two vanishing points in the to-be-corrected vanishing point pair and the value of the to-be-calibrated focal length of the camera can be adjusted in parallel to make the value of the orthogonality loss reach the convergence.

[0098] It can be seen that, by means of nonlinear optimization, the coordinates of the vanishing points are nonlinearly optimized. Since nonlinear optimization is a method for solving the target value of one or more nonlinear functions in the objective function or constraint condition, the target value can be the value of the variable that makes the objective function reach the minimum or maximum, and thus it is helpful to correct the coordinates of the two vanishing points in each initial orthogonal vanishing point pair, so as to obtain more accurate vanishing point coordinates and make the orthogonality of the two vanishing points in each orthogonal vanishing point pair more accurate, thereby improving the accuracy of the camera intrinsic parameters calculated based on the vanishing points.

[0099] The loss calculation algorithm and the nonlinear optimization strategy involved in the correction process of the initial orthogonal vanishing point pair can be deployed in the vanishing point optimization module shown in FIG. 2, and the server can call the vanishing point optimization module to perform the above-mentioned correction-related steps of the initial orthogonal vanishing point pair.

[0100] In step S102, the camera intrinsic parameters of the terminal device of the model are generated according to the corresponding orthogonal vanishing point pairs and the corresponding line segment groups of the terminal devices of the same model.

[0101] The model of the terminal device refers to the model or specification of the terminal device. In actual implementation scenarios, the hardware configuration information, size, parameters, and function description of the terminal device of the model can be represented by the model of the terminal device, for example, the model of any terminal device can correspond to the configuration information and parameters of the camera component integrated in the terminal device. The hardware configurations and parameters of terminal devices of different models can be partially or totally different. The model can be information describing the model of the terminal device, and can include the brand identifier, series identifier, and model of the terminal device, for example, the mobile phone model H magic 50, H refers to the brand identifier of the mobile phone, magic refers to the series identifier of the mobile phone, and 50 refers to the model of the mobile phone.

[0102] It should be understood that the model of the terminal device described above is only illustrative and does not limit the embodiments of the present disclosure. In actual implementation, the model can also include more or fewer information items, which are not limited herein.

[0103] The camera intrinsic parameters of the terminal device of the model can be the camera intrinsic parameters of the camera component in the terminal device of the model, for example, including the focal length (fx, fy) of the camera and the optical center (cx, cy) of the camera.

[0104] In some embodiments, the server can determine whether the number of orthogonal vanishing point pairs of the terminal device of the same model reaches a preset number threshold; if so, the server can execute the step of generating the camera intrinsic parameter of the terminal device of the same model according to the orthogonal vanishing point pairs and the corresponding line segment group of the terminal device of the same model. If not, the server can send a line segment collection instruction to the terminal device to receive the line segment collected by the terminal device in response to the instruction, and then continue to obtain the orthogonal vanishing point pairs and the corresponding line segment group of the terminal device of the same model from the line segment.

[0105] For example, the preset number threshold can be set in advance according to the empirical value of the accuracy of the camera intrinsic parameter. For example, the preset number threshold can be 1200.

[0106] Further, the server can call the algorithm in the intrinsic calibration module to calculate the camera intrinsic parameter corresponding to the model according to the preset number threshold of the orthogonal vanishing point pairs.

[0107] It should be noted that, assuming that p and q are a pair of orthogonal vanishing points, K is a camera intrinsic parameter matrix, and K includes the camera intrinsic parameter. For example, p=(p1, p2, 1); q=(q1, q2, 1).

[0108] According to the data principle of orthogonality, it can be inferred that (K -1 ·p) T ·(K -1 ·q)=0, i.e., p T ·(K -1 ) T K -1 ·q=0. Denote K -1 =Q, then Q T Q can be considered as a symmetric matrix, denoted as W, and p T ·Q T Q·q=p T ·W·q=0. Wherein,

[0109] Correspondingly, p T ·Q T Q·q=p T ·W·q=[p1,p2,1] T ·W·[q1,q2,1]=A·v=0. Wherein, v can be a group of equations with (v1,..., v6) as variables, and the matrix A is only related to p and q.

[0110] Based on the above derivation principle, v can be taken as a parameter of calculating an intrinsic variable matrix of the camera, the intrinsic variable matrix being a result of multiplying the inverse matrix of the camera intrinsic matrix by the symmetric matrix of the inverse matrix, and matrix A is taken as a parameter matrix for calculating v.

[0111] Based on this, the camera intrinsic of the terminal equipment of the same model can be generated according to the orthogonal vanishing point pairs and the corresponding line segment groups of the terminal equipment of the same model.

[0112] The coordinates of each vanishing point in the orthogonal vanishing point pairs according to the preset quantity threshold are used to construct a parameter matrix A.

[0113] The camera intrinsic variable matrix is decomposed from the parameter matrix A.

[0114] The camera intrinsic matrix is calculated according to the camera intrinsic variable matrix, and the camera intrinsic matrix includes the focal length and the principal point (i.e. the optical center) of the camera, as shown in the above formula K.

[0115] For example, the intrinsic variable matrix is decomposed from the parameter matrix A, which can be implemented by performing SVD decomposition on the parameter matrix A to obtain a left singular vector, a right singular vector and a singular value matrix, upgrading the dimension of the right singular vector, and the upgraded matrix being the intrinsic variable matrix.

[0116] The SVD decomposition is an algorithm for separating the characteristic vectors and characteristic values contained in a parameter matrix by decomposing the parameter matrix into three matrices. The three matrices include a left singular vector, a right singular vector and a singular value matrix. The left singular vector and the right singular vector are orthogonal matrices, and can represent the characteristic vectors of different dimensions of the parameter matrix. The singular value matrix is a diagonal matrix, and the elements on the diagonal are the characteristic values of the parameter matrix.

[0117] It should be noted that the right singular vector obtained by SVD decomposition is usually a one-dimensional vector with the smallest singular value, for example, a one-dimensional vector of 1*6. The one-dimensional vector can be considered as the above v. Further, the right singular vector obtained by decomposition can be upgraded to a 3*3 skew-symmetric matrix, which can be taken as the intrinsic variable matrix. Since the intrinsic variable matrix is equal to the result of multiplying the inverse matrix of the camera intrinsic matrix by the symmetric matrix of the inverse matrix, based on this, the camera intrinsic matrix can be calculated according to the intrinsic variable matrix, which can include: decomposing the intrinsic variable matrix based on symmetry to obtain the inverse matrix of the camera intrinsic matrix; and performing inverse operation on the inverse matrix of the camera intrinsic matrix to obtain the camera intrinsic matrix.

[0118] For example, the intrinsic variable matrix can be decomposed by LU decomposition to obtain the inverse matrix K of the camera intrinsic matrix. -1and the transpose matrix of the inverse matrix (K -1 ) T . Wherein, the upper triangular matrix U obtained after LU decomposition is the inverse matrix K -1 of the camera intrinsic matrix, and the lower triangular matrix L is the transpose matrix of the inverse matrix (K -1 ) T . The inverse operation is performed on the inverse matrix K -1 of the camera intrinsic matrix (i.e. the upper triangular matrix U), so that the camera intrinsic matrix including the intrinsic parameters of the camera can be obtained.

[0119] Wherein, the LU decomposition can decompose a matrix into two mutually symmetric matrices, one of which is a lower triangular matrix (L matrix), and the other is an upper triangular matrix (U matrix). Wherein, the L matrix and the U matrix are two mutually symmetric matrices, the elements below the diagonal elements of the L matrix are all 0, and the diagonal elements of the U matrix are all 1, and the elements above the diagonal elements are all 0.

[0120] For example, the preset quantity threshold is n, and in this example, n is greater than or equal to 3, and n sets of orthogonal vanishing point pairs are represented as: For example, it is recorded as: u i = (vp i1 [0], vp i1 [1], 1.0), v i = (vp i2 [0], vp i2 [1], 1.0)

[0121] The parameter matrix A can be obtained:

[0122] The SVD decomposition is performed on the parameter matrix A, the right singular vector v corresponding to the minimum singular value is taken, and the intrinsic variable matrix W is constructed according to the right singular vector v:

[0123] The LU decomposition is performed on the intrinsic variable matrix W to obtain the U matrix (the inverse matrix K -1 of the camera intrinsic matrix) and the L matrix (the transpose matrix of the inverse matrix (K -1 ) T ), and the inverse operation is performed on the U matrix, so that the camera intrinsic matrix can be obtained.

[0124] It can be seen that, in the embodiment of the present disclosure, for any type of terminal device, the server collects the crowdsourcing data of the corresponding type of terminal device, and processes and calculates based on the collected crowdsourcing data, thereby calculating the corresponding camera intrinsic parameters of the corresponding type. Wherein, the crowdsourcing data can be the crowdsourcing line segments extracted from the images collected by the corresponding type of terminal device. In this way, without the need for special personnel to collect images for calibration, the calibration efficiency can be significantly improved, and it is beneficial to maintain the coverage rate of the calibrated type at a high level.

[0125] Wherein, the preset number of threshold orthogonal vanishing point pairs can be pre-stored in the vanishing point storage module shown in FIG. 2. After calculating the camera intrinsic parameters corresponding to the type based on the preset number of orthogonal vanishing point pairs, the server can delete the orthogonal vanishing point pairs corresponding to the type from the vanishing point storage module. In this way, the occupation of the storage resources of the server can be reduced.

[0126] The embodiment shown in FIG. 3A is an angle of the implementation of the server generating the camera intrinsic parameters, and the description of the present technical solution. After generating the camera intrinsic parameters of the type of the terminal device, the terminal device can also be interacted with to use the camera intrinsic parameters.

[0127] Please refer to FIG. 3B, which is a second exemplary method flow diagram of a camera calibration method provided by an embodiment of the present disclosure. Referring to FIG. 3B, the camera calibration method can include the following steps:

[0128] In step S201, in response to the camera intrinsic parameter acquisition request sent by the terminal device, the camera intrinsic parameter corresponding to the type of the terminal device carried by the camera intrinsic parameter acquisition request is queried from the pre-stored camera intrinsic parameter calibration data.

[0129] Wherein, the terminal device sends an intrinsic parameter acquisition request to the server in the case of starting the AR function. The terminal device can be any type of electronic device, and the terminal device is integrated with a camera component and can pre-install an APP supporting the AR function, for example, the terminal device is installed with a map APP which can support the AR function.

[0130] The camera intrinsic parameter obtaining request can comprise a model of the terminal device. After receiving the model of the terminal device, the server can search the corresponding relationship between the model and the camera intrinsic parameter stored in the server, and determine whether the corresponding relationship of the model is included in the corresponding relationship between the model and the camera intrinsic parameter. If the corresponding relationship of the model is included, the server sends the camera intrinsic parameter corresponding to the model to the terminal device. That is, the camera intrinsic parameter is sent by the server when the server stores the camera intrinsic parameter corresponding to the model. If the corresponding relationship of the model is not included in the corresponding relationship between the model and the camera intrinsic parameter, the terminal device can obtain a line segment set and send the line segment set to the server, so that the server calculates the camera intrinsic parameter based on the line segment set.

[0131] For example, in the embodiment shown in FIG. 2, the server can traverse each model in the corresponding relationship between the model and the camera intrinsic parameter in the intrinsic parameter storage module. If the model of the terminal device is traversed, it indicates that the server stores the camera intrinsic parameter corresponding to the model. If the model of the terminal device is not traversed, it indicates that the server does not store the camera intrinsic parameter corresponding to the model.

[0132] The server generates the camera intrinsic parameter. For details, refer to the description of the foregoing embodiments, which will not be described here.

[0133] In step S202, the camera intrinsic parameter is sent to the terminal device, so that the terminal device calculates the pose of the terminal device according to the camera intrinsic parameter.

[0134] The terminal device can determine the pose of the terminal device according to the camera intrinsic parameter, and then generate and display an AR model for navigation according to the pose of the terminal device.

[0135] In this way, the accuracy of the AR model integrated into the perspective relationship of the real scene can be improved, thereby improving the user experience.

[0136] In some embodiments, if the camera intrinsic parameter is not received from the server, the terminal device can use the factory-set intrinsic parameter for AR positioning.

[0137] As can be seen, in the embodiments of the present disclosure, for any model of terminal device, the server can obtain the orthogonal vanishing point pair corresponding to the model of the terminal device and the line segment group corresponding to the orthogonal vanishing point pair based on the line segment set associated with the terminal device, so as to calculate the camera intrinsic parameter of the terminal device of the corresponding model according to the obtained data. Since the present disclosure is based on the line segment identified from the image captured by the terminal device, and the camera parameter of the terminal device is calibrated, it can be understood that the image capable of identifying the line segment does not need to be specially captured by a specific pattern, and therefore the calibration efficiency can be obviously improved, and the coverage rate of the calibrated model can be maintained at a high level.

[0138] The above embodiments introduce each implementation of the method provided by the embodiments of the present disclosure from the perspective of extracting vanishing points, extracting orthogonal vanishing point pairs, calibrating camera intrinsic parameters according to the orthogonal vanishing point pairs, and the like. It should be understood that, corresponding to the above processing steps, the embodiments of the present disclosure can implement the above functions in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0139] For example, if the above implementation steps implement the corresponding functions through a software module. As shown in FIG. 4A, FIG. 4A is an exemplary component schematic diagram of a camera calibration device provided by the embodiments of the present disclosure, which is applied to the server shown in FIG. 1 or FIG. 2, and can include an acquisition module 401 and a generation module 402. The camera calibration device can be used to perform part or all of the operations in the above FIG. 3A.

[0140] For example: the acquisition module 401 is configured to acquire, based on a line segment set associated with a terminal device, an orthogonal vanishing point pair corresponding to a model of the terminal device and a line segment group corresponding to the orthogonal vanishing point pair, the line segment set associated with the terminal device including a line segment identified based on an image captured by the terminal device; and the generation module 402 is configured to generate a camera intrinsic parameter of the terminal device of the model according to the orthogonal vanishing point pairs corresponding to the terminal devices of the same model and the line segment groups.

[0141] Optionally, the camera calibration device further includes a judgment module. The judgment module is configured to judge whether a number of the orthogonal vanishing point pairs of the terminal devices of the same model reaches a preset number threshold.

[0142] Optionally, the camera intrinsic parameter is a camera intrinsic parameter matrix, and the generation module 402 is specifically configured to construct a parameter matrix according to coordinates of each vanishing point in the orthogonal vanishing point pairs of the preset number threshold; decompose an intrinsic variable matrix from the parameter matrix; and calculate the camera intrinsic parameter matrix according to the intrinsic variable matrix, the camera intrinsic parameter matrix including a focal length and a principal point of the camera.

[0143] Optionally, the generation module 402 is specifically configured to perform singular value decomposition on the parameter matrix to obtain a left singular value vector, a right singular value vector, and a singular value matrix.

[0144] The right singular value vector is dimensioned, and a matrix after dimensioning is the intrinsic variable matrix.

[0145] Optionally, the generation module 402 is specifically configured to dimension the right singular value vector into a 3*3 anti-symmetric matrix to obtain the intrinsic variable matrix.

[0146] Optionally, the intrinsic variable matrix is equal to a result of multiplication of the inverse matrix of the camera intrinsic matrix and a symmetric matrix of the inverse matrix, the generating module 402 is specifically configured to decompose the intrinsic variable matrix based on symmetry to obtain the inverse matrix of the camera intrinsic matrix; and perform inverse operation on the inverse matrix of the camera intrinsic matrix to obtain the camera intrinsic matrix.

[0147] Optionally, the generating module 402 is specifically configured to perform LU decomposition on the intrinsic variable matrix to obtain an upper triangular matrix U and a lower triangular matrix L which are mutually symmetric; and determine the U matrix as the inverse matrix of the camera intrinsic matrix.

[0148] Optionally, the camera calibration apparatus further includes a sending module and a receiving module, the sending module is configured to

[0149] send a line segment collection instruction to the terminal device, so that the terminal device calls a camera carried thereby to shoot at least one image and acquire a line segment based on the image in response to the instruction; and the receiving module is configured to receive the line segment collected by the terminal device in response to the instruction.

[0150] In some other embodiments, as shown in FIG. 4B, which is an exemplary component schematic diagram two of a camera calibration apparatus provided by the embodiments of the present disclosure, the camera calibration apparatus can include a receiving module 411 and a sending module 412. The camera calibration apparatus can be used to perform part or all of the operations in FIG. 3B described above.

[0151] For example, the receiving module 411 queries the camera intrinsic parameter corresponding to the model of the terminal device from the pre-stored camera intrinsic parameter calibration data in response to a camera intrinsic parameter acquisition request sent by the terminal device, the camera intrinsic parameter is generated according to the embodiment shown in FIG. 3A; and the sending module 412 sends the camera intrinsic parameter to the terminal device, so that the terminal device calculates the pose of the terminal device according to the camera intrinsic parameter.

[0152] It can be understood that the division of each module in FIG. 4A and FIG. 4B is only a logical function division, and the functions of the above modules can be integrated into a hardware entity for actual implementation, for example, the receiving module 411 and the sending module 412 can be integrated into a transceiver, and the functions of the obtaining module 401 and the generating module 402 can be integrated into a processor, etc.

[0153] Referring to FIG. 5, FIG. 5 is an exemplary structural schematic diagram of an electronic device provided by the embodiments of the present disclosure, which can be implemented as the server involved in the above embodiments. The electronic device includes a processor 511, a transceiver 512 and a memory 513, which are connected and communicated through a communication bus 514. The transceiver 512 can be used for transmitting and receiving data, requests and instructions with the terminal device. The memory 513 is used for storing the program instructions of the programmable chip and the vanishing points generated in the camera calibration process, and when the program instructions are called, the processor 511 performs part or all of the operations of the corresponding device in FIGS. 3A and 3B.

[0154] For specific implementation process, please refer to the related description in FIGS. 3A and 3B, which will not be repeated here.

[0155] The embodiments of the present disclosure also provide a camera calibration system, which includes a terminal device and a server, and the terminal device is provided with a camera.

[0156] The terminal device is configured to capture an image and identify a line segment set of the image, and send the line segment set to the server; the server is configured to obtain, based on the line segment set associated with the terminal device, an orthogonal vanishing point pair corresponding to a model of the terminal device and a line segment group corresponding to the orthogonal vanishing point pair, and generate a camera intrinsic parameter of the terminal device of the model according to the orthogonal vanishing point pair and the line segment group corresponding to the terminal device of the same model; the terminal device is further configured to send a camera intrinsic parameter acquisition request to the server, the camera intrinsic parameter acquisition request carrying the model of the terminal device; the server is further configured to query, from pre-stored camera intrinsic parameter calibration data, the camera intrinsic parameter corresponding to the model of the terminal device carried by the request, and send the camera intrinsic parameter to the terminal device; and the terminal device is further configured to calculate a pose of the terminal device according to the camera intrinsic parameter.

[0157] The embodiments of the present disclosure also provide a computer readable storage medium, which stores a camera calibration instruction, and when the instruction is run on a computer, the computer executes part or all of the steps in the method described in the above embodiments.

[0158] The embodiments of the present disclosure also provide a computer program product including a camera calibration instruction, and when the instruction is run on a computer, the computer executes part or all of the steps in the method described in the above embodiments.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the above method embodiments, which will not be repeated here.

[0160] In several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method can be implemented in other manners. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is merely logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0161] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0162] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0163] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a smart phone, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0164] Although the optional embodiments of the present disclosure have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present disclosure.

[0165] The above detailed description of the specific implementation has further detailed the purposes, technical solutions and beneficial effects of the present disclosure. It should be understood that the above detailed description is only a specific implementation of the present disclosure and is not used to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A camera calibration method, wherein, The method includes: Based on the line segment set associated with the terminal device, obtain the orthogonal vanishing point pair corresponding to the model of the terminal device and the line segment group corresponding to the orthogonal vanishing point pair. The line segment set associated with the terminal device includes line segments identified based on the images captured by the terminal device. Based on the orthogonal vanishing point pairs and corresponding line segment groups corresponding to terminal devices of the same model, the camera intrinsic parameters of the terminal device of that model are generated.

2. The method according to claim 1, wherein, Before generating the camera intrinsic parameters of the terminal device of the same model based on the orthogonal vanishing point pairs and corresponding line segment groups of the terminal devices, the method further includes: Determine whether the number of orthogonal vanishing point pairs of terminal devices of the same model reaches a preset threshold. If this is achieved, then the step of generating the camera intrinsic parameters of the terminal device of the same model based on the orthogonal vanishing point pairs and corresponding line segment groups of the terminal devices of the same model is executed.

3. The method according to claim 2, wherein, The camera intrinsic parameters are a camera intrinsic parameter matrix. Generating the camera intrinsic parameters of the terminal device of the same model based on the orthogonal vanishing point pairs and corresponding line segment groups includes: Construct a parameter matrix based on the coordinates of each vanishing point in the orthogonal vanishing point pair with the preset quantity threshold; Decompose the intrinsic parameter variable matrix from the parameter matrix; The camera intrinsic parameter matrix is ​​calculated based on the intrinsic parameter variable matrix, which includes the camera's focal length and principal point.

4. The method according to claim 3, wherein, The step of decomposing the intrinsic parameter variable matrix from the parameter matrix includes: Perform singular value decomposition on the parameter matrix to obtain the left singular value vector, the right singular value vector, and the singular value matrix; The dimension of the right singular value vector is increased, and the matrix after the dimension increase is the intrinsic parameter variable matrix.

5. The method according to claim 4, wherein, The dimension-upgrading of the right singular value vector, resulting in the matrix of intrinsic parameters, includes: The right singular value vector is increased in dimension to a 3*3 antisymmetric matrix to obtain the intrinsic parameter variable matrix.

6. The method according to any one of claims 3-5, wherein, The intrinsic parameter variable matrix is ​​equal to the product of the inverse of the camera intrinsic parameter matrix and the symmetric matrix of the inverse matrix. The calculation of the camera intrinsic parameter matrix based on the intrinsic parameter variable matrix includes: Based on symmetry, the intrinsic parameter variable matrix is ​​decomposed to obtain the inverse matrix of the camera intrinsic parameter matrix; The inverse operation of the inverse matrix of the camera intrinsic parameter matrix is ​​performed to obtain the camera intrinsic parameter matrix.

7. The method according to claim 6, wherein, The decomposition of the intrinsic parameter variable matrix based on symmetry to obtain the inverse matrix of the camera intrinsic parameter matrix includes: The intrinsic parameter variable matrix is ​​decomposed by LU to obtain a mutually symmetric upper triangular U matrix and a lower triangular L matrix; The upper triangular U matrix is ​​determined as the inverse of the camera intrinsic parameter matrix.

8. The method according to any one of claims 1-7, wherein, The method further includes: Send a line segment acquisition command to the terminal device, so that the terminal device responds to the line segment acquisition command by calling its onboard camera to capture at least one image and acquire line segments based on the image; The terminal device receives the line segment collected in response to the line segment collection command.

9. A camera calibration method, wherein, The method includes: In response to a camera intrinsic parameter acquisition request sent by a terminal device, the camera intrinsic parameters corresponding to the model of the terminal device carried in the camera intrinsic parameter acquisition request are queried from the pre-stored camera intrinsic parameter calibration data. The camera intrinsic parameters are generated by the method according to any one of claims 1-8. The camera intrinsic parameters are sent to the terminal device so that the terminal device can calculate the pose of the terminal device based on the camera intrinsic parameters.

10. A camera calibration system, wherein, It includes a terminal device and a server, wherein the terminal device is equipped with a camera, The terminal device is used to capture images and identify a set of line segments in the images, and then send the set of line segments to the server. The server is configured to obtain, based on the line segment set associated with the terminal device, the orthogonal vanishing point pair corresponding to the model of the terminal device and the line segment group corresponding to the orthogonal vanishing point pair, and generate the camera intrinsic parameters of the terminal device of the same model according to the orthogonal vanishing point pair and the corresponding line segment group. The terminal device is also used to send a camera intrinsic parameter acquisition request to the server, the camera intrinsic parameter acquisition request carrying the model of the terminal device; The server is also configured to query the camera intrinsic parameters corresponding to the model of the terminal device carried in the request from the pre-stored camera intrinsic parameter calibration data, and send the camera intrinsic parameters to the terminal device. The terminal device is also used to calculate the pose of the terminal device based on the camera intrinsic parameters.

11. A computer-readable storage medium, wherein, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

12. A computer program product, wherein, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

Images