A large scene monitoring network system calibration method based on slam

Abstract

The application discloses a kind of big scene monitoring network system calibration methods based on SLAM, it is related to robot vision technical field;The method comprises the following steps: confirming the relative pose between calibration object and mobile object: the relative relationship of calibration tool coordinate system and mobile object body coordinate system is obtained;Mobile object real-time pose and scene mapping are obtained;Calculate the space coordinates of calibration object: the calibration object image obtained is paired with mobile object pose, and the relative pose of calibration tool coordinate system and mobile object body coordinate system is combined, the coordinates of the feature point of calibration object under the world coordinate system are calculated;Camera internal participation external parameter calibration: two-dimensional coordinates of feature points are extracted in the calibration object image, and the coordinates of the feature point of calibration object under the world coordinate system are combined, to obtain the internal participation external parameter of camera;The beneficial effects of the application are: in the absence of common field of view between multiple cameras, the external parameter calibration of all monitoring cameras in the same scene is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot vision technology, and more particularly, to a large scene monitoring network system calibration method based on SLAM. BACKGROUND

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that enables a robot to automatically create a map in an unknown environment and self-localize according to the estimation of its own state and the map. High-precision self-localization information is a prerequisite for autonomous mobile platforms to have intelligence and perform path planning, mapping, and other tasks. Inertial navigation algorithms have the advantages of high positioning accuracy, strong adaptability, and wide applicability, while vision has the characteristics of low sensor price, easy maintenance, and rich texture information, and has great advantages in repositioning and scene classification. Combining the advantages of multiple sensors, multi-sensor fusion technology has become a research hotspot in the field of SLAM.

[0003] In the prior art, multi-camera calibration technology usually requires cameras to have a common field of view between cameras, which can simultaneously observe the calibration object and solve the pose relationship between multiple cameras through the same calibration object. However, the layout of the monitoring network system in a large-scale real scene is more sparse, and there is no common field of view or the common field of view is small, which makes it difficult to solve the pose between multiple cameras in a large scene monitoring network. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the present application provides a large scene monitoring network system calibration method based on SLAM.

[0005] The technical scheme adopted by the present application to solve its technical problems is: a large scene monitoring network system calibration method based on SLAM, which improves in that the method comprises the following steps:

[0006] Confirming the relative pose between the calibration object and the mobile object: installing the calibration object on the mobile object to obtain the relative relationship between the calibration tool coordinate system and the mobile object coordinate system;

[0007] Obtaining the real-time pose of the mobile object and the scene mapping: when the mobile object carrying the calibration object moves in the scene, a panoramic three-dimensional map is constructed and the calibration object is positioned using the SLAM algorithm, a world coordinate system is constructed based on the initial pose of the mobile object, and the calibration object image is collected;

[0008] Calculating the spatial coordinates of the calibration object: pairing the obtained calibration object image and mobile object pose, and combining the relative pose of the calibration tool coordinate system and the mobile object coordinate system, calculating the coordinates of the feature points of the calibration object in the world coordinate system;

[0009] Calibration of camera intrinsic and extrinsic parameters: extracting the two-dimensional coordinates of feature points in the calibration object image, and combining the coordinates of the feature points of the calibration object in the world coordinate system to calculate the camera intrinsic and extrinsic parameters;

[0010] Monitor digital twin visualization and visualize the camera's position in the panoramic three-dimensional map.

[0011] Furthermore, the step of confirming the relative position between the calibration object and the moving object further includes:

[0012] The relative relationship between the calibration tool coordinate system and the moving object coordinate system is obtained by the external parameter calibration method of the calibration tool and the moving object coordinate system.

[0013] Furthermore, the obtaining of the real-time position and posture of the moving object and scene mapping also includes:

[0014] Collect i two-dimensional calibration object images;

[0015] Get the transformed pose of the calibration plate of frame i relative to the calibration plate of frame i-1

[0016] Get the pose of the moving object in the i-th frame relative to the pose of the moving object in the i-1-th frame

[0017] Construct the i-1 set of equations:

[0018] The simultaneous equations are obtained using SVD singular value decomposition

[0019] Furthermore, the acquisition of the real-time position and posture of the moving object and scene mapping includes the following steps:

[0020] Set the world coordinate to O w X w Y w Z w , take the initial pose of SLAM technology as the origin of the world coordinate system; set O c X c Y c Z c -m is the camera coordinate system of the mth camera in the panoramic 3D map.

[0021] Furthermore, the acquisition of the real-time position and posture of the moving object and scene mapping further includes the following steps:

[0022] When the moving object enters the field of view of the mth camera, n images of the calibration object are collected for calibrating the camera's intrinsic and extrinsic parameters.

[0023] Further, the camera internal participates in the calibration of the external parameter, including the following steps:

[0024] Detecting the two-dimensional coordinates of the feature points in the nth calibration object image And obtaining the three-dimensional coordinates of the feature points of the calibration object in the world coordinate system Constructing 3D-2D matching feature points:

[0025]

[0026] Wherein, The nth pose of the mobile object in the world coordinate system after entering the camera range, P is the pose of the calibration object relative to the mobile object, cali The prior three-dimensional coordinate point of the feature point of the calibration object in the self-coordinate system;

[0027] Based on the feature point pair, the camera internal parameter K is solved c , d c ;

[0028] Based on the feature point pair, the PnP problem is solved, and the pose of the camera in the world coordinate system is obtained

[0029] Further, the acquisition of the real-time pose of the mobile object and the scene mapping includes the step of driving a plurality of cameras to collect calibration object images.

[0030] Further, the calculation of the space coordinates of the calibration object includes the use of timestamps in combination with the calibration object images obtained by each camera and the use of SLAM algorithm to obtain the pose of the mobile object.

[0031] The application also provides an electronic device, comprising: at least one processor and at least one memory, wherein,

[0032] The memory has computer readable instructions stored thereon;

[0033] The computer readable instructions are executed by one or more processors, so that the electronic device implements the SLAM-based large scene monitoring network system calibration method as described above.

[0034] The application also provides a storage medium having computer readable instructions stored thereon, which are executed by one or more processors to implement the SLAM-based large scene monitoring network system calibration method as described above.

[0035] The application has the advantages that the application combines the advantages of SLAM technology in acquiring the whole scene position and the advantages of camera single calibration, realizes the external parameter calibration of all monitoring cameras in the same scene without the common view among the multiple cameras, realizes the display of all camera positions acquired by calibration in the three-dimensional scene established by the SLAM technology, and provides high-precision guidance for the multi-camera monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of a large scene monitoring network system calibration method based on SLAM.

[0037] Figure 2 、 Figure 3 A specific embodiment diagram of a large scene monitoring network system calibration method based on SLAM.

[0038] Figure 4 It is a hardware structure diagram of an electronic device according to an exemplary embodiment.

[0039] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] The application will be further described below in combination with the drawings and embodiments.

[0041] The concept, specific structure and generated technical effects of the application will be clearly and completely described below in combination with the embodiments and drawings, so as to fully understand the purposes, features and effects of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. Based on the embodiments of the application, other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application. In addition, all the coupling / connections involved in the patent do not mean that the components are directly connected, but means that the more optimal coupling structure can be composed by adding or reducing the coupling accessories according to the specific implementation situation. The technical features in the present application can be combined interactively without mutual contradiction and conflict.

[0042] Referring to Figure 1 The application discloses a large scene monitoring network system calibration method based on SLAM, and specifically, in the embodiment, the method comprises the following steps:

[0043] S10, confirming the relative position between the calibration object and the mobile object: installing the calibration object on the mobile object to acquire the relative relationship between the calibration tool coordinate system and the mobile object coordinate system;

[0044] In the embodiment, the relative pose between the calibration object and the mobile object further comprises:

[0045] The relative relationship between the calibration tool coordinate system and the mobile object body coordinate system is obtained through the external parameter calibration method of the calibration tool and the mobile object body coordinate system

[0046] Combined with Figure 2 As shown in the figure, it is a specific example of the application, in Figure 2 In the figure, 1 is the initial pose of the mobile object (in this embodiment, the mobile object is a car) during operation, which is also the world coordinate system; 2 is the trajectory generated by the mobile object during movement through sensors such as laser radars and IMUs using SLAM algorithm; 3 is the monitoring camera to be calibrated, which is the camera in this embodiment; 4 is the mobile object (in this embodiment, a car) moving in the scene; 5 is the calibration object fixed on the mobile object.

[0047] In the embodiment, the relative pose between the calibration object and the mobile object further comprises:

[0048] S101, collect i two-dimensional calibration object images;

[0049] S102, obtain the transformation pose of the i-th frame calibration board relative to the i-1th frame calibration board

[0050] S103, obtain the mobile object pose of the i-th frame relative to the mobile object pose of the i-1th frame

[0051] S104, construct i-1 equations:

[0052] S105, solve the equations using SVD singular value decomposition to obtain

[0053] S20, obtain the real-time pose of the mobile object and the scene mapping: when the mobile object carrying the calibration object moves in the scene, use the SLAM algorithm to construct a panoramic three-dimensional map and locate the calibration object, and construct a world coordinate system with the initial pose of the mobile object, and collect calibration object images; In addition, the step of driving a plurality of cameras to collect calibration object images is further included.

[0054] In the embodiment, the step of obtaining the real-time pose of the mobile object and the scene mapping comprises the following steps:

[0055] S201, refer to Figure 3 As shown in the figure, the world coordinate is set as O w X w Y w Zw , the initial pose of the SLAM technology is taken as the origin of the world coordinate system; O c X c Y c Z c -m is the camera coordinate system of the mth camera under the panoramic three-dimensional map;

[0056] S202, when the mobile object enters the field of view of the mth camera, n images of the calibration object are collected for the calibration of the camera internal and external parameters.

[0057] S30, calculating the spatial coordinates of the calibration object: pairing the acquired calibration object images and the mobile object pose, and combining the relative pose of the calibration tool coordinate system and the mobile object body coordinate system, the coordinates of the feature points of the calibration object in the world coordinate system are calculated.

[0058] In this embodiment, the calculation of the spatial coordinates of the calibration object includes using the timestamp to cooperate with the calibration object images acquired under each camera and using the SLAM algorithm to acquire the mobile object pose.

[0059] S40, calibration of camera internal and external parameters: extracting the two-dimensional coordinates of the feature points in the calibration object image, and combining the coordinates of the feature points of the calibration object in the world coordinate system to obtain the internal and external parameters of the camera; it should be noted that the camera is a monitoring camera.

[0060] In this embodiment, the step S40 of calibrating the camera internal and external parameters includes the following steps:

[0061] S401, detecting the two-dimensional coordinates of the feature points in the nth calibration object image and obtaining the three-dimensional coordinates of the feature points of the calibration object in the world coordinate system Constructing 3D-2D matching feature points:

[0062]

[0063] wherein, represents the nth pose of the mobile object in the world coordinate system after entering the camera range, is the pose of the calibration object relative to the mobile object, P cali is the prior three-dimensional coordinate point of the feature point of the calibration object in the self-coordinate system;

[0064] S402, based on the feature point pair, the camera intrinsic parameter K c , d c is solved;

[0065] S403, based on the feature point pair, the PnP problem is solved to obtain the pose of the camera in the world coordinate system

[0066]

[0067] S50, monitoring a digital twin visualization, visualizing the pose of the camera in the panoramic three-dimensional map.

[0068] The application discloses a large-scene monitoring network system calibration method based on SLAM.

[0069] The application combines the advantages of SLAM technology in acquiring the whole scene pose and the advantages of camera calibration, and realizes the extrinsic parameter calibration of all monitoring cameras in the same scene without a common view between the multiple cameras, and simultaneously realizes the display of all camera poses acquired by calibration in the three-dimensional scene established by the SLAM technology, thereby providing high-precision guidance for multi-camera monitoring.

[0070] Figure 4 According to an exemplary embodiment, a structure of an electronic device is shown.

[0071] It should be noted that the electronic device is only an example adapted to the application, and should not be considered as providing any limitation on the use range of the application. The electronic device should not be interpreted as being dependent on or necessarily having Figure 4 One or more components in the exemplary electronic device 2000 are shown.

[0072] The hardware structure of the electronic device 2000 can have great differences due to different configurations or performances, such as Figure 4 As shown, the electronic device 2000 includes a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0073] Specifically, the power supply 210 is configured to provide working voltage for each hardware device on the electronic device 2000.

[0074] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to the application, the interface 230 can further include at least one serial-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this is not a specific limitation.

[0075] The memory 250 can be a read-only memory, a random access memory, a magnetic disk, an optical disk, etc., and stores resources including an operating system 251, an application 253, data 255, etc. The storage manner can be temporary storage or permanent storage.

[0076] The operating system 251 is used to manage and control each hardware device and the application 253 on the electronic device 2000, so as to realize the operation and processing of the central processing unit 270 on the mass data 255 in the memory 250. The operating system 251 can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0077] The application 253 is computer readable instructions for completing at least one specific work based on the operating system 251. The application 253 can include at least one module (not shown), and each module can include computer readable instructions for the electronic device 2000. For example, the UDP-based hardware monitoring device can be regarded as an application 253 deployed on the electronic device 2000. Figure 4

[0078] The data 255 can be signal information, etc., and is stored in the memory 250.

[0079] The central processing unit 270 can include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus, so as to read the computer readable instructions stored in the memory 250, and then realize the operation and processing of the mass data 255 in the memory 250. For example, the UDP-based hardware monitoring method is completed by the central processing unit 270 reading a series of computer readable instructions stored in the memory 250.

[0080] In addition, the present application can also be realized by hardware circuit or hardware circuit combined with software, and therefore, the realization of the present application is not limited to any specific hardware circuit, software and combination of the two.

[0081] Please refer to Figure 5 , an electronic device 4000 is provided in the embodiment of the present application, and the electronic device 4000 can include a desktop computer, a notebook computer, a server, etc. with sensor identification capability.

[0082] In Figure 5 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0083] ​The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 can include a channel for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the middle, but it does not mean that there is only one bus or one type of bus.

[0084] Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0085] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the present disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0086] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program instructions or code in the form of instructions or data structures and that can be accessed by the electronic device 4000, but is not limited thereto.

[0087] The memory 4003 stores computer readable instructions, which can be read by the processor 4001 through the communication bus 4002.

[0088] The computer readable instructions are executed by the one or more processors 4001 to implement the above-described UDP-based hardware monitoring method in each embodiment.

[0089] In addition, the present embodiment provides a storage medium, which stores computer readable instructions, and the computer readable instructions are executed by one or more processors to implement the above-described SLAM-based large scene monitoring network system calibration method.

[0090] In the above-described embodiments provided in the present application, it should be understood that the disclosed method, device, computer readable storage medium and electronic device can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of components or modules can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between the components or modules can be indirect coupling or communication connection through some interfaces, devices or components, and can be electrical, mechanical or other forms.

[0091] The components described as separate components may or may not be physically separate, and the components shown as components may or may not be physical modules, i.e., they may be located in one place or distributed to multiple network modules. Some or all of the components can be selected as needed to achieve the purpose of the embodiment.

[0092] In addition, the functional modules in each embodiment of the application can be integrated into one processing module, or each component can be physically present separately, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0093] The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0094] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the application is not limited by the order of the described actions, because according to the application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the application.

[0095] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0096] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the described embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for calibrating a large scene monitoring network system based on SLAM, characterized in that, The method comprises the following steps: Confirming the relative pose between the calibration object and the mobile object: installing the calibration object on the mobile object, and obtaining the relative relationship between the calibration tool coordinate system and the mobile object body coordinate system; The confirmation of the relative pose between the calibration object and the mobile object further comprises: Collecting i two-dimensional calibration object images; obtaining a transformation pose of the i-th frame calibration board relative to the i-1-th frame calibration board acquiring the pose of the moving object in the ith frame relative to the pose of the moving object in the ith-1 frame Constructing i-1 group equation: Wherein, is the relative relationship between the calibration tool coordinate system and the moving object body coordinate system; Simultaneous equations, obtained using SVD singular value decomposition Obtaining the real-time pose of the mobile object and the scene mapping: when the mobile object carrying the calibration object moves in the scene, using the SLAM algorithm to construct a panoramic three-dimensional map and locate the calibration object, and constructing a world coordinate system based on the initial pose of the mobile object, and collecting calibration object images; Calculating the spatial coordinates of the calibration object: pairing the obtained calibration object images and mobile object poses, and combining the relative pose between the calibration tool coordinate system and the mobile object body coordinate system to calculate the coordinates of the feature points of the calibration object in the world coordinate system; Calibration of camera internal and external parameters: extracting feature point two-dimensional coordinates in the calibration object image, and combining the coordinates of the feature points of the calibration object in the world coordinate system to obtain the internal and external parameters of the camera; The calibration of the camera internal and external parameters comprises the following steps: detecting the two-dimensional coordinates of the feature points in the nth calibration object image and obtaining the three-dimensional coordinates of the feature points of the calibration object in the world coordinate system constructing the matching feature points of 3D-2D wherein, represents the n-th pose of the mobile object in the world coordinate system after entering the camera range, P is the pose of the calibration object relative to the mobile object, cali is the prior three-dimensional coordinate point of the feature point of the calibration object in the self-coordinate system; Based on the feature point pairs, the camera internal parameters are solved; Solve PnP problem based on feature point pairs to obtain pose of camera in world coordinate system Monitor digital twin visualization, visualize the pose of the camera in the panoramic three-dimensional map. 2.The method of claim 1, wherein, The confirmation of the relative pose between the calibration object and the mobile object further comprises: The relative relationship between the calibration tool coordinate system and the mobile object body coordinate system is obtained through the external parameter calibration method of the calibration tool and the mobile object body coordinate system 3. The method of claim 1, wherein, The obtaining of the real-time pose of the mobile object and the scene mapping comprises the following steps: Set the world coordinates as O w X w Y w Z w The initial pose of the SLAM technology is set as the origin of the world coordinate system; set O c X c Y c Z c -m is the camera coordinate system of the mth camera under the panoramic three-dimensional map.

4. The method of claim 3, wherein, The obtaining of the real-time pose of the mobile object and the scene mapping further comprises the following steps: When the mobile object enters the field of view of the mth camera, n calibration object images are collected for camera internal and external parameter calibration.

5. The method of claim 1, wherein, The obtaining of the real-time pose of the mobile object and the scene mapping comprises the step of driving multiple cameras to collect calibration object images.

6. The method of claim 5, wherein, The calculation of the spatial coordinates of the calibration object comprises using timestamps to match the calibration object images obtained by each camera and using the SLAM algorithm to obtain the mobile object pose.

7. An electronic device, comprising: Comprise: At least one processor and at least one memory, The memory has computer readable instructions stored thereon; The computer readable instructions are executed by one or more processors to enable the electronic device to implement the SLAM-based large scene monitoring network system calibration method according to any one of claims 1 to 6.

8. A storage medium having stored thereon computer readable instructions, characterized in that, The computer readable instructions are executed by one or more processors to implement the SLAM-based large scene monitoring network system calibration method according to any one of claims 1 to 6.

Images