Large-scale scene monitoring network system calibration method based on SLAM

Abstract

The invention discloses a large-scale scene monitoring network system calibration method based on SLAM, and relates to the technical field of robot vision. The method comprises the following steps: determining a relative pose between a calibration object and a moving object; obtaining a relative relation between a calibration tool coordinate system and a moving object body coordinate system; acquiring a real-time pose and scene mapping of the moving object; calculating the space coordinates of the calibration object: matching the obtained calibration object image and the pose of the moving object, and calculating the coordinates of the feature points of the calibration object in the world coordinate system in combination with the relative pose of the calibration tool coordinate system and the moving object body coordinate system; calibration of internal and external parameters of the camera: extracting two-dimensional coordinates of feature points in the calibration object image, and solving the internal and external parameters of the camera by combining the coordinates of the feature points of the calibration object in the world coordinate system; the method has the beneficial effects that the external parameter calibration of all monitoring cameras in the same scene is realized under the condition that the multiple cameras have no common visual field.

Description

Technical Field

[0001] The present invention relates to the field of robot vision technology, and more specifically, to a SLAM-based large-scene monitoring network system calibration method. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that enables robots to automatically create maps in unknown environments and perform self-localization based on estimates of their own states and maps. High-precision self-localization information is a prerequisite for autonomous mobile platforms to have intelligence and perform tasks such as path planning and mapping. Inertial navigation algorithms have the advantages of high positioning accuracy, strong adaptability, and a wide range of applicable environments. At the same time, vision sensors have the characteristics of low price, easy maintenance, and rich texture information, which has great advantages in relocalization and scene classification. Combining the advantages of multiple sensors, multi-sensor fusion technology has become a research hotspot in the field of SLAM.

[0003] In the prior art, multi-camera calibration technology usually requires that the cameras have a common field of view, can observe the calibration objects at the same time, and solve the pose relationship between multiple cameras through the calibration objects with the same name. However, the layout of the monitoring network system in real large-scale scenes is more sparse, and there may not be a common field of view or the common field of view is small, which makes it difficult to solve the pose between multiple cameras in a large-scale monitoring network. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a large-scene monitoring network system calibration method based on SLAM.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a large-scene monitoring network system calibration method based on SLAM, the improvement of which is that the method comprises the following steps:

[0006] Confirm the relative position and posture between the calibration object and the moving object: install the calibration object on the moving object and obtain the relative relationship between the calibration tool coordinate system and the moving object coordinate system;

[0007] Obtaining the real-time position and posture of moving objects and scene mapping: When a moving object carries a calibration object and moves in the scene, the SLAM algorithm is used to construct a panoramic three-dimensional map and locate the calibration object. The world coordinate system is constructed based on the initial position and posture of the moving object, and the calibration object image is collected.

[0008] Calculate the spatial coordinates of the calibration object: Pair the acquired calibration object image with the mobile object posture, and calculate the coordinates of the feature points of the calibration object in the world coordinate system in combination with the relative posture of the calibration tool coordinate system and the mobile object body coordinate system;

[0009] Calibration of camera intrinsics and extrinsic parameters: extracting two-dimensional coordinates of feature points in the calibration object image, and combining the coordinates of the feature points of the calibration object in the world coordinate system to calculate the camera intrinsics and extrinsic parameters;

[0010] Monitor the digital twin visualization and visualize the position and posture of the camera in the panoramic three-dimensional map.

[0011] Furthermore, the step of confirming the relative position between the calibration object and the moving object further includes:

[0012] The relative relationship between the calibration tool coordinate system and the moving object coordinate system is obtained by using the external parameter calibration method of the calibration tool and the moving object coordinate system.

[0013] Furthermore, the obtaining of the real-time position and posture of the moving object and scene mapping also includes:

[0014] Collect i two-dimensional calibration object images;

[0015] Get the transformed pose of the calibration plate of the i-th frame relative to the calibration plate of the i-1 frame

[0016] Get the position of the moving object in the i-th frame relative to the position of the moving object in the i-1th frame

[0017] Construct the i-1 set of equations:

[0018] The simultaneous equations are obtained using SVD singular value decomposition

[0019] Furthermore, the acquisition of the real-time position and posture of the moving object and scene mapping includes the following steps:

[0020] Set the world coordinate to O w X w Y w Z w , take the initial pose of SLAM technology as the origin of the world coordinate system; set O c X c Y c Z c -m is the camera coordinate system of the mth camera in the panoramic 3D map.

[0021] Furthermore, the acquisition of the real-time position and posture of the moving object and scene mapping also includes the following steps:

[0022] When the moving object enters the field of view of the mth camera, n images of the calibration object are collected for calibration of the camera's internal and external parameters.

[0023] Furthermore, the calibration of the camera's intrinsic and extrinsic parameters includes the following steps:

[0024] Detect the two-dimensional coordinates of the feature points in the nth calibration object image And obtain the three-dimensional coordinates of the characteristic points of the calibration object in the world coordinate system Construct 3D-2D matching feature points:

[0025]

[0026] in, It represents the nth position of the moving object in the world coordinate system after it enters the camera range. is the position of the calibration object relative to the moving object, P cali is the prior three-dimensional coordinate point of the characteristic point of the calibration object in the self-coordinate system;

[0027] Based on the feature point pairs, the camera intrinsic parameter K is solved c , d c ;

[0028] Based on the feature point pairs, solve the PnP problem and obtain the camera's position and posture in the world coordinate system

[0029] Furthermore, the acquisition of the real-time position and posture of the moving object and scene mapping includes the step of driving multiple cameras to capture images of the calibration object.

[0030] Furthermore, the calculation of the spatial coordinates of the calibration object includes using a timestamp to match the calibration object image obtained by each camera with the position and posture of the moving object obtained by using a SLAM algorithm.

[0031] The present invention also provides an electronic device, comprising: at least one processor and at least one memory, wherein:

[0032] The memory has computer-readable instructions stored thereon;

[0033] The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the SLAM-based large-scene monitoring network system calibration method as described above.

[0034] The present invention also provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the SLAM-based large-scene monitoring network system calibration method as described above.

[0035] The beneficial effects of the present invention are as follows: the present invention combines the advantages of full-scene posture acquisition of SLAM technology and the advantages of individual camera calibration, and realizes the external parameter calibration of all monitoring cameras in the same scene when there is no common field of view between multiple cameras. At the same time, it realizes the display of all camera postures obtained by calibration in the three-dimensional scene established under SLAM technology, providing high-precision guidance for multi-camera monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a flowchart of a large-scene monitoring network system calibration method based on SLAM.

[0037] Figure 2 , Figure 3 It is a specific embodiment diagram of a large-scene monitoring network system calibration method based on SLAM of the present invention.

[0038] Figure 4 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0039] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0041] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0042] Reference Figure 1 As shown, the present invention discloses a large-scene monitoring network system calibration method based on SLAM. Specifically, in this embodiment, the method includes the following steps:

[0043] S10, confirming the relative position and posture between the calibration object and the moving object: installing the calibration object on the moving object, and obtaining the relative relationship between the calibration tool coordinate system and the moving object body coordinate system;

[0044] In this embodiment, the step of confirming the relative position between the calibration object and the moving object further includes:

[0045] The relative relationship between the calibration tool coordinate system and the moving object coordinate system is obtained by using the external parameter calibration method of the calibration tool and the moving object coordinate system.

[0046] Combination Figure 2 As shown, it is a specific example of the present invention. Figure 2 In the figure, 1 is the initial posture of the moving object (the moving object in this embodiment is a car) during the movement, which is also used as the world coordinate system; 2 is the trajectory of the moving object generated by the SLAM algorithm using sensors such as lidar and imu during the movement of the moving object; 3 is the monitoring camera to be calibrated, that is, the camera described in this embodiment; 4 is the moving object moving in the scene (the car in this embodiment); 5 is the calibration object fixed on the moving object.

[0047] In this embodiment, the step of confirming the relative position between the calibration object and the moving object further includes:

[0048] S101, collecting i two-dimensional calibration object images;

[0049] S102, obtaining the transformed position of the calibration plate of the i-th frame relative to the calibration plate of the i-1 frame

[0050] S103, obtaining the position of the moving object in the i-th frame relative to the position of the moving object in the i-1-th frame

[0051] S104. Construct i-1 set of equations:

[0052] S105, simultaneous equations, obtained using SVD singular value decomposition

[0053] S20, obtaining the real-time position and posture of the moving object and scene mapping: When the moving object carries the calibration object and moves in the scene, the SLAM algorithm is used to construct a panoramic three-dimensional map and locate the calibration object, and the world coordinate system is constructed with the initial position of the moving object, and the calibration object image is collected; in addition, the obtaining of the real-time position and posture of the moving object and scene mapping also includes the step of driving multiple cameras to collect the calibration object image.

[0054] In this embodiment, the step of obtaining the real-time position and posture of the moving object and scene mapping includes the following steps:

[0055] S201, reference Figure 3 As shown, the world coordinate is set to O w X w Y w Zw , take the initial pose of SLAM technology as the origin of the world coordinate system; set O c X c Y c Z c -m is the camera coordinate system of the mth camera in the panoramic 3D map;

[0056] S202: When the moving object enters the field of view of the mth camera, n images of the calibration object are collected for calibration of the camera's internal and external parameters.

[0057] S30, calculating the spatial coordinates of the calibration object: pairing the acquired calibration object image with the mobile object posture, and combining the relative posture of the calibration tool coordinate system and the mobile object body coordinate system to calculate the coordinates of the feature points of the calibration object in the world coordinate system;

[0058] In this embodiment, the calculation of the calibration object spatial coordinates includes using a timestamp to match the calibration object image obtained by each camera with the moving object posture obtained by using a SLAM algorithm.

[0059] S40, calibration of camera intrinsic parameters and extrinsic parameters: extracting two-dimensional coordinates of feature points in the calibration object image, and combining the coordinates of the feature points of the calibration object in the world coordinate system to calculate the intrinsic parameters and extrinsic parameters of the camera; it should be noted that the camera is a surveillance camera;

[0060] In this embodiment, the step S40, the calibration of the camera intrinsic parameters and extrinsic parameters, includes the following steps:

[0061] S401, detecting the two-dimensional coordinates of the feature points in the nth calibration object image And obtain the three-dimensional coordinates of the characteristic points of the calibration object in the world coordinate system Construct 3D-2D matching feature points:

[0062]

[0063] in, It represents the nth position of the moving object in the world coordinate system after it enters the camera range. is the position of the calibration object relative to the moving object, P cali is the prior three-dimensional coordinate point of the characteristic point of the calibration object in the self-coordinate system;

[0064] S402: Based on the feature point pairs, solve the camera intrinsic parameter K c , d c ;

[0065] S403, based on the feature point pairs, solve the PnP problem and obtain the position and posture of the camera in the world coordinate system

[0066]

[0067] S50, monitoring digital twin visualization, visualizing the position and posture of the camera in the panoramic three-dimensional map.

[0068] The present invention provides a large-scale scene monitoring network system calibration method based on SLAM, and provides a large-scale scene monitoring network multi-camera calibration method combining SLAM global technology and single camera calibration technology under non-common view, including camera intrinsic parameters, camera posture in the scene, and solving the spatial relationship between cameras, so as to solve the problem that multiple cameras in real scenes have no common field of view and are difficult to calibrate.

[0069] The present invention combines the advantages of SLAM technology in acquiring full-scene posture and the advantages of individual camera calibration. In the case where there is no common field of view between multiple cameras, the external parameter calibration of all monitoring cameras in the same scene can be realized. At the same time, the postures of all cameras obtained by calibration can be displayed in the three-dimensional scene established under SLAM technology, providing high-precision guidance for multi-camera monitoring.

[0070] Figure 4 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.

[0071] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 4 One or more components of exemplary electronic device 2000 are shown.

[0072] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 4 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0073] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0074] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this is not intended to be a specific limitation.

[0075] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0076] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0077] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 4 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the UDP-based hardware monitoring device may be regarded as an application 253 deployed on the electronic device 2000.

[0078] The data 255 may be signal information, etc., stored in the memory 250 .

[0079] The central processor 270 may include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the hardware monitoring method based on UDP is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0080] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0081] See also Figure 5 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.

[0082] exist Figure 5 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0083] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0084] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0085] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0086] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0087] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0088] The computer-readable instructions are executed by one or more processors 4001 to implement the UDP-based hardware monitoring method in the above-mentioned embodiments.

[0089] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the SLAM-based large-scene monitoring network system calibration method as described above.

[0090] In the above embodiments provided in the present application, it should be understood that the disclosed methods, devices, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple components or modules can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or components or modules, which can be electrical, mechanical or other forms.

[0091] The components described as separate components may or may not be physically separated, and the components shown as components may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the components may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each component may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0093] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0094] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0095] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A large-scene monitoring network system calibration method based on SLAM, characterized in that: The method comprises the following steps: Confirm the relative position and posture between the calibration object and the moving object: install the calibration object on the moving object and obtain the relative relationship between the calibration tool coordinate system and the moving object coordinate system; Obtaining the real-time position and posture of moving objects and scene mapping: When a moving object carries a calibration object and moves in the scene, the SLAM algorithm is used to construct a panoramic three-dimensional map and locate the calibration object. The world coordinate system is constructed based on the initial position and posture of the moving object, and the calibration object image is collected. Calculate the spatial coordinates of the calibration object: Pair the acquired calibration object image with the mobile object posture, and calculate the coordinates of the feature points of the calibration object in the world coordinate system in combination with the relative posture of the calibration tool coordinate system and the mobile object body coordinate system; Calibration of camera intrinsics and extrinsic parameters: extracting two-dimensional coordinates of feature points in the calibration object image, and combining the coordinates of the feature points of the calibration object in the world coordinate system to calculate the camera intrinsics and extrinsic parameters; Monitor the digital twin visualization and visualize the position and posture of the camera in the panoramic three-dimensional map.

2. A SLAM-based large-scene monitoring network system calibration method according to claim 1, characterized in that: The step of confirming the relative position between the calibration object and the moving object further includes: The relative relationship between the calibration tool coordinate system and the moving object coordinate system is obtained by using the external parameter calibration method of the calibration tool and the moving object coordinate system.

3. A large-scene monitoring network system calibration method based on SLAM according to claim 2, characterized in that: The step of confirming the relative position between the calibration object and the moving object further includes: Collect i two-dimensional calibration object images; Get the transformed position of the calibration plate of the i-th frame relative to the calibration plate of the i-1 frame Get the position of the moving object in the i-th frame relative to the position of the moving object in the i-1th frame Construct the i-1 set of equations: The simultaneous equations are obtained using SVD singular value decomposition 4. A SLAM-based large-scene monitoring network system calibration method according to claim 3, characterized in that: The step of obtaining the real-time position and posture of the moving object and building a scene map comprises the following steps: Set the world coordinate to O w X w Y w Z w , take the initial pose of SLAM technology as the origin of the world coordinate system; set O c X c Y c Z c -m is the camera coordinate system of the mth camera in the panoramic 3D map.

5. A SLAM-based large-scene monitoring network system calibration method according to claim 4, characterized in that: The method of obtaining the real-time position and posture of the moving object and scene mapping also includes the following steps: When the moving object enters the field of view of the mth camera, n images of the calibration object are collected for calibration of the camera's internal and external parameters.

6. A SLAM-based large-scene monitoring network system calibration method according to claim 5, characterized in that: The calibration of the camera's internal and external parameters includes the following steps: Detect the two-dimensional coordinates of the feature points in the nth calibration object image And obtain the three-dimensional coordinates of the characteristic points of the calibration object in the world coordinate system Construct 3D-2D matching feature points: in, It represents the nth position of the moving object in the world coordinate system after it enters the camera range. is the position of the calibration object relative to the moving object, P cali is the prior three-dimensional coordinate point of the feature point of the calibration object in the self-coordinate system; Based on the feature point pairs, the camera intrinsic parameter K is solved c , d c ; Based on the feature point pairs, solve the PnP problem and obtain the camera's position and posture in the world coordinate system 7. A SLAM-based large-scene monitoring network system calibration method according to claim 1, characterized in that: The obtaining of the real-time position and posture of the moving object and scene mapping includes the step of driving multiple cameras to collect images of the calibration object.

8. A SLAM-based large-scene monitoring network system calibration method according to claim 7, characterized in that: The calculation of the calibration object spatial coordinates includes using the timestamp to match the calibration object image obtained by each camera with the moving object posture obtained by using the SLAM algorithm.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the SLAM-based large-scene monitoring network system calibration method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the SLAM-based large-scene monitoring network system calibration method as described in any one of claims 1 to 7.

Images