External parameter calibration method and device, electronic equipment, computer readable storage medium and computer program product

By setting up a calibration board on a mobile device and using automated data acquisition to calibrate and calibrate external parameters of distance sensors and image sensors, the problem of low manual calibration stability is solved, and high accuracy and consistency external parameters of calibration are achieved.

CN119068059APending Publication Date: 2024-12-03UBTECH ROBOTICS CORP LTD

Patent Information

Application Number
CN202411025248.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the external parameter calibration between the distance sensor and the image sensor relies on manual calibration, resulting in low stability, large influence of human factors, time-consuming and labor-consuming, poor repeatability and large errors.

Method used

By setting up N sets of calibration plates on the mobile device, the mobile device collects data from distance sensors and image sensors on the mobile path, uses point cloud data and calibration images to automatically calibrate the initial external parameters, and calibrates the external parameters through the calibration module to obtain Target external parameters to achieve automated calibration.

Benefits of technology

Without human participation, the stability and accuracy of external parameter calibration is improved, the uncertainty caused by human operation is reduced, and the calibration results of different equipment are consistent.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068059A_ABST
    Figure CN119068059A_ABST
Patent Text Reader

Abstract

The invention provides an external parameter calibration method and device, electronic equipment, a computer program product and a computer readable storage medium. The method comprises the following steps: determining a moving path of mobile equipment and N groups of calibration plates positioned on the moving path; wherein N is a positive integer, and poses corresponding to different groups of calibration plates are different; in the process that the mobile device moves on the moving path, point cloud data collected by a distance sensor for the N sets of calibration plates and calibration images collected by an image sensor for the N sets of calibration plates are obtained; based on the point cloud data and the calibration image, calibrating an external parameter between the distance sensor and the image sensor to obtain N initial external parameters; and based on the N initial external parameters, calibrating the external parameters between the distance sensor and the image sensor to obtain target external parameters. According to the invention, the stability of external parameter calibration can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to external parameter calibration technology, and in particular to an external parameter calibration method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of automation technology, distance sensors and image sensors have become important sensors in some automation equipment. Distance sensors can detect the distance and speed of objects and collect point cloud data of objects, while image sensors can provide high-resolution image information. Distance sensors and image sensors can help devices have environmental perception functions such as visual systems, navigation systems, and autonomous driving systems. Therefore, external parameter calibration between distance sensors and image sensors is crucial.

[0003] In the related art, manual calibration is usually relied on, and the calibration plate is rotated by holding it to generate more postures. However, this method is greatly affected by human factors, resulting in low stability of the external parameter calibration of the device. Summary of the invention

[0004] The embodiments of the present application provide an external parameter calibration method, device, electronic device, computer-readable storage medium and computer program product, which can enable a mobile device to automatically complete the calibration process without human intervention, thereby improving the stability of the external parameter calibration.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an external parameter calibration method, the method comprising:

[0007] Determine a moving path of a mobile device and N groups of calibration plates located on the moving path; wherein N is a positive integer, and different groups of calibration plates correspond to different postures; in the process of the mobile device moving on the moving path, obtain point cloud data collected by a distance sensor for the N groups of calibration plates, and calibration images collected by an image sensor for the N groups of calibration plates; based on the point cloud data and the calibration images, calibrate the extrinsic parameters between the distance sensor and the image sensor to obtain N initial extrinsic parameters; based on the N initial extrinsic parameters, calibrate the extrinsic parameters between the distance sensor and the image sensor to obtain target extrinsic parameters.

[0008] The present application provides an external parameter calibration device, including:

[0009] A determination module, used to determine a moving path of the mobile device and N groups of calibration plates located on the moving path; wherein N is a positive integer, and different groups of calibration plates correspond to different postures;

[0010] an acquisition module, configured to acquire point cloud data collected by a distance sensor for the N groups of calibration plates and calibration images collected by an image sensor for the N groups of calibration plates during movement of the mobile device on the movement path;

[0011] A calibration module, configured to calibrate the external parameters between the distance sensor and the image sensor based on the point cloud data and the calibration image to obtain N initial external parameters;

[0012] The calibration module is used to calibrate the extrinsic parameters between the distance sensor and the image sensor based on the N initial extrinsic parameters to obtain target extrinsic parameters.

[0013] In the above scheme, the N groups of calibration plates are arranged on both sides of the moving path based on a specified order; wherein each group of calibration plates includes two calibration plates with the same posture and the two calibration plates with the same posture are respectively arranged on both sides of the moving path.

[0014] In the above solution, the N groups of calibration plates are arranged on the same side of the moving path based on a specified order; wherein each group of calibration plates includes one calibration plate.

[0015] In the above scheme, the size of the calibration plate is not less than the size threshold; the distance between the center of the calibration plate and the center line of the moving path does not exceed the first distance threshold; the distance between two adjacent calibration plates located on the same side of the moving path is greater than the second distance threshold.

[0016] In the above scheme, the placement of the calibration plate includes at least one of the following: flat placement, three-dimensional placement, hanging placement, and dynamic placement; wherein the flat placement is used to indicate that the calibration plate is statically laid flat on the moving path; the three-dimensional placement is used to indicate that different groups of calibration plates have different heights or angles; the dynamic placement is used to indicate that the calibration plate rotates based on a preset speed, and the preset speed is associated with the moving speed of the mobile device on the moving path.

[0017] In the above solution, when the N groups of calibration plates are placed in the same manner, the distances between the centers of calibration plates in different groups and the center line of the moving path are different.

[0018] In the above scheme, the acquisition module is also used to obtain the position correspondence between different calibration plates and different sections in the moving path; according to the position correspondence, determine the target section in the moving path corresponding to each group of the calibration plates; when the mobile device moves on the target section, obtain the target point cloud data collected by the distance sensor for the calibration plates of the corresponding group, and the target calibration image collected by the image sensor for the calibration plates of the corresponding group; combine the target point cloud data corresponding to each group of the calibration plates obtained to obtain the point cloud data collected by the distance sensor for the N groups of calibration plates, and combine the target calibration images corresponding to each group of the calibration plates obtained to obtain the calibration images collected by the image sensor for the N groups of calibration plates.

[0019] In the above scheme, the calibration module is also used to divide the point cloud data and the calibration image according to the N groups of calibration plates to obtain target point cloud data corresponding to each group of calibration plates and a target calibration image corresponding to the target point cloud data; based on the target point cloud data and the target calibration image corresponding to each group of calibration plates, the external parameters between the distance sensor and the image sensor are calibrated to obtain N initial external parameters.

[0020] In the above scheme, the calibration module is also used to perform the following processing for each group of the calibration plates: extracting features from the target point cloud data and the target calibration image to obtain point cloud features and image features; performing coordinate mapping on the point cloud features and the image features to obtain initial external parameters corresponding to the calibration plate.

[0021] In the above scheme, the calibration module is also used to obtain the arrangement sequence number of the target calibration plate corresponding to the target calibration image on the moving path; based on the arrangement sequence number, determine the deviation data of the target calibration plate in the target calibration image; based on the deviation data, determine the cropping range for the target calibration image; based on the cropping range, crop the target calibration image, and the cropped target calibration image is used for the feature extraction.

[0022] In the above scheme, the calibration module is also used to perform the following processing for the target point cloud data and the target calibration image corresponding to each group of the calibration plate: based on the timing information of the target point cloud data and the target calibration image, determine M pairs of point cloud image data, wherein M is a positive integer, and each pair of the point cloud image data includes a frame of target point cloud data and a frame of target calibration image corresponding to the timing; based on each pair of the point cloud image data, calibrate the external parameters of the distance sensor and the image sensor to obtain M reference external parameters corresponding to each group of the calibration plate, and the M reference external parameters constitute the initial external parameters corresponding to each group of the calibration plate.

[0023] In the above scheme, the calibration module is also used to obtain the dot product error and calibration error of each of the initial external parameters, wherein the dot product error is used to indicate the first angle offset between the normal vector of the corresponding target point cloud data and the plane of the target calibration image, and the calibration error is used to indicate the second angle offset between the normal vector of the target point cloud data and the normal vector of the target calibration image; from the N initial external parameters, the initial external parameters whose first angle offset is greater than the first angle threshold and whose second angle offset is less than the second angle threshold are screened out, and the initial rotation matrix included in the screened initial external parameters is determined as the target rotation matrix, and the first angle threshold is greater than the second angle threshold; key points are extracted from the target point cloud data and the target calibration image to obtain n pairs of first key points, wherein n is a positive integer, and each pair of the first key points includes a point cloud key point and an image key point; based on the target rotation matrix and the n pairs of first key points, the external parameters between the distance sensor and the image sensor are calibrated to obtain target external parameters.

[0024] In the above scheme, the calibration module is also used to project the point cloud key points in the n pairs of first key points into the calibration image based on the initial external parameters to obtain n point cloud projection points, so as to combine the n image key points in the n pairs of first key points to form n pairs of second key points, each pair of the second key points includes a point cloud projection point and an image key point; obtain the first distance between the point cloud projection point and the image key point in each pair of the second key points, and average the n first distances to obtain the average distance; from the n pairs of second key points, select m pairs of second key points whose first distance does not exceed the average distance, wherein m is a positive integer not exceeding n; based on the target rotation matrix and the m pairs of second key points, select the target translation vector from the initial translation vectors included in the N initial external parameters; combine the target rotation matrix and the target translation vector to obtain the target external parameters.

[0025] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0026] A memory for storing computer executable instructions;

[0027] The processor is used to implement the external parameter calibration method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0028] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the external parameter calibration method provided in the embodiment of the present application when executed by a processor.

[0029] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the external parameter calibration method provided in the embodiment of the present application is implemented.

[0030] By applying the above embodiment, N groups of calibration plates are set on the moving path, and data is collected for the N groups of calibration plates when the mobile device moves on the moving path, that is, point cloud data is collected for the N groups of calibration plates by the distance sensor, and calibration images are collected for the N groups of calibration plates by the image sensor. Then, the external parameters between the distance sensor and the image sensor are automatically calibrated using the collected point cloud data and calibration images to obtain N initial external parameters, and then the external parameters are calibrated based on the N initial external parameters to obtain the target external parameters. In this way, it is possible to replace manual movement with autonomous movement of the mobile device, and the entire process does not require human participation, thereby reducing the uncertainty caused by human operation, and because the position of the calibration plate in the moving path is fixed, the posture displayed to the same type of mobile device is the same, which can ensure that the external parameter calibration results of different mobile devices of the same type are consistent, thereby improving the stability of the external parameter calibration of the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of a person holding a calibration board in the related art provided in an embodiment of the present application;

[0032] Figure 2 It is a structural diagram of the external parameter calibration system architecture provided in the embodiment of the present application;

[0033] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 4A It is a flow chart of the external parameter calibration method provided in the embodiment of the present application;

[0035] Figure 4B It is a schematic diagram of the process of obtaining calibration plate data provided in an embodiment of the present application;

[0036] Figure 5A This is a first schematic diagram of the placement of the calibration plate provided in the embodiment of the present application;

[0037] Figure 5B is a second schematic diagram of the calibration plate placement method provided in the embodiment of the present application;

[0038] Figure 5C is a third schematic diagram of the calibration plate placement method provided in the embodiment of the present application;

[0039] Figure 5D is a fourth schematic diagram of the calibration plate placement method provided in the embodiment of the present application;

[0040] Figure 6 It is a schematic diagram of the conversion between point cloud data and calibration images provided in an embodiment of the present application.

[0041] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0043] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0046] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0049] 1) Distance sensor is a device that can measure distance. It uses various physical principles to sense the distance between an object and the sensor. The working principle of a distance sensor usually involves emitting a certain form of energy (such as ultrasound, laser or infrared light), and then measuring the time difference or phase difference between the energy beam being emitted and reflected from the object being measured. According to different physical principles, distance sensors can be divided into the following categories: ultrasonic sensors, laser sensors, infrared sensors, and radar sensors.

[0050] 2) Image sensors, also known as photosensitive elements, are electronic devices used to capture light images. They are widely used in digital cameras, camcorders, scanners, cameras and industrial automation systems to convert light signals into electrical signals.

[0051] 3) Point cloud data is a data set consisting of a collection of points in three-dimensional space, which are usually obtained through laser scanning, structured light scanning, light field photography or other forms of three-dimensional imaging technology. Each point represents a sampling point on the surface of an object, contains the spatial coordinates (x, y, z) of the point, and sometimes contains other attributes such as color, intensity or reflectivity.

[0052] 4) External parameters. External parameters refer to the position and direction relationship between a sensor (such as a camera, lidar, etc.) and another coordinate system. This relationship describes the placement of the sensor in space and its orientation, which is essential for converting the data from the sensor to the same coordinate system for fusion and processing. External parameters usually consist of two parts: a rotation matrix and a translation vector: a rotation matrix describes the rotation of the sensor coordinate system relative to the global coordinate system, and the rotation matrix can be used to convert a vector from one coordinate system to another coordinate system; a translation vector describes the translation of the sensor coordinate system relative to the global coordinate system, and the translation vector can be used to correct the position deviation between the sensor coordinate system and the global coordinate system. In an embodiment of the present application, external parameters are used for the conversion between the coordinate system of the distance sensor and the coordinate system of the image sensor.

[0053] With the development of automation technology, distance sensors and image sensors have become important sensors in some automation equipment. Distance sensors can detect the distance and speed of objects and collect point cloud data of objects, while image sensors can provide high-resolution image information. Distance sensors and image sensors can help devices have environmental perception functions such as visual systems, navigation systems, and autonomous driving systems. Therefore, external parameter calibration between distance sensors and image sensors is crucial.

[0054] In the related technology, manual calibration is usually relied on, and the calibration plate is rotated by holding it to generate more poses. However, this method is greatly affected by human factors, resulting in low stability of the device external parameter calibration. Figure 1 , Figure 1 A schematic diagram of manual calibration using a hand-held calibration plate is shown. This method can display fewer calibration plate poses to the mobile device and is too dependent on human operation factors. Different mobile devices of the same type use different calibration plate poses for external parameter calibration, which brings great uncertainty and error. In addition, reliance on manual calibration will also have the following problems: time-consuming and labor-intensive, requiring a lot of time and manpower investment, and the calibration process may be very cumbersome; human errors, the operator's skills and experience level may affect the accuracy of the calibration; poor repeatability, and the calibration results of different people may be different; high workload intensity, which may impose a heavy workload on the operator.

[0055] In order to reduce the uncertainty and errors caused by human operation and improve the stability of external parameter calibration, the embodiments of the present application utilize device movement instead of human movement to realize data collection on the calibration plate of the mobile device. The embodiments of the present application provide an external parameter calibration method, apparatus, device, computer-readable storage medium and computer program product, which can improve the stability of automatic calibration of external parameters of mobile devices.

[0056] See also Figure 2 , Figure 2 It is a schematic diagram of the architecture of the external parameter calibration system 100 provided in an embodiment of the present application. To support an external parameter calibration application, the terminal 401 is connected to the server 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0057] Terminal 401 may be a terminal set on a mobile device, or may be a terminal that communicates with the mobile device in real time. The mobile device may be an autonomous driving vehicle, a mobile robot with a navigation system, an augmented reality (AR) and virtual reality (VR) device, a device with a visual system in industrial automation, an unmanned aerial vehicle and aerospace equipment, a monitoring device, a medical imaging device, a smart phone, a smart camera and other devices. The mobile device is provided with a distance sensor and an image sensor, which can collect point cloud data of the calibration plate through the distance sensor and collect calibration images of the calibration plate through the image sensor during movement on the moving path. Terminal 401 can obtain point cloud data collected by the distance sensor of the mobile device and calibration images collected by the image sensor in real time, and send the obtained point cloud data and calibration images to the server 200 through the network 300.

[0058] The terminal 401 is used to send an external parameter calibration request to the server 200 when external parameter calibration is required.

[0059] The server 200 is used to, when receiving an external parameter calibration request, respond to the external parameter calibration request, determine the moving path of the mobile device and N groups of calibration plates located on the moving path; when the mobile device moves on the moving path, obtain the point cloud data collected by the distance sensor for the N groups of calibration plates, and the calibration images collected by the image sensor for the N groups of calibration plates; based on the point cloud data and the calibration image, calibrate the external parameters between the distance sensor and the image sensor to obtain N initial external parameters; based on the N initial external parameters, calibrate the external parameters between the distance sensor and the image sensor to obtain target external parameters, and send the external parameter calibration result (i.e., the target external parameter) to the terminal 401.

[0060] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 401 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0061] In some embodiments, terminal 401 can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, a car terminal, etc., and can also be implemented as a server.

[0062] It should be noted that the external parameter calibration method implemented by the server 200 in the external parameter calibration system 100 can also be implemented on the terminal 401 .

[0063] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present application, Figure 3 The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 3 Various buses are labeled as bus system 440 .

[0064] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0066] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0067] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0069] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0070] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0071] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0072] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0073] In some embodiments, the external parameter calibration device provided in the embodiments of the present application can be implemented in a software manner. Figure 3 The external parameter calibration device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, including the following software modules: a determination module 4551, an acquisition module 4552, a calibration module 4553, and a calibration module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0074] In other embodiments, the external parameter calibration device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the external parameter calibration method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0075] The external parameter calibration method provided in the embodiment of the present application will be described below in conjunction with the accompanying drawings. As mentioned above, the electronic device implementing the external parameter calibration method in the embodiment of the present application can be a terminal 401, a server 200, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0076] The external parameter calibration method of the embodiment of the present application is described by taking the execution subject as the server 200 as an example. Figure 4A , Figure 4A is a flow chart of the external parameter calibration method provided in the embodiment of the present application, which will be combined with Figure 4A The steps shown are explained.

[0077] In step 101, a moving path of a mobile device and N groups of calibration plates located on the moving path are determined.

[0078] Among them, N is a positive integer, and different groups of calibration plates correspond to different postures.

[0079] In the embodiments of the present application, the mobile device may be an autonomous vehicle, a mobile robot with a navigation system, an augmented reality (AR) and virtual reality (VR) device, a device with a visual system in industrial automation, a drone and aerospace equipment, a monitoring device, a medical imaging device, a smart phone, and other devices. For devices that can move on their own, such as vehicles, drones, and aerospace equipment, the mobile device may be these devices themselves; for devices that cannot move on their own, such as medical imaging equipment and smart phones, these devices may be placed on a movable base to form a mobile device, which moves on a moving path through the mobile base, so that a mobile phone or other device can collect data from the calibration plate. The following description will be made using a vehicle as an example of a mobile device.

[0080] In the embodiment of the present application, one or more moving paths are pre-set, and N groups of calibration plates are set on the moving paths, so that when the mobile device moves on the moving paths, data of the calibration plates is collected through distance sensors and image sensors.

[0081] Among them, the distance sensor can be a laser radar sensor, or a pre-calibrated structured light camera, stereo vision camera and other different types of sensors. As long as the sensor can obtain the point cloud data of the calibration plate, it can be used as the distance sensor of this application and is applicable to this solution. The image sensor can be a camera, a video camera and other sensors. As long as the calibration image of the calibration plate can be obtained, it can be used as the image sensor of this application and is applicable to this solution.

[0082] Since different mobile devices have image sensors and distance sensors installed at different locations, the embodiment of the present application may set N groups of calibration plates on one side or both sides of the moving path.

[0083] In some embodiments, N groups of calibration plates are arranged on the same side of the moving path based on a specified order, wherein each group of calibration plates includes one calibration plate.

[0084] In this case, the side of the moving path where the N sets of calibration plates are arranged can be determined based on the installation positions of the distance sensor and the image sensor on the mobile device to be calibrated with external parameters, and the orientation of the two sensors when the mobile device moves on the moving path. Here, a set of calibration plates includes only one calibration plate, and multiple calibration plates are arranged on the same side of the moving path. Different calibration plates have different postures. This arrangement can reduce the calibration cost.

[0085] For example, see Figure 5A , multiple calibration plates can be arranged in a specified order along the left side of the vehicle's forward direction on the moving path, and the number of calibration plates can be determined according to actual conditions. Figure 5A Six calibration plates are displayed, and by rotating different calibration plates at different angles, six postures of the calibration plates are provided for the vehicle.

[0086] In some embodiments, N groups of calibration plates are arranged on both sides of the moving path based on a specified order, wherein each group of calibration plates includes two calibration plates with the same posture, and the two calibration plates with the same posture are arranged on both sides of the moving path respectively.

[0087] In order to accommodate sensors in different orientations, calibration plates can also be placed directly on both sides of the moving path. Figure 5BEach group of calibration plates includes two calibration plates, which are arranged on both sides of the moving path respectively, and the two calibration plates in the same group of calibration plates have the same posture. This arrangement method can ensure that no matter where the sensor is facing on the mobile device and where the mobile device is moving on the moving path, the relevant data of the calibration plates under different set postures can be collected.

[0088] In some embodiments, the size of the calibration plate is not less than a size threshold; the distance between the center of the calibration plate and the center line of the moving path does not exceed a first distance threshold; the distance between two adjacent calibration plates on the same side of the moving path is greater than a second distance threshold.

[0089] In an embodiment of the present application, in order to enable the distance sensor and image sensor on the mobile device to clearly capture the calibration plate, a size threshold can be pre-set for the calibration plate so that the size of the calibration plate is not less than the size threshold, and a first distance threshold is set so that the distance between the center of the calibration plate and the center line of the moving path does not exceed the first distance threshold. The size threshold and the first distance threshold here can be set according to different types of mobile devices without specific limitation.

[0090] In order to reduce the situation where the calibration image taken by the image sensor includes multiple calibration plate images due to the distance between adjacent calibration plates being too close, or the situation where a frame of point cloud data collected by the distance sensor includes data points of multiple calibration plates, the embodiment of the present application also sets a second distance threshold for the distance between two adjacent calibration plates located on the same side of the moving path, so that the distance between the two adjacent calibration plates located on the same side of the moving path is greater than the second distance threshold.

[0091] In some embodiments, the placement of the calibration plate includes at least one of the following: flat placement, three-dimensional placement, hanging placement, and dynamic placement.

[0092] Among them, the tiling placement is used to indicate that the calibration plate is statically laid out on the moving path. By tiling placement, the distance sensor and the image sensor can capture the calibration plate from multiple angles, and the calibration plate in multiple postures can be obtained.

[0093] Stereoscopic placement is used to indicate that different groups of calibration plates have different heights or angles. Through stereoscopic placement, various situations encountered by mobile devices in actual applications can be simulated. For example, a vehicle may encounter objects at different heights or angles during driving. This placement method helps to simulate data collection for calibration plates in a real environment, thereby improving the external parameter calibration effect.

[0094] Dynamic placement is used to instruct the calibration plate to rotate based on a preset speed. The preset speed is associated with the moving speed of the mobile device on the moving path, and the preset speed and the moving speed of the mobile device can affect each other. For example, if the moving speed of the mobile device is relatively fast, in order to clearly capture the calibration image or point cloud data of the rotating calibration plate, the calibration plate can be rotated at a relatively low speed. This placement method can collect data of the calibration plate in different motion states and improve the external parameter calibration effect.

[0095] In actual implementation, different groups of calibration plates may be placed in different ways, but the same group of calibration plates may be placed in the same way; or, N groups of calibration plates may be placed in the same way; or the placement methods may be randomly combined according to actual conditions.

[0096] As an example, take N sets of calibration plates set on the same side of the moving path as an example, see Figure 5C , calibration plates 501 and 502 can be laid flat, and the laying postures of calibration plates 501 and 502 are different; calibration plate 503 can be dynamically laid, so that calibration plate 503 rotates at a constant speed according to a preset speed; calibration plates 504, 505, and 506 can be laid in three dimensions, and the three have different heights or angles. By combining multiple laying methods, the distance sensor and image sensor of the mobile device can capture the data of the calibration plate in more postures, thereby greatly improving the accuracy of external parameter calibration.

[0097] In some embodiments, when N groups of calibration plates are placed in the same manner, the distances between the centers of calibration plates in different groups and the center line of the moving path are different.

[0098] In actual implementation, if N groups of calibration plates are placed in the same way, the distances between the centers of different groups of calibration plates and the center line of the moving path can be different, but the distances between the centers of the calibration plates and the center line of the moving path do not exceed the first distance threshold. In this way, objects at different distances in actual applications of mobile devices can be simulated, thereby capturing data of calibration plates at different distances and improving the external parameter calibration effect.

[0099] For example, see Figure 5D The center of calibration plate 601 to calibration plate 603 is getting farther and farther away from the center line of the moving path, while the center of calibration plate 603 to calibration plate 606 is getting closer and closer to the center line of the moving path, and different calibration plates are placed in different postures.

[0100] Continue reading Figure 4A , and the description will continue with the above step 101.

[0101] In step 102, while the mobile device is moving on the moving path, point cloud data collected by the distance sensor for the N groups of calibration plates and calibration images collected by the image sensor for the N groups of calibration plates are obtained.

[0102] In some embodiments, see Figure 4B , Figure 4A The step 102 shown can be implemented by following the steps 1021 to 1024 .

[0103] In step 1021, the position correspondence between different calibration plates and different sections in the moving path is obtained.

[0104] In actual implementation, since the positions of the N groups of calibration plates arranged on the moving path are fixed, the position correspondence can be set according to the positions of the different calibration plates arranged on the moving path and the corresponding sections in the moving path.

[0105] For example, if the moving path is 100 meters, 5 calibration plates are set on the same side of the moving path, the spacing between each calibration plate is 10 meters, and the width of each calibration plate is 2 meters. Then, according to the forward direction of the mobile device in the moving path, the 0-17 meter section in the moving path is used as the section corresponding to the first calibration plate, and the 17-39 meter section is used as the section corresponding to the second calibration plate, so as to set the corresponding position relationship between the two.

[0106] By setting the position correspondence, the identity of the calibration plate captured by the mobile device can be determined according to the position of the mobile device in the moving path.

[0107] In step 1022, the target section corresponding to each set of calibration plates in the moving path is determined according to the position correspondence.

[0108] According to the moving path adopted by the mobile device and the corresponding position relationship of the calibration plate on the moving path, the target section corresponding to each set of calibration plates on the moving path can be determined. For example, for the second set of calibration plates, if the corresponding target section is a section of 17-39 meters, when the mobile device moves in the section of 17-39 meters, data is collected for the second set of calibration plates.

[0109] In step 1023, when the mobile device moves on the target road section, target point cloud data collected by the distance sensor for the calibration plate of the corresponding group and target calibration images collected by the image sensor for the calibration plate of the corresponding group are obtained.

[0110] When the mobile device moves on the target road section corresponding to each group of calibration plates, the distance sensor can be used to collect multiple frames of point cloud data for the grouped calibration plates corresponding to the target road section, and the multiple frames of point cloud data collected by the distance sensor constitute the target point cloud data of the grouped calibration plates; the image sensor can be used to collect multiple frames of calibration images for the grouped calibration plates corresponding to the target road section, and the multiple frames of calibration images collected by the image sensor constitute the character calibration images of the grouped calibration plates.

[0111] In step 1024, the target point cloud data corresponding to each group of calibration plates obtained are combined to obtain the point cloud data collected by the distance sensor for the N groups of calibration plates, and the target calibration images corresponding to each group of calibration plates obtained are combined to obtain the calibration images collected by the image sensor for the N groups of calibration plates.

[0112] The target point cloud data collected by the distance sensor for each set of calibration plates are combined to obtain N sets of target point cloud data, each set of target point cloud data includes multiple frames of point cloud data, and the N sets of target point cloud data are point cloud data collected by the distance sensor for the N sets of calibration plates. The target calibration images collected by the image sensor for each set of calibration plates are combined to obtain N sets of target calibration images, each set of target calibration images includes multiple frames of calibration images, and the N sets of target calibration images are calibration images collected by the image sensor for the N sets of calibration plates.

[0113] Continue reading Figure 4A , and the description will continue with the above step 102.

[0114] In step 103, based on the point cloud data and the calibration image, the extrinsic parameters between the distance sensor and the image sensor are calibrated to obtain N initial extrinsic parameters.

[0115] In some embodiments, Figure 4A Step 103 in the above method can be implemented in the following manner.

[0116] According to N groups of calibration plates, the point cloud data and calibration images are divided to obtain target point cloud data corresponding to each group of calibration plates and target calibration images corresponding to the target point cloud data; based on the target point cloud data and target calibration images corresponding to each group of calibration plates, the extrinsic parameters between the distance sensor and the image sensor are calibrated to obtain N initial extrinsic parameters.

[0117] In actual implementation, in order to ensure the accuracy of external parameter calibration, it is necessary to use time-correlated point cloud data and calibration images to calibrate the external parameters of the image sensor and distance sensor. Through the process of obtaining point cloud data and calibration images in the above embodiment, it can be seen that the target point cloud data and target calibration images collected by the mobile device on the target road section for the corresponding grouped calibration plates are time-correlated. Therefore, the embodiment of the present application can divide the collected point cloud data and calibration images based on N groups of calibration plates.

[0118] As an example, for the first group of calibration plates, the target point cloud data corresponding to the first group of calibration plates can be determined in the point cloud data as the target point cloud data of the first group of calibration plates, and the target calibration images corresponding to the first group of calibration plates can be determined in the calibration images as the target calibration images of the first group of calibration plates. The target point cloud data and the target calibration images are derived from the same road section and the same time period, and have a time series correspondence.

[0119] After determining the target point cloud data and target calibration image corresponding to each set of calibration plates, the target point cloud data and target calibration image corresponding to each set of calibration plates are used to calibrate the external parameters between the distance sensor and the image sensor, and N initial external parameters are obtained, and one set of calibration plates corresponds to one initial external parameter. Since the target point cloud data corresponding to each set of calibration plates may include multiple frames of point cloud data, and the target calibration image includes multiple frames of calibration images, the initial external parameter here can be an external parameter set, including multiple reference external parameters, and each reference external parameter is determined by a frame of point cloud data and a frame of calibration image corresponding to the frame of point cloud data.

[0120] In some embodiments, the above “calibrating the external parameters between the distance sensor and the image sensor based on the target point cloud data and the target calibration image corresponding to each group of calibration plates to obtain N initial external parameters” can be achieved in the following way.

[0121] The following processing is performed for each set of calibration plates: feature extraction is performed on the target point cloud data and the target calibration image to obtain point cloud features and image features; coordinate mapping is performed on the point cloud features and image features to obtain the initial external parameters corresponding to the calibration plate.

[0122] In actual implementation, coordinate mapping can be performed between the key points of the positions corresponding to the target point cloud data and the target calibration image to obtain the initial external parameters for converting the key points from the coordinate system of the distance sensor corresponding to the target point cloud data to the coordinate system of the image sensor corresponding to the target calibration image.

[0123] As an example, for each set of calibration plates, feature extraction is performed on the target point cloud data and the target calibration image, and key point extraction can be performed on the target point cloud data and the target calibration image, and the obtained point cloud features are the point cloud key points in the point cloud data, and the obtained image features are the image key points in the calibration image. The positions of the point cloud key points and the image key points on the calibration plate image correspond to each other. The difference is that the positions of the point cloud key points are the coordinate positions in the distance sensor coordinate system, while the positions of the image key points are the coordinate positions in the image sensor coordinate system. By performing coordinate mapping of the point cloud key points and image key points at corresponding positions, the initial external parameters corresponding to the calibration plate can be obtained. In the above manner, the initial external parameters corresponding to each set of calibration plates can be obtained.

[0124] In some embodiments, before the above-mentioned feature extraction is performed on the target point cloud data and the target calibration image, the target calibration image may be cropped, and feature extraction is performed based on the cropped target calibration image, which may be specifically achieved in the following manner.

[0125] Obtain the arrangement sequence number of the target calibration plate corresponding to the target calibration image on the moving path; determine the deviation data of the target calibration plate in the target calibration image based on the arrangement sequence number; determine the cropping range for the target calibration image based on the deviation data; crop the target calibration image based on the cropping range, and the cropped target calibration image is used for feature extraction.

[0126] In an embodiment of the present application, since different groups of calibration plates are set at different positions, there may be a certain shooting deviation error between the mobile device and different calibration plates during the movement of the mobile path. Therefore, before extracting features from the target calibration image, the embodiment of the present application can also determine a deviation data based on the arrangement sequence number of the target calibration plate corresponding to the target calibration image on the moving path, and crop the target calibration data through the deviation data, so that the cropped target calibration image can exclude other interference items while retaining the target calibration plate.

[0127] As an example, if the original size of the target calibration image is (h, w), where h represents the height of the target calibration image, w represents the width of the target calibration image, and f represents the arrangement number. The deviation coefficient can be determined based on the empirical data of the image sensor. Assuming that the deviation coefficient is 1 / 4, the deviation data can be Since the arrangement position of the calibration plate has little effect on the height position of the calibration plate in the calibration image, the cropping range in width can be determined using only the deviation data. In one example, the cropping range in height of the target calibration image can be set to to The cropping range of the target calibration image in width can be set to to The target calibration image is cropped according to the determined cropping range, so as to perform feature extraction through the cropped target calibration image.

[0128] In some embodiments, Figure 4A Step 103 in the above method can also be implemented in the following manner.

[0129] For the target point cloud data and target calibration images corresponding to each set of calibration plates, the following processing is performed: based on the timing information of the target point cloud data and the target calibration images, M pairs of point cloud image data are determined, where M is a positive integer; based on each pair of point cloud image data, the extrinsic parameters of the distance sensor and the image sensor are calibrated to obtain M reference extrinsic parameters corresponding to each set of calibration plates, and the M reference extrinsic parameters constitute the initial extrinsic parameters corresponding to each set of calibration plates.

[0130] As an example, if it is assumed that when a mobile device moves on a target road section, multiple frames of point cloud data are collected from the calibration plate corresponding to the target road section through a distance sensor, and multiple frames of calibration images are collected from the calibration plate corresponding to the target road section through an image sensor, then when determining the initial external parameters corresponding to each group of calibration plates, it is necessary to determine the reference external parameters respectively based on a frame of point cloud data collected for the group of calibration plates and a corresponding frame of calibration images, so as to obtain the initial external parameters corresponding to the group of calibration plates.

[0131] As an example, for each set of target point cloud data and target calibration images corresponding to the calibration plate, M pairs of point cloud image data are determined according to their corresponding timing information. It should be noted that the target point cloud data and the target calibration image may include more than M elements, and here M elements can be selected according to the actual calculation accuracy. Each pair of point cloud image data includes a frame of target point cloud data and a frame of target calibration image corresponding to the timing. The timing correspondence can be understood as a frame of target point cloud data and a frame of target calibration image in the pair of point cloud image data being collected at the same time, or at a similar time.

[0132] For example, see Figure 6 , Figure 6 The point cloud 701 in represents a frame of point cloud data, and the image 702 represents a frame of calibration image corresponding to the time sequence of the point cloud 701. By extracting features from the point cloud 701, point cloud key points (such as point cloud key points 7011 and point cloud key points 7012) are obtained, and by extracting features from the image 702, image key points (such as image key points 7021 and image key points 7022) are obtained. The corresponding relationship between the key points is that the position of the point cloud key point 7011 corresponds to the position of the image key point 7021 on the calibration board, and the position of the point cloud key point 7012 corresponds to the position of the image key point 7021 on the calibration board. By referring to the reference rotation matrix R and the reference translation vector T of the external parameters, the point cloud key point 7011 can be transformed from the coordinate system of the distance sensor to the position of the image key point 7021 in the coordinate system of the image sensor. Therefore, when the coordinate positions of the point cloud key point 7011 and the image key point 7021 are determined, the reference external parameters for the transformation of the two can be calculated.

[0133] As an example, for each pair of point cloud image data, the image key points in the target calibration image can be extracted and the coordinate positions of the image key points can be determined in the following way: the key corner points (i.e., image key points) in the target calibration image are identified by a corner point recognition algorithm, and the pixel coordinates of the key corner points in the pixel coordinate system of the target calibration image are obtained, wherein the corner point recognition algorithm can adopt the findChessboardCorners(InputArray image, SizepatternSize, OutputArray corners) function in the opencv library, and the three parameters of the function are the target calibration image (image), the size of the target calibration image (patternSize), and the key corner point array (corners).

[0134] Then, the coordinates of the key corner points in the target calibration image in the coordinate system of the image sensor are obtained through the first conversion relationship, that is, the coordinate positions of the key points of the image in the coordinate system of the image sensor are obtained.

[0135] Among them, the first conversion relationship is:

[0136]

[0137] Where (u, v) represents the pixel coordinates in the pixel coordinate system, (X c , Y c , Z c ) represents the coordinates in the coordinate system of the image sensor; M represents the intrinsic parameters of the image sensor (such as the intrinsic parameters of the camera).

[0138] As an example, for each pair of point cloud image data, the coordinate position of the point cloud key point in the target point cloud data in the coordinate system of the distance sensor can be determined in the following way: plane fitting is performed on the target point cloud data to obtain the point cloud plane corresponding to the target point cloud data. The expression of the obtained plane is:

[0139] Ax1+By1+Cz1+D=0;

[0140] Among them, A, B, C, and D are the parameters of plane fitting, (A, B, C) is the normal vector of the plane, D is the offset of the plane, and (x1, y1, z1) represents the coordinates in the point cloud plane.

[0141] In actual implementation, before plane fitting is performed on the target point cloud data, the target point cloud data can also be filtered out, including the point cloud at the location of the calibration plate. The point cloud retention position is: height (0, +∞), width is a specified length (such as 1.5 meters), and length is (a-0.05*f, b-0.05*f); among them, a and b are constants set according to actual conditions.

[0142] As an example, for each pair of point cloud image data, after determining the coordinates of the point cloud key points in the target point cloud data and the coordinates of the image key points in the target calibration image in the above manner, the following formula can be used to perform external parameter calibration to obtain the reference external parameters of each pair of point cloud image data:

[0143] [X c Y c Z c ] T =[R t t t ][x1 y1 z1] T ;

[0144] Among them, (X c , Y c , Z c ) represents the coordinates in the image sensor’s coordinate system, (x1, y1, z1) represents the coordinates in the point cloud plane, and R t Represents the 3*3 reference rotation matrix, t t Represents the 3D reference translation vector.

[0145] Based on each pair of point cloud image data in the M pairs of point cloud image data, the extrinsic parameters of the distance sensor and the image sensor are calibrated using the above method to obtain M reference extrinsic parameters corresponding to each set of calibration plates, each reference extrinsic parameter includes a reference rotation matrix and a reference translation vector, and the M reference extrinsic parameters constitute the initial extrinsic parameters corresponding to each set of calibration plates.

[0146] Continue reading Figure 4A , and continue with the above step 103 for explanation.

[0147] In step 104, based on the N initial extrinsic parameters, the extrinsic parameters between the distance sensor and the image sensor are calibrated to obtain target extrinsic parameters.

[0148] In some embodiments, Figure 4A The illustrated step 104 may be implemented in the following manner.

[0149] Obtain the dot product error and calibration error of each initial extrinsic parameter; select the initial extrinsic parameters whose first angle offset is greater than the first angle threshold and whose second angle offset is less than the second angle threshold from the N initial extrinsic parameters, and determine the initial rotation matrix included in the selected initial extrinsic parameters as the target rotation matrix, wherein the first angle threshold is greater than the second angle threshold; perform key point extraction on the target point cloud data and the target calibration image to obtain n pairs of first key points; calibrate the extrinsic parameters between the distance sensor and the image sensor based on the target rotation matrix and the n pairs of first key points to obtain the target extrinsic parameters.

[0150] Wherein, n is a positive integer, each pair of first key points includes a point cloud key point and an image key point; the dot product error is used to indicate the first angle offset between the normal vector of the corresponding target point cloud data and the plane of the target calibration image, and the calibration error is used to indicate the second angle offset between the normal vector of the target point cloud data and the normal vector of the target calibration image.

[0151] As an example, the dot product error e can be determined by the following formula d and calibration error e r :

[0152]

[0153] Where N represents different postures of the calibration plate, and one posture corresponds to a reference external parameter in the initial external parameters; Represents the coordinates of the two image key points in the target calibration image corresponding to the calibration plate at the i-position; the normal vector of the point cloud plane P1 of the target point cloud data corresponding to the calibration plate at the i-position The reference rotation matrix R t After transformation, we can get Right now

[0154] It can be seen that the dot product error e d It is used to indicate the first angle offset between the normal vector of the target point cloud data corresponding to the calibration plate corresponding to the initial external parameter and the plane of the target calibration image, and the dot product error e d Ideally, it should be equal to 0, and the first angle offset is ideally 90°. Calibration error e r It is used to indicate the second angle offset between the normal vector of the target point cloud data and the normal vector of the target calibration image, the calibration error e r Ideally, it should be equal to 1, and the second angle offset should be equal to 0. It can be seen that the dot product error e d The closer it is to 0, the more the calibration error e r The closer it is to 1, the more consistent the directions of the image sensor and the distance sensor are, and the more accurate the reference rotation matrix of the reference extrinsic parameter in the corresponding initial extrinsic parameter is.

[0155] Each initial extrinsic parameter includes M reference extrinsic parameters, each reference extrinsic parameter includes a reference rotation matrix and a reference translation vector, and the M reference rotation matrices constitute an initial rotation matrix of the initial extrinsic parameter.

[0156] Therefore, based on the above formula, for each initial external parameter, the reference rotation matrix of each reference external parameter in the initial external parameter is substituted, and the dot product error and calibration error under each reference rotation matrix are calculated. According to the value of the dot product error and the calibration error, the reference external parameters whose first angle offset is greater than the first angle threshold and whose second angle offset is less than the second angle threshold are screened out from the reference external parameters included in the N initial external parameters, and the reference rotation matrix included in the screened reference external parameters is determined as the target rotation matrix. Here, the first angle threshold is greater than the second angle threshold, the first angle threshold is close to 90 degrees, and the second angle threshold is close to 0 degrees.

[0157] Then, the target point cloud data and the target calibration image are subjected to key point extraction to obtain n pairs of first key points. The image key points and the point cloud key points in each pair of first key points correspond to each other on the corresponding calibration plate. The key point extraction can refer to the above feature extraction process, which will not be repeated here. The target translation vector is determined based on the n pairs of first key points and the N initial external parameters. The target translation vector and the target rotation matrix constitute the target external parameters.

[0158] In some embodiments, the above step of "calibrating the extrinsic parameters between the distance sensor and the image sensor based on the target rotation matrix and n pairs of first key points to obtain the target extrinsic parameters" can be implemented in the following manner.

[0159] Based on the initial extrinsic parameters, the point cloud key points in the n pairs of first key points are projected into the calibration image to obtain n point cloud projection points, which are combined with the n image key points in the n pairs of first key points to form n pairs of second key points, each pair of second key points includes a point cloud projection point and an image key point; the first distance between the point cloud projection point and the image key point in each pair of second key points is obtained, and the n first distances are averaged to obtain the average distance; from the n pairs of second key points, m pairs of second key points whose first distances do not exceed the average distance are screened out, where m is a positive integer not exceeding n; based on the target rotation matrix and the m pairs of second key points, the target translation vector is screened out from the initial translation vectors included in the N initial extrinsic parameters; the target rotation matrix and the target translation vector are combined to obtain the target extrinsic parameters.

[0160] In the embodiment of the present application, the average distance may be the average Euclidean distance between the point cloud projection point and the corresponding image key point, and the average Euclidean distance may be calculated by the following formula:

[0161]

[0162] Where N represents different postures of the calibration plate, and one posture corresponds to a reference external parameter in the initial external parameters; is the point cloud projection point of the jth image key point on the calibration image corresponding to the i-th pose and the jth point cloud key point on the i-th pose Indicates the first distance.

[0163] is the jth point cloud key point on the point cloud plane corresponding to the i-th pose, and The variance of the Euclidean distance is:

[0164]

[0165] Determine the variance v among the M initial external parameters t The reference extrinsic parameter with the smallest value is used to recalculate the average Euclidean distance between the point cloud projection point of the point cloud key point and the image key point. The first key point whose Euclidean distance is greater than the average Euclidean distance between the image key point and the point cloud key point is removed. The remaining point cloud key point set is denoted as O1, and the remaining image key point set is denoted as O c , that is, screen out m pairs of second key points whose first distance does not exceed the average distance.

[0166] Each initial extrinsic parameter includes M reference extrinsic parameters, each reference extrinsic parameter includes a reference rotation matrix and a reference translation vector, and the M reference translation vectors constitute an initial translation vector of the initial extrinsic parameter.

[0167] Based on the target rotation matrix determined above and the m pairs of second key points, the target translation vector is selected from the initial translation vectors included in the N initial external parameters, which can be achieved by the following formula:

[0168]

[0169] The mean(·) function represents the row-wise average. represents the target translation vector, is the target rotation matrix determined above; the point cloud key point set O1 includes m point cloud key points in m pairs of second key points; the image key point set O c The m image key points in the m pairs of second key points are included; the m point cloud key points and the m image key points constitute the m pairs of second key points, and the positions of the m image key points and the m point cloud key points on the calibration plate are corresponding.

[0170] The target rotation matrix and the target translation vector are combined to obtain the target extrinsic parameters.

[0171] By applying the above embodiment, N groups of calibration plates are set on the moving path, and data is collected for the N groups of calibration plates set by the mobile device during the movement of the mobile path, that is, point cloud data is collected for the N groups of calibration plates by the distance sensor, and calibration images are collected for the N groups of calibration plates by the image sensor. Then, the external parameters between the distance sensor and the image sensor are automatically calibrated using the collected point cloud data and calibration images to obtain N initial external parameters, and then the external parameters are calibrated based on the N initial external parameters to obtain the target external parameters. Thus, it is possible to replace manual movement with autonomous movement of the mobile device, and the entire process does not require human participation, reducing the uncertainty caused by human operation, and because the position of the calibration plate in the moving path is fixed, the posture displayed to the same type of mobile device is the same, which can ensure that the external parameter calibration results of different mobile devices of the same type are consistent, thereby improving the stability of the external parameter calibration of the mobile device. In addition, by applying the above external parameter calibration and external parameter calibration process, the target external parameter can be made more accurate to improve the calibration accuracy between the distance sensor and the image sensor.

[0172] The following further describes the exemplary structure of the external parameter calibration device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 3 As shown, the software modules stored in the external parameter calibration device 455 of the memory 440 may include:

[0173] The determination module 4551 is used to determine the moving path of the mobile device and N groups of calibration plates located on the moving path.

[0174] Among them, N is a positive integer, and different groups of calibration plates correspond to different postures.

[0175] The acquisition module 4552 is used to acquire point cloud data collected by the distance sensor for N groups of calibration plates and calibration images collected by the image sensor for N groups of calibration plates when the mobile device moves on the moving path.

[0176] The calibration module 4553 is used to calibrate the external parameters between the distance sensor and the image sensor based on the point cloud data and the calibration image to obtain N initial external parameters.

[0177] The calibration module 4554 is used to calibrate the extrinsic parameters between the distance sensor and the image sensor based on the N initial extrinsic parameters to obtain the target extrinsic parameters.

[0178] In some embodiments, N groups of calibration plates are arranged on both sides of the moving path based on a specified order; wherein each group of calibration plates includes two calibration plates with the same posture and the two calibration plates with the same posture are arranged on both sides of the moving path respectively.

[0179] In some embodiments, N groups of calibration plates are arranged on the same side of the moving path based on a specified order; wherein each group of calibration plates includes one calibration plate.

[0180] In some embodiments, the size of the calibration plate is not less than a size threshold; the distance between the center of the calibration plate and the center line of the moving path does not exceed a first distance threshold; the distance between two adjacent calibration plates on the same side of the moving path is greater than a second distance threshold.

[0181] In some embodiments, the placement of the calibration plate includes at least one of the following: flat placement, three-dimensional placement, hanging placement, and dynamic placement. Among them, flat placement is used to indicate that the calibration plate is statically laid flat on the moving path; three-dimensional placement is used to indicate that different groups of calibration plates have different heights or angles; dynamic placement is used to indicate that the calibration plate rotates based on a preset speed, and the preset speed is associated with the moving speed of the mobile device on the moving path.

[0182] In some embodiments, when N groups of calibration plates are placed in the same manner, the distances between the centers of calibration plates in different groups and the center line of the moving path are different.

[0183] In some embodiments, the acquisition module 4552 is also used to obtain the position correspondence between different calibration plates and different sections in the moving path; determine the target section in the moving path corresponding to each group of calibration plates based on the position correspondence; when the mobile device moves on the target section, obtain the target point cloud data collected by the distance sensor for the corresponding group of calibration plates, and the target calibration images collected by the image sensor for the corresponding group of calibration plates; combine the target point cloud data corresponding to each group of calibration plates obtained to obtain the point cloud data collected by the distance sensor for N groups of calibration plates, and combine the target calibration images corresponding to each group of calibration plates obtained to obtain the calibration images collected by the image sensor for N groups of calibration plates.

[0184] In some embodiments, the calibration module 4553 is also used to divide the point cloud data and the calibration image according to N groups of calibration plates to obtain target point cloud data corresponding to each group of calibration plates and a target calibration image corresponding to the target point cloud data; based on the target point cloud data and the target calibration image corresponding to each group of calibration plates, the external parameters between the distance sensor and the image sensor are calibrated to obtain N initial external parameters.

[0185] In some embodiments, the calibration module 4553 is also used to perform the following processing for each set of calibration plates: extract features from the target point cloud data and the target calibration image to obtain point cloud features and image features; perform coordinate mapping on the point cloud features and image features to obtain the initial external parameters corresponding to the calibration plate.

[0186] In some embodiments, the calibration module 4553 is also used to obtain the arrangement sequence number of the target calibration plate corresponding to the target calibration image on the moving path; based on the arrangement sequence number, determine the deviation data of the target calibration plate in the target calibration image; based on the deviation data, determine the cropping range for the target calibration image; based on the cropping range, crop the target calibration image, and the cropped target calibration image is used for feature extraction.

[0187] In some embodiments, the calibration module 4553 is also used to perform the following processing for the target point cloud data and the target calibration image corresponding to each group of calibration plates: based on the timing information of the target point cloud data and the target calibration image, determine M pairs of point cloud image data, where M is a positive integer, and each pair of point cloud image data includes a frame of target point cloud data and a frame of target calibration image corresponding to the timing; based on each pair of point cloud image data, calibrate the external parameters of the distance sensor and the image sensor to obtain M reference external parameters corresponding to each group of calibration plates, and the M reference external parameters constitute the initial external parameters corresponding to each group of calibration plates.

[0188] In some embodiments, the calibration module 4554 is also used to obtain the dot product error and calibration error of each initial extrinsic parameter, wherein the dot product error is used to indicate the first angle offset between the normal vector of the corresponding target point cloud data and the plane of the target calibration image, and the calibration error is used to indicate the second angle offset between the normal vector of the target point cloud data and the normal vector of the target calibration image; from the N initial extrinsic parameters, the initial extrinsic parameters whose first angle offset is greater than the first angle threshold and whose second angle offset is less than the second angle threshold are screened out, and the initial rotation matrix included in the screened initial extrinsic parameters is determined as the target rotation matrix, and the first angle threshold is greater than the second angle threshold; key points are extracted from the target point cloud data and the target calibration image to obtain n pairs of first key points, wherein n is a positive integer, and each pair of first key points includes a point cloud key point and an image key point; based on the target rotation matrix and the n pairs of first key points, the extrinsic parameters between the distance sensor and the image sensor are calibrated to obtain the target extrinsic parameters.

[0189] In some embodiments, the calibration module 4554 is also used to project the point cloud key points in the n pairs of first key points into the calibration image based on the initial external parameters to obtain n point cloud projection points, so as to combine the n image key points in the n pairs of first key points to form n pairs of second key points, each pair of second key points including a point cloud projection point and an image key point; obtain the first distance between the point cloud projection point and the image key point in each pair of second key points, and average the n first distances to obtain the average distance; from the n pairs of second key points, select m pairs of second key points whose first distance does not exceed the average distance, where m is a positive integer not exceeding n; based on the target rotation matrix and the m pairs of second key points, select the target translation vector from the initial translation vectors included in the N initial external parameters; combine the target rotation matrix and the target translation vector to obtain the target external parameters.

[0190] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device 400 reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device 400 performs the external parameter calibration method described above in the embodiment of the present application.

[0191] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the external parameter calibration method provided in the present application embodiment, for example, Figure 4A The external parameter calibration method is shown.

[0192] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0193] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0194] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0195] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0196] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for calibrating an external parameter, characterized in that: The method comprises: Determine a moving path of the mobile device and N groups of calibration plates located on the moving path; wherein N is a positive integer, and different groups of calibration plates correspond to different postures; In the process of the mobile device moving on the moving path, acquiring point cloud data collected by a distance sensor for the N groups of calibration plates, and calibration images collected by an image sensor for the N groups of calibration plates; Based on the point cloud data and the calibration image, calibrate the extrinsic parameters between the distance sensor and the image sensor to obtain N initial extrinsic parameters; Based on the N initial extrinsic parameters, the extrinsic parameters between the distance sensor and the image sensor are calibrated to obtain target extrinsic parameters.

2. The method according to claim 1, characterized in that: The N groups of calibration plates are arranged on both sides of the moving path based on a specified order; Wherein, each group of calibration plates includes two calibration plates with the same posture, and the two calibration plates with the same posture are respectively arranged on both sides of the moving path.

3. The method according to claim 1, characterized in that The N groups of calibration plates are arranged on the same side of the moving path based on a specified order; Wherein, each set of calibration plates includes one calibration plate.

4. The method according to claim 2 or 3, characterized in that: The size of the calibration plate is not less than a size threshold; the distance between the center of the calibration plate and the center line of the moving path does not exceed a first distance threshold; The distance between two adjacent calibration plates located on the same side of the moving path is greater than a second distance threshold.

5. The method according to claim 2 or 3, characterized in that: The placement of the calibration plate includes at least one of the following: flat placement, three-dimensional placement, hanging placement, and dynamic placement; Among them, the flat placement is used to indicate that the calibration plate is statically laid flat on the moving path; the three-dimensional placement is used to indicate that different groups of calibration plates have different heights or angles; the dynamic placement is used to indicate that the calibration plate rotates based on a preset speed, and the preset speed is associated with the moving speed of the mobile device on the moving path.

6. The method according to claim 2 or 3, characterized in that: When the N groups of calibration plates are placed in the same manner, the distances between the centers of calibration plates in different groups and the center line of the moving path are different.

7. The method according to any one of claims 1 to 3, characterized in that: The step of acquiring point cloud data collected by the distance sensor for the N groups of calibration plates and calibration images collected by the image sensor for the N groups of calibration plates includes: Obtain the position correspondence between different calibration plates and different sections in the moving path; According to the position correspondence, determining the target section in the moving path corresponding to each group of the calibration plates; When the mobile device moves on the target road section, the target point cloud data collected by the distance sensor for the calibration plate of the corresponding group and the target calibration image collected by the image sensor for the calibration plate of the corresponding group are obtained; The target point cloud data corresponding to each group of the calibration plates obtained are combined to obtain the point cloud data collected by the distance sensor for the N groups of calibration plates, and the target calibration images corresponding to each group of the calibration plates obtained are combined to obtain the calibration images collected by the image sensor for the N groups of calibration plates.

8. The method according to any one of claims 1 to 3, characterized in that: The extrinsic parameters between the distance sensor and the image sensor are calibrated based on the point cloud data and the calibration image to obtain N initial extrinsic parameters, including: According to the N groups of calibration plates, the point cloud data and the calibration images are divided to obtain target point cloud data corresponding to each group of calibration plates and a target calibration image corresponding to the target point cloud data; Based on the target point cloud data and the target calibration image corresponding to each group of the calibration plates, the external parameters between the distance sensor and the image sensor are calibrated to obtain N initial external parameters.

9. The method according to claim 8, characterized in that The extrinsic parameters between the distance sensor and the image sensor are calibrated based on the target point cloud data and the target calibration image corresponding to each group of the calibration plates to obtain N initial extrinsic parameters, including: The following processing is performed for each set of calibration plates: Performing feature extraction on the target point cloud data and the target calibration image to obtain point cloud features and image features; Coordinate mapping is performed on the point cloud features and the image features to obtain initial external parameters corresponding to the calibration plate.

10. The method according to claim 9, characterized in that Before extracting features from the target point cloud data and the target calibration image, the method further includes: Obtaining an arrangement sequence number of the target calibration plate corresponding to the target calibration image on the moving path; Based on the arrangement sequence number, determining deviation data of the target calibration plate in the target calibration image; Determining a cropping range for the target calibration image based on the deviation data; The target calibration image is cropped based on the cropping range, and the cropped target calibration image is used for the feature extraction.

11. The method according to claim 8, characterized in that The extrinsic parameters between the distance sensor and the image sensor are calibrated based on the target point cloud data and the target calibration image corresponding to each group of the calibration plates to obtain N initial extrinsic parameters, including: For each set of the target point cloud data and the target calibration image corresponding to the calibration plate, the following processing is performed: Based on the time sequence information of the target point cloud data and the target calibration image, determine M pairs of point cloud image data, where M is a positive integer, and each pair of the point cloud image data includes a frame of target point cloud data and a frame of target calibration image corresponding to the time sequence; Based on each pair of the point cloud image data, the external parameters of the distance sensor and the image sensor are calibrated to obtain M reference external parameters corresponding to each group of the calibration plates, and the M reference external parameters constitute the initial external parameters corresponding to each group of the calibration plates.

12. The method according to any one of claims 1 to 3, characterized in that: The initial extrinsic parameters include an initial rotation matrix and an initial translation vector. Based on the N initial extrinsic parameters, calibrating the extrinsic parameters between the distance sensor and the image sensor to obtain target extrinsic parameters includes: Acquire a dot product error and a calibration error of each of the initial extrinsic parameters, wherein the dot product error is used to indicate a first angle offset between a normal vector of the corresponding target point cloud data and a plane of a target calibration image, and the calibration error is used to indicate a second angle offset between a normal vector of the target point cloud data and a normal vector of the target calibration image; From the N initial external parameters, select the initial external parameters whose first angle offset is greater than a first angle threshold and whose second angle offset is less than a second angle threshold, and determine the initial rotation matrix included in the selected initial external parameters as the target rotation matrix, wherein the first angle threshold is greater than the second angle threshold; Extract key points from the target point cloud data and the target calibration image to obtain n pairs of first key points, where n is a positive integer and each pair of the first key points includes a point cloud key point and an image key point; Based on the target rotation matrix and the n pairs of first key points, the extrinsic parameters between the distance sensor and the image sensor are calibrated to obtain target extrinsic parameters.

13. The method according to claim 12, characterized in that The step of calibrating the extrinsic parameters between the distance sensor and the image sensor based on the target rotation matrix and the n pairs of first key points to obtain the target extrinsic parameters includes: Based on the initial external parameters, projecting the point cloud key points in the n pairs of first key points into the calibration image to obtain n point cloud projection points, so as to form n pairs of second key points in combination with the n image key points in the n pairs of first key points, each pair of the second key points including a point cloud projection point and an image key point; Obtaining a first distance between a point cloud projection point and an image key point in each pair of the second key points, and averaging n of the first distances to obtain an average distance; Filtering out m pairs of second key points whose first distance does not exceed the average distance from the n pairs of second key points, wherein m is a positive integer not exceeding n; Based on the target rotation matrix and the m pairs of second key points, a target translation vector is selected from the initial translation vectors included in the N initial extrinsic parameters; The target rotation matrix and the target translation vector are combined to obtain target extrinsic parameters.

14. An external parameter calibration device, characterized in that: The device comprises: A determination module, used to determine a moving path of the mobile device and N groups of calibration plates located on the moving path; wherein N is a positive integer, and different groups of calibration plates correspond to different postures; an acquisition module, configured to acquire point cloud data collected by a distance sensor for the N groups of calibration plates and calibration images collected by an image sensor for the N groups of calibration plates during movement of the mobile device on the movement path; A calibration module, configured to calibrate the external parameters between the distance sensor and the image sensor based on the point cloud data and the calibration image to obtain N initial external parameters; The calibration module is used to calibrate the extrinsic parameters between the distance sensor and the image sensor based on the N initial extrinsic parameters to obtain target extrinsic parameters.

15. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the method according to any one of claims 1 to 13 when executing the computer executable instructions stored in the memory.

16. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Citation Information

Patent Citations

  • Vehicle-mounted camera auxiliary calibration method, device and equipment and medium

    CN110276807A

  • Multi-camera-device external parameter calibration equipment and method, storage medium and electronic equipment

    CN113345031A

  • Calibration method and device

    CN117141359A

  • Partial calibration target detection for improved vehicle sensor calibration

    US20210051317A1

  • Method and system for calibrating extrinsic parameters between depth camera and visible light camera

    US20220092819A1

Cited By

  • Warehouse goods data processing method and device

    CN120782841A