A method and apparatus for obtaining a calibration parameter, and an electronic device

Abstract

Embodiments of the present application disclose a method and device for acquiring calibration parameters and electronic equipment. The method comprises: acquiring motion trajectories of multiple targets under a first mode and a second mode; performing coarse matching on the motion trajectories of the targets under the first mode and the second mode to obtain preliminary matching results corresponding to the motion trajectories of the targets; acquiring point pair information of the targets according to the preliminary matching results corresponding to the motion trajectories of the targets and time information; and performing fine matching on the motion trajectories of the targets under the first mode and the second mode according to the point pair information of the targets to obtain final matching results corresponding to the motion trajectories of the targets. Through the present application, the technical problem of poor multi-modal information fusion effect caused by the failure of a multi-modal device to timely and accurately implement registration correction is solved, and the technical effect of improving the real-time performance and accuracy of the multi-modal device to implement registration correction and ensuring the effectiveness of multi-modal information fusion is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a method, apparatus, and electronic device for obtaining calibration parameters. Background Technology

[0002] Multimodal devices are typically calibrated before leaving the factory. This usually involves obtaining information about the same set of data for each modality based on a calibration board, then performing mapping calculations to obtain a mapping matrix, thereby achieving registration between modalities. However, in practice, transportation and installation processes often cause physical deviations. For example, even though multimodal devices are generally calibrated at the factory, slight deviations in the positions of each modality may occur during installation due to various reasons, leading to registration errors in the detection of some distant objects. Furthermore, as the device operates over time, larger modal deviations can occur. For instance, during use, the registration accuracy of multimodal devices may gradually decrease due to factors such as the aging of mechanical components. For example, in RGB / Thermal, when registration is good, the two modal views overlap; after deviation, the two modes shift, resulting in ghosting. To maintain good operation, manual calibration or even returning the device to the factory for calibration is often necessary. In theory, by acquiring two images with strong texture features in each of the two RGB modes, image registration can be achieved, i.e., online calibration. However, if the device is installed in a relatively open area lacking texture features, good registration correction cannot be achieved. Furthermore, since modern multimodal systems are often cross-modal, such as RGB and Thermal, RGB and LiDAR, traditional image-based registration is essentially ineffective. Due to the significant differences between modes, existing single-modal registration methods cannot effectively achieve correction.

[0003] No effective solution has yet been proposed to address the above problems. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for obtaining calibration parameters, in order to at least solve the technical problem of poor multimodal information fusion effect caused by the inability of multimodal devices to perform registration correction in a timely and accurate manner.

[0005] According to one aspect of the present invention, a method for obtaining calibration parameters is provided, comprising: acquiring the motion trajectories of multiple targets in a first mode and a second mode; performing coarse matching on the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results corresponding to the motion trajectories of each target; acquiring point pair information of each target based on the preliminary matching results and time information, wherein the point pair information is the coordinates of the center point of a target detection box that appears simultaneously in the first mode and the second mode; performing fine matching on the motion trajectories of each target in the first mode and the second mode based on the point pair information of each target to obtain final matching results corresponding to the motion trajectories of each target, wherein the final matching results corresponding to the motion trajectories of each target are used as calibration parameters between the first mode and the second mode.

[0006] Optionally, acquiring the motion trajectories of multiple targets in a first mode and a second mode includes: acquiring feature point information of each target in the first mode and the second mode, wherein the feature point information includes the center point coordinates and time information of the target detection box; processing the feature point information of each target in the first mode and the second mode to obtain the motion trajectory of each target in the first mode and the second mode.

[0007] Optionally, the feature point information of each target in the first mode and the second mode is processed to obtain the motion trajectory of each target in the first mode and the second mode, including: target tracking of each target based on the feature point information of each target in the first mode and the second mode to obtain the initial motion trajectory of each target in the first mode and the second mode; and smoothing the initial motion trajectory of each target in the first mode and the second mode to obtain the motion trajectory of each target in the first mode and the second mode.

[0008] Optionally, coarse matching is performed on the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results corresponding to the motion trajectories of each target. This includes: obtaining trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode, wherein the trajectory texture features include target pose information and movement direction, the target pose information is the coordinates of the center point of the target detection box, the movement direction is the tangent direction of the motion trajectory, and the feature descriptor is used to characterize the regional feature information formed by the intersection of multiple motion trajectories; feature matching is performed on the trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results corresponding to the motion trajectories of each target.

[0009] Optionally, feature matching is performed on the trajectory texture features corresponding to the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results for the motion trajectories of each target. This includes: pairing the target pose information corresponding to the motion trajectories of each target in the first mode and the second mode to obtain the transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode; adjusting the transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode according to the movement direction corresponding to the motion trajectories of each target in the first mode and the second mode to obtain the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode; calculating the overlap degree corresponding to the motion trajectories of each target in the first mode and the second mode according to the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode respectively; filtering the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode according to the overlap degree corresponding to the motion trajectories of each target in the first mode and the second mode to determine the preliminary matching results for the motion trajectories of each target.

[0010] Optionally, the target transformation relationship corresponding to the motion trajectory of each target in the first mode and the second mode is filtered according to the overlap degree corresponding to the motion trajectory of each target in the first mode and the second mode to determine the preliminary matching result corresponding to the motion trajectory of each target, including: filtering the overlap degree corresponding to the motion trajectory of each target in the first mode and the second mode to determine the optimal overlap degree of each target; and taking the target transformation relationship corresponding to the optimal overlap degree of each target as the preliminary matching result corresponding to the motion trajectory of each target.

[0011] Optionally, feature matching is performed based on the feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results corresponding to the motion trajectories of each target. This includes: registering the feature descriptors corresponding to the motion trajectories of each target in the first mode and the feature descriptors corresponding to the motion trajectories in the second mode to obtain preliminary matching results corresponding to the motion trajectories of each target.

[0012] According to another aspect of the present invention, a calibration parameter acquisition device is also provided, comprising: a first acquisition module, configured to acquire the motion trajectories of a plurality of targets in a first mode and a second mode; a coarse matching module, configured to coarsely match the motion trajectories of each target in the first mode and the second mode to obtain a preliminary matching result corresponding to the motion trajectory of each target; a second acquisition module, configured to acquire point pair information of each target based on the preliminary matching result corresponding to the motion trajectory of each target and time information, wherein the point pair information is the coordinates of the center point of a target detection box that appears simultaneously in the first mode and the second mode; and a fine matching module, configured to finely match the motion trajectories of each target in the first mode and the second mode based on the point pair information of each target to obtain a final matching result corresponding to the motion trajectory of each target, wherein the final matching result corresponding to the motion trajectory of each target is used as a calibration parameter between the first mode and the second mode.

[0013] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the calibration parameter acquisition method described in any one of the preceding embodiments.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the calibration parameter acquisition method described in any one of the above descriptions.

[0015] In this embodiment of the invention, the following steps are taken: acquiring the motion trajectories of multiple targets in a first mode and a second mode; performing coarse matching on the motion trajectories of each target in the first and second modes to obtain preliminary matching results corresponding to the motion trajectories of each target; acquiring point-pair information of each target based on the preliminary matching results and time information, wherein the point-pair information is the coordinates of the center point of the target detection box that appears simultaneously in the first and second modes; and performing fine matching on the motion trajectories of each target in the first and second modes based on the point-pair information of each target to obtain the final matching results corresponding to the motion trajectories of each target, wherein the final matching results corresponding to the motion trajectories of each target are used as calibration parameters between the first and second modes. In other words, the embodiments of the present invention do not require factory calibration, do not require manual control of the target, and do not require setting thresholds. By acquiring the motion trajectories of multiple targets in multimodal modes, acquiring feature information unrelated to modal characteristics, and establishing connections between modes, the online automatic calibration of calibration parameters is achieved. This solves the technical problem of poor multimodal information fusion effect caused by the inability of multimodal devices to achieve registration correction in a timely and accurate manner. The invention achieves the technical effect of improving the real-time performance and accuracy of multimodal devices in achieving registration correction and ensuring the effectiveness of multimodal information fusion. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 A flowchart illustrating a method for obtaining calibration parameters provided in an embodiment of the present invention;

[0018] Figure 2 A schematic diagram illustrating registration correction for a multimodal device provided in an optional embodiment of the present invention;

[0019] Figure 3 A schematic diagram of the motion trajectories of multiple targets in a real-world scenario, provided as an optional embodiment of the present invention;

[0020] Figure 4 A schematic diagram of multiple target motion trajectories detected in mode A and mode B, provided as an optional embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram illustrating feature matching based on feature descriptors in modal A and modal B, provided as optional embodiments of the present invention.

[0022] Figure 6 A schematic diagram showing the coarse matching before and after in mode A and mode B, provided for an optional embodiment of the present invention;

[0023] Figure 7 This is a schematic diagram illustrating the acquisition of target information corresponding to mode A and mode B based on coarse matching results, provided as an optional embodiment of the present invention.

[0024] Figure 8 A schematic diagram illustrating the extraction of trajectory texture features in modes A and B, provided as an optional embodiment of the present invention;

[0025] Figure 9 A schematic diagram of trajectory texture features extracted in mode A and mode B, provided for an optional embodiment of the present invention;

[0026] Figure 10 A schematic diagram illustrating the poor overlap matching in modes A and B, provided for an optional embodiment of the present invention;

[0027] Figure 11 A schematic diagram illustrating the optimal overlap matching in modes A and B, provided for an optional embodiment of the present invention;

[0028] Figure 12 This is a schematic diagram of a calibration parameter acquisition device provided in an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish different objects, rather than to limit a specific order.

[0031] For ease of description, some of the nouns or terms appearing in this invention will be explained in detail below.

[0032] Multimodal: Contains two or more modes, such as RGB-T bimodal, which means RGB and Thermal modes.

[0033] Modal alignment refers to calibrating two or more modalities, obtaining calibration parameters, and performing affine transformations based on the calibration parameters to achieve better registration of bimodal data / images.

[0034] According to one aspect of the present invention, a method for obtaining calibration parameters is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] Figure 1 A flowchart of a method for obtaining calibration parameters provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes the following steps:

[0036] Step S102: Obtain the motion trajectories of multiple targets in the first mode and the second mode;

[0037] The first mode is a selected mode from multiple modes, and the second mode is any mode other than the first mode. Furthermore, the first and second modes can be the same mode or different modes; these modes include, but are not limited to, RGB, LiDAR, and Thermal. The multiple targets can be of the same type or different types.

[0038] Step S104: Perform coarse matching of the motion trajectories of each target in the first mode and the second mode to obtain the preliminary matching results corresponding to the motion trajectories of each target;

[0039] In other words, a coarse match is performed between the trajectories of each target in the first mode and the trajectories of each target in the second mode to obtain a preliminary matching result for each target's trajectory. For any target, the preliminary matching result for its trajectory includes the transformation relationship between the target's trajectory in the first mode and its trajectory in the second mode.

[0040] Step S106: Based on the preliminary matching results and time information corresponding to the motion trajectory of each target, obtain the point pair information of each target, wherein the point pair information is the coordinates of the center point of the target detection box that appears simultaneously in the first mode and the second mode;

[0041] For any target, by combining the preliminary matching results corresponding to the target's motion trajectory with time information, the point-to-point information of the target can be obtained. That is, the coordinates of the center point of the detection box of the same target in different modalities at the same time; similarly, the point-to-point information of multiple other targets can be obtained.

[0042] Step S108: Based on the point-to-point information of each target, perform fine matching on the motion trajectory of each target in the first mode and the second mode to obtain the final matching result corresponding to the motion trajectory of each target. The final matching result corresponding to the motion trajectory of each target is used as the calibration parameter between the first mode and the second mode.

[0043] In other words, based on the point-pair information of each target, a fine-grained matching is performed on the motion trajectories of each target in the first mode and the second mode. This establishes a more accurate transformation relationship between the motion trajectories of each target in the first mode and the second mode, serving as the final matching result for each target's motion trajectory. This final matching result can be used as a calibration parameter between the first and second modes to achieve registration correction between multimodal devices. Therefore, in multimodal applications, only a certain visual intersection between any two modes is required to achieve modal alignment. Furthermore, a multimodal device is a device containing two or more modes.

[0044] It should be noted that the above method can specifically achieve online alignment of data across multiple modalities. Its practical applications include dual-modal scenarios such as RGB and LiDAR, RGB and Thermal, RGB and RGB, and LiDAR and Thermal; it can also achieve tri-modal or higher alignment. Specifically, using one modality as a reference, such as RGB / Thermal / LiDAR tri-modal, this method first obtains the extrinsic parameters for RGB and Thermal registration, then obtains the extrinsic parameters for Thermal and LiDAR registration, using Thermal as the reference, thereby obtaining the extrinsic parameter information for the RGB / Thermal / LiDAR tri-modal, achieving better registration and correction between them. In practical scenarios, if multi-modal devices are not calibrated at the factory, or if significant problems arise during actual use leading to large deviations, this method can automatically calibrate online without requiring manual control of the target or setting thresholds.

[0045] In this embodiment of the invention, the following steps are taken: acquiring the motion trajectories of multiple targets in a first mode and a second mode; performing coarse matching on the motion trajectories of each target in the first and second modes to obtain preliminary matching results corresponding to the motion trajectories of each target; acquiring point-pair information of each target based on the preliminary matching results and time information, wherein the point-pair information is the coordinates of the center point of the target detection box that appears simultaneously in the first and second modes; and performing fine matching on the motion trajectories of each target in the first and second modes based on the point-pair information of each target to obtain the final matching results corresponding to the motion trajectories of each target, wherein the final matching results corresponding to the motion trajectories of each target are used as calibration parameters between the first and second modes. In other words, the embodiments of the present invention do not require factory calibration, do not require manual control of the target, and do not require setting thresholds. By acquiring the motion trajectories of multiple targets in multimodal modes, acquiring feature information unrelated to modal characteristics, and establishing connections between modes, the online automatic calibration of calibration parameters is achieved. This solves the technical problem of poor multimodal information fusion effect caused by the inability of multimodal devices to achieve registration correction in a timely and accurate manner. The invention achieves the technical effect of improving the real-time performance and accuracy of multimodal devices in achieving registration correction and ensuring the effectiveness of multimodal information fusion.

[0046] In one exemplary implementation, acquiring the motion trajectories of multiple targets in a first mode and a second mode includes: acquiring feature point information of each target in the first mode and the second mode, wherein the feature point information includes the center point coordinates and time information of the target detection box; and processing the feature point information of each target in the first mode and the second mode to obtain the motion trajectory of each target in the first mode and the second mode.

[0047] Optionally, feature point information of each target in the first modality and feature point information in the second modality can be obtained through a detection algorithm. It should be noted that the aforementioned feature point information includes, but is not limited to, the center point coordinates of the target detection box and time information; in actual scene target detection, each target detection box and target have a one-to-one correspondence; the center point coordinates of the target detection box are used to characterize the target's trajectory, and the center point of the target's detection box is also the target's feature point in any modality; the time information is the absolute time corresponding to the detection of the center point of the target detection box, which can be represented using a timestamp.

[0048] Furthermore, the motion trajectories of each target in the first mode are generated based on the feature point information of each target in the first mode; the motion trajectories of each target in the second mode are generated based on the feature point information of each target in the second mode. Since there is no human assistance and the correspondence of feature points of the targets in each mode is unknown (i.e., it is unknown which feature point in the first mode should correspond to which feature point in the second mode), introducing time information allows multiple motion trajectory lines of targets to appear in each mode at that time. In practical implementation, for any target, connecting multiple feature points representing the target's trajectory according to the time information forms the target's trajectory. It should be noted that in the process of acquiring the target's motion trajectory, the target's feature points are not considered; only the coordinates of the center point of the target detection box and the time information are relevant.

[0049] In one exemplary embodiment, the feature point information of each target in the first mode and the second mode is processed to obtain the motion trajectory of each target in the first mode and the second mode, including: target tracking of each target based on the feature point information of each target in the first mode and the second mode to obtain the initial motion trajectory of each target in the first mode and the second mode; and smoothing the initial motion trajectory of each target in the first mode and the second mode to obtain the motion trajectory of each target in the first mode and the second mode.

[0050] In one exemplary implementation, for any target, target tracking is performed, and after a period of time, the original motion trajectory of the target in each mode can be obtained; then, the original motion trajectory is smoothed to obtain the smoothed original motion trajectory, which is the motion trajectory of the target in each mode. The above smoothing can be performed using methods such as moving average smoothing or SG filtering smoothing based on local polynomial least squares fitting.

[0051] In the above embodiments of the present invention, the target is tracked using the feature point information of the target in each mode to obtain the initial motion trajectory of the target in each mode; then the initial motion trajectory of the target in each mode is smoothed, and finally the smoothed original motion trajectory is taken as the motion trajectory of the target in each mode.

[0052] In one exemplary implementation, coarse matching is performed on the motion trajectories of each target in the first and second modes to obtain preliminary matching results corresponding to the motion trajectories of each target. This includes: acquiring trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first and second modes, wherein the trajectory texture features include target pose information and movement direction, the target pose information is the coordinates of the center point of the target detection box, the movement direction is the tangent direction of the motion trajectory, and the feature descriptor is used to characterize the regional feature information formed by the intersection of multiple motion trajectories; and feature matching is performed on the trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first and second modes to obtain preliminary matching results corresponding to the motion trajectories of each target.

[0053] The trajectory feature information of the motion trajectory includes, but is not limited to, the trajectory texture features and feature descriptors corresponding to the motion trajectory; the feature descriptors include, but are not limited to, scale-invariant feature transform (SIFT), speed-up robust features (SURF), and ORB (OrientedFAST and Rotated BRIEF), etc.

[0054] The motion trajectory of the aforementioned target can be identified using category information, which represents the target's category attribute, such as a vehicle. In practice, category information can be introduced into the coarse matching process, serving as a dimension to increase feature information and improve the accuracy of coarse matching.

[0055] The above step S104 can be implemented in multiple ways. The embodiments of the present invention mainly provide three implementation schemes:

[0056] Option 1: Obtain the trajectory texture features corresponding to the motion trajectory of each target in the first mode and the second mode; perform feature matching on the trajectory texture features corresponding to the motion trajectory of each target in the first mode and the second mode to obtain the preliminary matching results corresponding to the motion trajectory of each target.

[0057] Option 2: Obtain the feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode; perform feature matching on the feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode to obtain the preliminary matching results corresponding to the motion trajectories of each target.

[0058] Option 3: Obtain the trajectory texture features and feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode; perform feature matching between the trajectory texture features and feature descriptors corresponding to the motion trajectories of each target in the first mode and the trajectory texture features and feature descriptors corresponding to the motion trajectories in the second mode to obtain the preliminary matching results corresponding to the motion trajectories of each target.

[0059] In other words, Scheme 1 uses trajectory texture features for feature matching to achieve coarse matching; Scheme 2 uses feature descriptors for feature matching to achieve coarse matching; Scheme 3 uses a combination of trajectory texture features and feature descriptors for feature matching to achieve coarse matching, which is essentially a combination of Scheme 1 and Scheme 2.

[0060] It should be noted that all three methods described above can yield relatively accurate preliminary matching results for the motion trajectories of each target. The choice of which method to use depends on the specific scenario.

[0061] In the above embodiments of the present invention, the motion trajectories of each target in the first mode and the motion trajectories in the second mode are coarsely matched to obtain the preliminary matching results corresponding to the motion trajectories of each target, which can realize the preliminary alignment between multimodal devices.

[0062] Furthermore, feature matching is performed on the trajectory texture features corresponding to the motion trajectories of each target in the first and second modes to obtain preliminary matching results for the motion trajectories of each target. This includes: pairing the target pose information corresponding to the motion trajectories of each target in the first and second modes to obtain the transformation relationship of the motion trajectories of each target in the first and second modes; adjusting the transformation relationship of the motion trajectories of each target in the first and second modes according to the movement direction of the motion trajectories of each target in the first and second modes to obtain the target transformation relationship of the motion trajectories of each target in the first and second modes; calculating the overlap degree of the motion trajectories of each target in the first and second modes according to the target transformation relationship of the motion trajectories of each target in the first and second modes respectively; and filtering the target transformation relationship of the motion trajectories of each target in the first and second modes according to the overlap degree of the motion trajectories of each target in the first and second modes to determine the preliminary matching results for the motion trajectories of each target.

[0063] Optionally, firstly, the target pose information corresponding to the motion trajectories of each target in the two modes is paired to obtain the transformation relationship of the motion trajectories of each target in the two modes; then, the transformation relationship of the motion trajectories of each target in the two modes is adjusted according to the movement direction corresponding to the motion trajectories of each target in the two modes to obtain the target transformation relationship of the motion trajectories of each target in the two modes; next, the overlap degree of the motion trajectories of each target in the two modes is calculated according to the target transformation relationship of the motion trajectories of each target in the two modes; finally, the target transformation relationship of the motion trajectories of each target in the two modes is filtered according to the overlap degree of the motion trajectories of each target in the two modes to determine the preliminary matching result of the motion trajectory of each target.

[0064] In the above embodiments of the present invention, feature matching is performed using the trajectory texture features corresponding to the motion trajectories of each target in two modes to achieve coarse matching, thereby obtaining the preliminary matching results corresponding to the motion trajectories of each target.

[0065] Furthermore, based on the overlap degree of the motion trajectories of each target in the first and second modes, the target transformation relationship corresponding to the motion trajectories of each target in the first and second modes is screened to determine the preliminary matching result corresponding to the motion trajectory of each target. This includes: screening the overlap degree of the motion trajectories of each target in the first and second modes to determine the optimal overlap degree of each target; and using the target transformation relationship corresponding to the optimal overlap degree of each target as the preliminary matching result corresponding to the motion trajectory of each target.

[0066] To obtain the preliminary matching results corresponding to the optimal motion trajectories of each target, the overlap between the motion trajectories of each target in the first mode and the motion trajectories in the second mode can be filtered to select the optimal overlap between each target. Then, the target transformation relationship corresponding to the optimal overlap between each target is used as the preliminary matching result corresponding to the motion trajectory of each target, thereby effectively avoiding poor matching situations.

[0067] Furthermore, feature matching is performed based on the feature descriptors corresponding to the motion trajectories of each target in the first and second modes to obtain preliminary matching results for the motion trajectories of each target. This includes: registering the feature descriptors corresponding to the motion trajectories of each target in the first and second modes to obtain preliminary matching results for the motion trajectories of each target.

[0068] In other words, by using the feature descriptors corresponding to the motion trajectories of each target in the first mode and the feature descriptors corresponding to the motion trajectories in the second mode for registration, coarse matching is achieved, thereby obtaining the preliminary matching results corresponding to the motion trajectories of each target.

[0069] For any target, the preliminary matching result based on the target's motion trajectory can be used to determine that a certain motion trajectory in the first mode should correspond to a certain trajectory in the second mode, that is, to obtain the motion trajectory of the same target in two modes.

[0070] The optional embodiments of the present invention will be described in detail using a multimodal device as an example.

[0071] Figure 2 A schematic diagram illustrating registration correction for a multimodal device provided in an optional embodiment of the present invention, as shown below. Figure 2 As shown, it can realize the joint calibration of multimodal devices with two or more modes. The core is to first realize pairwise calibration (such as mode A and mode B). After mode A and mode B are registered, mode B and mode C are registered in the same way. This process is extended, and finally, joint calculation is performed to realize the joint calibration of multimodal devices.

[0072] Furthermore, from modality A, target information (corresponding to the aforementioned feature point information) is obtained through detection algorithms (including traditional algorithms and network learning-based methods, etc.); similarly, from modality B, target information is obtained using a similar method. The target information includes at least three types of information: the position and time information of the feature points in their respective modalities [x, y, t], and may also include category information [person, car, etc.].

[0073] It should be noted that in mode A, the motion trajectories of multiple different types of vehicles are obtained through a detection and tracking algorithm; in mode B, the motion trajectories of multiple different types of vehicles are obtained through a detection and tracking algorithm specific to this mode (the motion trajectories have been smoothed). The original point forming the motion trajectory is the coordinate of the center point of the target detection box.

[0074] Based on the detection and tracking of targets, multiple target motion trajectories (i.e., the motion trajectory of the targets) can be obtained through cumulative recording over a certain period of time. Figure 3 A schematic diagram of the motion trajectories of multiple targets in a real-world scenario is provided for an optional embodiment of the present invention, such as... Figure 3 As shown, the motion trajectories of different types of (vehicle) targets in a real-world scenario are recorded within a certain time period. Figure 3 A total of 7 target trajectories (① to ⑦) were recorded.

[0075] Due to the differences between modalities, there are deviations in the perspectives between modalities, specifically including rotation, translation, scaling, affine transformations, etc. Figure 4 The diagram illustrates the motion trajectories of multiple targets detected in modes A and B, as provided in an optional embodiment of the present invention. Figure 4 As shown, for a real-world scenario, the result of obtaining the target's motion trajectory in mode A is: Figure 4The left image shows the result of obtaining the target trajectory in mode B. Figure 4 The image on the right. Figure 4 There are obvious differences between the two images on the left and right, and since they are not assisted by humans, the specific correspondence of the motion trajectories is unknown; they are all motion trajectories without labels or distinctions.

[0076] After obtaining the target motion trajectory in modality A and modality B, a coarse matching between modal A and modal B is performed based on the trajectory information. A problem with existing image matching methods is that the differences between modalities within the same scene are significant, making it difficult to effectively obtain information about corresponding feature points across modalities. This invention addresses this problem effectively. The target motion trajectory generates a large number of texture features. Trajectory texture features in modal A and modal B are extracted, and / or the intersection of multiple trajectories forms a large amount of regional feature information as feature descriptors (specifically, SIFT descriptors, etc.) for matching. Finally, matching calculations are performed based on the trajectory texture features and / or feature descriptors to achieve preliminary registration between modal A and modal B, i.e., coarse matching. Furthermore, category information [person, car, etc.] can be introduced to increase the dimensionality of feature information and improve the accuracy of coarse matching.

[0077] Furthermore, by using trajectory texture features under modes A and B as a coarse matching method, we can obtain target pose information, target category information, and target movement direction at a series of different time points. Figure 8 The schematic diagram provided for an optional embodiment of the present invention illustrates the extraction of trajectory texture features in modes A and B, as shown below. Figure 8 As shown, the trajectory texture feature extraction process of two modalities at a certain moment is as follows: if the detected target is only of two types [person, car], person is represented by a triangle and car is represented by a square; the target pose information in each modality is the coordinates of the center point of the target detection box; the target's movement direction is compared with the target in the previous frame, which is the tangent direction of the trajectory line. Figure 9 The diagram illustrates the trajectory texture features extracted in mode A and mode B, as provided in an optional embodiment of the present invention. Figure 9 As shown, after extracting the trajectory texture features at that moment, a matching is performed based on the trajectory texture features under mode A and mode B. That is, a point of the same type in mode A, such as a person, is paired with a feature point of the same type in mode B, such as a person. Based on the direction information of the corresponding point, the transformation relationship between mode A and mode B is adjusted (which can be extended to line pair matching with directional points and triangulation matching, etc.). The overlap of other feature points is checked (after adjusting the transformation relationship, the overlap of feature information between mode A and mode B can be calculated by counting the Euclidean distance between the nearest feature points of the same category [person, car] in different modes after mode A and mode B overlap). Figure 10This is a schematic diagram illustrating the poor overlap matching situation in modes A and B, provided for an optional embodiment of the present invention. Figure 11 The diagram illustrates the optimal overlap matching in modes A and B, as provided in an optional embodiment of the present invention. Figure 10 , 11 As shown, after matching all points, the transformation matrix with the best overlap at that moment is obtained. Similarly, data from mode A and mode B at different times are used for verification and correction to achieve preliminary registration, thereby obtaining the trajectory correspondence between mode A and mode B.

[0078] If a large amount of regional feature information is generated by the interlacing of multiple trajectories as a feature descriptor, then SIFI descriptors can be used for registration. Figure 5 The diagram illustrates feature matching based on feature descriptors in modal A and modal B, as provided in an optional embodiment of the present invention. Figure 5 As shown, a preliminary registration was achieved between mode A and mode B. Figure 5 The horizontal line in the diagram represents a feature point in mode A that, after being calculated by the algorithm, corresponds to a point in mode B.

[0079] Figure 6 Schematic diagrams before and after coarse matching in mode A and mode B, provided for optional embodiments of the present invention, as shown below. Figure 6 As shown, Figure 6 The left image shows the acquisition of the target's motion trajectory in mode A and mode B. The default direct registration between the two modes shows significant misregistration between modes A and B. After performing coarse matching based on feature matching, modes A and B roughly overlap after registration, achieving coarse matching and thus obtaining the correspondence between the trajectory lines of modes A and B.

[0080] Figure 7 A schematic diagram illustrating the acquisition of target information corresponding to mode A and mode B based on coarse matching results, provided as an optional embodiment of the present invention, is shown below. Figure 7 As shown, the corresponding targets for multiple target trajectories in modes A and B can be obtained. Trajectory ③ in mode A corresponds to trajectory ③ in mode B, and both are the motion trajectories of the same target. Combining this with information about a specific target in mode A, the information of target trajectory ③ over a certain period of time is obtained. For example, the target information of trajectory ③ in modes A and B is as follows:

[0081] In mode A: ...[X1,Y1,t1],[X2,Y2,t2],[X3,Y3,t3],[X4,Y4,t4],[X5,Y5,t5],[X6,Y6,t6]...

[0082] In mode B: ...[x0,y0,t0],[x1,y1,t1],[x2,y2,t2],[x3,y3,t3],[x4,y4,t4],[x5,y5,t5]...

[0083] The target information [X,Y,t] has the following specific meanings:

[0084] The x-axis pixel coordinate of the center point of the target detection box is X;

[0085] The vertical pixel coordinate of the center point of the target detection box is Y;

[0086] The absolute time at which the target box was detected is t.

[0087] It should be noted that the feature points of each modality are the center points of the target detection box. For example, the feature points in the RGB modality are the center points of the 2D box, and the feature points in the Lidar modality are the center points of the 3D box.

[0088] Because of the differences between Mode A and Mode B, the relevant information of trajectory ③ is inconsistent in Mode A and Mode B. That is, the target may appear in Mode A but not yet in Mode B, or the target may have left the view of Mode A but not yet left the view of Mode B. Based on the time information, the target information that appears in both Mode A and Mode B is selected. The processed trajectory information of ③ is as follows:

[0089] In mode A: ...[X1,Y1],[X2,Y2],[X3,Y3],[X4,Y4],[X5,Y5] ...

[0090] In mode B: ...[x1,y1],[x2,y2],[x3,y3],[x4,y4],[x5,y5] ...

[0091] Obtaining the target's motion trajectory in modality A and modality B involves acquiring multiple trajectories for various target types. This allows for the acquisition of a large amount of matching data; that is, information about a specific target in modality A can be correlated with information about the corresponding target in modality B, resulting in a large number of corresponding feature matching pairs. Finally, matrix operations are combined to complete the A / B modality registration correction.

[0092] According to another aspect of the present invention, a device for acquiring calibration parameters is also provided. Figure 12 This is a schematic diagram of a calibration parameter acquisition device provided in an embodiment of the present invention, as shown below. Figure 12 As shown, the calibration parameter acquisition device includes: a first acquisition module 1202, a coarse matching module 1204, a second acquisition module 1206, and a fine matching module 1208. The calibration parameter acquisition device will be described in detail below.

[0093] The first acquisition module 1202 is used to acquire the motion trajectories of multiple targets in the first mode and the second mode;

[0094] The coarse matching module 1204 is connected to the first acquisition module 1202 mentioned above, and is used to coarsely match the motion trajectories of each target in the first mode and the second mode to obtain the preliminary matching results corresponding to the motion trajectory of each target.

[0095] The second acquisition module 1206 is connected to the coarse matching module 1204 mentioned above. It is used to acquire point-to-point information of each target based on the preliminary matching results and time information corresponding to the motion trajectory of each target. The point-to-point information is the coordinates of the center point of the target detection box that appears simultaneously in the first mode and the second mode.

[0096] The fine matching module 1208 is connected to the second acquisition module 1206 and is used to perform fine matching on the motion trajectory of each target in the first mode and the second mode according to the point pair information of each target, so as to obtain the final matching result corresponding to the motion trajectory of each target. The final matching result corresponding to the motion trajectory of each target is used as the calibration parameter between the first mode and the second mode.

[0097] In this embodiment of the invention, the device for acquiring calibration parameters does not require factory calibration, manual target control, or threshold setting. By acquiring the motion trajectories of multiple targets in multimodal modes, it obtains feature information unrelated to modal characteristics and establishes connections between modes, thereby achieving online automatic calibration of calibration parameters. This solves the technical problem of poor multimodal information fusion due to the inability of multimodal devices to perform registration correction in a timely and accurate manner. It achieves the technical effect of improving the real-time performance and accuracy of multimodal devices in performing registration correction and ensuring the effectiveness of multimodal information fusion.

[0098] It should be noted that the first acquisition module 1202, coarse matching module 1204, second acquisition module 1206 and fine matching module 1208 mentioned above correspond to steps S102 to S108 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment.

[0099] In one exemplary embodiment, the first acquisition module 1202 includes: a first acquisition unit, configured to acquire feature point information of each target in a first mode and a second mode, wherein the feature point information includes the center point coordinates and time information of the target detection box; and a processing unit, configured to process the feature point information of each target in the first mode and the second mode to obtain the motion trajectory of each target in the first mode and the second mode.

[0100] In one exemplary embodiment, the processing unit includes: an acquisition subunit, configured to perform target tracking on each target based on feature point information of each target in a first mode and a second mode, and acquire the initial motion trajectory of each target in the first mode and the second mode; and a processing subunit, configured to smooth the initial motion trajectory of each target in the first mode and the second mode, and obtain the motion trajectory of each target in the first mode and the second mode.

[0101] In one exemplary embodiment, the coarse matching module 1204 includes: a second acquisition unit, configured to acquire trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in a first mode and a second mode, wherein the trajectory texture features include target pose information and movement direction, the target pose information is the coordinates of the center point of the target detection box, the movement direction is the tangent direction of the motion trajectory, and the feature descriptor is used to characterize the regional feature information formed by the intersection of multiple motion trajectories; and a feature matching unit, configured to perform feature matching on the trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results corresponding to the motion trajectories of each target.

[0102] In one exemplary embodiment, the feature matching unit includes: a pairing subunit, configured to pair the target pose information corresponding to the motion trajectories of each target in the first mode and the second mode to obtain the transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode; an adjustment subunit, configured to adjust the transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode according to the movement direction corresponding to the motion trajectories of each target in the first mode and the second mode to obtain the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode; a calculation subunit, configured to calculate the overlap degree corresponding to the motion trajectories of each target in the first mode and the second mode according to the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode; and a filtering subunit, configured to filter the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode according to the overlap degree corresponding to the motion trajectories of each target in the first mode and the second mode to determine the preliminary matching result corresponding to the motion trajectories of each target.

[0103] In one exemplary embodiment, the above-mentioned screening subunit includes: a first determining subunit, used to screen the overlap degree corresponding to the motion trajectory of each target in the first mode and the second mode, and determine the optimal overlap degree of each target; and a second determining subunit, used to take the target transformation relationship corresponding to the optimal overlap degree of each target as the preliminary matching result corresponding to the motion trajectory of each target.

[0104] In one exemplary embodiment, the feature matching unit includes a registration subunit, used to register the feature descriptors corresponding to the motion trajectories of each target in the first mode and the feature descriptors corresponding to the motion trajectories in the second mode, to obtain preliminary matching results corresponding to the motion trajectories of each target.

[0105] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform a calibration parameter acquisition method of any of the above.

[0106] Optionally, the aforementioned electronic device can be any one of the electronic devices in the group of electronic devices. Alternatively, the aforementioned electronic device can be replaced with a terminal device such as a mobile terminal.

[0107] Optionally, the aforementioned electronic device may include one or more processors and memory.

[0108] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the calibration parameter acquisition method of any of the above.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A method for obtaining calibration parameters, characterized in that, include: Acquire the motion trajectories of multiple targets in the first and second modes; The motion trajectories of each target in the first mode and the second mode are coarsely matched to obtain the preliminary matching results corresponding to the motion trajectories of each target; Based on the preliminary matching results and time information corresponding to the motion trajectories of each target, the point pair information of each target is obtained, wherein the point pair information is the coordinates of the center point of the target detection box that appears simultaneously in the first mode and the second mode; Based on the point-to-point information of each target, fine matching is performed on the motion trajectories of each target in the first mode and the second mode to obtain the final matching result corresponding to the motion trajectory of each target. The final matching result corresponding to the motion trajectory of each target is used as the calibration parameter between the first mode and the second mode. Specifically, the motion trajectories of each target in the first mode and the second mode are coarsely matched to obtain preliminary matching results corresponding to the motion trajectories of each target, including: The trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode are obtained. The trajectory texture features include target pose information and movement direction. The target pose information is the coordinates of the center point of the target detection box. The movement direction is the tangent direction of the motion trajectory. The feature descriptors are used to characterize the regional feature information formed by the intersection of multiple motion trajectories. The trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode are matched to obtain the preliminary matching results corresponding to the motion trajectories of each target.

2. The method for obtaining calibration parameters according to claim 1, characterized in that, Acquire the motion trajectories of multiple targets in the first and second modes, including: The feature point information of each target in the first mode and the second mode is obtained, wherein the feature point information includes the center point coordinates of the target detection box and time information; The feature point information of each target in the first mode and the second mode is processed to obtain the motion trajectory of each target in the first mode and the second mode.

3. The method for obtaining calibration parameters according to claim 2, characterized in that, The feature point information of each target in the first mode and the second mode is processed to obtain the motion trajectory of each target in the first mode and the second mode, including: Based on the feature point information of each target in the first mode and the second mode, target tracking is performed on each target to obtain the initial motion trajectory of each target in the first mode and the second mode; The initial motion trajectories of each target in the first mode and the second mode are smoothed to obtain the motion trajectories of each target in the first mode and the second mode.

4. The method for obtaining calibration parameters according to claim 1, characterized in that, The trajectory texture features corresponding to the motion trajectories of each target in the first mode and the second mode are matched to obtain preliminary matching results for the motion trajectories of each target, including: The target pose information corresponding to the motion trajectories of each target in the first mode and the second mode are paired to obtain the transformation relationship of the motion trajectories of each target in the first mode and the second mode; The transformation relationship of the motion trajectory of each target in the first mode and the second mode is adjusted according to the movement direction corresponding to the motion trajectory of each target in the first mode and the second mode, so as to obtain the target transformation relationship corresponding to the motion trajectory of each target in the first mode and the second mode; Based on the target transformation relationship corresponding to the motion trajectory of each target in the first mode and the second mode, calculate the overlap degree corresponding to the motion trajectory of each target in the first mode and the second mode respectively; Based on the overlap of the motion trajectories of each target in the first mode and the second mode, the target transformation relationship corresponding to the motion trajectories of each target in the first mode and the second mode is filtered to determine the preliminary matching result corresponding to the motion trajectory of each target.

5. The method for obtaining calibration parameters according to claim 4, characterized in that, Based on the overlap of the motion trajectories of each target in the first mode and the second mode, the target transformation relationships corresponding to the motion trajectories of each target in the first mode and the second mode are filtered to determine the preliminary matching results corresponding to the motion trajectories of each target, including: The overlap of the motion trajectories of each target in the first mode and the second mode is filtered to determine the optimal overlap of each target; The target transformation relationship corresponding to the optimal overlap degree of each target is used as the preliminary matching result of the motion trajectory of each target.

6. The method for obtaining calibration parameters according to claim 1, characterized in that, Feature matching is performed based on the feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode to obtain preliminary matching results for the motion trajectories of each target, including: The feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode are registered to obtain the preliminary matching results corresponding to the motion trajectories of each target.

7. A device for acquiring calibration parameters, characterized in that, include: The first acquisition module is used to acquire the motion trajectories of multiple targets in the first mode and the second mode; The coarse matching module is used to coarsely match the motion trajectories of each target in the first mode and the second mode to obtain the preliminary matching results corresponding to the motion trajectories of each target; The second acquisition module is used to acquire point-pair information of each target based on the preliminary matching results and time information corresponding to the motion trajectory of each target, wherein the point-pair information is the coordinates of the center point of the target detection box that appears simultaneously in the first mode and the second mode; The fine matching module is used to perform fine matching on the motion trajectories of each target in the first mode and the second mode based on the point-pair information of each target, and to obtain the final matching result corresponding to the motion trajectory of each target, wherein the final matching result corresponding to the motion trajectory of each target is used as the calibration parameter between the first mode and the second mode; The coarse matching module includes: The second acquisition unit is used to acquire trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode, wherein the trajectory texture features include target pose information and movement direction, the target pose information is the coordinates of the center point of the target detection box, the movement direction is the tangent direction of the motion trajectory, and the feature descriptor is used to characterize the regional feature information formed by the intersection of multiple motion trajectories; The feature matching unit is used to perform feature matching on the trajectory texture features and / or feature descriptors corresponding to the motion trajectories of each target in the first mode and the second mode, so as to obtain the preliminary matching results corresponding to the motion trajectories of each target.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method for obtaining calibration parameters according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the calibration parameter acquisition method according to any one of claims 1 to 6.

Images