Calibration parameter failure detection method, device and equipment and readable storage medium

Abstract

The invention provides a calibration parameter failure detection method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining a sparse depth map obtained through the current collection of a target object by an active depth camera, and obtaining a visual image obtained through the shooting of the target object at the same time by a visual camera; in response to a trigger operation for detecting the calibration external parameter, determining a target depth map under a visual angle of a visual camera corresponding to the sparse depth map according to the calibration external parameter to be detected; and according to the target depth map and the visual image, determining whether the current calibration external parameter is invalid. According to the method provided by the invention, online external parameter failure detection can be effectively realized.

Description

Technical Field

[0001] This application relates to the field of parameter failure detection, and in particular, to a calibration parameter failure detection method, device, equipment, and readable storage medium. Background Art

[0002] A lidar, which is also known as an optical radar by laser detection and ranging; among them, the working principle of the lidar: detecting objects by emitting, reflecting, and receiving infrared light beams. However, although the lidar can obtain the spatial information of the environment and is independent of external light, it is difficult to obtain color information, and the points obtained by the lidar are relatively sparse. While a vision camera (such as an RGB camera) is highly dependent on ambient light, the information obtained is relatively rich, color information can also be obtained, and the overall amount of information is larger. Therefore, it is a common practice to fuse the outputs of the two sensors.

[0003] To ensure the accuracy of products equipped with a camera system, it is necessary to ensure that the depth estimation result is not invalidated due to being dropped or loosened, and thus cannot meet the quality requirements of the depth map for downstream algorithms. Therefore, for the calibration parameters on the product's entire machine production line, especially the external parameter calibration relationship between the active lidar data used in the depth map and the RGB camera, it is necessary to detect whether they are invalid.

[0004] However, in the prior art, the external parameter calibration and failure detection between the lidar camera and the RGB camera are completed on the entire machine production line and the process is relatively complex, and it is not suitable for detecting the failure of the calibrated external parameters during the use of the product. Therefore, online external parameter failure detection cannot be effectively achieved. Summary of the Invention

[0005] This application provides a calibration parameter failure detection method, device, equipment, and readable storage medium to solve the problem that the prior art cannot effectively achieve online external parameter failure detection.

[0006] In a first aspect, an embodiment of this application provides a calibration parameter failure detection method, which is applied to a calibration parameter failure detection device. The calibration parameter failure detection device is used to online detect whether the calibrated external parameters of a camera system fail. The camera system includes a vision camera and an active depth camera. The calibrated external parameters include the transformation matrix between the vision camera and the active depth camera. The calibration parameter failure detection device and the camera system are configured in the same device; the method includes:

[0007] Obtain the sparse depth map currently collected by the active depth camera for a target object, and obtain the visual image captured by the vision camera for the target object at the same moment;

[0008] In response to a trigger operation for detecting the calibration extrinsic parameters, a target depth map corresponding to the visual camera view of the sparse depth map is determined according to the calibration extrinsic parameters to be detected;

[0009] According to the target depth map and the visual image, it is determined whether the current calibration extrinsic parameters are invalid.

[0010] In a possible design, the determining, according to the calibration extrinsic parameters to be detected, a target depth map corresponding to the visual camera view of the sparse depth map includes:

[0011] Densify the sparse depth map to obtain a dense depth map containing semantic information and / or visual information;

[0012] According to the calibration extrinsic parameters, project the dense depth map onto the visual camera view to obtain a target depth map containing semantic and / or visual information.

[0013] In a possible design, before the responding to the trigger operation for detecting the calibration extrinsic parameters, the method further includes:

[0014] If one or more preset semantic targets are detected in both the dense depth map and the visual image, then trigger the operation for detecting the calibration extrinsic parameters.

[0015] In a possible design, the determining, according to the target depth map and the visual image, whether the current calibration extrinsic parameters are invalid includes:

[0016] Perform foreground target detection and segmentation on the target depth map to obtain a first mask;

[0017] Perform foreground target detection and segmentation on the visual image to obtain a second mask;

[0018] According to the edge contours of the first mask and the edge contours of the second mask, determine whether the current calibration extrinsic parameters are invalid by calculating the overlap rate or difference between the first mask and the second mask.

[0019] In a possible design, the determining, according to the edge contours of the first mask and the edge contours of the second mask, whether the current calibration extrinsic parameters are invalid by calculating the overlap rate or difference between the first mask and the second mask includes:

[0020] Respectively determine the intersection and union of the edge contours of the first mask and the edge contours of the second mask;

[0021] Take the ratio of the intersection and the union as the coincidence rate of the first mask and the second mask, and / or, take the value obtained by subtracting the coincidence rate from one as the difference value;

[0022] If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, it is determined that the current calibrated external parameter fails; wherein, the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

[0023] In a possible design, the determining whether the current calibrated external parameter fails by calculating the overlap rate or difference between the first mask and the second mask according to the edge contours of the first mask and the second mask includes:

[0024] According to the edge contours of the first mask and the second mask, determine the number of elements in the intersection of the edge contours of the first mask and the edge contours of the second mask, the number of elements in the edge contours of the first mask, and the number of elements in the edge contours of the second mask;

[0025] Take the ratio of twice the number of elements in the intersection to the sum of the number of elements in the edge contours of the first mask and the number of elements in the edge contours of the second mask as the coincidence rate of the first mask and the second mask, and / or take the value obtained by subtracting the coincidence rate from 1 as the difference value;

[0026] If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, it is determined that the current calibrated external parameter fails; wherein, the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

[0027] In a possible design, the method further includes:

[0028] After determining that the current calibrated external parameter fails, the device is used to prompt that the calibrated external parameter fails.

[0029] In a second aspect, an embodiment of the present application provides a calibration parameter failure detection device, which is used to online detect whether the calibrated external parameter of a camera system fails. The camera system includes a vision camera and an active depth camera, and the calibrated external parameter includes a conversion matrix between the vision camera and the active depth camera. The calibration parameter failure detection device and the camera system are configured in the same device; the device includes:

[0030] An acquisition module, configured to acquire the sparse depth map obtained by the active depth camera currently capturing a target object, and acquire the visual image obtained by the vision camera capturing the target object at the same moment;

[0031] A projection module, configured to respond to a triggering operation for detecting the extrinsic calibration parameters, and determine a target depth map in a visual camera view corresponding to the sparse depth map according to the to-be-detected extrinsic calibration parameters;

[0032] A detection module, configured to determine whether the current extrinsic calibration parameters are invalid according to the target depth map and the visual image.

[0033] In a possible design, the visual camera includes any one of the following: RGB camera, micro camera, infrared camera, grayscale camera; the active depth camera includes any one of the following: lidar camera, ToF camera, structured light camera.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0035] The memory stores computer-executable instructions;

[0036] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the method according to any one of the first aspect is implemented.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method according to any one of the first aspect is implemented.

[0039] The calibration parameter failure detection method, device, equipment and readable storage medium provided in this embodiment first obtain the sparse depth map currently collected by the active depth camera for the target object, and obtain the visual image captured by the visual camera for the target object at the same moment; in response to a triggering operation for detecting the extrinsic calibration parameters, determine a target depth map in a visual camera view corresponding to the sparse depth map according to the to-be-detected extrinsic calibration parameters; then determine whether the current extrinsic calibration parameters are invalid according to the target depth map and the visual image. Therefore, based on the currently collected sparse depth map and visual image, when the detection of the extrinsic calibration parameters is triggered, the sparse depth map and the visual image are aligned, and the online (during the use of the device) failure detection of the extrinsic calibration parameters is realized, without the need for offline detection, which does not affect the current operation of the device configured with this camera system, and can effectively detect whether the extrinsic parameters are invalid online. Description of the Drawings

[0040] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0041] Figure 1 A schematic diagram of the scenario of the calibration parameter failure detection method provided by the embodiment of the present application;

[0042] Figure 2 A schematic flowchart of the calibration parameter failure detection method provided by another embodiment of the present application;

[0043] Figure 3 A schematic flowchart of the calibration parameter failure detection method provided by another embodiment of the present application;

[0044] Figure 4 A schematic diagram of the calculation method based on IoU provided by the embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of the calibration parameter failure detection device provided by the embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0048] In the description, claims and the above drawings of this application, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0049] Currently, the static calibration scheme on existing products is relatively complex. The external parameter calibration between camera systems is completed on the whole machine production line, usually requiring a special calibration process and calibration environment. During the user's use, due to relative position changes caused by dropping, loosening, etc., depth estimation errors occur. Moreover, this calibration process and calibration environment on the whole machine production line are not suitable for users to perform self-detection and recalibration of external parameter failure. Therefore, the prior art cannot effectively achieve online external parameter failure detection.

[0050] Therefore, in view of the above problems, the technical concept of this application is to obtain a sparse depth map from the current acquisition of the target object by an active depth camera, and a visual image obtained by a visual camera capturing the target object at the same moment, perform alignment processing on the sparse depth map and the visual image to determine the target depth map from the perspective of the visual camera corresponding to the sparse depth map, and then based on the aligned target depth map combined with the visual image, determine whether the current calibrated external parameters are invalid, thereby effectively achieving online external parameter failure detection, without the need for the device equipped with this camera system to perform offline detection, without affecting the current operation of the device, and improving the user experience.

[0051] In practical applications, as shown in Figure 1 shown, Figure 1A schematic diagram of a scenario of a calibration parameter failure detection method provided in an embodiment of the present application. The scenario includes a device configured with a camera system 101 (the device may be a wearable device, such as an Extended Reality (XR) device) and a calibration parameter failure detection device 102. The camera system 101 may include a visual camera and an (active) depth camera, wherein the visual camera includes but is not limited to: an RGB camera, a miniature camera (mono camera) infrared camera, and the active depth camera includes but is not limited to: a LiDAR camera, a ToF camera, and a structured light camera; wherein the camera system may be any combination of a visual camera and an active depth camera, for example: the camera system is composed of an RGB camera and a LiDAR camera, which are not listed one by one here.

[0052] The calibration parameter failure detection device can be configured in the device configured with the camera system (that is, the calibration parameter failure detection device and the camera system are configured in the same device), and can also be configured in a server or an electronic device including a processor, which is not specifically limited here.

[0053] Taking the device equipped with the camera system 101 as an XR device as an example, in Example 1, the calibration parameter failure detection device is configured in the XR device. Through the internal communication logic in the XR device, the calibration parameter failure detection device can respectively obtain the sparse depth map of the target object collected by the active depth camera and the visual image of the target object taken by the visual camera at the same time, and then determine the target depth map under the perspective of the visual camera corresponding to the sparse depth map according to the calibration extrinsic parameters to be detected, and determine whether the current calibration extrinsic parameters are invalid according to the target depth map and the visual image. The calibration extrinsic parameters here include the transformation matrix between the visual camera and the active depth camera.

[0054] Combination Figure 2 As shown in the figure, the process of detecting whether the calibration external parameters are invalid is as follows: first, the obtained sparse depth map is sparsely densified, that is, the sparse depth map without semantic information of the active depth camera (such as LiDAR camera, ToF camera, structured light camera, etc.) is converted into a dense depth map containing semantic and visual information, and then the dense depth map is combined with the visual image (such as RGB map, not limited to RGB camera, can be mono camera, infrared camera and other visual cameras) to judge the overlapping area to detect whether the current external parameter (here refers to the calibration external parameter) is valid. The algorithm process is simpler, more robust and efficient. It is suitable for online detection and judgment, without the need for additional hardware devices and complex process algorithms. In addition, if it is detected that the current calibration external parameter is invalid, the external parameter can be re-roughly calibrated, and then the online failure detection method is used to determine whether the re-roughly calibrated external parameter is correct.

[0055] Specifically, in step 1, the sparse depth of the lidar is densely filled to obtain a dense depth map. Among them, the densification based on the sparse depth map includes, but is not limited to, depth filling methods based on traditional methods and deep learning methods.

[0056] In step 2, according to the calibration parameter to be detected (i.e., the extrinsic calibration parameter) T, the (densified) depth map is projected onto the RGB view.

[0057] In step 3, foreground object segmentation is performed on the (dense) depth map projected onto the RGB view to obtain foreground Mask1. Among them, the methods for foreground object segmentation include, but are not limited to, object segmentation based on traditional methods and deep learning methods.

[0058] In step 4, foreground object segmentation is performed on the RGB image to obtain foreground Mask2. Among them, the methods for foreground object segmentation include, but are not limited to, object segmentation based on traditional methods and deep learning methods.

[0059] In step 5, the coincidence and alignment of the edge contours of Mask1 and Mask2 are compared. Among them, the coincidence judgment algorithms include, but are not limited to, calculation methods such as the Intersection over Union (IoU) and common segmentation performance metrics and methods such as the Dice coefficient. The IoU calculates the ratio of the intersection and union of the "predicted bounding box" and the "true bounding box"; the Dice coefficient is a set similarity metric function, usually used to calculate the similarity of two samples, and its value range is [0, 1].

[0060] In step 6, if the coincidence error (or overlap error) is less than the threshold t, then this extrinsic parameter (here referring to the calibration parameter T) is valid. Conversely, that is, if there is a misalignment phenomenon, it indicates that there is a large error in the extrinsic parameter calibration result, and the extrinsic parameter fails (or is invalid) and cannot be used.

[0061] Among them, the validity or invalidity of this extrinsic parameter can be prompted by displaying or broadcasting a prompt message. The output form of the result of whether the extrinsic parameter is valid is not specifically limited here.

[0062] Exemplary 2, the calibration parameter failure detection device is configured in a server or electronic device that can communicate with the XR device. When the operation of detecting the extrinsic calibration parameter is triggered, the sparse depth map of the target object collected by the active depth camera and the visual image of the target object captured by the visual camera at the same moment can be uploaded or sent to the server or electronic device of the calibration parameter failure detection device. The calibration parameter failure detection device processes according to the acquired image data to determine whether the current extrinsic calibration parameter fails; among them, the process of detecting whether the current extrinsic calibration parameter fails can refer to the embodiment described in Exemplary 1, which will not be elaborated here.

[0063] Therefore, based on the currently acquired sparse depth map and visual image, when the detection of the calibrated external parameters is triggered, the sparse depth map and the visual image are aligned to achieve online (during the use of the device) failure detection of the calibrated external parameters, without the need for offline detection, which does not affect the current operation of the device equipped with this camera system and can effectively detect online whether the external parameters fail, improving the user experience.

[0064] The technical solutions of the present application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0065] Figure 3 The flowchart of the calibration parameter failure detection method provided by another embodiment of the present application is shown. The calibration parameter failure detection method may include:

[0066] S301. Obtain the sparse depth map obtained by the active depth camera currently collecting the target object, and obtain the visual image obtained by the visual camera capturing the target object at the same moment.

[0067] In this embodiment, the execution subject may be a calibration parameter failure detection device, which is used to detect online whether the calibrated external parameters of the camera system fail. The camera system includes a visual camera and an active depth camera. The visual camera here includes, but is not limited to: RGB camera, micro camera, infrared camera, grayscale camera. The active depth camera includes, but is not limited to: lidar camera, ToF camera, structured light camera. The calibrated external parameters include the transformation matrix between the visual camera and the active depth camera. The calibration parameter failure detection device and the camera system may be configured in the same device. Taking an XR device as an example below, the calibration parameter failure detection method will be described in detail.

[0068] Among them, taking the depth camera as a lidar camera and the visual camera as an RGB camera as an example, the sparse depth map of the current target object can be collected by the lidar camera, and the visual image (here refers to the RGB image) of the target object at the same moment can be collected by the RGB camera.

[0069] S302. In response to the trigger operation for detecting the calibrated external parameters, determine the target depth map in the visual camera view corresponding to the sparse depth map according to the calibrated external parameters to be detected.

[0070] In this embodiment, the triggering condition for detecting and calibrating the extrinsic parameters can be when a preset scene is detected (for example, when a semantic target of interest is detected (such as: people, animals, desks, chairs, sofas, etc.)) or when a preset condition is met (for example, setting a detection period, when the time interval is one period, that is, at regular time intervals (such as monthly)), a self-detection request for whether the extrinsic parameters are invalid is made, and the operation of detecting and calibrating the extrinsic parameters can be automatically triggered; it can also be that the user triggers the operation of detecting and calibrating the extrinsic parameters by gesture interaction, clicking a detection control, or voice-activated detection function, etc. according to needs.

[0071] Specifically, after the function of detecting and calibrating the extrinsic parameters is triggered, the sparse depth map can be converted into a target depth map in the visual field of the visual camera according to the extrinsic parameters to be detected (for example: the rotation matrix representing the correspondence between the lidar camera and the RGB camera). The implementation process is as follows:

[0072] Through the intrinsic parameters of the camera system, the coordinates of the dense depth map in the lidar camera coordinate system are converted into coordinates in the world coordinate system; through the extrinsic parameters, the coordinates in the world coordinate system are converted into the coordinates of the depth map in the perspective of the RGB camera to obtain the target depth map.

[0073] In a possible design, before the triggering operation for detecting the extrinsic parameters, the method further includes:

[0074] If one or more preset semantic targets are detected in both the dense depth map and the visual image, the operation for detecting the extrinsic parameters is triggered.

[0075] In this embodiment, in the application scenario, when semantic targets of interest (such as: people, animals, desks, chairs, sofas, etc.) are detected in both the dense depth map and the RGB image, the detection of the extrinsic parameters is triggered. For example, the sub-detection program of the extrinsic parameters is started. The self-detection of whether the extrinsic parameters are correct (valid) can be realized. On the one hand, the detection period and activation process of the self-detection of the extrinsic parameters are optimized. On the other hand, the self-detection of the invalidity of the extrinsic parameters on the target of interest can also prevent the situation where the self-detection program of the invalidity of the extrinsic parameters cannot work properly in complex scenes or unknown foreground target scenes, improving the robustness.

[0076] S303. Determine whether the current extrinsic parameters are invalid according to the target depth map and the visual image.

[0077] In this embodiment, according to the target depth map and the visual depth map, the overlap error between the two is calculated, and whether the extrinsic parameters of the camera system are invalid is judged through the overlap error. Among them, the calculation method of the overlap error can be the calculation method of IoU or the Dice coefficient and other common segmentation performance indicators and methods, and it is not limited here.

[0078] The calibration parameter failure detection method provided by the embodiment of the present application first obtains the sparse depth map obtained by the active depth camera currently collecting the target object, and obtains the visual image obtained by the visual camera shooting the target object at the same moment; in response to the trigger operation for detecting the external calibration parameters, according to the external calibration parameters to be detected, the target depth map corresponding to the visual camera view of the sparse depth map is determined; then, according to the target depth map and the visual image, it is determined whether the current external calibration parameters fail. Therefore, based on the currently collected sparse depth map and visual image, when the detection of the external calibration parameters is triggered, the sparse depth map and the visual image are aligned to realize the online (during the use of the device) failure detection of the external calibration parameters, without the need for offline detection, which does not affect the current operation of the device equipped with the camera system and can effectively detect whether the external parameters fail online.

[0079] In a possible design, the determining the target depth map corresponding to the visual camera view of the sparse depth map according to the external calibration parameters to be detected may include:

[0080] Densify the sparse depth map to obtain a dense depth map containing semantic information and / or visual information;

[0081] According to the external calibration parameters, project the dense depth map onto the visual camera view to obtain a target depth map containing semantics and / or visual information.

[0082] Specifically, after the function of detecting the external calibration parameters is triggered, according to the external calibration parameters to be detected (for example, the rotation matrix representing the corresponding relationship between the lidar camera and the RGB camera), the densified sparse depth map containing semantic information and / or visual information can be converted into the target depth map under the view of the visual camera. The implementation process is as follows:

[0083] Through the internal parameters of the camera system, the coordinates of the dense depth map containing semantic information and / or visual information in the lidar camera coordinate system are converted into the coordinates in the world coordinate system; through the external calibration parameters, the coordinates in the world coordinate system are converted into the coordinates of the depth map under the view of the RGB camera to obtain the target depth map.

[0084] In this embodiment, first densifying the sparse depth map can obtain more depth points, making the depth information in the image more complete, and then aligning the dense depth map with the RGB image can improve the accuracy of detection.

[0085] Among them, the sparse depth without semantic information of the lidar is transformed into a densified depth containing semantic or visual information through algorithms including depth densification, and it is easier to implement the external parameter failure judgment and information fusion with RGB.

[0086] In a possible design, determining whether the current calibrated external parameter fails according to the target depth map and the visual image may include:

[0087] Performing foreground object detection and segmentation on the target depth map to obtain a first mask;

[0088] Performing foreground object detection and segmentation on the visual image to obtain a second mask;

[0089] Determining whether the current calibrated external parameter fails according to the edge contours of the first mask and the second mask by calculating the overlap rate or difference between the first mask and the second mask.

[0090] In this embodiment, foreground object segmentation method (for example, object segmentation based on deep learning method) is used to perform foreground object segmentation on the densified depth map projected onto the RGB view to obtain a first mask, which is referred to as foreground Mask1 here. Foreground object segmentation method (for example, object segmentation based on deep learning method) is used to perform foreground object segmentation in the RGB image to obtain a second mask, which is referred to as foreground Mask2 here. Compare the coincidence and alignment of the edge contours of Mask1 and Mask2. Among them, the coincidence judgment algorithms include, but are not limited to, common segmentation performance indicators and methods such as the calculation method of IoU and Dice coefficient. If the coincidence error (or overlap error) is less than the threshold t, then the external parameter (here referring to the calibration parameter T) is valid. On the contrary, that is, there is a dislocation phenomenon, indicating that there is a large error in the external parameter calibration result, and the external parameter fails (or is invalid) and cannot be used.

[0091] Among them, the foreground object detection uses the densified depth map as the input and does not use visual information features such as RGB. It performs the detection and segmentation of foreground objects based on the continuity and consistency of depth values, including but not limited to the depth image object segmentation based on traditional algorithms and the foreground object detection and segmentation of images based on deep learning methods. It can also use deep learning-based methods to segment foreground objects in the dense depth map and complete the extraction of semantic information on the depth map, which can further improve and optimize the accuracy of the final detection failure. In this embodiment, the method for online detecting whether the external parameters are invalid does not require using information between multiple frames to determine whether the RGB and the dense depth map are spatially aligned, and can relatively easily implement real-time online calibration of external parameter failure detection and give real-time feedback results. When the external parameters are correct, it can be used as a self-detection program. When the external parameters are incorrect, it can be used as a failure detection and give corresponding feedback and prompts. Furthermore, it ensures that the user can timely understand the status of the device.

[0092] By leveraging the densified depth map and RGB map, and using the information of foreground segmentation to detect whether the RGB camera and the lidar camera (or depth cameras such as Lidar, spot iToF, spot dToF, etc.) are completely aligned, using the results of semantic segmentation, the detection accuracy is relatively high, and the detection algorithm has good robustness.

[0093] In a possible design, determining whether the current calibrated external parameters are invalid by calculating the overlap rate or difference between the first mask and the second mask according to the edge contours of the first mask and the edge contours of the second mask includes:

[0094] Respectively determine the intersection and union of the edge contours of the first mask and the edge contours of the second mask;

[0095] Take the ratio of the intersection and the union as the coincidence rate of the first mask and the second mask, and / or take the value of one minus the coincidence rate as the difference value;

[0096] If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, then determine that the current calibrated external parameters are invalid; where the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

[0097] In this embodiment, the Mask is based on segmentation. Mask1 is the segmentation Mask implemented on the dense depth map, and Mask2 is the segmentation Mask implemented on the RGB / grayscale map. Therefore, the calculation method for comparing the overlapping areas or differences between the two Masks can compare the IoU of Mask1 and Mask2, which is used to measure the fitting degree of the two frames. Among them, the overlap rate here refers to the coincidence rate.

[0098] Specifically, in combination with Figure 4 As shown, the calculation method of IoU: By using the ratio of the intersection to the union of two bounding boxes, the intersection over union can be obtained. Among them, A is the ground truth box and B is the predicted box. The formula is as follows:

[0099]

[0100] Among them, the value range of IoU is [0, 1]. The larger the IoU value, the better the coincidence of the two boxes. For IoU, the judgment threshold is usually taken as 0.5. For example, when the threshold t = 0.5, it is used to determine whether the predicted box is correct or incorrect. When the IoU of the two boxes is greater than 0.5, it is considered an effective detection (here it means the extrinsic parameters are effective), otherwise it belongs to an invalid match (here it means the extrinsic parameters are invalid).

[0101] In a possible design, determining whether the current calibrated extrinsic parameters are invalid by calculating the overlap rate or difference between the first mask and the second mask according to the edge contours of the first mask and the edge contours of the second mask includes:

[0102] According to the edge contours of the first mask and the edge contours of the second mask, determine the number of elements in the intersection between the edge contours of the first mask and the edge contours of the second mask, the number of elements in the edge contour of the first mask, and the number of elements in the edge contour of the second mask;

[0103] Take the ratio of twice the number of elements in the intersection to the sum of the number of elements in the edge contour of the first mask and the number of elements in the edge contour of the second mask as the coincidence rate of the first mask and the second mask, and / or, take the value obtained by subtracting the coincidence rate from 1 as the difference value;

[0104] If the coincidence rate is less than the first preset threshold, and / or, the difference value is greater than the second preset threshold, then determine that the current calibrated extrinsic parameters are invalid; where the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

[0105] In this embodiment, the index Dice coefficient. The Dice coefficient is a function for measuring the similarity of sets and is usually used to calculate the similarity of two samples, with a value range of [0, 1]. The formula is as follows:

[0106]

[0107] Among them, |X∩Y| is the number of elements in the intersection between X and Y, |X| and |Y| respectively represent the number of elements in X and Y, where X represents the edge contour of the first mask, and Y represents the edge contour of the second mask. The value of s can be used to judge the overlapping degree of the two masks. The overlapping rate here refers to the coincidence rate.

[0108] Among them, the calculation method of IoU and the calculation method of Dice coefficient can also be used in combination, and the specific algorithm for failure detection is not limited here. It should be noted that the setting of the threshold can be determined according to experimental results and empirical values, so as to accurately give when the calibration parameters are correctly available and when the calibration parameters fail and are unavailable. Most of the reasons for the failure of the external parameters are that the equipment is impacted, shocked, dropped, collided, etc., resulting in the unavailability of the calibration parameters. Therefore, the diagnostic conclusion obtained according to the judgment threshold has a high accuracy.

[0109] In order to further improve the detection accuracy and precision, a combination of multiple judgment methods can be used. Therefore, the methods for judging the alignment of edge contours include but are not limited to using foreground object detection and segmentation, edge contour extraction, and a combination of multiple methods. For example: perform foreground object detection and segmentation on the densified depth map, and at the same time perform operations such as detection, extraction, and post-processing of the edge contours on the foreground segmentation result to obtain a closed edge contour detection region. Similarly, perform foreground object detection and segmentation in the RGB image, and at the same time perform operations such as detection, extraction, and post-processing of the edge contours in the RGB image to obtain a closed edge contour detection region. When the judgment of the overlapping region of the segmented masks (here referring to the calculation result of IoU) and the overlapping region of the edge contour detection (here referring to the calculation result of Dice coefficient) both meet the threshold conditions, then it is determined whether the current calibration external parameter to be detected is correct. On the one hand, the combination of multiple judgment conditions can improve the detection accuracy and precision. On the other hand, it can give a gradient-based grading of the accuracy of whether the external parameter is correct, for example: the current external parameter is correct, available, average, unavailable, incorrect, etc., more multi-level judgments, which is convenient for users to decide whether to continue using it or send it to after-sales for repair, etc. It realizes the improvement of the accuracy and precision of the detection method, and thus improves the user experience.

[0110] In a possible design, the method further includes:

[0111] After determining that the current calibration external parameter fails, the device is used to prompt that the calibration external parameter fails.

[0112] In this embodiment, after determining that the current calibration external parameter fails, the device can prompt whether the external parameter is valid or invalid by displaying or broadcasting a prompt message, so as to ensure that the user can timely understand the state of the device. The output form of the result of whether the external parameter is valid is not specifically limited here.

[0113] Among them, after determining that the external parameters have failed, the external parameters can be coarsely calibrated again, and then the online failure detection method can be used to determine whether the coarsely calibrated external parameters are correct. Exemplarily, by optimizing the segmentation algorithm and results, the online calibration and correction of the sparse depth map and RGB can be realized. Whether the calibration parameters are accurate depends on the coincidence of two masks. If the two masks perfectly coincide or are far higher than the coincidence threshold, then it can be considered that the corrected parameters are correct.

[0114] Therefore, by using the sparse depth map to perform real-time online self-detection on whether the external parameters of the device are valid (correct), and using the above-mentioned various judgment methods (the calculation method of IoU or the calculation method of Dice coefficient, which can be in a combined manner), the detection accuracy of whether the external parameters are effective can be greatly improved, and the error result of the calibration parameters can be given. Thus, the external parameter calibration of the camera system composed of the depth camera and the vision camera can be realized online on the device. The process of external parameter calibration is greatly simplified, changing from the original offline production line calibration to online calibration on the terminal device.

[0115] To implement the calibration parameter failure detection method, this embodiment provides a calibration parameter failure detection device. The calibration parameter failure detection device is used to online detect whether the calibration external parameters of the camera system fail. The camera system includes a vision camera and an active depth camera. The calibration external parameters include the transformation matrix between the vision camera and the active depth camera. The calibration parameter failure detection device and the camera system are configured in the same device.

[0116] See Figure 5 , Figure 5 is a schematic structural diagram of the calibration parameter failure detection device provided by the embodiment of the present application; the calibration parameter failure detection device includes: an acquisition module 501, a projection module 502, and a detection module 503.

[0117] Among them, the acquisition module 501 is used to acquire the sparse depth map obtained by the active depth camera currently collecting the target object, and acquire the visual image obtained by the vision camera shooting the target object at the same moment;

[0118] The projection module 502 is used to respond to the trigger operation for detecting the calibration external parameters, and determine the target depth map in the visual camera view corresponding to the sparse depth map according to the calibration external parameters to be detected;

[0119] The detection module 503 is used to determine whether the current calibration external parameters fail according to the target depth map and the visual image.

[0120] In this embodiment, through the acquisition module 501, the projection module 502, and the detection module 503, it is used to obtain the sparse depth map obtained by the active depth camera for the current acquisition of the target object, and obtain the visual image obtained by the visual camera for photographing the target object at the same moment; in response to the trigger operation for detecting the calibration external parameters, according to the calibration external parameters to be detected, determine the target depth map under the visual camera view corresponding to the sparse depth map; then, according to the target depth map and the visual image, determine whether the current calibration external parameters are invalid. Therefore, based on the currently acquired sparse depth map and visual image, when the detection of the calibration external parameters is triggered, the sparse depth map and the visual image are aligned, realizing the online (during the device usage process) detection of the invalidation of the calibration external parameters, without the need for offline detection, without affecting the current operation of the device equipped with this camera system, and being able to effectively detect online whether the external parameters are invalid.

[0121] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here in this embodiment.

[0122] In a possible design, the visual camera includes any one of the following: RGB camera, micro camera, infrared camera, grayscale camera; the active depth camera includes any one of the following: lidar camera, ToF camera, structured light camera.

[0123] In a possible design, the projection module 502 is specifically used for:

[0124] Densify the sparse depth map to obtain a dense depth map containing semantic information and / or visual information;

[0125] According to the calibration external parameters, project the dense depth map into the visual camera view to obtain a target depth map containing semantics and / or visual information.

[0126] In a possible design, the device further includes: a trigger module; the trigger module is used for:

[0127] Before the response to the trigger operation for detecting the calibration external parameters, if one or more preset semantic targets are detected in both the dense depth map and the visual image, then trigger the operation for detecting the calibration external parameters.

[0128] In a possible design, the detection module includes: a first detection unit and a second detection unit;

[0129] The first detection unit is used to perform foreground target detection and segmentation on the target depth map to obtain a first mask;

[0130] The first detection unit is further configured to perform foreground object detection and segmentation on the visual image to obtain a second mask;

[0131] The second detection unit is configured to determine whether the current calibrated external parameters are invalid by calculating the overlap rate or difference between the first mask and the second mask according to the edge contours of the first mask and the second mask.

[0132] In a possible design, the second detection unit is specifically configured to:

[0133] Determine the intersection and union of the edge contours of the first mask and the edge contours of the second mask respectively;

[0134] Use the ratio of the intersection to the union as the coincidence rate of the first mask and the second mask, and / or use the value obtained by subtracting the coincidence rate from 1 as the difference value;

[0135] If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, it is determined that the current calibrated external parameters are invalid; wherein, the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

[0136] In a possible design, the second detection unit is specifically configured to:

[0137] According to the edge contours of the first mask and the edge contours of the second mask, determine the number of elements in the intersection between the edge contours of the first mask and the edge contours of the second mask, the number of elements in the edge contour of the first mask, and the number of elements in the edge contour of the second mask;

[0138] Use the ratio of twice the number of elements in the intersection to the sum of the number of elements in the edge contour of the first mask and the number of elements in the edge contour of the second mask as the coincidence rate of the first mask and the second mask, and / or use the value obtained by subtracting the coincidence rate from 1 as the difference value;

[0139] If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, it is determined that the current calibrated external parameters are invalid; wherein, the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

[0140] In a possible design, the device further includes: a prompt module; the prompt module is configured to:

[0141] After determining that the current calibrated external parameters are invalid, prompt the invalidation of the calibrated external parameters through the device.

[0142] To implement the method of the above embodiments, this embodiment provides an electronic device. Figure 6 It is a schematic structural diagram of the electronic device provided in the embodiments of the present application. As Figure 6 shown, the electronic device 60 of this embodiment includes: a processor 601 and a memory 602; among them, the memory 602 is used to store computer-executable instructions; the processor 601 is used to execute the computer-executable instructions stored in the memory to implement each step executed in the above embodiments. For details, reference can be made to the relevant descriptions in the foregoing method embodiments.

[0143] The embodiments of the present application also provide a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the above method is implemented.

[0144] The embodiments of the present application also provide a computer program product, including a computer program, which implements the above method when executed by the processor.

[0145] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, indirect couplings or communication connections of devices or modules, and can be in electrical, mechanical or other forms. In addition, in each embodiment of the present application, the functional modules can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in a unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0146] The integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above-mentioned software functional modules stored in a storage medium include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute some steps of the methods of the various embodiments of the present application. It should be understood that the above-mentioned processor may be a central processing unit (English: Central Processing Unit, abbreviated as: CPU), or may also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated as: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated as: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by a hardware processor, or can be executed and completed by a combination of hardware and software modules in the processor.

[0147] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus. The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0148] An exemplary storage medium is coupled to a processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a master device.

[0149] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting the failure of calibration parameters, characterized in that, it is applied to a calibration parameter failure detection device for online detecting whether the external calibration parameters of a camera system fail. The camera system includes a vision camera and an active depth camera, and the external calibration parameters include a transformation matrix between the vision camera and the active depth camera. The calibration parameter failure detection device and the camera system are configured in the same device. The method includes: obtaining a sparse depth map obtained by the active depth camera currently capturing a target object, and obtaining a vision image obtained by the vision camera capturing the target object at the same moment; in response to a trigger operation for detecting the external calibration parameters, determining a target depth map in the visual camera view corresponding to the sparse depth map according to the external calibration parameters to be detected; determining whether the current external calibration parameters fail according to the target depth map and the vision image.

2. The method according to claim 1, characterized in that, the determining a target depth map in the visual camera view corresponding to the sparse depth map according to the external calibration parameters to be detected includes: densifying the sparse depth map to obtain a dense depth map containing semantic information and / or visual information; projecting the dense depth map to the visual camera view according to the external calibration parameters to obtain a target depth map containing semantics and / or visual information.

3. The method according to claim 2, characterized in that, before the responding to the trigger operation for detecting the external calibration parameters, the method further includes: if one or more preset semantic targets are detected in both the dense depth map and the vision image, triggering the operation for detecting the external calibration parameters.

4. The method according to any one of claims 1-3, characterized in that, the determining whether the current external calibration parameters fail according to the target depth map and the vision image includes: performing foreground object detection and segmentation on the target depth map to obtain a first mask; performing foreground object detection and segmentation on the vision image to obtain a second mask; determining whether the current external calibration parameters fail according to the edge contours of the first mask and the second mask by calculating the overlap rate or difference between the first mask and the second mask.

5. The method according to claim 4, characterized in that, the determining whether the current external calibration parameters fail according to the edge contours of the first mask and the second mask by calculating the overlap rate or difference between the first mask and the second mask includes: respectively determining the intersection and union of the edge contours of the first mask and the edge contours of the second mask; taking the ratio of the intersection and the union as the coincidence rate of the first mask and the second mask, and / or taking the value obtained by subtracting the coincidence rate from one as the difference value; If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, it is determined that the current calibrated external parameter fails; wherein, the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

6. The method according to claim 4, wherein, the determining whether the current calibrated external parameter fails by calculating the overlap rate or difference between the first mask and the second mask according to the edge contours of the first mask and the second mask includes: determining the number of elements in the intersection between the edge contour of the first mask and the edge contour of the second mask, the number of elements in the edge contour of the first mask, and the number of elements in the edge contour of the second mask according to the edge contour of the first mask and the edge contour of the second mask; taking the ratio of twice the number of elements in the intersection to the sum of the number of elements in the edge contour of the first mask and the number of elements in the edge contour of the second mask as the coincidence rate of the first mask and the second mask, and / or taking the value obtained by subtracting the coincidence rate from 1 as the difference value; If the coincidence rate is less than a first preset threshold, and / or the difference value is greater than a second preset threshold, it is determined that the current calibrated external parameter fails; wherein, the first preset threshold is greater than or equal to the second preset threshold, and the sum of the first preset threshold and the second preset threshold is 1.

7. The method according to any one of claims 1-3, wherein, the method further includes: after determining that the current calibrated external parameter fails, prompting the failure of the calibrated external parameter through the device.

8. A calibration parameter failure detection device, wherein, the calibration parameter failure detection device is used to online detect whether the calibrated external parameter of a camera system fails. The camera system includes a vision camera and an active depth camera, and the calibrated external parameter includes a conversion matrix between the vision camera and the active depth camera. The calibration parameter failure detection device and the camera system are configured in the same device; the device includes: an acquisition module, configured to acquire a sparse depth map obtained by the active depth camera currently capturing a target object, and acquire a vision image obtained by the vision camera capturing the target object at the same moment; a projection module, configured to, in response to a trigger operation for detecting the calibrated external parameter, determine a target depth map corresponding to the sparse depth map from the perspective of the vision camera according to the calibrated external parameter to be detected; a detection module, configured to determine whether the current calibrated external parameter fails according to the target depth map and the vision image.

9. The device according to claim 8, wherein, the vision camera includes any one of the following: RGB camera, micro camera, infrared camera, grayscale camera; the active depth camera includes any one of the following: lidar camera, ToF camera, structured light camera.

10. An electronic device, wherein, includes: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images