A calibration block corner point detection method, device and medium based on deep learning

Abstract

The application relates to a kind of calibration block corner point detection method, equipment and medium based on deep learning, the method includes: the calibration block image for camera external parameter calibration is collected for being shot by vehicle-mounted camera, and the point and edge of the component calibration block in it are marked, and training set is constructed by the calibration block image after marking;Calibration block component line segment detection model is constructed, and the model is trained by training set;After the calibration block image to be detected is input into the trained model, the candidate line segment set contained in the calibration block image to be detected is obtained;From the candidate line segment set, select four line segments L, L1, L2 and L3 that can surround calibration block;The four corners of the quadrangle surrounded by the four line segments are used as the corner points of the calibration block in the calibration block image to be detected.The application improves the precision of corner point detection and has better robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a calibration block corner point detection method based on deep learning, a device and a medium. BACKGROUND

[0002] At present, computer vision technology has been widely applied to various advanced driving assistance systems. As an important function, the panoramic surround view system can help the driver to maximize the elimination of visual blind area and improve the safety of driving.

[0003] The external parameter calibration is an important link of panoramic surround view image generation. It uses a corner point detection algorithm to extract various feature corner points in the calibration site for modeling, and uses a calibration algorithm to solve the pose parameters of the camera for splicing to generate panoramic images. The robustness and detection accuracy of the corner point detection algorithm determine the success rate and accuracy of the external parameter calibration. The existing method has the following technical defects:

[0004] (1) In the offline calibration scene, there are situations where the corner points cannot be correctly detected due to complex site background or light interference. In the after-sales calibration scene, the environmental interference is more complex. The traditional algorithm based on segmentation or edge detection cannot well cover various scenes;

[0005] (2) The traditional corner point detection algorithm is easily affected by environmental interference, which affects the calibration accuracy;

[0006] (3) Developing different corner point detection methods for different sites is time-consuming and inefficient. SUMMARY

[0007] In order to solve the above problems, the present application provides a calibration block corner point detection method based on deep learning, a terminal device and a storage medium.

[0008] The specific scheme is as follows:

[0009] A calibration block corner point detection method based on deep learning, comprising the following steps:

[0010] S1: Collecting a calibration block image for camera external parameter calibration shot by a vehicle-mounted camera, and labeling the points and edges of the calibration block, and constructing a training set through the labeled calibration block image;

[0011] S2: Constructing a calibration block component line segment detection model, and training the model through the training set;

[0012] The model includes a feature extraction module, a candidate line segment detection module and a line segment extraction module, wherein:

[0013] The feature extraction module receives a model input image and performs feature extraction, and outputs a corresponding feature map;

[0014] The candidate line segment detection module includes an endpoint detection unit, a line segment detection unit, and a line segment and endpoint matching unit. The endpoint detection unit receives the feature map output by the feature extraction module and outputs detected endpoints. The line segment detection unit receives the feature map output by the feature extraction module and outputs detected line segments. The line segment and endpoint matching unit receives the endpoints output by the endpoint detection unit and the line segments output by the line segment detection unit, matches the endpoints with the line segments based on the Euclidean distance between the two endpoints of the line segments, and outputs line segments to which both endpoints can be matched.

[0015] The line segment extraction module receives the line segment set output by the line segment detection unit and the feature map output by the feature extraction module, uniformly samples the line segment set, calculates the features of the sampling points in the feature map, inputs all the calculated features into a classifier composed of a fully connected layer and a sigmoid activation function layer, and outputs a candidate line segment set contained in the model input image.

[0016] S3: After inputting the to-be-detected calibration block image into the trained model, a candidate line segment set contained in the to-be-detected calibration block image is obtained.

[0017] S4: Four line segments L, L1, L2, and L3 are selected from the candidate line segment set, and the four line segments satisfy: the distance between the first endpoint of the line segment L1 and the first endpoint of the line segment L is less than a first distance threshold, the distance between the first endpoint of the line segment L2 and the second endpoint of the line segment L is less than the first distance threshold, the distance between the first endpoint of the line segment L3 and the second endpoint of the line segment L1 is less than the first distance threshold, the distance between the second endpoint of the line segment L3 and the second endpoint of the line segment L2 is less than the first distance threshold, and the difference between the distance between the first endpoint of the line segment L1 and the first endpoint of the line segment L2 and the distance between the second endpoint of the line segment L1 and the second endpoint of the line segment L2 is less than a second distance threshold.

[0018] S5: The four corner points of the quadrilateral surrounded by the line segments L, L1, L2, and L3 are taken as the corner points of the calibration block in the to-be-detected calibration block image.

[0019] Further, the vehicle-mounted camera adopts a fisheye camera.

[0020] Further, the annotation content includes a coordinate set of four points constituting the calibration block and a coordinate set of four edges constituting the calibration block.

[0021] Further, in step S1, the annotated data is further converted by using a Bezier curve representation method.

[0022] Further, before step S4, it is further included that whether the number of candidate line segments contained in the candidate line segment set is less than the minimum number threshold, if yes, it is determined that the corner point detection cannot be performed, and step S4 is not performed, and ends.

[0023] Further, the model further outputs the scores of the candidate line segments when outputting the candidate line segment set; before step S4, it is further included that after filtering out the line segments in the candidate line segment set with scores less than the score threshold, it is judged whether the number of the remaining line segments is less than the minimum number threshold, if yes, it is determined that the corner point detection cannot be performed, and step S4 is not performed, and ends.

[0024] Further, before step S3, it is further included that whether the quadrilateral surrounded by the line segments L, L1, L2 and L3 meets the requirements of area and gray value, if yes, the four corner points of the quadrilateral surrounded by the line segments L, L1, L2 and L3 are taken as the corner points of the calibration block in the calibration block image to be detected.

[0025] A terminal device for detecting corner points of a calibration block based on deep learning, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to the embodiments of the present application when executing the computer program.

[0026] A computer readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the steps of the method according to the embodiments of the present application.

[0027] The technical scheme is adopted, the accuracy of the corner point detection is improved, and better robustness is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Fig. 1 shows a flowchart of a method according to an embodiment of the present application.

[0029] Figure 2 Fig. 2 shows a schematic diagram of a selected line segment L in this embodiment.

[0030] Figure 3 Fig. 3 shows a schematic diagram of selected line segments L1 and L2 in this embodiment.

[0031] Figure 4 Fig. 4 shows a schematic diagram of a selected line segment L3 in this embodiment. DETAILED DESCRIPTION

[0032] To further illustrate the embodiments, the present application provides accompanying drawings. These drawings are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. With reference to these contents, those skilled in the art should understand other possible implementations and advantages of the present application.

[0033] The present application is further illustrated in conjunction with the accompanying drawings and specific embodiments.

[0034] Embodiment one:

[0035] The embodiment of the present application provides a calibration block corner point detection method based on deep learning, as shown in Figure 1 The method comprises the following steps:

[0036] S1: Collecting a calibration block image for camera extrinsic parameter calibration shot by a vehicle-mounted camera, and labeling the points and edges of the calibration block in the calibration block image, and constructing a training set through the labeled calibration block image.

[0037] In this embodiment, the vehicle-mounted camera is a fisheye camera.

[0038] The calibration block is usually quadrilateral, so when labeling, the label obtained should include the coordinate set of the four points constituting the calibration block and the coordinate set of the four edges constituting the calibration block.

[0039] Further, since the camera can adopt different models, the output data format of the camera can include multiple formats, in order to facilitate subsequent model training and use, the format of the data is normalized in this embodiment, that is, the original data (coordinates of points and edges) collected is data converted, and the data labeled is data converted using Bezier curve representation method.

[0040] S2: Constructing a calibration block component line segment detection model, and training the model through the training set.

[0041] The model in this embodiment is composed of three parts: 1, a feature extraction module; 2, a candidate line segment detection module; 3, a line segment extraction module.

[0042] 1, Feature extraction module

[0043] The feature extraction module receives the model input image and performs feature extraction, and uses convolutional layers and cascaded networks to perform feature extraction on the model input image (RGB image), and outputs the feature map obtained by feature extraction.

[0044] 2, Candidate line segment detection module

[0045] The candidate line segment detection module includes three parts: an endpoint detection unit, a line segment detection unit, and a line segment and endpoint matching unit.

[0046] The endpoint detection unit receives the feature map output by the feature extraction module, and performs grid processing on the feature map. The endpoint detection unit uses a convolutional layer to extract features from the divided grid, and predicts the confidence of the endpoint in the grid and the offset of the endpoint relative to the center of the grid. The cross-entropy loss function and the L1 loss function are used for the confidence and the offset, respectively.

[0047] The line segment detection unit receives the feature map output by the feature extraction module, and performs grid processing on the feature map. The line segment detection unit uses a convolutional layer to extract features from the divided grid, and predicts the confidence of the line segment center in the grid, the offset of the line segment center to the center of the grid, and the offset of the midpoint of the line segment to the center of the grid. The cross-entropy loss function is used for the confidence, and the L1 loss function is used for the offset.

[0048] The line segment and endpoint matching unit receives the endpoints output by the endpoint detection unit and the line segments output by the line segment detection unit, and matches the endpoints and the line segments based on the Euclidean distance between the two endpoints of the line segment. The matching information is used to eliminate incorrect endpoints and line segments. The output is a line segment that can match both endpoints, and the distance between the endpoints of the line segment and the matched endpoints is less than the first distance threshold. The size of the first distance threshold can be set by the person skilled in the art, and is not limited herein.

[0049] 3. Line segment extraction module

[0050] The line segment extraction module receives the line segment set output by the line segment detection unit and the feature map output by the feature extraction module, uniformly samples the line segment set, and calculates the features of the sampling points in the feature map. After inputting all the calculated features into the classifier composed of a fully connected layer and a sigmoid activation function layer, the model outputs the candidate line segment set contained in the input image.

[0051] The present embodiment also outputs the score of each candidate line segment when outputting the candidate line segment set.

[0052] S3: After inputting the to-be-detected calibration block image into the trained model, the candidate line segment set contained in the to-be-detected calibration block image is obtained.

[0053] S4: selecting four line segments L, L1, L2 and L3 from the candidate line segment set, and the four line segments satisfy: the distance between the first end point of the line segment L1 and the first end point of the line segment L is less than the first distance threshold, the distance between the first end point of the line segment L2 and the second end point of the line segment L is less than the first distance threshold, the distance between the first end point of the line segment L3 and the second end point of the line segment L1 is less than the first distance threshold, the distance between the second end point of the line segment L3 and the second end point of the line segment L2 is less than the first distance threshold, and the difference between the distance between the first end point of the line segment L1 and the first end point of the line segment L2 and the distance between the second end point of the line segment L1 and the second end point of the line segment L2 is less than the second distance threshold.

[0054] In order to select the above four line segments from the candidate line segment set, three steps are set in the embodiment:

[0055] Step one: selecting a line segment L from the candidate line segment set, as shown in FIG. 4. Figure 2

[0056] Step two: selecting two line segments L1 and L2 from the candidate line segment set, and the two line segments satisfy: the distance between the first end point of the line segment L1 and the first end point of the line segment L is less than the first distance threshold, the distance between the first end point of the line segment L2 and the second end point of the line segment L is less than the first distance threshold, and the difference between the distance between the first end point of the line segment L1 and the first end point of the line segment L2 and the distance between the second end point of the line segment L1 and the second end point of the line segment L2 is less than the second distance threshold (for limiting the line segments L1 and L2 to be approximately parallel and located on the same side of the line segment L), as shown in FIG. 5. Figure 3

[0057] Step three: selecting a line segment L3 from the candidate line segment set, and the line segment satisfies: the distance between the first end point of the line segment and the second end point of the line segment L1 is less than the first distance threshold, and the distance between the second end point of the line segment and the second end point of the line segment L2 is less than the first distance threshold, as shown in FIG. 6. Figure 4

[0058] If the selection of the line segment L cannot complete the selection of steps two and three, it is indicated that the selected line segment is not the line segment constituting the calibration block, and a line segment is reselected from the candidate line segment set.

[0059] Further, before step S4, it further includes judging whether the number of the candidate line segments contained in the candidate line segment set is less than a minimum number threshold, if yes, it is determined that the corner point detection cannot be performed, and step S4 is not performed, and the process is ended. In the embodiment, the minimum number threshold is set to 4.

[0060] ​​​In addition, since the model in the embodiment outputs the scores of the candidate line segments when outputting the set of candidate line segments, before step S4, the following is further included: filtering out the line segments in the set of candidate line segments whose scores are less than a score threshold, and then determining whether the number of the remaining line segments is less than a minimum number threshold. If yes, it is determined that the corner point detection cannot be performed, and step S4 is not performed, and the process ends. The size of the score threshold can be set by the person skilled in the art according to experimental data, and is not limited herein.

[0061] S5: taking the four corner points of the quadrilateral surrounded by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected.

[0062] When the corner points are stored, the coordinates of the corner points need to be stored.

[0063] Before step S3, the following is further included: determining whether the quadrilateral surrounded by the line segments L, L1, L2 and L3 meets the requirements of area and gray value. If both meet, the four corner points of the quadrilateral surrounded by the line segments L, L1, L2 and L3 are taken as the corner points of the calibration block in the calibration block image to be detected. The area requirement is that the area of the quadrilateral meets a set area range, and the gray value requirement is that the gray values of the pixels in the quadrilateral meet a set gray value range. The sizes of the area range and the gray value range can be set by the person skilled in the art according to experimental data, and are not limited herein.

[0064] The method for corner point detection based on deep learning adopted in the embodiment has good robustness, can cover various scenes including online calibration and after-sales calibration, and the deep learning neural network can directly realize extraction of sub-pixel corner points, thereby ensuring the detection accuracy.

[0065] Embodiment Two

[0066] The application further provides a terminal device for calibration block corner point detection based on deep learning, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps in the above method embodiments of the first embodiment of the application when executing the computer program.

[0067] Further, as an executable solution, the deep learning-based calibration block corner point detection terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The deep learning-based calibration block corner point detection terminal device can include, but is not limited to, a processor, a memory. Those skilled in the art can understand that the above-mentioned composition structure of the deep learning-based calibration block corner point detection terminal device is only an example of the deep learning-based calibration block corner point detection terminal device, and does not constitute a limitation on the deep learning-based calibration block corner point detection terminal device, and can include more or fewer components than the above, or combine certain components, or different components, for example, the deep learning-based calibration block corner point detection terminal device can also include an input and output device, a network access device, a bus, and the like, and the embodiments of the present application do not limit this.

[0068] Further, as an executable solution, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like, and the processor is the control center of the deep learning-based calibration block corner point detection terminal device, and connects various parts of the deep learning-based calibration block corner point detection terminal device through various interfaces and lines.

[0069] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the deep learning-based calibration block corner point detection terminal device by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the mobile phone, and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0070] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method of the above-mentioned embodiments of the application.

[0071] The module / unit integrated with the deep learning-based calibration block corner detection terminal equipment can be stored in a computer readable storage medium if it is realized in the form of a software function unit and sold or used as an independent product. Based on this understanding, all or part of the processes in the above-mentioned embodiments of the method of the application can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0072] Although the application is specifically shown and described in connection with the preferred embodiments, those skilled in the art should understand that various changes in form and details can be made to the application without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A method for detecting a corner point of a calibration block based on deep learning, characterized in that, The method comprises the following steps: S1: collecting a calibration block image photographed by a vehicle-mounted camera for camera extrinsic parameter calibration, and labeling points and edges of the calibration block in the calibration block image, and constructing a training set by using the labeled calibration block image; S2: constructing a line segment detection model of the calibration block, and training the model by using the training set; The model comprises a feature extraction module, a candidate line segment detection module, and a line segment extraction module, wherein: The feature extraction module receives a model input image and performs feature extraction, and outputs a corresponding feature mapping image; The candidate line segment detection module comprises an endpoint detection unit, a line segment detection unit, and a line segment and endpoint matching unit; the endpoint detection unit receives the feature mapping image output by the feature extraction module, and outputs detected endpoints; the line segment detection unit receives the feature mapping image output by the feature extraction module, and outputs detected line segments; the line segment and endpoint matching unit receives the endpoints output by the endpoint detection unit and the line segments output by the line segment detection unit, matches the endpoints with the line segments based on the Euclidean distance between the two endpoints of the line segments, and outputs line segments to which both endpoints can be matched in the input; The line segment extraction module receives the line segment set output by the line segment detection unit and the feature mapping image output by the feature extraction module, uniformly samples the line segment set, calculates the features of the sampling points in the feature mapping image, inputs all the calculated features into a classifier composed of a full connection layer and a sigmoid activation function layer, and outputs a candidate line segment set contained in the model input image; S3: inputting a to-be-detected calibration block image into the trained model to obtain a candidate line segment set contained in the to-be-detected calibration block image; S4: selecting four line segments L, L1, L2, and L3 from the candidate line segment set, and the four line segments satisfy: the distance between the first endpoint of the line segment L1 and the first endpoint of the line segment L is less than a first distance threshold, the distance between the first endpoint of the line segment L2 and the second endpoint of the line segment L is less than the first distance threshold, the distance between the first endpoint of the line segment L3 and the second endpoint of the line segment L1 is less than the first distance threshold, the distance between the second endpoint of the line segment L3 and the second endpoint of the line segment L2 is less than the first distance threshold, and the difference between the distance between the first endpoint of the line segment L1 and the first endpoint of the line segment L2 and the distance between the second endpoint of the line segment L1 and the second endpoint of the line segment L2 is less than a second distance threshold; S5: taking the four corner points of the quadrilateral surrounded by the line segments L, L1, L2, and L3 as the corner points of the calibration block in the to-be-detected calibration block image.

2. The deep learning-based calibration block corner point detection method according to claim 1, wherein: The vehicle-mounted camera is a fisheye camera.

3. The deep learning based calibration block corner point detection method of claim 1, wherein: The labeling content comprises a coordinate set of four points constituting the calibration block and a coordinate set of four edges constituting the calibration block.

4. The deep learning-based calibration block corner point detection method according to claim 1, characterized in that: In step S1, the labeled data is further converted by using a Bezier curve representation method.

5. The deep learning based calibration block corner point detection method according to claim 1, wherein: Before step S4, it is further determined whether the number of candidate line segments contained in the candidate line segment set is less than a minimum number threshold, if yes, it is determined that the corner point detection cannot be performed, and step S4 is not performed, and the process is ended.

6. The deep learning based calibration block corner point detection method according to claim 1, characterized in that: The model outputs scores of the candidate line segments when outputting the set of candidate line segments; before step S4, it further includes filtering out line segments with scores less than a score threshold from the set of candidate line segments, and then determining whether the number of remaining line segments is less than a minimum number threshold; if yes, it is determined that corner point detection cannot be performed, and step S4 is not performed, and the process ends.

7. The deep learning based calibration block corner point detection method according to claim 1, characterized in that: Before step S3, it further includes determining whether the quadrilateral formed by the line segments L, L1, L2 and L3 meets the requirements of area and gray value; if yes, the four corner points of the quadrilateral formed by the line segments L, L1, L2 and L3 are taken as the corner points of the calibration block in the calibration block image to be detected. 8.A terminal device for calibrating a block corner point based on deep learning, characterized in that: A computer program product comprising a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1-7 when running the computer program.

9. A computer readable storage medium, the computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-7.

Images