Calibration block corner detection method and device based on deep learning and medium

Abstract

The invention relates to a calibration block angular point detection method and device based on deep learning and a medium, and the method comprises the steps: collecting a calibration block image which is shot by a vehicle-mounted camera and is used for carrying out camera external parameter calibration, marking points and edges forming a calibration block, and constructing a training set through the marked calibration block image; constructing calibration blocks to form a line segment detection model, and training the model through a training set; inputting a to-be-detected calibration block image into the trained model to obtain a candidate line segment set contained in the to-be-detected calibration block image; selecting four line segments L, L1, L2 and L3 which can enclose a calibration block from the candidate line segment set; and taking four corner points of a quadrangle defined by the four line segments as corner points of the calibration block in the calibration block image to be detected. According to the method, the corner detection precision is improved, and better robustness is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a calibration block corner point detection method, device and medium based on deep learning. Background Art

[0002] At present, computer vision technology has been widely used in various advanced driver assistance systems. As an important function, the panoramic view system can help drivers eliminate visual blind spots to the greatest extent and improve driving safety.

[0003] External parameter calibration is an important part of panoramic surround image generation. It uses the corner detection algorithm to extract various characteristic corner points in the calibration site for modeling, and uses the calibration algorithm to solve the camera's pose parameters for stitching to generate panoramic images. The robustness and detection accuracy of the corner detection algorithm determine the success rate and accuracy of external parameter calibration. The existing methods have the following technical defects:

[0004] (1) In offline calibration scenarios, corner points may not be detected correctly due to complex background or light interference. In after-sales calibration scenarios, environmental interference is even more complex. Traditional algorithms based on segmentation or edge detection cannot cover various scenarios well.

[0005] (2) Traditional corner detection algorithms are easily affected by environmental interference, resulting in poor feature extraction and affecting calibration accuracy;

[0006] (3) Developing different corner detection methods for different sites is time-consuming and inefficient. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a calibration block corner point detection method, terminal device and storage medium based on deep learning.

[0008] The specific plan is as follows:

[0009] A method for detecting corner points of a calibration block based on deep learning comprises the following steps:

[0010] S1: Collect the calibration block images taken by the vehicle camera for camera extrinsic calibration, and mark the points and edges that make up the calibration block, and construct a training set through the marked calibration block images;

[0011] S2: Construct a line segment detection model composed of calibration blocks and train the model through the training set;

[0012] The model includes a feature extraction module, a candidate line segment detection module and a line segment extraction module, where:

[0013] The feature extraction module receives the model input image and performs feature extraction, and outputs the corresponding feature map;

[0014] The candidate line segment detection module includes an endpoint detection unit, a line segment detection unit, and a line segment and endpoint matching unit; the endpoint detection unit receives the feature map output by the feature extraction module and outputs the detected endpoints; the line segment detection unit receives the feature map output by the feature extraction module and outputs the detected line segments; the line segment and endpoint matching unit receives the endpoints output by the endpoint detection unit and the line segments output by the line segment detection unit, and matches the endpoints with the line segments based on the Euclidean distance between the two endpoints of the endpoint and the line segment, and outputs a line segment whose two endpoints can both match the input endpoints;

[0015] After receiving the line segment set output by the line segment detection unit and the feature map output by the feature extraction module, the line segment extraction module uniformly samples the line segment set and calculates the features of the sampling points in the feature map. After inputting all the calculated features into the classifier composed of the fully connected layer and the sigmoid activation function layer, the candidate line segment set contained in the model input image is output;

[0016] S3: After inputting the image of the calibration block to be detected into the trained model, a set of candidate line segments contained in the image of the calibration block to be detected is obtained;

[0017] S4: Select four line segments L, L1, L2 and L3 from the candidate line segment set, and the four line segments satisfy: the distance between the first endpoint of line segment L1 and the first endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L2 and the second endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L3 and the second endpoint of line segment L1 is less than the first distance threshold, the distance between the second endpoint of line segment L3 and the second endpoint of line segment L2 is less than the first distance threshold, and the difference between the distance between the first endpoint of line segment L1 and the first endpoint of line segment L2 and the distance between the second endpoint of line segment L1 and the second endpoint of line segment L2 is less than the second distance threshold;

[0018] S5: taking the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected.

[0019] Furthermore, the vehicle-mounted camera adopts a fisheye camera.

[0020] Furthermore, the annotation content includes a coordinate set of four points constituting the calibration block and a coordinate set of four edges constituting the calibration block.

[0021] Furthermore, step S1 also includes converting the marked data using a Bezier curve representation method.

[0022] Furthermore, before performing step S4, it is also included to determine whether the number of candidate line segments included in the candidate line segment set is less than a minimum number threshold. If so, it is determined that corner point detection cannot be performed, and step S4 is no longer performed, and the process ends.

[0023] Furthermore, the model also outputs the score of each candidate line segment when outputting the candidate line segment set; before performing step S4, it also includes filtering out the line segments in the candidate line segment set whose scores are less than the score threshold, and then judging whether the number of remaining line segments is less than the minimum number threshold. If so, it is determined that corner point detection cannot be performed, and step S4 is no longer performed, and the process ends.

[0024] Furthermore, before taking the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected in step S3, it also includes judging whether the quadrilateral enclosed by the line segments L, L1, L2 and L3 meets the requirements of area and grayscale value. If both are satisfied, the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 are taken as the corner points of the calibration block in the calibration block image to be detected.

[0025] A terminal device for detecting corner points of calibration blocks based on deep learning comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in the embodiment of the present invention are implemented.

[0026] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0027] The present invention adopts the above technical solution to improve the accuracy of corner point detection and has better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Shown is a flow chart of a method according to a first embodiment of the present invention.

[0029] Figure 2 Shown is a schematic diagram of the line segment L selected in this embodiment.

[0030] Figure 3 Shown is a schematic diagram of line segments L1 and L2 selected in this embodiment.

[0031] Figure 4 Shown is a schematic diagram of the line segment L3 selected in this embodiment. DETAILED DESCRIPTION

[0032] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0033] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0034] Embodiment 1:

[0035] The embodiment of the present invention provides a method for detecting corner points of a calibration block based on deep learning. Figure 1 As shown, the method comprises the following steps:

[0036] S1: Collect the calibration block images taken by the vehicle-mounted camera for camera extrinsic calibration, and annotate the points and edges that make up the calibration block, and construct a training set through the annotated calibration block images.

[0037] In this embodiment, the vehicle-mounted camera adopts a fisheye camera.

[0038] The calibration block is usually a quadrilateral, so when annotating, the obtained label should contain the coordinate set of the four points constituting the calibration block and the coordinate set of the four edges constituting the calibration block.

[0039] Furthermore, since the camera can adopt different models, the output data format may include multiple types. In order to facilitate the training and use of subsequent models, the data format is normalized in this embodiment, that is, the collected original data (coordinates of points and edges) are converted, and the Bezier curve representation method is used to convert the labeled data.

[0040] S2: Construct a line segment detection model composed of calibration blocks and train the model through the training set.

[0041] In this embodiment, the model consists of three parts: 1. Feature extraction module; 2. Candidate line segment detection module; 3. Line segment extraction module.

[0042] 1. Feature extraction module

[0043] The feature extraction module receives the model input image and performs feature extraction, uses convolutional layers and cascade networks to extract features from the model input image (RGB image), and outputs a feature map obtained by feature extraction.

[0044] 2. Candidate line segment detection module

[0045] The candidate line segment detection module consists of three parts: endpoint detection unit, line segment detection unit, and line segment and endpoint matching unit.

[0046] The endpoint detection unit receives the feature map output by the feature extraction module, grids the feature map, uses a convolutional layer to extract features from the divided grid, predicts the confidence of the endpoint in the grid and the offset relative to the center of the grid, and uses the cross entropy loss function and L1 loss function for the confidence and offset respectively.

[0047] The line segment detection unit receives the feature map output by the feature extraction module, grids the feature map, uses a convolutional layer to extract features from the divided grid, predicts the confidence of the line segment center in the grid, the offset from the line segment center to the grid center, and the offset from the average point of the line segment to the grid center, uses the cross entropy loss function for the confidence, and uses the L1 loss function for the offset.

[0048] The line segment and endpoint matching unit receives the endpoint output by the endpoint detection unit and the line segment output by the line segment detection unit, and matches the endpoint with the line segment based on the Euclidean distance between the endpoint and the two endpoints of the line segment, eliminates the wrong endpoint and line segment through matching information, and outputs a line segment whose two endpoints can match the input endpoint, that is, the distance between the endpoint of the line segment and the matched endpoint is less than the first distance threshold. The size of the first distance threshold can be set by those skilled in the art, and is not limited here.

[0049] 3. Line segment extraction module

[0050] After receiving the line segment set output by the line segment detection unit and the feature map output by the feature extraction module, the line segment extraction module uniformly samples the line segment set and calculates the features of the sampling points in the feature map. After inputting all the calculated features into the classifier composed of a fully connected layer and a sigmoid activation function layer, the candidate line segment set contained in the model input image is output.

[0051] This embodiment also outputs the score of each candidate line segment when outputting the candidate line segment set.

[0052] S3: After the calibration block image to be detected is input into the trained model, a set of candidate line segments contained in the calibration block image to be detected is obtained.

[0053] S4: Select four line segments L, L1, L2 and L3 from the candidate line segment set, and the four line segments satisfy: the distance between the first endpoint of line segment L1 and the first endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L2 and the second endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L3 and the second endpoint of line segment L1 is less than the first distance threshold, the distance between the second endpoint of line segment L3 and the second endpoint of line segment L2 is less than the first distance threshold, and the difference between the distance between the first endpoint of line segment L1 and the first endpoint of line segment L2 and the distance between the second endpoint of line segment L1 and the second endpoint of line segment L2 is less than the second distance threshold.

[0054] In order to select the above four line segments from the candidate line segment set, three steps are set in this embodiment:

[0055] Step 1: Select a line segment L from the candidate line segment set, such as Figure 2 shown.

[0056] Step 2: Select two line segments L1 and L2 from the candidate line segment set, and the two line segments satisfy: the distance between the first endpoint of line segment L1 and the first endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L2 and the second endpoint of line segment L is less than the first distance threshold, and the difference between the distance between the first endpoint of line segment L1 and the first endpoint of line segment L2 and the distance between the second endpoint of line segment L1 and the second endpoint of line segment L2 is less than the second distance threshold (used to define that line segments L1 and L2 are approximately parallel and located on the same side of line segment L), as shown in FIG. Figure 3 shown.

[0057] Step 3: Select a line segment L3 from the candidate line segment set, which satisfies: the distance between the first endpoint of the line segment and the second endpoint of the line segment L1 and the distance between the second endpoint of the line segment and the second endpoint of the line segment L2 are both less than the first distance threshold, such as Figure 4 shown.

[0058] If the selection in step 2 and step 3 cannot be completed based on any line segment L, it means that the selected line segment is not a line segment constituting the calibration block, and then a new line segment is selected from the candidate line segment set.

[0059] Furthermore, before performing step S4, it is also included to determine whether the number of candidate line segments included in the candidate line segment set is less than a minimum number threshold. If so, it is determined that corner point detection cannot be performed, and step S4 is not performed, and the process ends. In this embodiment, the minimum number threshold is set to 4.

[0060] In addition, since the model in this embodiment also outputs the score of each candidate line segment when outputting the candidate line segment set, before performing step S4, it also includes filtering out the line segments in the candidate line segment set whose scores are less than the score threshold, and then judging whether the number of remaining line segments is less than the minimum number threshold. If so, it is judged that corner point detection cannot be performed, and step S4 is no longer performed, and the process ends. The size of the score threshold can be set by a person skilled in the art according to experimental data, and is not limited here.

[0061] S5: taking the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected.

[0062] When storing corner points, the coordinates of each corner point need to be stored.

[0063] Before using the four corner points of the quadrilateral surrounded by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected in step S3, it also includes judging whether the quadrilateral surrounded by the line segments L, L1, L2 and L3 meets the requirements of area and grayscale value. If both are satisfied, the four corner points of the quadrilateral surrounded by the line segments L, L1, L2 and L3 are used as the corner points of the calibration block in the calibration block image to be detected. Meeting the area requirement means that the area of ​​the enclosed quadrilateral meets the set area range, and meeting the grayscale value requirement means that the grayscale value of each pixel point in the enclosed quadrilateral meets the set grayscale value range. The size of the area range and the grayscale value range can be set by a person skilled in the art according to experimental data, and no limitation is made here.

[0064] The method for corner detection based on deep learning adopted in the embodiment of the present invention has good robustness and can cover various scenarios including offline calibration and after-sales calibration. In addition, the deep learning neural network can directly realize the extraction of sub-pixel corner points to ensure the accuracy of detection.

[0065] Embodiment 2:

[0066] The present invention also provides a calibration block corner point detection terminal device based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of the first embodiment of the present invention when executing the computer program.

[0067] Further, as an executable solution, the calibration block corner point detection terminal device based on deep learning can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The calibration block corner point detection terminal device based on deep learning may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the calibration block corner point detection terminal device based on deep learning is only an example of the calibration block corner point detection terminal device based on deep learning, and does not constitute a limitation on the calibration block corner point detection terminal device based on deep learning. It may include more or less components than the above, or a combination of certain components, or different components. For example, the calibration block corner point detection terminal device based on deep learning may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.

[0068] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the calibration block corner point detection terminal device based on deep learning, and uses various interfaces and lines to connect various parts of the entire calibration block corner point detection terminal device based on deep learning.

[0069] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the calibration block corner point detection terminal device based on deep learning by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0070] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0071] If the module / unit integrated in the terminal device for detecting corner points of calibration blocks based on deep learning is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0072] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A method for detecting corner points of a calibration block based on deep learning, characterized in that: The following steps are involved: S1: Collect the calibration block images taken by the vehicle camera for camera extrinsic calibration, and mark the points and edges that make up the calibration block, and construct a training set through the marked calibration block images; S2: Construct a line segment detection model composed of calibration blocks and train the model through the training set; The model includes a feature extraction module, a candidate line segment detection module and a line segment extraction module, where: The feature extraction module receives the model input image and performs feature extraction, and outputs the corresponding feature map; The candidate line segment detection module includes an endpoint detection unit, a line segment detection unit, and a line segment and endpoint matching unit; the endpoint detection unit receives the feature map output by the feature extraction module and outputs the detected endpoints; the line segment detection unit receives the feature map output by the feature extraction module and outputs the detected line segments; the line segment and endpoint matching unit receives the endpoints output by the endpoint detection unit and the line segments output by the line segment detection unit, and matches the endpoints with the line segments based on the Euclidean distance between the two endpoints of the endpoint and the line segment, and outputs a line segment whose two endpoints can both match the input endpoints; After receiving the line segment set output by the line segment detection unit and the feature map output by the feature extraction module, the line segment extraction module uniformly samples the line segment set and calculates the features of the sampling points in the feature map. After inputting all the calculated features into the classifier composed of the fully connected layer and the sigmoid activation function layer, the candidate line segment set contained in the model input image is output; S3: After inputting the image of the calibration block to be detected into the trained model, a set of candidate line segments contained in the image of the calibration block to be detected is obtained; S4: Select four line segments L, L1, L2 and L3 from the candidate line segment set, and the four line segments satisfy: the distance between the first endpoint of line segment L1 and the first endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L2 and the second endpoint of line segment L is less than the first distance threshold, the distance between the first endpoint of line segment L3 and the second endpoint of line segment L1 is less than the first distance threshold, the distance between the second endpoint of line segment L3 and the second endpoint of line segment L2 is less than the first distance threshold, and the difference between the distance between the first endpoint of line segment L1 and the first endpoint of line segment L2 and the distance between the second endpoint of line segment L1 and the second endpoint of line segment L2 is less than the second distance threshold; S5: taking the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected.

2. The method for detecting corner points of a calibration block based on deep learning according to claim 1, characterized in that: The on-board camera uses a fisheye camera.

3. The method for detecting corner points of calibration blocks based on deep learning according to claim 1, characterized in that: The annotation content includes the coordinate sets of the four points that make up the calibration block and the coordinate sets of the four edges that make up the calibration block.

4. The method for detecting corner points of a calibration block based on deep learning according to claim 1, characterized in that: Step S1 also includes converting the marked data using a Bezier curve representation method.

5. The method for detecting corner points of calibration blocks based on deep learning according to claim 1, characterized in that: Before performing step S4, it is also included to determine whether the number of candidate line segments included in the candidate line segment set is less than a minimum number threshold. If so, it is determined that corner point detection cannot be performed, and step S4 is no longer performed, and the process ends.

6. The method for detecting corner points of a calibration block based on deep learning according to claim 1, characterized in that: When outputting the candidate line segment set, the model also outputs the score of each candidate line segment; before performing step S4, it also includes filtering out the line segments in the candidate line segment set whose scores are less than the score threshold, and then judging whether the number of remaining line segments is less than the minimum number threshold. If so, it is determined that corner point detection cannot be performed, and step S4 is no longer performed, and the process ends.

7. The method for detecting corner points of calibration blocks based on deep learning according to claim 1, characterized in that: Before taking the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 as the corner points of the calibration block in the calibration block image to be detected in step S3, it is also included to determine whether the quadrilateral enclosed by the line segments L, L1, L2 and L3 meets the requirements of area and grayscale value. If both are satisfied, the four corner points of the quadrilateral enclosed by the line segments L, L1, L2 and L3 are taken as the corner points of the calibration block in the calibration block image to be detected.

8. A calibration block corner point detection terminal device based on deep learning, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images