Corner point detection method and corner point detection model training method and device

Abstract

The embodiment of the invention provides a corner detection method and a corner detection model training method and device, and belongs to the technical field of computer vision. The method comprises the following steps: processing an original image to obtain a canvas image; candidate angular points and confidence coefficients are obtained through the trained angular point detection model according to the canvas image, the angular point detection model adopts a target loss function based on angular point regression, and the candidate angular points are in one-to-one correspondence with the confidence coefficients; and performing non-maximum suppression on the candidate angular point according to the distance threshold value and the confidence coefficient to obtain a target angular point. According to the embodiment of the invention, redundant angular points can be removed, and the angular point detection precision is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a corner detection method, a corner detection model training method and a device. Background Art

[0002] In the related technology, camera calibration by detecting the corner points of the chessboard calibration plate is a common camera calibration method. In the prior art, there are methods based on image grayscale changes and local features for detection, such as Harris, SUSAN, etc., and there are also methods based on neural network models combined with screening rules for detection. The above methods have the following technical problems:

[0003] When the camera pixel is low or the calibration plate in the image is small, there will be blurred pixel blocks at the local corners of the target, which makes it easy for the algorithm to detect redundant points in the area near the target.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a corner point detection method, a corner point detection model training method and a device, aiming to remove redundant corner points and improve the accuracy of corner point detection.

[0006] To achieve the above objective, an embodiment of the present application provides a corner point detection method, which includes the following steps:

[0007] Processing the original image to obtain a canvas image;

[0008] Obtaining candidate corner points and confidence levels according to the canvas image through a trained corner point detection model, wherein the corner point detection model adopts a target loss function based on corner point regression, and the candidate corner points correspond to the confidence levels one by one;

[0009] Non-maximum suppression is performed on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points.

[0010] In some embodiments, the objective loss function based on corner regression is expressed as follows:

[0011]

[0012] in, is the objective loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λ coord is the loss weight of coordinate regression when the grid has real corner points, λ classis the loss weight of sub-region row classification and column classification when the grid has real corner points, S is the size of the grid, B is the number of predicted corner points of each grid, Is the indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ij (c y ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c y The probability of being in the sub-region of the column position, the grid belongs to the feature map output by the corner point detection model based on the training image, the predicted corner point, and The training image includes the real corner points by calculating based on the grid and the corner point detection model through a logic function.

[0013] In some embodiments, performing non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points includes:

[0014] Sort the candidate corner points from large to small according to the corresponding confidence levels to obtain a corner point sequence;

[0015] Taking the first candidate corner point in the corner point sequence as the current corner point;

[0016] Taking the corner point after the current corner point in the corner point sequence as a comparison corner point;

[0017] If the distance between the comparison corner point and the current corner point is less than a distance threshold, the comparison corner point is deleted from the corner point sequence.

[0018] In some embodiments, performing non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points further includes:

[0019] Taking the first corner point after the current corner point in the corner point sequence as the current corner point;

[0020] Return to the step of using the corner points after the current corner point in the corner point sequence as comparison corner points until there is no node after the current corner point.

[0021] To achieve the above objective, another aspect of an embodiment of the present application provides a corner detection model training method, the method comprising the following steps:

[0022] Label and expand the sample images to obtain training images of preset sizes;

[0023] Obtaining a feature map according to the training image through a corner detection model, wherein the feature map includes a plurality of grids, the grids include a plurality of sub-regions, and the grids are used to regress a plurality of corner points;

[0024] Calculating a target loss function based on corner regression using the feature map and the training image;

[0025] Return to the step of obtaining a feature map according to the training image through the corner detection model, until the target loss function is less than the preset loss threshold, to obtain the trained corner detection model, and the trained corner detection model is used to perform corner detection through the above-mentioned corner detection method.

[0026] In some embodiments, the training image includes the real corner points, and the target loss function based on corner point regression is expressed as follows:

[0027]

[0028] in, is the objective loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λ coord is the loss weight of coordinate regression when the grid has real corner points, λ class is the loss weight of sub-region row classification and column classification when the grid has real corner points, S is the size of the grid, B is the number of predicted corner points of each grid, is an indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ij (c y ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c yThe probability of the predicted corner point in the sub-region of the column position, and Calculated based on the grid and corner detection model through a logical function.

[0029] In some embodiments, the expression of the predicted corner point is as follows:

[0030]

[0031] Where biases represents the sub-region offset, n is the sub-region number, (biases x(n) ,biases y(n) ) includes (0,0), (0.5,0), (0,0.5) and (0.5,0.5), w is the width of the sub-region, h is the height of the sub-region, x logistic Indicates the horizontal and y coordinate offset of the grid logistic Indicates the vertical offset of the grid.

[0032] To achieve the above objective, another aspect of an embodiment of the present application provides a corner point detection device, the device comprising:

[0033] An image processing module, used for processing the original image to obtain a canvas image;

[0034] A model output module, used to obtain candidate corner points and confidence levels according to the canvas image through a trained corner point detection model, wherein the corner point detection model adopts a target loss function based on corner point regression, and the candidate corner points correspond to the confidence levels one by one;

[0035] The corner point processing module is used to perform non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned corner point detection method when executing the computer program.

[0037] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above corner point detection method is implemented.

[0038] The embodiments of the present application include at least the following beneficial effects: the present application provides a corner detection method and device, an electronic device and a storage medium, the scheme obtains a canvas image by processing the original image; obtains candidate corner points and confidence levels according to the canvas image through a trained corner detection model, the corner detection model adopts a target loss function based on corner regression, the candidate corner points correspond to the confidence levels one-to-one, and solves the problem of false detection of non-target corner points in complex backgrounds through the learning ability of the neural network; non-maximum suppression is performed on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points, which can effectively remove duplicate candidate corner points, retain only important target corner points, and improve the accuracy of corner detection.

[0039] The present application also provides a corner detection model training method, which obtains a training image of a preset size by annotating and expanding a sample image; obtains a feature map based on the training image through the corner detection model, the feature map includes a number of grids, the grid includes a number of sub-regions, and the grid is used to regress a number of corner points; calculates a target loss function based on corner regression through the feature map and the training image; returns to the step of obtaining the feature map based on the training image through the corner detection model until the target loss function is less than a preset loss threshold, and solves the problem of false detection of non-target corners in complex backgrounds through the learning ability of the neural network. The corner detection model can improve the accuracy of corner detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of a corner point detection method provided in an embodiment of the present application;

[0041] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0042] Figure 3 is a flowchart of a corner point detection method provided in an embodiment of the present application;

[0043] Figure 4 This is an optional flow chart of applying the corner detection method provided in the embodiment of the present application to the YOLO model;

[0044] Figure 5 It is a sample schematic diagram provided in the embodiment of the present application;

[0045] Figure 6 It is a schematic diagram of the tiny-yolo-voc network structure provided in an embodiment of the present application;

[0046] Figure 7 It is a schematic diagram of a characteristic diagram provided in an embodiment of the present application;

[0047] Figure 8 is a schematic diagram of a sub-region provided in an embodiment of the present application;

[0048] Fig. 9 is a schematic diagram of redundant points provided in an embodiment of the present application;

[0049] Fig.10 is a flow chart of the non-maximum suppression method provided in an embodiment of the present application;

[0050] Fig.11 This is a processing effect diagram of the non-maximum suppression method provided in an embodiment of the present application;

[0051] Fig.12 It is a detection effect diagram of the Harris corner point method provided in the embodiment of the present application;

[0052] Fig.13 is a detection effect diagram of the corner point detection method provided in an embodiment of the present application;

[0053] Fig.14 is a structural schematic diagram of a corner point detection device provided in an embodiment of the present application;

[0054] Fig.15 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0056] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0057] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0060] 1) Cross-platform application framework: A development framework that helps developers build applications on multiple different platforms without having to write application code separately for each platform.

[0061] In the related technology, camera calibration by detecting the corner points of the chessboard calibration plate is a common camera calibration method. In the prior art, there are methods based on image grayscale changes and local features for detection, such as Harris, SUSAN, etc., and there are also methods based on neural network models combined with screening rules for detection. The above methods have the following technical problems:

[0062] When the camera pixel is low or the calibration plate in the image is small, there will be blurred pixel blocks at the local corners of the target, which makes it easy for the algorithm to detect redundant points in the area near the target.

[0063] In summary, the technical problems existing in the relevant technologies need to be improved.

[0064] In view of this, a corner detection method, apparatus, device and medium are provided in an embodiment of the present application. The scheme obtains a canvas image by processing the original image; obtains candidate corner points and confidence levels according to the canvas image through a trained corner detection model, and the corner detection model adopts a target loss function based on corner regression. The candidate corner points correspond to the confidence levels one by one, and the problem of false detection of non-target corner points in complex backgrounds is solved through the learning ability of the neural network; non-maximum suppression is performed on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points, which can effectively remove duplicate candidate corner points and retain only important target corner points, thereby improving the accuracy of corner detection.

[0065] The present application also provides a corner detection model training method, which obtains a training image of a preset size by annotating and expanding a sample image; obtains a feature map based on the training image through the corner detection model, the feature map includes a number of grids, the grid includes a number of sub-regions, and the grid is used to regress a number of corner points; calculates a target loss function based on corner regression through the feature map and the training image; returns to the step of obtaining the feature map based on the training image through the corner detection model until the target loss function is less than a preset loss threshold, and solves the problem of false detection of non-target corners in complex backgrounds through the learning ability of the neural network. The corner detection model can improve the accuracy of corner detection.

[0066] The corner detection method and corner detection model training method provided in the embodiment of the present application relate to the field of computer vision technology. The corner detection method and corner detection model training method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the corner detection method, etc., but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0069] Figure 1 is an optional flowchart of the corner point detection method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.

[0070] Step S101, processing the original image to obtain a canvas image.

[0071] In some embodiments, the original image is preprocessed so as to be better input into the trained corner detection model. Image preprocessing helps to unify the scale and features of the image, reduce noise interference, and improve the detection accuracy of the model.

[0072] Exemplarily, image preprocessing includes resizing, cropping, normalization, and denoising.

[0073] In some embodiments, the original image is processed by a fixed-length-width strategy, wherein a fixed-length-width canvas is determined; the length-width ratio of the original image is fine-tuned; the image is scaled proportionally while maintaining the fine-tuned length-width ratio; the canvas is offset horizontally and vertically respectively so that the coordinates of the upper left corner of the canvas coincide with the coordinates of the upper left corner of the original image; the transformed coordinates of the corner points in the canvas area are recorded, and the canvas image is saved.

[0074] Specifically, the pseudo code for fine-tuning the length and width ratio of the original image is as follows:

[0075] dw=jitter*orig_w

[0076] dh=jitter*orig_h

[0077] new_ar=(orig_w+rand_uniform(-dw,dw)) / (orig_h+rand_uniform(-dh,dh))

[0078] Among them, orig_w represents the width of the original image, orig_h represents the length of the original image; dw represents the width of the fine-tuned image, dh represents the length of the fine-tuned image, jitter is the jitter fine-tuning parameter, which is used to control the sample expansion without distortion; rand_uniform is a uniformly distributed random number generation function; new_ar is the length-to-width ratio of the fine-tuned image.

[0079] Optionally, in order to improve the generalization ability of the model, data augmentation techniques such as random rotation, flipping, shearing, etc. can be used to increase the diversity of training samples.

[0080] In this embodiment, the original image is processed to obtain a canvas image, which can unify the image input format and enable the corner detection model to better identify the features of the image.

[0081] Step S102, obtaining candidate corner points and confidence levels according to the canvas image through the trained corner point detection model.

[0082] Specifically, the corner detection model adopts a target loss function based on corner regression, and the candidate corners correspond to the confidence levels one by one.

[0083] It should be noted that the target detection model will perform target rectangular box regression in each grid, and this application transforms it into corner point regression.

[0084] Specifically, the objective loss function based on corner regression is expressed as follows:

[0085]

[0086]

[0087] in, is the target loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λ coord is the loss weight of coordinate regression when the grid has real corner points, λ class is the loss weight of sub-region row classification and column classification when the grid has real corner points, S is the size of the grid, B is the number of predicted corner points of each grid, Is the indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ik (cy ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c y The probability of the grid belonging to the sub-region of the column position is that the corner point detection model predicts the corner points based on the feature map output by the training image. and The training images include real corner points calculated based on the grid and corner detection model through the logic function.

[0088] It can be understood that if the real corner point is in the sub-region of the grid, then λ obj , coord With λ class If the true corner point is not in the sub-region of the grid, then λ is used noobj The loss weight.

[0089] In some embodiments, the trained corner detection model predicts the processed canvas image and outputs the positions of the candidate corner points and the corresponding confidence levels.

[0090] Optionally, the corner detection model may adopt a YOLO model.

[0091] It is understandable that the candidate corner points output by the corner detection model contain many irrelevant corner points and may also have redundant corner points.

[0092] In this embodiment, the trained corner detection model obtains candidate corner points and confidence levels according to the canvas image, which can effectively extract key information in the image and prepare for subsequent non-maximum suppression.

[0093] Step S103, performing non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points.

[0094] In some embodiments, redundant corner points are removed by non-maximum suppression.

[0095] Among them, non-maximum suppression includes sorting, threshold determination, screening and looping.

[0096] Specifically, the candidate corner points are sorted from large to small according to the corresponding confidence levels to obtain a corner point sequence; the first candidate corner point in the corner point sequence is used as the current corner point; the corner point after the current corner point in the corner point sequence is used as the comparison corner point; if the distance between the comparison corner point and the current corner point is less than a threshold, the comparison corner point is deleted from the corner point sequence; the first corner point after the current corner point in the corner point sequence is used as the current corner point; return to the step of using the corner point after the current corner point in the corner point sequence as the comparison corner point until there is no node after the current corner point.

[0097] It is understandable that the distance threshold and confidence in non-maximum suppression affect the selected target corner points. A lower distance threshold and a higher confidence can effectively reduce redundant corner points and retain the most accurate predictions.

[0098] Optionally, non-maximum suppression can be performed using the Soft-NMS method, which retains some overlapping corner points based on confidence attenuation, thereby retaining more effective information.

[0099] Alternatively, a density-based non-maximum suppression method may be used, and the distance threshold may be optimized according to the distribution density of the corner points in the image. For example, a looser distance threshold may be used for a dense corner point region, while a stricter threshold may be used for a sparse region.

[0100] In this embodiment, non-maximum suppression is performed on candidate corner points according to a distance threshold and a confidence level to obtain target corner points, which can effectively remove duplicate candidate corner points and retain only important target corner points, thereby improving corner point detection accuracy.

[0101] In steps S101 to S103 shown in the embodiment of the present application, a canvas image is obtained by processing the original image; candidate corner points and confidence levels are obtained based on the canvas image through the trained corner detection model, and the corner detection model adopts a target loss function based on corner regression. The candidate corner points correspond to the confidence levels one-to-one, and the problem of false detection of non-target corner points in complex backgrounds is solved through the learning ability of the neural network; non-maximum suppression is performed on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points, which can effectively remove duplicate candidate corner points and retain only important target corner points, thereby improving the accuracy of corner detection.

[0102] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S206:

[0103] Step S201, sorting the candidate corner points from large to small according to the corresponding confidence levels to obtain a corner point sequence.

[0104] In step S201 of some embodiments, quick sort, heap sort or other sorting algorithms are used to sort the candidate corner points from large to small according to the corresponding confidence levels to obtain a corner point sequence.

[0105] In this embodiment, the candidate corner points are sorted from large to small according to the corresponding confidence levels to obtain a corner point sequence so that in subsequent processing steps, corner points with high confidence levels are given priority, which helps to improve the efficiency and accuracy of corner point filtering.

[0106] Step S202: taking the first candidate corner point in the corner point sequence as the current corner point.

[0107] In step S202 of some embodiments, the first candidate corner point in the corner point sequence, that is, the candidate corner point with the highest confidence, is taken as the current corner point.

[0108] Step S203: taking the corner point after the current corner point in the corner point sequence as a comparison corner point.

[0109] In step S203 of some embodiments, other corner points are selected from positions after the current corner point as comparison corner points to be compared with the current corner point.

[0110] It can be understood that, since the comparison corner point is located after the current corner point, the confidence of the comparison corner point is smaller than the confidence of the current corner point.

[0111] Step S204: if the distance between the comparison corner point and the current corner point is less than the distance threshold, the comparison corner point is deleted from the corner point sequence.

[0112] In step S204 of some embodiments, the distance between the current corner point and the comparison corner point is calculated to obtain a distance threshold.

[0113] Furthermore, if the distance between the comparison corner point and the current corner point is less than a distance threshold, the comparison corner point is deleted from the corner point sequence.

[0114] Step S205: taking the first corner point after the current corner point in the corner point sequence as the current corner point.

[0115] In step S205 of some embodiments, the first corner point after the current corner point in the corner point sequence, that is, the node whose confidence is second only to the node after the current corner point, is taken as the current corner point.

[0116] Step S206, returning to the step of using the corner points after the current corner point in the corner point sequence as comparison corner points, until there is no node after the current corner point.

[0117] It is understandable that if there is no node after the current corner point, no comparison filtering is required.

[0118] Figure 3is an optional flowchart of the corner detection model training method provided in an embodiment of the present application. Figure 3 The method may include but is not limited to steps S301 to S304.

[0119] Step S301 , annotating and expanding the sample image to obtain a training image of a preset size.

[0120] Specifically, the training image includes the real corner points.

[0121] In step S301 of some embodiments, a length and width fixed strategy is adopted when training the corner detection model. Fine-tune the length and width ratio of the sample image; keep the length and width ratio after fine-tuning, scale the image proportionally, and use bilinear interpolation. Randomly offset the canvas in the horizontal and vertical directions respectively. The coordinates of the upper left corner of the starting position of the canvas coincide with the coordinates of the upper left corner of the sample image. Record the transformed coordinates of the corner points in the canvas area and save them as training images.

[0122] Step S302, obtaining a feature map according to the training image through a corner detection model.

[0123] Specifically, the feature map includes several grids, the grid includes several sub-regions, and the grid is used to regress several corner points.

[0124] In step S302 of some embodiments, the training image is input into a corner detection model, and the last convolutional layer of the corner detection model generates a feature map of a 13*13 grid after processing.

[0125] It can be understood that each grid contains the features of a specific area. Each sub-region in the grid is responsible for regressing the corner points associated with that region. Each grid predicts the location of a corner point and its confidence.

[0126] The corner points of grid regression refer to the corner point coordinates predicted by the model at each grid position. The regression task can optimize the parameters in the network by calculating the loss function.

[0127] Step S303, calculating a target loss function based on corner regression using the feature map and the training image.

[0128] Specifically, the objective loss function based on corner regression is expressed as follows:

[0129]

[0130] in, is the target loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λcoord is the loss weight of coordinate regression when the grid has real corner points, λ class is the loss weight of sub-region row classification and column classification when the grid has real corner points, S is the size of the grid, B is the number of predicted corner points of each grid, Is the indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ij (c y ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c y The probability of the column position in the sub-region, predicting the corner point, and Calculated based on grid and corner detection model through logical function.

[0131] It can be understood that the loss weight of confidence regression when the grid has no real corner points and the loss weight of confidence regression when the grid has real corner points are the confidence regression loss, and the loss weight of sub-region row classification and the loss weight of column classification when the grid has real corner points are the sub-region classification loss.

[0132] Step S304, returning to the step of obtaining a feature map according to the training image through the corner detection model, until the target loss function is less than a preset loss threshold.

[0133] In step S304 of some embodiments, after each round of training, the current target loss function value is calculated and compared with a preset loss threshold. If the loss value is lower than the set threshold, the training is stopped and it is considered that the model has achieved a sufficiently good effect.

[0134] It is understandable that if the target loss function does not reach the preset threshold, the model continues to be trained by continuously optimizing the regression task in the corner detection task until convergence.

[0135] In steps S301 to S304 shown in the embodiment of the present application, a training image of a preset size is obtained by annotating and expanding the sample image; a feature map is obtained according to the training image through a corner detection model, the feature map includes a number of grids, the grid includes a number of sub-regions, and the grid is used to regress a number of corner points; a target loss function based on corner regression is calculated through the feature map and the training image; the step of obtaining a feature map according to the training image through the corner detection model is returned until the target loss function is less than a preset loss threshold. The problem of false detection of non-target corners in complex backgrounds is solved through the learning ability of the neural network, and the corner detection model can improve the accuracy of corner detection.

[0136] Figure 4 is an optional flowchart of applying the corner detection method provided in the embodiment of the present application to the YOLO model. Figure 4 The method may include but is not limited to the following steps.

[0137] Step 1: Sample annotation and expansion.

[0138] In some embodiments, the horizontal and vertical coordinates of each target corner point of each group of chessboards in the original image are recorded and normalized in the horizontal and vertical directions respectively. The chessboard corner point sample set is expanded by jitter fine-tuning, scaling and translation. The specific operations include:

[0139] a. Set the canvas: This embodiment adopts a fixed length and width strategy when training the YOLO corner detection model. For example, the length and width are both 512, so a canvas with a length and width of 512 is defined.

[0140] b. Fine-tune the length and width ratio of the original image. The pseudo code is as follows:

[0141] dw=jitter*orig_w

[0142] dh=jitter*orig_h

[0143] new_ar=(orig_w+rand_uniform(-dw,dw)) / (orig_h+rand_uniform(-dh,dh))

[0144] Among them, orig_w and orig_h represent the length and width of the original image; jitter is the jitter fine-tuning parameter, which is used to control the sample expansion without distortion; rand_uniform is the uniformly distributed random number generation function; new_ar is the length-to-width ratio of the fine-tuned image.

[0145] c. Keep the length and width ratio after fine-tuning and scale the image proportionally. This application uses bilinear interpolation.

[0146] d. Randomly shift the canvas horizontally and vertically. The coordinates of the upper left corner of the canvas start position coincide with the coordinates of the upper left corner of the original image.

[0147] e. Record the transformed coordinates of the corner points in the canvas area and save the canvas image.

[0148] For example, the sample schematic diagram is as follows Figure 5 shown.

[0149] Step 2: Neural network structure design.

[0150] In some embodiments, the corner detection model is trained using the tiny-yolo-voc network structure of YOLOv2-Darknet, which inherits the advantage of fast inference speed of YOLO. For example, the schematic diagram of the tiny-yolo-voc network structure is shown in Figure 6 .

[0151] Step 3: loss function design.

[0152] In some embodiments, the key to transforming YOLO object detection into corner detection is the loss function design. After the input image is processed by the last convolution layer of tiny-yolo-voc, a 13*13 grid feature map is generated. For example, the feature map diagram is as follows Figure 7 As shown in the figure, the original YOLO will perform target rectangle regression in each grid. This application transforms it into corner point regression, setting each grid to regress 4 corner points. The regression information includes horizontal and vertical coordinates and confidence probability. In order to make the prediction area of ​​the 4 corner points in each grid non-repeated, this method further divides the grid into 2 rows and 2 columns, a total of 4 sub-areas. For example, the schematic diagram of the sub-areas is as follows Figure 8 As shown. Among them, each prediction point is responsible for 1 sub-region, and the loss function expression is as follows:

[0153]

[0154] in, is the objective loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λ coord is the loss weight of coordinate regression when the grid has real corner points, λ class is the loss weight of the sub-region row classification and the loss weight of the column classification when the grid has real corner points, S is the size of the grid 13, B is the number of predicted corner points of each grid 4, is an indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ij (c y ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c y The probability of the predicted corner point in the sub-region of the column position, and Calculated based on the grid and corner detection model through a logical function.

[0155] Specifically, the expression for predicting corner points is as follows:

[0156]

[0157] Where biases represents the sub-region offset, n is the sub-region number, (biases x(n) ,biases y(n) ) includes (0,0), (0.5,0), (0,0.5) and (0.5,0.5), w is the width of the sub-region, h is the height of the sub-region, x logistic and logistic Indicates the horizontal and vertical coordinate offsets of each anchor predicted by YOLO relative to its true position.

[0158] It can be understood that the YOLO corner detection model generates a feature map containing 13*13 grids, each grid corresponds to an anchor box, and this embodiment transforms the target rectangular box regression into corner regression.

[0159] Step 4: Corner point non-maximum suppression.

[0160] In some embodiments, when the camera pixel is low or the checkerboard calibration plate in the image is small, irregular fuzzy pixel blocks will appear around the target corners. At this time, whether using classical methods such as Harris and SUSAN or neural network methods, redundant points may be detected in the area near the target. The main reason is that the fixed threshold cannot be applied to all scenarios. For example, the schematic diagram of redundant points is as follows: Fig. 9To solve this problem, the present application designs a non-maximum suppression method for corner points: first, all candidate corner points are sorted from large to small according to the confidence of the neural network output; then, the candidate points whose distance to the current point is less than the threshold and whose confidence is less than the current point are deleted in a loop. For example, the flowchart of the non-maximum suppression method can be found in Fig.10 , see the effect diagram of non-maximum suppression method Fig.11 .

[0161] For example, a fisheye camera is used to capture a calibration plate image including a background. Fig.12 This is the detection effect diagram of the Harris corner point method. Fig.13 The detection effect diagram of the corner point detection method of this application is shown in FIG. By comparison, it is found that this application can effectively eliminate the background influence and will not generate redundant points in the vicinity of the target.

[0162] It should be noted that the present application can also normally detect scenes in which the camera image only contains the calibration plate and no background. The main reason is that the sample expansion stage adds images containing only the calibration plate to the training set, making the detection model more universal.

[0163] In this embodiment, the core of the corner detection method of this application is the design idea of ​​the loss function. Through the learning ability of the neural network, the problem of false detection of non-target corners in complex backgrounds is solved, providing a technical basis for end-to-end automatic camera calibration. Among them, the core of the corner non-maximum suppression method is to merge and filter the prior parameters of the target corner confidence and distance threshold learned by the neural network, which can effectively remove redundant points in the area near the target corner.

[0164] See also Fig.14 The embodiment of the present application further provides a corner point detection device, which can implement the above corner point detection method, and the device includes:

[0165] The image processing module 1401 is used to process the original image to obtain a canvas image;

[0166] A model output module 1402 is used to obtain candidate corner points and confidence levels according to the canvas image through a trained corner point detection model, wherein the corner point detection model adopts a target loss function based on corner point regression, and the candidate corner points correspond to the confidence levels one by one;

[0167] The corner point processing module 1403 is used to perform non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain target corner points.

[0168] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above corner point detection method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0170] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0171] See also Fig.15 , Fig.15 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0172] The processor 1501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0173] The memory 1502 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1502 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1502, and the processor 1501 calls and executes the corner point detection method of the embodiment of this application;

[0174] Input / output interface 1503, used to implement information input and output;

[0175] Communication interface 1504, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0176] A bus 1505 that transmits information between the various components of the device (e.g., the processor 1501, the memory 1502, the input / output interface 1503, and the communication interface 1504);

[0177] The processor 1501 , the memory 1502 , the input / output interface 1503 and the communication interface 1504 are connected to each other in communication within the device via the bus 1505 .

[0178] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned corner point detection method is implemented.

[0179] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The corner detection method, corner detection device, electronic device and storage medium provided in the embodiments of the present application obtain a canvas image by processing the original image; obtain candidate corner points and confidence levels according to the canvas image through a trained corner detection model, the corner detection model adopts a target loss function based on corner regression, the candidate corner points correspond to the confidence levels one-to-one, and the problem of false detection of non-target corner points in complex backgrounds is solved through the learning ability of the neural network; non-maximum suppression is performed on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points, which can effectively remove duplicate candidate corner points, retain only important target corner points, and improve the accuracy of corner detection.

[0182] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0183] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0186] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0187] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0192] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A corner point detection method, characterized in that: The method comprises the following steps: Processing the original image to obtain a canvas image; Obtaining candidate corner points and confidence levels according to the canvas image through a trained corner point detection model, wherein the corner point detection model adopts a target loss function based on corner point regression, and the candidate corner points correspond to the confidence levels one by one; Non-maximum suppression is performed on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points.

2. The method according to claim 1, characterized in that: The objective loss function based on corner regression is expressed as follows: in, is the objective loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λ coord is the loss weight of coordinate regression when the grid has real corner points, λ class is the loss weight of sub-region row classification and column classification when the grid has real corner points, S is the size of the grid, B is the number of predicted corner points of each grid, Is the indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ij (c y ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c y The probability of being in the sub-region of the column position, the grid belongs to the feature map output by the corner point detection model based on the training image, the predicted corner point, and The training image includes the real corner points by calculating based on the grid and the corner point detection model through a logic function.

3. The method according to claim 1, characterized in that: The step of performing non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points includes: Sort the candidate corner points from large to small according to the corresponding confidence levels to obtain a corner point sequence; Taking the first candidate corner point in the corner point sequence as the current corner point; Taking the corner point after the current corner point in the corner point sequence as a comparison corner point; If the distance between the comparison corner point and the current corner point is less than a distance threshold, the comparison corner point is deleted from the corner point sequence.

4. The method according to claim 3, characterized in that The step of performing non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points further includes: Taking the first corner point after the current corner point in the corner point sequence as the current corner point; Return to the step of using the corner points after the current corner point in the corner point sequence as comparison corner points until there is no node after the current corner point.

5. A corner point detection model training method, characterized in that: The method comprises the following steps: Label and expand the sample images to obtain training images of preset sizes; Obtaining a feature map according to the training image through a corner detection model, wherein the feature map includes a plurality of grids, the grids include a plurality of sub-regions, and the grids are used to regress a plurality of corner points; Calculating a target loss function based on corner regression using the feature map and the training image; Return to the step of obtaining a feature map according to the training image through the corner detection model, until the target loss function is less than a preset loss threshold, to obtain the trained corner detection model, and the trained corner detection model is used to perform corner detection through the method according to any one of claims 1 to 4.

6. The method according to claim 5, characterized in that The training image includes the real corner points, and the target loss function based on corner point regression is expressed as follows: in, is the objective loss function, λ noobj is the loss weight of confidence regression when the grid has no real corner points, λ obj is the loss weight of confidence regression when the grid has real corner points, λ coord is the loss weight of coordinate regression when the grid has real corner points, λ class is the loss weight of sub-region row classification and column classification when the grid has real corner points, S is the size of the grid, B is the number of predicted corner points of each grid, is an indicator function of whether there is a corner point in the grid, tx is the horizontal coordinate of the real corner point, ty is the vertical coordinate of the real corner point, is the horizontal coordinate of the predicted corner point, is the ordinate of the predicted corner point, is the confidence probability of the corner point prediction, p ij (c x ), p ij (c y ) refers to whether the real corner point is in the grid c x Line, c y The probability of being within the sub-region of the column position; It means the predicted corner point is in the grid c x Line, c y The probability of the predicted corner point in the sub-region of the column position, and Calculated based on the grid and corner detection model through a logical function.

7. The method according to claim 6, characterized in that The expression of the predicted corner point is as follows: Where biases represents the sub-region offset, n is the sub-region number, (biases x(n) ,biases y(n) ) includes (0,0), (0.5,0), (0,0.5) and (0.5,0.5), w is the width of the sub-region, h is the height of the sub-region, x logistic Indicates the horizontal and y coordinate offset of the grid logistic Indicates the vertical offset of the grid.

8. A corner point detection device, characterized in that: The device comprises: An image processing module, used for processing the original image to obtain a canvas image; A model output module, used to obtain candidate corner points and confidence levels according to the canvas image through a trained corner point detection model, wherein the corner point detection model adopts a target loss function based on corner point regression, and the candidate corner points correspond to the confidence levels one by one; The corner point processing module is used to perform non-maximum suppression on the candidate corner points according to the distance threshold and the confidence level to obtain the target corner points.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images