Vanishing point detection methods, devices, robots, and storage media

Abstract

This application relates to the field of image processing technology and provides a vanishing point detection method, apparatus, robot, and storage medium. The vanishing point detection method includes: performing line detection on an image to be detected to obtain N candidate lines in the image; inputting the image to be detected into a vanishing point detection network to obtain target location regions of candidate vanishing points in the image; filtering the N candidate lines in the image based on the target location regions to obtain M target lines from the N candidate lines; and determining the target vanishing points based on the M target lines. This solution improves the accuracy of vanishing point detection and localization by improving the quality of the target lines.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a vanishing point detection method, device, robot, and storage medium. Background Technology

[0002] Cameras are crucial tools for robots to perceive their environment. However, due to prolonged use, vibrations, and hardware aging, camera positions can easily shift, altering their extrinsic parameters. Changes in the extrinsic parameters of robot cameras can lead to difficulties in correctly projecting and matching point cloud data and image data during multi-sensor fusion. For example, when fusion is performed between LiDAR and cameras, if the camera's extrinsic parameters (such as pitch angle) change, the image captured with the changed pitch angle will differ from the image captured with the correct pitch angle. This results in a different depth estimate for the image captured with the changed pitch angle compared to the image captured with the correct pitch angle. Consequently, the LiDAR point cloud data may not be correctly projected onto the camera's image plane, and the depth formed by the LiDAR point cloud data on the camera's image plane may not be correctly matched with the estimated depth of the image captured with the changed pitch angle.

[0003] Among the calibration parameters of a camera, the calibration of its pitch angle is the most important. A commonly used online calibration method for pitch angle is to detect vanishing points in the image captured by the camera and then calculate the camera's pitch angle. Therefore, the accuracy of vanishing point detection and positioning directly affects the accuracy of obtaining the robot's camera extrinsic parameters through online calibration methods. Summary of the Invention

[0004] This application provides a method, apparatus, robot, and storage medium for vanishing point detection, which can improve the accuracy of vanishing point detection and positioning.

[0005] A first aspect of this application provides a vanishing point detection method, the vanishing point detection method comprising:

[0006] Line detection is performed on the image to be detected to obtain N candidate lines in the image to be detected, where N is an integer greater than 1;

[0007] The image to be detected is input into the vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected;

[0008] Based on the target location region, N candidate lines in the image to be detected are filtered to obtain M target lines from the N candidate lines, where M is an integer greater than 1 and less than or equal to N;

[0009] Based on the M target straight lines, determine the target vanishing point.

[0010] A second aspect of this application provides a vanishing point detection device, the vanishing point detection device comprising:

[0011] The line detection module is used to perform line detection on the image to be detected, and obtain N candidate lines in the image to be detected, where N is an integer greater than 1;

[0012] The network detection module is used to input the image to be detected into the vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected;

[0013] The line filtering module is used to filter N candidate lines in the image to be detected based on the target location region, and obtain M target lines from the N candidate lines, where M is an integer greater than 1 and less than or equal to N;

[0014] The vanishing point determination module is used to determine the target vanishing point based on the M target straight lines.

[0015] A third aspect of this application provides a robot, including: a camera mounted on the robot, a memory, and a processor; the camera is used to acquire an image to be detected, and the memory is used to store a computer program, the processor being used to execute the computer program to implement the vanishing point detection method described in the first aspect.

[0016] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the vanishing point detection method described in the first aspect.

[0017] A fifth aspect of this application provides a computer program product that, when run on a robot, causes the robot to perform the vanishing point detection method described in the first aspect.

[0018] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment first obtains N candidate lines in the image to be detected by performing line detection on the image to be detected. Then, the image to be detected is input into the vanishing point detection network, and the target position region of the candidate vanishing point in the image to be detected can be obtained. Based on the target position region, the N candidate lines in the image to be detected can be filtered to obtain M target lines. Finally, based on the M target lines, the target vanishing point is determined. The above scheme filters the candidate lines in the image to be detected according to the target position region output by the vanishing point detection network, thereby improving the vanishing point detection efficiency and positioning accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a vanishing point detection method provided in Embodiment 1 of this application;

[0021] Figure 2 This is a schematic diagram of candidate lines in the image to be detected;

[0022] Figure 3 It is the candidate location region of the candidate vanishing point in the second image;

[0023] Figure 4 This is a schematic diagram of the target straight line in the image to be detected;

[0024] Figure 5 This is a flowchart illustrating a vanishing point detection method provided in Embodiment 2 of this application;

[0025] Figure 6 This is a schematic diagram of the structure of a vanishing point detection device provided in Embodiment 3 of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a robot provided in Embodiment 4 of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] Research has revealed that commonly used vanishing point detection methods in existing technologies can be divided into two main categories: one category uses line detection algorithms to detect straight lines in an image and calculates vanishing points based on the detected lines. This method is susceptible to noise, making it difficult to iteratively converge when calculating vanishing points. The other category is based on deep learning detection methods, which train convolutional neural networks to learn the location of vanishing points in an image. This method has a faster model convergence speed, but can only provide coarse localization of vanishing points with insufficient accuracy.

[0034] To address the problems in the aforementioned vanishing point detection methods and improve the accuracy of vanishing point detection and localization, this application provides a vanishing point detection method. The specific implementation process of this method is as follows: First, line detection is performed on the image to be detected to obtain N candidate lines. Second, the image to be detected is input into a vanishing point detection network to obtain the target location region of the candidate vanishing point in the image. Based on this target location region, the N candidate lines in the image are filtered to obtain M target lines from the N candidate lines, where M is an integer greater than 1 and less than or equal to N. Finally, based on the M target lines, the target vanishing point is determined. This vanishing point detection method filters candidate lines in the image to be detected based on the target location region output by the vanishing point detection network, thereby improving the efficiency of vanishing point detection and the accuracy of localization.

[0035] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0036] To illustrate the technical solution of this application, specific embodiments are described below.

[0037] Reference Figure 1 The diagram illustrates a flow chart of a vanishing point detection method provided in Embodiment 1 of this application. Figure 1 As shown, the vanishing point detection method may include the following steps:

[0038] Step 101: Perform line detection on the image to be detected to obtain N candidate lines in the image to be detected.

[0039] In this embodiment of the application, the Hough line detection algorithm or the line segment detection algorithm (LSD) can be used to detect the straight lines in the image to be detected, so as to obtain N candidate straight lines in the image to be detected, where N is an integer greater than zero.

[0040] For example, taking the Hough line detection algorithm as an example, the specific process of line detection in the image to be detected can be as follows: First, the image to be detected is converted to grayscale, from a color image to a grayscale image. Then, Gaussian kernels are used to denoise the converted grayscale image to eliminate Gaussian noise and obtain a smoother grayscale image to be detected. Second, the grayscale image to be detected is binarized, that is, the grayscale value of each pixel in the grayscale image to be detected is set to 0 or 255. The grayscale image to be detected after binarization exhibits a clear black and white effect. The Canny edge detection algorithm is used to detect the contour lines in the grayscale image to be detected. Finally, the contour points on the binarized contour lines are mapped to the Hough space through the Hough transform. An accumulator is assigned to each unit in the Hough space, and each unit corresponds to a line parameter. If multiple contour points are mapped to the same unit in the Hough space, it is determined that multiple contour points correspond to the same line parameter (i.e., multiple contour points are located on a straight line in the grayscale image to be detected). After all contour points are mapped, the value of the accumulator corresponding to each unit is obtained. By setting a threshold, the N units with the largest values ​​in the accumulator are selected, and the line parameters corresponding to these units are obtained. Then, based on the N sets of line parameters, N candidate lines are obtained, thus obtaining the N candidate lines in the image to be detected. Figure 2 As shown, the candidate lines in the figure are the candidate lines in the image to be detected (i.e., ...) detected by the Hough line detection algorithm. Figure 2 (The black lines in the text).

[0041] It should be understood that any line detection algorithm can be used to perform line detection on the image to be detected, such as deep learning-based line detection algorithms, Hough line detection algorithms, LSD line detection algorithms, etc., and this application does not limit it.

[0042] Step 102: Input the image to be detected into the vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected.

[0043] In this context, a vanishing point detection network refers to a network capable of identifying the target location regions of candidate vanishing points in an image to be detected. The vanishing point detection network is trained on a neural network using labeled sample images; for example, the neural network could be a convolutional neural network, and the labeled sample images could be those marked with vanishing points, with the labels indicating the vanishing points in the sample images.

[0044] In this embodiment, since the vanishing point detection network is a trained network, inputting the image to be detected into the vanishing point detection network can directly obtain the target location region of the candidate vanishing point in the image. The convolutional neural network model used by the vanishing point detection network can be a classification network model or a regression network model. Since the number of neurons in the output layer is finite (i.e., corresponding to multiple regions in the image to be detected) regardless of whether it is a classification network model or a regression network model, the position of the candidate vanishing point can be determined according to the multiple regions divided in the image to be detected, thereby obtaining the target location region of the candidate vanishing point in the image to be detected, thus coarsely locating the candidate vanishing point.

[0045] In one possible implementation, the image to be detected is input into a vanishing point detection network to obtain the target location region of candidate vanishing points in the image to be detected, including:

[0046] When the original size of the image to be detected is smaller than or larger than the preset size, the original size of the image to be detected is adjusted to the preset size to obtain the first image;

[0047] The first image is divided into grids to obtain the second image;

[0048] The second image is input into the vanishing point detection network to obtain the candidate location regions of candidate vanishing points in the second image;

[0049] Based on the position of the candidate location region in the second image and the conversion relationship between the size of the second image and the original size of the image to be detected, the target location region of the candidate vanishing point in the image to be detected is determined.

[0050] In this embodiment of the application, in order to balance the calculation speed of the vanishing point detection network and improve the accuracy of the vanishing point detection network, the images to be detected input to the vanishing point detection network can be unified to the same size. That is, when the original size of the image to be detected is greater than or less than the preset size, the original size of the image to be detected is adjusted to the preset size to obtain the first image, wherein the size of the first image is the preset size.

[0051] In this embodiment of the application, in order to evenly distribute multiple regions in the image to be detected, the image to be detected (i.e., the first image) after size adjustment can be divided into grids, dividing the first image into K×K grids of the same size to obtain the second image, where K is an integer greater than zero, such as... Figure 3The image shown is the image to be detected after resizing and meshing (i.e., the second image). Inputting the second image into the vanishing point detection network reveals potential candidate vanishing points within the corresponding meshes of the second image. This means that the region corresponding to that mesh is identified as the candidate location area for the vanishing point in the second image. Since the second image is a resized image to be detected, the candidate location area of ​​its vanishing points in the second image cannot be directly used as the target location area of ​​the vanishing point in the image to be detected. A size conversion is required between the second image and the image to be detected to restore the size of the second image to the original size of the image to be detected before determining the target location area of ​​the vanishing point in the image to be detected.

[0052] For example, assuming the original size of the image to be detected is 1280×720, and the preset size is 300×300, and K equals 15, after size adjustment and grid division, the image is obtained as follows: Figure 3 The second image shown has 225 grids. From this, we can obtain the size conversion relationship between the second image and the image to be detected. Specifically, one grid in the second image corresponds to 48 pixels in the image to be detected in terms of height, and one grid in the second image corresponds to 85 pixels in the image to be detected in terms of width. Therefore, when determining candidate vanishing points in... Figure 3 The candidate location area shown ( Figure 3 After determining the grid where the pentagram is located, the candidate vanishing point can be determined based on the above conversion relationship. Figure 4 The target location area shown ( Figure 4 (The grid containing the central pentagram).

[0053] In one possible implementation, the second image is input into a vanishing point detection network to obtain candidate location regions of candidate vanishing points in the second image, including:

[0054] The second image is input into the vanishing point detection network to obtain the probability value of each grid in the second image having a candidate vanishing point;

[0055] The grid with the highest probability value of the candidate vanishing point among all grids in the second image is determined as the candidate location region.

[0056] In this embodiment of the application, if the second image is divided into a 15×15 grid, that is, 225 grids, the fully connected layer of the vanishing point detection network can be set to 225 neurons. The second image is input into the vanishing point detection network, and the probability value that each grid in the 225 grids may have a candidate vanishing point can be obtained. The magnitude of the probability value that each grid may have a candidate vanishing point is determined, and the grid corresponding to the highest probability value is determined as the candidate location region where the candidate vanishing point is located.

[0057] It should be understood that when determining the candidate location region of a candidate vanishing point, a one-hot encoding format can also be used, causing the vanishing point detection network to output a 225-dimensional vector (corresponding to 225 grids). The vector elements of the grid containing the candidate vanishing point have a value of 1, while the other vector elements have a value of 0. Figure 3 The vector element value of the grid (7th row, 8th column) where the pentagram is located is 1, that is, the 98th element in the output vector has a value of 1, and the rest of the elements have a value of 0.

[0058] It should also be understood that the vanishing point detection network can adopt a VGG (Visual Geometry Group Network, VGGNet) structure, a deep residual network (ResNet) structure, an AlexNet structure, etc., and the number of neurons in the fully connected layer can be changed according to the number of grids. This application does not limit the specific form of the vanishing point detection structure or the number of neurons in the fully connected layer of the vanishing point detection network structure.

[0059] Step 103: Based on the target location region, filter the N candidate lines in the image to be detected to obtain M target lines from the N candidate lines.

[0060] Where M is an integer greater than 1 and less than or equal to N.

[0061] Since the vanishing point is most likely located in an area where a large number of lines converge in the image to be detected, and many of the candidate lines detected may not pass through areas where a large number of lines converge, in order to improve the efficiency of vanishing point detection, candidate lines that do not pass through areas where a large number of lines converge are identified as redundant and interfering lines, which interfere with the calculation of the vanishing point. Therefore, before performing the vanishing point calculation, these redundant and interfering lines should be filtered out to improve the efficiency of the target line for vanishing point calculation, thereby improving the accuracy of vanishing point detection and localization.

[0062] In this embodiment of the application, when filtering redundant and interfering lines, candidate lines that are far from the target location area can be filtered out by calculating the distance between the candidate lines and the target location area, so as to filter out redundant and interfering lines in the image to be detected.

[0063] In one possible implementation, based on the target location region, N candidate lines in the image to be detected are filtered to obtain M target lines from the N candidate lines, including:

[0064] The coordinates of the center point of the target location region are determined as the position coordinates of the candidate vanishing point in the image to be detected;

[0065] Calculate the distances from the candidate vanishing point's location coordinates to each of the N candidate lines;

[0066] Based on the distances from the candidate vanishing point's location coordinates to the N candidate lines, the N candidate lines in the image to be detected are filtered to obtain the M target lines in the image to be detected.

[0067] In this embodiment, the center point of the target location region obtained by the coarse localization of the vanishing point detection network is used as the position coordinates of the candidate vanishing point. All N candidate lines obtained by the line detection algorithm are traversed, and the distances from the position coordinates of the candidate vanishing point to each of the N candidate lines are calculated. By setting a distance threshold, redundant and interfering candidate lines in the image are filtered out, leaving the target line. Figure 4 The diagram shows the target line among N candidate lines. Figure 4 The white lines in the diagram represent redundant and interfering lines (located far from the target area), while the black lines represent the target line.

[0068] In one possible implementation, based on the distances from the candidate vanishing point's location coordinates to the N candidate lines, the N candidate lines in the image to be detected are filtered to obtain M target lines from the N candidate lines, including:

[0069] The distances from the location coordinates of the candidate vanishing point to any of the N candidate lines are compared with the distance thresholds.

[0070] The candidate lines whose position coordinates from the candidate vanishing point are less than a distance threshold from any candidate line among the N candidate lines are identified as target lines, thus obtaining M target lines from the N candidate lines.

[0071] In this embodiment, the distance threshold can be set to the radius of the circumcircle or incircle of the target location region (i.e., the grid where the candidate vanishing point is located), or it can be a preset value. Assuming the distance threshold is d, the distances from the position coordinates of the candidate vanishing point to the N candidate lines are d1, d2, d3, ... d n d1, d2, d3, ... d n Each value is compared with a distance threshold d. If d1, d2, d3, ... d n If there are lines with a distance less than d, then the lines with a distance less than d are extracted and used as target lines, thereby determining the M target lines in the image to be detected.

[0072] It should be understood that when calculating the distances from the candidate vanishing point's position coordinates to the N candidate lines, the position coordinates of any point in the grid where the candidate vanishing point is located can also be used as the position coordinates of the candidate vanishing point. Furthermore, when calculating the distances from the candidate vanishing point's position coordinates to the N candidate lines, the distance calculation method from any point to the line can be used, such as the Euclidean distance calculation method. This application does not limit this.

[0073] Step 104: Determine the target vanishing point based on the M target straight lines.

[0074] In this embodiment of the application, the vanishing point of the target based on M target straight lines can be determined by using Hough transform or the Random Sampling Consensus (RANSAC) algorithm to calculate the vanishing point.

[0075] In one possible implementation, when M is greater than 2, determining the target vanishing point based on M target straight lines includes:

[0076] Choose K groups of target lines from M target lines, where K is an integer greater than 1. Each group of target lines in the K groups includes two intersecting target lines.

[0077] The intersection point of two intersecting target lines in each group of target lines is determined as the first predicted vanishing point of each group of target lines. The remaining M-2 target lines are traversed, and the number of target lines passing through the first predicted vanishing point of each group of target lines is calculated.

[0078] The first predicted vanishing point with the most target line data among all the first predicted vanishing points passing through the K groups of target lines is determined as the target vanishing point.

[0079] For example, assuming M equals 5, when K is greater than 1, the specific process for determining the target vanishing point can be as follows: arbitrarily select K groups of target lines from the 5 target lines. Assuming arbitrarily select 5 groups of target lines, the target vanishing point can be determined in the following way: Firstly, simultaneously determine the first predicted vanishing point (i.e., intersection point) of the 5 groups of target lines as the first vanishing point, second vanishing point, third vanishing point, fourth vanishing point, and fifth vanishing point. Then, when determining the first, second, third, fourth, and fifth vanishing points, traverse and determine the remaining M-2 target lines after each group of vanishing points, respectively, and obtain the number of target lines passing through the first, second, third, fourth, and fifth vanishing points. Take the vanishing point with the most target lines passing through it as the target vanishing point. The second method involves first calculating the first predicted vanishing point (i.e., intersection point) of any target line in any of the 5 groups as the first vanishing point, and then using the first vanishing point as the target vanishing point. This process iterates through the remaining M-2 target lines after each vanishing point to obtain the target lines passing through the first vanishing point. Next, the method calculates the second vanishing point of any target line in any of the remaining groups in the 5 groups, iterates through the remaining M-2 target lines after each vanishing point to obtain the target lines passing through the second vanishing point. If the target line passing through the second vanishing point is greater than the target line passing through the first vanishing point, the second vanishing point is used as the target vanishing point. The remaining target lines in the 5 groups are then processed sequentially in this manner to obtain the final target vanishing point.

[0080] In one possible implementation, after determining the target vanishing point among the first predicted vanishing points of all target lines in the K groups of target lines, the method further includes:

[0081] Choose H groups of target lines from M target lines, where H is an integer greater than 1. Each group of target lines in the H groups includes two intersecting target lines.

[0082] The intersection point of two intersecting target lines in each group of target lines in group H is determined as the second predicted vanishing point of each group of target lines in group H.

[0083] Traverse the remaining M-2 target lines and calculate the number of target lines that pass through the second predicted vanishing point of each of the H groups of target lines;

[0084] The second predicted vanishing point with the most target line data among the second predicted vanishing points passing through the target line of group H is determined as the vanishing point to be confirmed.

[0085] When the number of target lines passing through the target vanishing point is less than the number of target lines passing through the vanishing point to be confirmed, the vanishing point to be confirmed will be taken as the new target vanishing point.

[0086] In this embodiment, if H is greater than 1, H groups of target lines are arbitrarily selected from the M target lines. Based on the selected H groups of target lines, H second predicted vanishing points are obtained for each group. Then, when H second predicted vanishing points are determined, the remaining M-2 target lines after each group of vanishing points are traversed, and the number of target lines passing through the H second predicted vanishing points is calculated. The vanishing point with the most target lines passing through it is taken as the vanishing point to be confirmed. The method for determining the vanishing point to be confirmed can also adopt the method described above for confirming target vanishing points.

[0087] In addition, after identifying the vanishing point to be confirmed, it is necessary to compare the number of target lines passing through the vanishing point to be confirmed with the number of target lines passing through the target vanishing point. If the number of target lines passing through the target vanishing point is less than the number of target lines passing through the vanishing point to be confirmed, the vanishing point to be confirmed is the latest target vanishing point. In one possible implementation, when M is greater than 2, determining the target vanishing point based on M target lines includes:

[0088] Step S1: Select any two intersecting target lines from the M target lines, and determine the intersection point of the two intersecting target lines as the target vanishing point. Traverse the remaining M-2 target lines from the M target lines and calculate the number of target lines that pass through the target vanishing point.

[0089] Repeat step S2: Select any two intersecting target lines from the M target lines, determine the intersection of the two intersecting target lines as the predicted vanishing point, traverse the remaining M-2 target lines, and calculate the number of target lines that pass through the predicted vanishing point.

[0090] Step S3: When the number of target lines passing through the target vanishing point is less than the number of target lines passing through the predicted vanishing point, the predicted vanishing point is taken as the new target vanishing point.

[0091] After repeating the preset round steps S2 and S3, the latest target vanishing point is determined.

[0092] In this embodiment, the RANSAC algorithm can be used to calculate the vanishing point. The specific process is as follows: First, randomly select two intersecting lines from the M target lines and obtain the intersection point of these two lines as the target vanishing point P*. Then, traverse the remaining M-2 target lines and calculate the number of target lines passing through the target vanishing point P*, denoted as Q*. Next, randomly select two intersecting lines from the M target lines and obtain the intersection point of these two lines as the predicted vanishing point P. Traverse the remaining M-2 target lines and calculate the number of target lines passing through the predicted vanishing point P, denoted as Q. If Q* is less than Q, then the predicted vanishing point P is taken as the target vanishing point, and steps S2 and S3 are executed. A certain number of iterations are set, and the vanishing point output in the last iteration is determined as the target vanishing point. This target vanishing point is the one that passes through the most target lines. Since the target lines are all lines passing through the target location region, they are of high quality, resulting in both fast iteration speed and high accuracy when calculating the vanishing point.

[0093] In one possible implementation, after determining the target vanishing point, the vanishing point detection method further includes: obtaining the position coordinates of the target vanishing point; and marking the position coordinates of the target vanishing point in the image to be detected based on the position coordinates of the target vanishing point, thereby obtaining the marked image to be detected.

[0094] In this embodiment, firstly, line detection is performed on the image to be detected to obtain N candidate lines. Secondly, the image to be detected is input into the vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected. Based on the target location region, the N candidate lines in the image to be detected can be filtered to obtain M target lines from the N candidate lines. Finally, based on the M target lines, the target vanishing point is determined. The above scheme filters the candidate lines in the image to be detected according to the target location region output by the vanishing point detection network, thereby improving the vanishing point detection efficiency and positioning accuracy.

[0095] See Figure 5The diagram illustrates a flow chart of a vanishing point detection method provided in Embodiment 2 of this application. Figure 5 As shown, the vanishing point detection method may include the following steps:

[0096] Step 501: Perform line detection on the image to be detected to obtain N candidate lines in the image to be detected.

[0097] Step 502: Input the image to be detected into the vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected.

[0098] Step 503: Based on the target location region, filter the N candidate lines in the image to be detected to obtain M target lines from the N candidate lines.

[0099] Step 504: Determine the target vanishing point based on the M target straight lines.

[0100] Steps 501-504 in this embodiment are the same as steps 101-104 in the previous embodiment, and can be referred to each other. This embodiment will not repeat them here.

[0101] Step 505: Obtain the location coordinates of the target vanishing point.

[0102] Step 506: Based on the position coordinates of the target vanishing point, mark the corresponding position coordinates of the target vanishing point in the image to be detected, and obtain the marked image to be detected.

[0103] In this embodiment of the application, after determining the target vanishing point based on M target straight lines, the robot can mark the location in the image to be detected at the position coordinates corresponding to the target vanishing point, for example... Figure 4 The pentagram markers in the image are used to meet the user's need to view the vanishing point location in the image to be detected. Users can also perform visual inspection based on the marked position of the target vanishing point. When the visual inspection result differs significantly from the marked position in the image to be detected, it is convenient for users to check the modules in the robot that perform each step of the vanishing point detection algorithm.

[0104] Compared to Embodiment 1, Embodiment 2 adds a step of displaying the target vanishing point. This step allows users to visually inspect the position of the target vanishing point in the image to be detected, and when the visual inspection result differs significantly from the marked position in the image to be detected, the various modules in the robot can be checked in a timely manner.

[0105] See Figure 6 The diagram shows a structural schematic of a vanishing point detection device provided in Embodiment 3 of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0106] The extinction point detection device may specifically include the following modules:

[0107] The line detection module 601 is used to perform line detection on the image to be detected, and obtain N candidate lines in the image to be detected, where N is an integer greater than 1;

[0108] The network detection module 602 is used to input the image to be detected into the vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected;

[0109] The line filtering module 603 is used to filter N candidate lines in the image to be detected based on the target location region, and obtain M target lines from the N candidate lines, where M is an integer greater than 1 and less than or equal to N;

[0110] The vanishing point determination module 604 is used to determine the vanishing point of the target based on M target straight lines.

[0111] In this embodiment, the line filtering module 603 may specifically include the following sub-modules:

[0112] The coordinate determination submodule is used to determine the coordinates of the center point of the target location region as the position coordinates of the candidate vanishing point in the image to be detected.

[0113] The distance calculation submodule is used to calculate the distances from the position coordinates of the candidate vanishing point to each of the N candidate lines;

[0114] The target acquisition submodule is used to filter the N candidate lines in the image to be detected based on the distances from the position coordinates of the candidate vanishing point to the N candidate lines, and obtain the M target lines from the N candidate lines.

[0115] In this embodiment of the application, the target acquisition submodule may specifically include the following units:

[0116] The comparison unit is used to compare the distance from the position coordinates of the candidate vanishing point to any of the N candidate lines with a distance threshold.

[0117] The judgment unit is used to determine the candidate lines whose position coordinates from the candidate vanishing point to any candidate line among the N candidate lines are less than a distance threshold as target lines, so as to obtain M target lines in the image to be detected.

[0118] In this embodiment, the network detection module 602 may specifically include the following sub-modules:

[0119] The size adjustment submodule is used to adjust the original size of the image to be detected to the preset size when the original size of the image to be detected is smaller or larger than the preset size, so as to obtain the first image;

[0120] The grid partitioning submodule is used to partition the first image into a grid to obtain the second image;

[0121] The vanishing point detection submodule is used to input the second image into the vanishing point detection network to obtain the candidate location regions of candidate vanishing points in the second image;

[0122] The size conversion submodule is used to determine the target location region of the candidate vanishing point in the image to be detected based on the position of the candidate location region in the second image and the conversion relationship between the size of the second image and the original size of the image to be detected.

[0123] In this embodiment of the application, the vanishing point detection submodule may specifically include the following units:

[0124] The probability acquisition unit is used to input the second image into the vanishing point detection network to obtain the probability value of each grid in the second image having a candidate vanishing point;

[0125] The candidate region determination unit is used to determine the grid with the highest probability value of the candidate vanishing point among all grids in the second image as the candidate location region.

[0126] In this embodiment of the application, when M is greater than 2, the vanishing point determination module 604 may specifically include the following sub-modules:

[0127] The first selection submodule is used to arbitrarily select K groups of target lines from M target lines, where K is an integer greater than 1, and each group of target lines in the K groups of target lines includes two intersecting target lines.

[0128] The first traversal submodule is used to determine the intersection point of two intersecting target lines in each group of target lines as the first predicted vanishing point of each group of target lines, traverse the remaining M-2 target lines, and calculate the number of target lines that pass through the first predicted vanishing point of each group of target lines.

[0129] The first target determination submodule is used to determine the first predicted vanishing point with the most target line data among the first predicted vanishing points that pass through K sets of target lines as the target vanishing point.

[0130] In this embodiment of the application, the vanishing point determination module 604 may further include the following sub-modules:

[0131] The second selection submodule arbitrarily selects H groups of target lines from M target lines, where H is an integer greater than 1, and each group of target lines in the H groups includes two intersecting target lines.

[0132] The prediction vanishing point determination submodule is used to determine the intersection point of two intersecting target lines in each group of target lines in the H group as the second prediction vanishing point of each group of target lines in the H group.

[0133] The second traversal submodule is used to traverse the remaining M-2 target lines and calculate the number of target lines that pass through the second predicted vanishing point of each of the H groups of target lines.

[0134] The submodule for determining the vanishing point to be confirmed is used to determine the second predicted vanishing point with the most target line data among the second predicted vanishing points that pass through the target line of group H as the vanishing point to be confirmed.

[0135] The second target determination submodule is used to identify the target extinction point as the target extinction point when the number of target lines passing through the target extinction point is less than the number of target lines passing through the extinction point to be confirmed.

[0136] In this embodiment of the application, the vanishing point prediction device may further include the following modules:

[0137] The coordinate acquisition module is used to obtain the position coordinates of the target vanishing point;

[0138] The vanishing point marking module is used to mark the vanishing point in the image to be detected at the same coordinates as the target vanishing point, thus obtaining the marked image to be detected.

[0139] The vanishing point detection device provided in this application embodiment can be applied in the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0140] Figure 7 This is a schematic diagram of the robot provided in Embodiment 4 of this application. Figure 7 As shown, the robot 700 of this embodiment includes: at least one processor 710 ( Figure 7 (Only one is shown in the diagram) a processor, a memory 720, and a computer program 721 stored in the memory 720 and executable on the at least one processor 710, wherein the processor 710 executes the computer program 721 to implement the steps in the above-described vanishing point detection method embodiments.

[0141] The robot 700 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The robot may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will understand that... Figure 7 This is merely an example of robot 700 and does not constitute a limitation on robot 700. It may include more or fewer parts than shown, or combine certain parts, or different parts. For example, it may also include input / output devices, network access devices, cameras mounted on the robot, etc.

[0142] The processor 710 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0143] In some embodiments, the memory 720 may be an internal storage unit of the robot 700, such as a hard disk or memory of the robot 700. In other embodiments, the memory 720 may be an external storage device of the robot 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the robot 700. Furthermore, the memory 720 may include both internal and external storage units of the robot 700. The memory 720 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 720 can also be used to temporarily store data that has been output or will be output.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0145] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0146] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] In the embodiments provided in this application, it should be understood that the disclosed devices / robots and methods can be implemented in other ways. For example, the device / robot embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0150] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0151] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on the robot, the robot can implement the steps in the various method embodiments described above.

[0152] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting vanishing points, characterized in that, The vanishing point detection method includes: Line detection is performed on the image to be detected to obtain N candidate lines in the image to be detected, where N is an integer greater than 1; The process of inputting the image to be detected into a vanishing point detection network to obtain the target location region of the candidate vanishing point in the image to be detected includes: when the original size of the image to be detected is smaller than or larger than a preset size, adjusting the original size of the image to be detected to the preset size to obtain a first image; dividing the first image into grids to obtain a second image; inputting the second image into the vanishing point detection network to obtain the probability value of each grid in the second image having the candidate vanishing point; determining the grid with the highest probability value of the candidate vanishing point among all grids in the second image as the candidate location region; and determining the target location region of the candidate vanishing point in the image to be detected based on the position of the candidate location region in the second image and the conversion relationship between the size of the second image and the original size of the image to be detected. Based on the target location region, N candidate lines in the image to be detected are filtered to obtain M target lines from the N candidate lines, where M is an integer greater than 1 and less than or equal to N; Based on the M target lines, the target vanishing point is determined, including: arbitrarily selecting K groups of target lines from the M target lines, where K is an integer greater than 1, and each group of target lines includes two intersecting target lines; determining the intersection point of the two intersecting target lines in each group as the first predicted vanishing point of each group; traversing the remaining M-2 target lines and calculating the number of target lines passing through the first predicted vanishing points of each group; and determining the first predicted vanishing point with the most target line data among all the first predicted vanishing points passing through the K groups as the target vanishing point. The step of filtering N candidate lines in the image to be detected based on the target location region to obtain M target lines from the N candidate lines includes: The coordinates of the center point of the target location region are determined as the position coordinates of the candidate extinction point in the image to be detected; Calculate the distances from the position coordinates of the candidate vanishing points to the N candidate straight lines; Using the circumcircle radius and / or incircle radius of the target location region as a distance threshold, the distance from the position coordinates of the candidate vanishing point to any of the N candidate straight lines is compared with the distance threshold. The candidate lines whose position coordinates from the candidate vanishing point are less than the distance threshold among the N candidate lines are determined as target lines, so as to obtain M target lines among the N candidate lines.

2. The vanishing point detection method as described in claim 1, characterized in that, The position coordinates of the candidate extinction point in the image to be detected are the coordinates of any point in the grid where the candidate extinction point is located.

3. The vanishing point detection method as described in claim 1, characterized in that, Before performing line detection on the image to be detected, the vanishing point detection method further includes: The image to be detected is sequentially processed by grayscale conversion, noise reduction, binarization, and edge detection.

4. The vanishing point detection method as described in claim 1, characterized in that, After determining the target vanishing point from the first predicted vanishing point of all target lines in the K groups of target lines, the method further includes: H groups of target lines are randomly selected from the M target lines, where H is an integer greater than 1, and each group of target lines in the H groups includes two intersecting target lines. The intersection point of two intersecting target lines in each group of target lines in the H group is determined as the second predicted vanishing point of each group of target lines in the H group. Traverse the remaining M-2 target lines and calculate the number of target lines that pass through the second predicted vanishing point of each of the H groups of target lines; The second predicted vanishing point with the most target line data among all the second predicted vanishing points passing through the target line of the H group is determined as the vanishing point to be confirmed. If the number of target lines passing through the target vanishing point is less than the number of target lines passing through the vanishing point to be confirmed, the vanishing point to be confirmed will be taken as the new target vanishing point.

5. The vanishing point detection method as described in claim 1, characterized in that, After determining the target vanishing point, the vanishing point detection method further includes: Obtain the position coordinates of the target vanishing point; Based on the position coordinates of the target vanishing point, a mark is made in the image to be detected at the same position coordinates as the target vanishing point, thus obtaining the marked image to be detected.

6. A vanishing point detection device, characterized in that, The vanishing point detection device includes: The line detection module is used to perform line detection on the image to be detected, and obtain N candidate lines in the image to be detected, where N is an integer greater than 1; A network detection module is used to input the image to be detected into a vanishing point detection network to obtain the target location region of candidate vanishing points in the image to be detected. The network detection module includes the following sub-modules: a size adjustment sub-module, used to adjust the original size of the image to be detected to a preset size when the original size of the image to be detected is smaller or larger than a preset size, to obtain a first image; a grid division sub-module, used to divide the first image into grids to obtain a second image; a vanishing point detection sub-module, used to input the second image into the vanishing point detection network to obtain the candidate location region of candidate vanishing points in the second image; and a size conversion sub-module, used to determine the target location region of candidate vanishing points in the image to be detected based on the position of the candidate location region in the second image and the conversion relationship between the size of the second image and the original size of the image to be detected. The vanishing point detection sub-module includes the following units: a probability acquisition unit, used to input the second image into the vanishing point detection network to obtain the probability value of each grid in the second image having a candidate vanishing point; and a candidate region determination unit, used to determine the grid with the highest probability value of candidate vanishing points among all grids in the second image as the candidate location region. The line filtering module is used to filter N candidate lines in the image to be detected based on the target location region, and obtain M target lines from the N candidate lines, where M is an integer greater than 1 and less than or equal to N; The vanishing point determination module is used to determine the target vanishing point based on the M target lines. When M is greater than 2, the vanishing point determination module includes the following sub-modules: a first selection sub-module, used to arbitrarily select K groups of target lines from the M target lines, where K is an integer greater than 1, and each group of target lines includes two intersecting target lines; a first traversal sub-module, used to determine the intersection of the two intersecting target lines in each group of target lines as the first predicted vanishing point of each group of target lines, traversing the remaining M-2 target lines, and calculating the number of target lines passing through the first predicted vanishing point of each group of target lines; and a first target determination sub-module, used to determine the first predicted vanishing point with the most target line data among the first predicted vanishing points passing through the K groups of target lines as the target vanishing point. The linear filtering module is also used for: The coordinates of the center point of the target location region are determined as the position coordinates of the candidate extinction point in the image to be detected; Calculate the distances from the position coordinates of the candidate vanishing points to the N candidate straight lines; Using the circumcircle radius and / or incircle radius of the target location region as a distance threshold, the distance from the position coordinates of the candidate vanishing point to any of the N candidate straight lines is compared with the distance threshold. The candidate lines whose position coordinates from the candidate vanishing point are less than the distance threshold among the N candidate lines are determined as target lines, so as to obtain M target lines among the N candidate lines.

7. A robot, characterized in that, The system includes a camera, a memory, and a processor mounted on the robot; the camera is used to acquire an image to be detected, and the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images