A computer vision-based light beam irradiation point prediction method and device

Abstract

The application discloses a kind of light beam irradiation point prediction method and device based on computer vision, belong to computer vision positioning technical field.This technical scheme includes: the image obtained by camera shooting is input to visual positioning device, obtains the image coordinate of at least one target in image;Image features of target are extracted by computer vision method;Combining the image features extracted, the physical parameters of target and the parameters of camera are estimated the distance between target and camera lens;According to the distance information between target and camera lens and the spatial position relationship between camera and light beam generating device, the image coordinates of at least one light beam irradiation point are estimated;The image coordinates of light beam irradiation point calculated and the image coordinates of target are used to predict the final light beam irradiation point.The application uses computer vision technology to realize a kind of low-cost light beam irradiation point prediction scheme, cooperate with light source scheduling in practical application scene, unnecessary energy consumption and light source pollution can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision positioning, and particularly relates to a light beam irradiation point prediction method and device based on computer vision. BACKGROUND

[0002] The propagation characteristics of light beams make them widely used in positioning, energy transmission and other fields. However, the potential harm caused by light pollution and laser burns due to improper use of lasers is increasingly apparent. Therefore, there is an urgent need for an external technology to predict the light beam irradiation point, so that the light beam generator works within a controllable range, thereby reducing unnecessary energy consumption and light source pollution.

[0003] At present, although the spatial modeling technology represented by three-dimensional point clouds has relatively mature development, the cost of equipment is high, the calculation amount is large, and the real-time performance is poor, so it is difficult to be used in small and medium-sized actual scenes. Point cloud mapping often needs professional binocular cameras or expensive laser radars, supplemented by professional computing acceleration equipment, to realize real-time reconstruction of the environment. However, for light beam irradiation point prediction and calculation, point cloud mapping calculates a large amount of unnecessary information and does not fully utilize the relationship between existing devices. SUMMARY

[0004] The main purpose of the present application is to provide a light beam irradiation point prediction method and device based on computer vision, which can realize a low-cost light beam irradiation point prediction scheme, cooperate with light source scheduling in actual application scenes, and avoid unnecessary energy consumption and light source pollution.

[0005] In order to achieve the above purpose, the present application provides a light beam irradiation point prediction method and device based on computer vision, comprising the following steps:

[0006] S1, capturing image information of a light beam irradiation area by a camera;

[0007] S2, inputting the image information captured by the camera into a visual positioning device, the visual positioning device judging whether a target irradiated exists in the image, and if the target irradiated exists, outputting image position information of at least one target; the image position information includes image coordinates of the target and non-missing and non-occluded bounding box image coordinate information;

[0008] S3, extracting image features around the at least one target image by using a computer vision method according to the image position information of the target;

[0009] S4, estimating a distance between the at least one target and a camera lens according to the extracted image features around the at least one target image, physical parameters of the at least one target and parameters of the camera.

[0010] S5, estimating image coordinates of at least one light beam irradiation point according to distance information between the at least one target and the camera lens and spatial position relationship between the camera and the light beam generating device;

[0011] S6, calculating pixel distance between each target image coordinate and image coordinate of the corresponding light beam irradiation point and converting the pixel distance into actual distance, regarding the target as a possible light beam irradiation point if the distance is less than the light beam radius, and selecting the target close to the light beam generating device as the final light beam irradiation point according to the light propagation principle if there are multiple targets being possible light beam irradiation points.

[0012] Further, the visual positioning device is equipped with a deep neural network model, which can judge whether there is a potential irradiation target in the image in real time through pre-extracted features.

[0013] Further, the image features around the target image include one or more of color features, shape features, scale features and contour features of the target, and the corresponding feature atlas is output.

[0014] Further, the distance between the target and the camera lens is estimated through camera imaging modeling, three-dimensional modeling and sensor method according to the color features, shape features, scale features and contour features of the target and the parameters of the camera.

[0015] Further, the image coordinate estimation of the light beam irradiation point includes the following steps:

[0016] S5-1, obtaining distance information between the target and the camera lens, parameters and pose information of the camera;

[0017] S5-2, obtaining spatial position relationship between the camera axis and the light beam generating device optical axis, including angle and distance information;

[0018] S5-3, using camera imaging principle, combining the above data, and calculating the image coordinates of the light beam irradiation point on the two-dimensional image through the actual spatial position relationship.

[0019] The application also provides a light beam irradiation point prediction device based on computer vision, characterized by comprising an identification unit, a distance measuring unit and a calculation unit.

[0020] The identification unit is used for collecting image information of a light beam irradiation region, judging whether there is a target laser receiving device in the image, and returning image position information of at least one target if there is.

[0021] The ranging unit is used for extracting the image features and estimating the distance between the target and the camera lens according to the physical parameters of the target and the parameters of the camera.

[0022] The computing unit calculates the candidate image coordinates of the light beam irradiation point and the image coordinate predicted final irradiation point of the target according to the distance information between the target and the camera lens and the spatial position relationship between the camera and the light beam generating device.

[0023] Further, the recognition unit comprises a visual positioning device, carries a deep neural network model, and judges whether there is a potential target image in the image in real time through the pre-extracted features.

[0024] The present application has the following beneficial effects:

[0025] The present application realizes a low-cost light beam irradiation point prediction scheme by using computer vision technology, which can avoid unnecessary energy consumption and light source pollution in cooperation with light source scheduling in actual application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the application scene diagram of the light beam irradiation point prediction method based on computer vision in the embodiment of the present application.

[0027] Figure 2 is the flowchart of the light beam irradiation point prediction based on computer vision in the embodiment of the present application.

[0028] Figure 3 is the flowchart of the target image coordinate estimation in the embodiment of the present application.

[0029] Figure 4 is the principle model diagram of the target image coordinate estimation in the embodiment of the present application.

[0030] Figure 5 is the structural diagram of the light beam irradiation point prediction based on computer vision in the embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] In the following description, the suffixes such as "module", "part", or "unit" used for an element are used only for convenience of explanation of the present application, and have no particular meaning by themselves. Thus, "module", "part", or "unit" can be mixedly used.

[0033] As shown in Figure 1 , it is an application scenario diagram of a computer vision-based light beam irradiation point prediction method in an embodiment of the present application.

[0034] Specifically, the computer vision-based light beam irradiation point prediction method in the present application is described below by taking the remote laser charging of a UAV as an example, which does not mean that the present application is only applicable to the remote laser charging of a UAV. The present application can be used for the prediction of any light beam irradiation point. The laser receiving device 2 is suspended below the UAV 1, and the laser generator 3 and the camera 5 are installed on the two-degree-of-freedom holder 4. Through the present application, the prediction of the position of the laser irradiation point during the remote laser charging process can be realized. When the laser irradiation point is not on the laser receiver, the system can actively turn off the laser generator, thereby avoiding unnecessary energy consumption and light source pollution.

[0035] As shown in Figure 2 and Figure 3 , the computer vision-based light beam irradiation point prediction method can include:

[0036] Step S1: capturing image information of the area where the laser receiving device is located by a camera.

[0037] For example, the image of the UAV is captured by the camera installed on the automatic tracking holder. The automatic tracking holder should actively adjust the pose orientation of the holder as the UAV moves, so that the UAV and its laser receiving device always remain in the camera picture.

[0038] Step S2: inputting the image captured by the camera into a visual positioning device. The visual positioning device should determine whether the target laser receiving device exists in the image. If it exists, the image position information and the length and width size of the target laser receiving device should be outputted.

[0039] Specifically, the visual positioning device can include a parallel acceleration device that supports instant reasoning, such as a GPU, and a laser receiving device recognition model obtained by training a fine-grained target detection deep learning network model (such as the RCNN series, SSD, YOLO series, etc.). For example, first, image data sets of the laser receiving device under different illuminations, different backgrounds, and different angles are collected, and the boundary box label data of the laser receiving device are labeled. Then, the image data sets are input into the target detection deep learning network model for training until the network parameters converge. After obtaining reliable laser receiving device recognition model parameters, the image information of the region where the laser receiving device is located obtained in step S1 is input into the model for reasoning to obtain the size and position information of the boundary box of the laser receiving device.

[0040] In step S3, image features around the image of the laser receiving device are extracted by using a computer vision method according to the image position information of the target laser receiving device. The image features should represent the contour, color, and size information of the laser receiving device.

[0041] For example, for extracting the size information of the cylindrical laser receiving device, the image features around the image of the laser receiving device can be extracted by using edge detection, dilation, erosion, and straight line detection operations to extract the contour image of the laser receiving device, or by using binarization image, dilation, and erosion operations to extract the vertices around the image of the laser receiving device.

[0042] The edge detection can use a Gaussian filter to smooth the image, then use a first-order partial derivative finite difference to calculate the gradient amplitude and direction, then perform non-maximum suppression on the gradient amplitude, and finally use a double-threshold algorithm to detect and connect the edges to obtain a binarized contour image.

[0043] Dilation and erosion belong to morphological operations. The dilation operation expands and enlarges the bright white region in the image by adding pixels to the perceived boundary of the object in the image. The erosion operation removes pixels along the object boundary and reduces the size of the object. The combination of the two operations can remove noise points and connect breakpoints from the binarized contour image.

[0044] The straight line detection can use Hough transform. The main principle is that for each pixel point (x0, y0) of the edge, all straight lines y=kx+b that can pass through it are mapped to the Hough space, and then voting is performed. For the two extracted contour straight lines, the diameter of the laser receiving device can be obtained by calculating the distance between the two straight lines.

[0045] In step S4, the distance between the laser receiving device and the camera lens is estimated according to the extracted image features, the physical parameters of the laser receiving device, and the parameters of the camera.

[0046] The distance between the laser receiving device and the camera lens can be directly obtained by a sensor or calculated by a computer vision method. For example, the diameter of the laser receiving device obtained in step S3 The measured diameter D of the laser receiving device, the measured focal length f of the camera, and the distance dist between the high-definition camera and the laser receiving device are calculated by using the formula.

[0047]

[0048] In step S5, the image coordinates of the laser charging are estimated according to the distance information between the laser receiving device and the camera lens and the spatial position relationship between the camera and the laser generating device.

[0049] Specifically, please combine Figure 4 In an embodiment, step S5 can be the image coordinate estimation of the laser charging by mapping three-dimensional space to a two-dimensional plane using camera imaging principles, specifically including:

[0050] In step S5-1, the distance information between the laser receiving device and the camera lens, the parameters and pose information of the camera are obtained.

[0051] In step S5-2, the spatial position relationship between the camera axis and the laser generating device optical axis is obtained, including the included angle and distance information.

[0052] In the above embodiment, the laser generating device and the camera are installed in parallel on both sides of the two-degree-of-freedom holder, wherein the laser generating device is installed on the left side and the camera is installed on the right side. The included angle of the laser generating device optical axis in the horizontal direction is always consistent with the included angle of the camera axis in the horizontal direction.

[0053] In step S5-3, the image coordinates of the laser charging on the two-dimensional image are calculated by using the camera imaging principle and combining the above data through the actual spatial position relationship.

[0054] Specifically, according to the camera imaging principle, the optical axis of the camera always remains in the center (0, 0) of the image. According to the installation position of step S5-2, the image coordinates of the laser charging always remain on the left side of the optical axis coordinates of the camera. The distance dist between the high-definition camera and the laser receiving device obtained in step S4, the distance between the camera axis and the laser generating device optical axis obtained in step S5-2 The measured focal length f of the camera, and the image coordinates (m, n) of the laser charging are calculated by using the formula.

[0055]

[0056] (m, n) = (0-bias, 0)

[0057] Step S6, calculate the distance between the predicted image coordinates and the image coordinates of the laser receiving device, if the distance is greater than the preset threshold or no target laser receiving device is detected in step 2, turn off the laser generating device; if the distance is less than the preset threshold, turn on the laser generating device.

[0058] Reference Figure 5 , for an embodiment of the application, a module structure schematic diagram of a computer vision-based light beam irradiation point prediction device 100.

[0059] The structure of the computer vision-based light beam irradiation point prediction device 100 comprises:

[0060] The recognition unit 101 comprises a camera 1011 and a visual positioning device 1012, which is used to collect image information of the area where the laser receiving device is located by the camera 1011, judge whether there is a target laser receiving device in the image, and if there is, output the image position information of the target laser receiving device.

[0061] Specifically, the visual positioning device 1012 can comprise a parallel acceleration device that can support instant reasoning, such as a GPU; and a laser receiving device recognition model obtained after training a fine-grained target detection deep learning network model (such as RCNN series, SSD, YOLO series, etc.). For example, first, collect image data sets of laser receiving devices under different illuminations, different backgrounds, and different angles, and label the boundary box label data of the laser receiving device, then input to the target detection deep learning network model for training until the network parameters converge, obtain reliable laser receiving device recognition model parameters, input the image information of the area where the laser receiving device is located obtained by the camera 1011 into the model for reasoning, and obtain the size and position information of the laser receiving device boundary box.

[0062] The distance measuring unit 102 is used to extract image features around the laser receiving device image and estimate the distance between the laser receiving device and the camera lens. The specific estimation method has been described above and will not be repeated here.

[0063] The calculation unit 103 is used to estimate the image coordinates of the laser charging and calculate the distance between the predicted image coordinates and the image coordinates of the laser receiving device. The specific estimation method has been described above and will not be repeated here. In addition, it is also used to turn on or turn off the laser generating device, if the distance is greater than the preset threshold or no target laser receiving device is detected, turn off the laser generating device; if the distance is less than the preset threshold, turn on the laser generating device.

[0064] The above embodiments are the preferred embodiments of the present application, and do not limit the patent scope of the present application, and any equivalent structural changes made under the concept of the present application, using the content of the present application specification and drawings, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A computer vision-based beam spot prediction method, characterized in that, The method comprises the following steps: S1, capturing image information of a light beam irradiation area by a camera; S2, inputting the image information into a visual positioning device, the visual positioning device judging whether there is a target irradiated in the image, and if there is, outputting image position information of at least one target; the image position information comprises image coordinates of the target and non-missing and non-occluded bounding box image coordinate information; S3, extracting image features around the at least one target image according to the image position information of the target by using a computer vision method; S4, estimating the distance between the at least one target and the camera lens according to the extracted image features around the at least one target image, the physical parameters of the at least one target and the parameters of the camera; S5, estimating the image coordinates of at least one light beam irradiation point according to the distance information between the at least one target and the camera lens and the spatial position relationship between the camera and the light beam generating device; S5-1, obtaining the distance information between the target and the camera lens, the parameters and pose information of the camera; S5-2, obtaining the spatial position relationship between the camera axis and the optical axis of the light beam generating device, including the angle and distance information; S5-3, using the camera imaging principle, combining the above data, and calculating the image coordinates of the light beam irradiation point on the two-dimensional image through the actual spatial position relationship; (m, n) = (0-bias, 0) wherein: dist is the distance between the at least one target and the camera lens, is the distance between the camera axis and the optical axis of the light beam generating device, f is the measured camera focal length, (m, n) are the image coordinates of the light beam impact point on the two-dimensional image; S6, calculating the pixel distance between each target image coordinate and its corresponding light beam irradiation point image coordinate and converting it into actual distance, if the distance is less than the light beam radius, regarding the target as a possible light beam irradiation point, if there are multiple targets that are possible light beam irradiation points, selecting the target close to the light beam generating device as the final light beam irradiation point according to the propagation principle of light.

2. The computer vision-based beam spot prediction method of claim 1, wherein: The visual positioning device is equipped with a deep neural network model, which can judge whether there is a potential irradiated target in the image in real time through pre-extracted features.

3. The computer vision-based beam spot prediction method of claim 1, wherein, The image features around the target image include one or more of the color features, shape features, size features and contour features of the target, and the corresponding feature atlas is outputted.

4. The computer vision-based beam spot prediction method of claim 3, wherein: The distance between the target and the camera lens is estimated through camera imaging modeling, three-dimensional modeling and sensor method according to the color features, shape features, size features and contour features of the target and the parameters of the camera.

5. A computer vision based beam spot prediction apparatus, characterized by: It comprises a recognition unit, a distance measuring unit and a calculation unit; The recognition unit comprises a camera, which is used for collecting image information of a light beam irradiation area, judging whether there is a target irradiated in the image, and returning image position information of at least one target if there is; The distance measuring unit is used for extracting the image features and estimating the distance between the target and the camera lens according to the physical parameters of the target and the parameters of the camera; The computing unit calculates the pixel distance between each target image coordinate and its corresponding light beam irradiation point image coordinate according to the distance information between the target and the camera lens and the spatial position relationship between the camera and the light beam generating device, and converts it into an actual distance. If the distance is less than the light beam radius, the target is considered as a possible light beam irradiation point. If there are multiple targets that are possible light beam irradiation points, according to the principle of light propagation, the target close to the light beam generating device is selected as the final light beam irradiation point. Specifically, the calculation of the pixel distance between each target image coordinate and its corresponding light beam irradiation point image coordinate according to the distance information between the target and the camera lens and the spatial position relationship between the camera and the light beam generating device is as follows: S1, obtaining the distance information between the target and the camera lens, the parameters and pose information of the camera; S2, obtaining the spatial position relationship between the camera axis and the light beam generating device optical axis, including the included angle and distance information; S3, using the camera imaging principle, combining the above data, and calculating the image coordinate of the light beam irradiation point on the two-dimensional image through the actual spatial position relationship; (m, n) = (0-bias, 0) wherein: dist is the distance between the at least one target and the camera lens, is the distance between the camera axis and the optical axis of the light beam generating device, f is the measured camera focal length, and (m, n) are the image coordinates of the light beam impact point on the two-dimensional image.

6. The computer vision-based beam spot prediction apparatus of claim 5, wherein, The recognition unit further comprises a visual positioning device carrying a deep neural network model, which can judge in real time whether there is a potential target in the image through the pre-extracted features.

Images