Automatic control method and system for vehicle-mounted pan-tilt camera based on lane line recognition

Abstract

The present invention relates to an automatic control method and system for a vehicle-mounted pan-tilt camera based on lane line recognition. The lane lines in a specific frame are detected and analyzed to obtain the linear equations of each lane line in the picture. The horizontal and vertical coordinates of the vanishing points of the lane lines in the picture are calculated according to the linear equations. The deviation of the camera in the vertical direction is obtained according to the deviation between the vertical coordinate and the vertical coordinate of the center point of the specific frame. According to the horizontal coordinate of the center point of the focused lane during shooting, the deviation between the horizontal coordinate of the center point and the horizontal coordinate of the center point of the specific frame is calculated to obtain the deviation of the camera in the horizontal direction. Finally, the deviations of the camera in the vertical and horizontal directions are adjusted respectively according to the two deviation values ​​to realize automatic correction of the camera. In the above method, the camera will adjust the camera position in real time according to the collected picture with the help of the calculation of the vehicle-mounted terminal to ensure that the camera shooting area is always focused on the current lane.

Description

Technical Field

[0001] The present invention relates to the field of automatic control, and in particular to a method and system for automatic control of a vehicle-mounted pan-tilt camera based on lane line recognition. Background Art

[0002] The front-end image acquisition device of the vehicle-mounted road inspection product, namely the vehicle-mounted pan-tilt camera, needs to focus the image on the road lane, but the existing vehicle-mounted pan-tilt camera can only fix the shooting position or manually adjust the position, which brings great changes to the collection of road data, especially after the inspection vehicle equipped with the vehicle-mounted pan-tilt camera changes lanes, the original pan-tilt camera image will change. If the pan-tilt camera position is not adjusted manually in time, the vehicle-mounted pan-tilt camera will not be able to collect the required road lane image. Based on this, there is an urgent need for an automatic adjustment method and system for a vehicle-mounted pan-tilt camera to automatically, timely and accurately adjust the position of the vehicle-mounted pan-tilt camera. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for automatic control of a vehicle-mounted pan-tilt camera based on lane line recognition, which can automatically adjust the position of the vehicle-mounted pan-tilt camera in real time so that the camera shooting picture is always focused on the lane that needs attention.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for automatic deployment of a vehicle-mounted pan-tilt camera based on lane line recognition, comprising:

[0006] Detecting lane lines in a specific frame to obtain a binary image of the lane lines in the specific frame, where the specific frame is a picture taken by a camera at a preset time point;

[0007] Analyze the binary image of the lane line to obtain a straight line equation of each lane line in the binary image;

[0008] Calculating the vanishing point coordinates of the lane lines in the picture taken by the camera according to the straight line equations of the lane lines, and calculating the deviation between the vanishing point coordinates and the longitudinal coordinates of the center point of the specific frame picture, to obtain a first deviation value of the camera in the vertical direction;

[0009] Calculating the horizontal coordinate of the center point of the focused lane according to the straight line equation of the two lane lines of the focused lane when the camera is shooting, and calculating the deviation between the horizontal coordinate of the center point of the focused lane and the horizontal coordinate of the center point of the specific frame image, to obtain a second deviation value of the camera in the horizontal direction, wherein the focused lane is the lane that the lens of the camera is aimed at when shooting;

[0010] Adjust the vertical angle and the horizontal angle of the camera according to the first deviation value and the second deviation value, and jump to the step of "detecting the lane line in the specific frame image" until the first deviation value and the second deviation value are both zero, and then stop jumping.

[0011] The present invention also provides a vehicle-mounted pan-tilt camera automatic control system based on lane line recognition, comprising:

[0012] A binary image acquisition module, used to detect the lane line in a specific frame picture to obtain a binary image of the lane line in the specific frame picture, wherein the specific frame picture is a picture taken by a camera at a preset time point;

[0013] An analysis module, used to analyze the binary image of the lane line to obtain a straight line equation of each lane line in the binary image;

[0014] a first deviation value calculation module, configured to calculate the coordinates of the vanishing points of the lane lines in the image captured by the camera according to the linear equations of the lane lines, and to calculate the deviation between the coordinates of the vanishing points and the longitudinal coordinates of the center point of the specific frame image, so as to obtain a first deviation value of the camera in the vertical direction;

[0015] a second deviation value calculation module, configured to calculate the horizontal coordinate of the center point of the focused lane according to the straight line equations of the two lane lines of the focused lane when the camera is shooting, and calculate the deviation between the horizontal coordinate of the center point of the focused lane and the horizontal coordinate of the center point of the specific frame image, to obtain a second deviation value of the camera in the horizontal direction, wherein the focused lane is the lane that the camera lens is aimed at when shooting;

[0016] An adjustment module is used to adjust the vertical angle and the horizontal angle of the camera according to the first deviation value and the second deviation value, and jump to the step of "detecting the lane line in a specific frame image" until the first deviation value and the second deviation value are both zero, and then stop jumping.

[0017] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0018] The present invention provides a method and system for automatic deployment of a vehicle-mounted pan-tilt camera based on lane line recognition. First, the lane lines in a specific frame are detected to obtain a binary image in the frame. Then, the binary image is analyzed to obtain the linear equations of each lane line in the binary image. The vanishing points of the lane lines in the shooting picture are calculated according to the linear equations. Then, the deviation between the ordinate of the vanishing point and the ordinate of the center point of the specific frame is calculated to obtain the first deviation value of the camera in the vertical direction. The horizontal coordinate of the center point of the focused lane is calculated according to the linear equations of the two lane lines of the focused lane when the camera is shooting, and the deviation between the horizontal coordinate of the center point and the horizontal coordinate of the center point of the specific frame is calculated to obtain the second deviation value of the camera in the horizontal direction. Finally, the deviations of the camera in the vertical and horizontal directions are adjusted respectively according to the first deviation value and the second deviation value to realize automatic camera correction. In the above method of the present invention, the pan-tilt camera will adjust the position of the pan-tilt camera in real time according to the collected picture with the help of the calculation of the vehicle-mounted terminal, so that the picture is always focused on the lane that needs to be paid attention to, and the interference of vehicle turning, lane changing or external displacement of the vehicle-mounted pan-tilt camera caused by external force is eliminated. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flow chart of a method for automatic deployment of a vehicle-mounted PTZ camera based on lane line recognition provided in Example 1 of the present invention;

[0021] Figure 2 A picture showing a lane line in a specific frame picture provided by Embodiment 1 of the present invention converted into a binary image;

[0022] Figure 3 The lane line is processed into a straight line provided in Embodiment 1 of the present invention, and the straight line is located in the image of the x and y coordinate axes;

[0023] Figure 4 A schematic diagram of an area surrounded by two lane lines and a vanishing point of a focused lane provided in Embodiment 1 of the present invention;

[0024] Figure 5 This is a structural diagram of the vehicle-mounted pan-tilt camera automatic control system based on lane line recognition provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The purpose of the present invention is to provide a method and system for automatic control of a vehicle-mounted pan-tilt camera based on lane line recognition, which can adjust the shooting area and position of the vehicle-mounted pan-tilt camera in real time through a vehicle-mounted terminal to ensure that the shooting area of ​​the vehicle-mounted pan-tilt camera is always focused on the current lane.

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1

[0029] This embodiment provides a method for automatic deployment of a vehicle-mounted PTZ camera based on lane line recognition. Figure 1 , the method comprising:

[0030] S1. Detect lane lines in a specific frame image to obtain a binary image of the lane lines in the specific frame image, where the specific frame image is an image captured by a camera at a preset time point.

[0031] Specifically, in this embodiment, the vehicle-mounted terminal can be used to detect the lane lines in the specific frame image through a lane line recognition algorithm based on deep learning to obtain a binary image of the lane lines in the specific frame image.

[0032] The lane line recognition algorithm includes segmentation, clustering and fitting algorithms, and finally outputs a binary image with the lane line area being white and the background being black. The converted image is as follows: Figure 2 shown.

[0033] As an optional implementation, the method further includes receiving a video captured by a camera, and cutting frames of the video according to a preset frequency.

[0034] Specifically, the video stream captured by the PTZ camera can be set to 25 frames per second, and the video stream is transmitted to the vehicle terminal. The vehicle terminal takes the last frame of 1 second as a specific frame for analysis.

[0035] As another optional implementation, before detecting the lane line in the specific frame, the method further includes:

[0036] Set the camera shooting angle: the camera focuses on lane r (which can be lane 1, 2, 3, or 4) in the horizontal direction of the shooting picture, and the vertical midpoint of the shooting picture is consistent with the ordinate of the vanishing point of the lane line in the picture.

[0037] S2. Analyze the binary image of the lane lines to obtain a straight line equation of each lane line in the binary image.

[0038] Optionally, step S2 includes:

[0039] S21. The vehicle-mounted terminal denoises the binary image of the lane line using an image denoising algorithm to obtain a denoised image.

[0040] Specifically, the image denoising algorithm removes noise points according to the size of the connected domain.

[0041] S22, using a thinning algorithm to perform thinning processing on the denoised image to obtain a thinned image.

[0042] The Zhang Suen algorithm can be used as the thinning algorithm.

[0043] S23, using the RANSAC line detection algorithm to process each lane line in the refined image into a straight line, and obtain a straight line equation of each lane line.

[0044] Among them, the straight line equation of the lane line is Y = K n X+B n , n is the lane line number, K n and B n For known coefficients, the above process presets the coordinate system X-axis and Y-axis, such as Figure 3 shown.

[0045] S3. Calculate the vanishing point coordinates of the lane lines in the picture taken by the camera according to the straight line equations of the lane lines, and calculate the deviation between the vanishing point coordinates and the vertical coordinates of the center point of the specific frame picture to obtain a first deviation value of the camera in the vertical direction.

[0046] The vanishing point refers to the point where two parallel lines in the real world intersect in the image. Figure 3 The four lane lines that are originally parallel in reality will eventually intersect at one point, which is the vanishing point. According to the linear equation Y=K corresponding to each lane line in step S23 n X+B n Calculate the coordinates of the intersection point. Take any combination of two straight line equations and solve them to calculate the coordinates of the vanishing point in the figure (X vp , Y vp ).

[0047] Since the camera shooting angle is set, the midpoint of the vehicle-mounted gimbal camera image (i.e. Figure 3 The ordinate of the center point of the camera screen should be consistent with the ordinate of the vanishing point. If they are not consistent, it means that the vehicle-mounted gimbal camera has a position deviation in the vertical direction. C It can be calculated (for example, assuming the image is 1920*1080, the coordinates of the midpoint of the screen are (960, 540), that is, the ordinate Y of the midpoint C is 540). Finally, the vertical deviation E of the vehicle-mounted gimbal camera is calculated by the formula v =|Y vp -Y c |.

[0048] S4. Calculate the horizontal coordinate of the center point of the focused lane according to the straight line equation of the two lane lines of the focused lane when the camera is shooting, and calculate the deviation between the horizontal coordinate of the center point of the focused lane and the horizontal coordinate of the center point of the specific frame image to obtain a second deviation value of the camera in the horizontal direction. The focused lane is the lane that the lens of the camera is aimed at when shooting.

[0049] When the camera is used for shooting, the lane that the camera focuses on is preset to be lane r. Figure 3 The detection of the middle lane line, counting from the left side of the figure, the area enclosed by the 1st-2nd lane lines is the 1st lane, the area enclosed by the 2nd-3rd lane lines is the 2nd lane, and so on. The preset focus lane (assuming that the preset lane r is 3 here) is surrounded by the two adjacent lane lines and the vanishing point. Figure 4 As shown in the shaded area, the horizontal coordinate of the center point of the shaded area is X r It can be obtained by calculation, and the calculation process is as follows:

[0050] The equations of the lines on both sides of the shaded area are Y=K r X+B r and Y = K r+1 X+B r+1 , the abscissa of the intersection of the straight line r and the image is X lr =-B r / K r Y coordinate lr =0, and similarly obtain the horizontal coordinate X of the intersection of the straight line r+1 and the X axis. lr+1 =-B r+1 / K r+1 , vertical coordinate Y lr+1 =0, and finally the horizontal coordinate X of the center point of the shadow area can be obtained by calculation. r =(X lr +X lr+1 ) / 2=-(B r / K r +Br+1 / K r+1 ) / 2.

[0051] The lane where the center point of the shadow area is located is the lane that the vehicle-mounted gimbal camera needs to focus on, and the center point of the image is the current focus position of the vehicle-mounted gimbal camera. Therefore, the deviation between the center point of the shadow area and the center point of the image on the X-axis is the deviation of the vehicle-mounted gimbal camera in the horizontal direction, which is the position that needs to be adjusted in the horizontal direction. Finally, the horizontal deviation E of the vehicle-mounted gimbal camera can be calculated by the formula h =|X vp -X r |.

[0052] S5. Adjust the vertical angle and the horizontal angle of the camera according to the first deviation value and the second deviation value, and jump to the step of "detecting the lane line in the specific frame image" until the first deviation value and the second deviation value are both zero, and then stop jumping.

[0053] The vertical deviation E output from the above steps S3 and S4 is v and horizontal deviation E h If both values ​​are 0, there is no need to change the current vehicle-mounted PTZ camera shooting angle, and the vehicle-mounted terminal does not need to send a control signal; if either value is not 0, then the formula A v =K v *E v and A h =K h *E h Calculate the vertical and horizontal angles that the vehicle-mounted gimbal camera needs to move, where K v and K h If the specific coefficient is known, then return to step S3 and repeat the above steps until both values ​​are 0.

[0054] The solution provided in this embodiment is mainly used for vehicle-mounted road inspection products. The front-end image acquisition device of the product, that is, the vehicle-mounted pan-tilt camera, needs to focus the picture on the road lane. After the above technical method is implemented, the vehicle-mounted pan-tilt camera will adjust the position of the pan-tilt camera in real time according to the collected picture with the help of calculation by the vehicle-mounted terminal, so that the picture is always focused on the lane that needs attention, eliminating interference from vehicle turning, lane changing, or position displacement of the vehicle-mounted pan-tilt camera due to external forces.

[0055] Example 2

[0056] This embodiment provides a vehicle-mounted PTZ camera automatic deployment control system based on lane line recognition, please refer to Figure 5 ,include:

[0057] A binary image acquisition module M1 is used to detect the lane line in a specific frame image to obtain a binary image of the lane line in the specific frame image, where the specific frame image is an image taken by a camera at a preset time point;

[0058] An analysis module M2 is used to analyze the binary image of the lane line to obtain a straight line equation of each lane line in the binary image;

[0059] A first deviation value calculation module M3, used to calculate the vanishing point coordinates of the lane lines in the picture taken by the camera according to the straight line equations of the lane lines, and calculate the deviation between the vanishing point coordinates and the longitudinal coordinates of the center point of the specific frame picture, to obtain a first deviation value of the camera in the vertical direction;

[0060] A second deviation value calculation module M4 is used to calculate the horizontal coordinate of the center point of the focused lane according to the straight line equations of the two lane lines of the focused lane when the camera is shooting, and calculate the deviation between the horizontal coordinate of the center point of the focused lane and the horizontal coordinate of the center point of the specific frame image, to obtain a second deviation value of the camera in the horizontal direction, wherein the focused lane is the lane that the lens of the camera is aimed at when shooting;

[0061] The adjustment module M5 is used to adjust the vertical angle and the horizontal angle of the camera according to the first deviation value and the second deviation value, and jump to the step of "detecting the lane line in a specific frame image" until the first deviation value and the second deviation value are both zero, and then stop jumping.

[0062] As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0063] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for automatic deployment of vehicle-mounted PTZ cameras based on lane line recognition, characterized in that: include: Detecting lane lines in a specific frame to obtain a binary image of the lane lines in the specific frame, where the specific frame is a picture taken by a camera at a preset time point; Analyze the binary image of the lane line to obtain a straight line equation of each lane line in the binary image; Calculating the vanishing point coordinates of the lane lines in the picture taken by the camera according to the straight line equations of the lane lines, and calculating the deviation between the vanishing point coordinates and the longitudinal coordinates of the center point of the specific frame picture, to obtain a first deviation value of the camera in the vertical direction; Calculating the horizontal coordinate of the center point of the focused lane according to the straight line equation of the two lane lines of the focused lane when the camera is shooting, and calculating the deviation between the horizontal coordinate of the center point of the focused lane and the horizontal coordinate of the center point of the specific frame image, to obtain a second deviation value of the camera in the horizontal direction, wherein the focused lane is the lane that the lens of the camera is aimed at when shooting; The vertical angle and the horizontal angle of the camera are adjusted according to the first deviation value and the second deviation value, and the process jumps to the step of "detecting the lane line in the specific frame" until the first deviation value and the second deviation value are both zero, and then the process stops jumping; the vertical angle and the horizontal angle of the camera are adjusted according to the first deviation value and the second deviation value as follows: v =K v *E v and A h =K h *E h Calculate the vertical and horizontal angles that the vehicle-mounted gimbal camera needs to move; where A v A is the vertical angle that the camera needs to move; h K is the horizontal angle that the camera needs to move; v and K h is a known specific coefficient; E v is the vertical deviation; E h The horizontal deviation is used to adjust the shooting area and position of the vehicle-mounted PTZ camera in real time through the vehicle-mounted terminal to ensure that the shooting area of ​​the vehicle-mounted PTZ camera is always focused on the current lane; Analyzing the binary image of the lane line to obtain the straight line equation of each lane line in the binary image specifically includes: De-noising the binary image of the lane line using an image denoising algorithm to obtain a denoised image; Image denoising algorithms remove noise points based on the size of connected domains; Using a thinning algorithm to perform thinning processing on the denoised image to obtain a thinned image; The Zhang Suen algorithm is used as the thinning algorithm; The RANSAC line detection algorithm is used to process each lane line in the refined image into a straight line, and the straight line equation of each lane line is obtained. Any two straight line equations are combined and solved to calculate the vanishing point coordinates.

2. The method according to claim 1, characterized in that The method also includes receiving a video captured by a camera, and cutting frames of the video according to a preset frequency.

3. The method according to claim 1, characterized in that Before detecting the lane line in the specific frame, the method further includes: Set the camera shooting angle: the camera focuses on lane r in the horizontal direction of the shooting picture, and the vertical midpoint of the shooting picture is consistent with the longitudinal coordinate of the vanishing point of the lane line in the picture.

4. The method according to claim 1, characterized in that The detecting of the lane line in the specific frame to obtain the binary image of the lane line in the specific frame specifically includes: A lane line recognition algorithm based on deep learning is used to detect the lane lines in the specific frame image to obtain a binary image of the lane lines in the specific frame image.

5. The method according to claim 1, characterized in that The calculating the horizontal coordinate of the center point of the focused lane according to the straight line equations of the two lane lines of the focused lane when the camera is shooting specifically includes: Calculate a first horizontal coordinate of a coordinate of an intersection of one of the lane lines and the x-axis according to a straight line equation of one of the lane lines of the focused lane; Calculating a second horizontal coordinate of the intersection point of another lane line and the x-axis according to a straight line equation of another lane line of the focused lane; An average value of the first horizontal coordinate and the second horizontal coordinate is calculated to obtain the horizontal coordinate of the center point of the focused lane.

6. A vehicle-mounted PTZ camera automatic deployment control system based on lane line recognition, which operates according to the vehicle-mounted PTZ camera automatic deployment control method based on lane line recognition according to any one of claims 1 to 5, characterized in that: include: A binary image acquisition module, used to detect the lane line in a specific frame picture to obtain a binary image of the lane line in the specific frame picture, wherein the specific frame picture is a picture taken by a camera at a preset time point; An analysis module, used to analyze the binary image of the lane line to obtain a straight line equation of each lane line in the binary image; a first deviation value calculation module, configured to calculate the coordinates of the vanishing points of the lane lines in the image captured by the camera according to the linear equations of the lane lines, and to calculate the deviation between the coordinates of the vanishing points and the longitudinal coordinates of the center point of the specific frame image, so as to obtain a first deviation value of the camera in the vertical direction; a second deviation value calculation module, configured to calculate the horizontal coordinate of the center point of the focused lane according to the straight line equations of the two lane lines of the focused lane when the camera is shooting, and calculate the deviation between the horizontal coordinate of the center point of the focused lane and the horizontal coordinate of the center point of the specific frame image, to obtain a second deviation value of the camera in the horizontal direction, wherein the focused lane is the lane that the camera lens is aimed at when shooting; an adjustment module, configured to adjust the vertical angle and the horizontal angle of the camera according to the first deviation value and the second deviation value, and jump to the step of "detecting the lane line in the specific frame image" until the first deviation value and the second deviation value are both zero, and then stop jumping; Analyzing the binary image of the lane line to obtain the straight line equation of each lane line in the binary image specifically includes: De-noising the binary image of the lane line using an image denoising algorithm to obtain a denoised image; Using a thinning algorithm to perform thinning processing on the denoised image to obtain a thinned image; Each lane line in the refined image is processed into a straight line using the RANSAC straight line detection algorithm, and a straight line equation of each lane line is obtained.

7. The system according to claim 6, characterized in that The system further comprises: The frame cutting module is used to receive the video shot by the camera and cut the frame of the video according to a preset frequency.

8. The system according to claim 6, characterized in that The system also includes: a camera shooting angle setting module, which is used to set the camera shooting angle before detecting the lane line in the specific frame: the horizontal direction of the camera shooting picture is focused on the lane, and the midpoint of the shooting picture in the vertical direction is consistent with the longitudinal coordinate of the vanishing point of the lane line in the picture.

9. The system according to claim 6, characterized in that The binary image acquisition module comprises: The binary image acquisition submodule is used to detect the lane lines in the specific frame image based on a lane line recognition algorithm of deep learning to obtain a binary image of the lane lines in the specific frame image.

Images