Method, device and electronic equipment for determining pitch angle of vehicle-mounted camera

Abstract

The present invention provides a method, device and electronic device for determining the pitch angle of a vehicle-mounted camera. First, based on the current image captured by the vehicle-mounted camera, the lane area of ​​the target vehicle on which the vehicle-mounted camera is fixed in the current image is determined; then, based on the image attribute information of the lane area of ​​the target vehicle in the current image, the initial pitch angle corresponding to the vehicle-mounted camera when capturing the current image is determined; finally, based on the initial pitch angle and the historical pitch angle associated with the initial pitch angle, the pitch angle corresponding to the vehicle-mounted camera when capturing the current image is determined. In the present invention, the pitch angle of the vehicle-mounted camera when capturing the image is initially determined by the image attribute information of the lane area; and then, based on the initially determined pitch angle and the associated historical pitch angle, the pitch angle corresponding to the vehicle-mounted camera when capturing the current image is accurately determined, thereby improving the accuracy of distance measurement based on the image captured by the vehicle-mounted camera.

Description

Technical Field

[0001] The present invention relates to the technical field of visual ranging, and in particular to a method, device and electronic equipment for determining the pitch angle of a vehicle-mounted camera. Background Art

[0002] In the related art, the monocular vision ranging method usually determines the distance information of obstacles around the current vehicle based on the pitch angle of the on-board camera calibrated when stationary and the pictures it takes. However, when the vehicle is driving, the pitch angle of the on-board camera will change due to the shaking of the vehicle, resulting in inaccurate ranging results. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method, device and electronic equipment for determining the pitch angle of a vehicle-mounted camera, so as to accurately determine the pitch angle corresponding to the vehicle-mounted camera when shooting the current image, and improve the accuracy of ranging based on the pictures taken by the vehicle-mounted camera.

[0004] In a first aspect, an embodiment of the present invention provides a method for determining the pitch angle of a vehicle-mounted camera, comprising: determining a lane area of ​​a target vehicle in the current image based on a current image captured by the vehicle-mounted camera; the vehicle-mounted camera is fixed to the target vehicle; determining a preliminary pitch angle corresponding to the vehicle-mounted camera when capturing the current image based on image attribute information of the lane area of ​​the target vehicle in the current image; determining a pitch angle corresponding to the vehicle-mounted camera when capturing the current image based on the preliminary pitch angle and a historical pitch angle associated with the preliminary pitch angle.

[0005] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein the lane area includes an image area of ​​a set portion of the self-vehicle lane including the target vehicle; the set portion of the self-vehicle lane includes a portion of the self-vehicle lane whose distance to the target vehicle is less than or equal to a preset distance; based on the current image captured by the acquired vehicle-mounted camera, the step of determining the lane area of ​​the target vehicle in the current image includes: recognizing the current image captured by the acquired vehicle-mounted camera based on a preset image recognition method to obtain the self-vehicle lane position information of the target vehicle in the current image; based on the self-vehicle lane position information and the preset distance, obtaining the image area of ​​the set portion of the self-vehicle lane including the target vehicle.

[0006] In combination with the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the lane area includes a first lane and a second lane; the step of determining a preliminary pitch angle corresponding to the vehicle-mounted camera when shooting the current image based on the image attribute information of the lane area of ​​the target vehicle in the current image comprises: determining a coordinate set of the first lane and a coordinate set of the second lane in the lane area based on the image attribute information of the lane area of ​​the target vehicle; fitting and generating a first straight line corresponding to the first lane based on the coordinate set of the first lane; fitting and generating a second straight line corresponding to the second lane based on the coordinate set of the second lane; and calculating the preliminary pitch angle corresponding to the vehicle-mounted camera when shooting the current image based on the intersection of the first straight line and the second straight line.

[0007] In combination with the second possible implementation of the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the step of determining a coordinate set of a first lane in a lane area and a coordinate set of a second lane based on the image attribute information of the lane area of ​​the target vehicle comprises: dividing the lane area of ​​the target vehicle into a set number of sub-areas; based on the image attribute information of the sub-areas, filtering the divided sub-areas to obtain filtered sub-areas; the filtered sub-areas include a portion of the first lane and a portion of the second lane; for each filtered sub-area, binarizing the sub-area to obtain a binary image corresponding to the sub-area; in the binary image, the pixel value of the pixel point in the area of ​​the partial lane is a first pixel value, and the pixel value of the pixel point in the area other than the area of ​​the partial lane is a second pixel value; according to the first pixel value and the second pixel value of the pixel in the binary image, determining the coordinates of the first lane in the sub-area and the coordinates of the second lane; determining the coordinates of the first lane of all filtered sub-areas as the coordinate set of the first lane; determining the coordinates of the second lane of all filtered sub-areas as the coordinate set of the second lane.

[0008] In combination with the third possible implementation of the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the image attribute information includes the grayscale value of the pixel; based on the image attribute information of the sub-region, the divided sub-region is screened to obtain the screened sub-region, including: obtaining the grayscale value histogram of the sub-region; filtering the grayscale value histogram; based on a preset grayscale threshold, determining the number of peaks and peak positions in the filtered grayscale value histogram; if the pixel value corresponding to the peak position is greater than the preset peak threshold and the number of peaks is 2, the sub-region is determined as a screened sub-region.

[0009] In combination with the fourth possible implementation of the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the grayscale threshold includes a peak threshold and a neighboring threshold; based on the preset grayscale threshold, the step of determining the number of peaks and the peak position in the filtered grayscale value histogram includes: determining the peak in the grayscale value histogram that is greater than or equal to the preset peak threshold as a preliminary peak; based on the preliminary peak and the preset neighboring threshold, merging the preliminary peaks to obtain merged peaks; determining the number of merged peaks as the number of peaks in the filtered grayscale value histogram, and determining the position of the merged peak as the peak position in the filtered grayscale value histogram.

[0010] In combination with the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein, based on the preliminary pitch angle and the historical pitch angle associated with the preliminary pitch angle, the step of determining the pitch angle corresponding to the vehicle camera when shooting the current image includes: judging whether the preliminary pitch angle is within a preset pitch angle range; if so, performing weighted summation processing on the preliminary pitch angle and the historical pitch angle according to set weights to obtain the pitch angle corresponding to the vehicle camera when shooting the current image; if not, determining the pitch angle corresponding to the vehicle camera when shooting the current image based on the historical pitch angles.

[0011] In combination with the first aspect, an embodiment of the present invention provides a seventh possible implementation of the first aspect, wherein the historical pitch angles include multiple; the historical pitch angles are determined by the lane area information of the target vehicle in the historical images taken by the vehicle-mounted camera; the method also includes: if the preliminary pitch angle is within a preset pitch angle range, the preliminary pitch angle is determined as the historical pitch angle; if the number of historical pitch angles is greater than a preset number threshold, the historical pitch angle corresponding to the historical image with the earliest shooting time among the multiple historical pitch angles is deleted.

[0012] In a second aspect, an embodiment of the present invention further provides a device for determining a pitch angle of a vehicle-mounted camera, comprising: a lane area determination module, for determining a lane area of ​​a target vehicle in a current image based on a current image captured by the vehicle-mounted camera; the vehicle-mounted camera is fixed to the target vehicle; a preliminary pitch angle determination module, for determining a preliminary pitch angle corresponding to the vehicle-mounted camera when capturing the current image based on image attribute information of the lane area of ​​the target vehicle in the current image; a pitch angle determination module, for determining a pitch angle corresponding to the vehicle-mounted camera when capturing the current image based on the preliminary pitch angle and a historical pitch angle associated with the preliminary pitch angle.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above method.

[0014] In a fourth aspect, an embodiment of the present invention further provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above method.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] The present invention provides a method, device and electronic device for determining the pitch angle of a vehicle-mounted camera. First, based on the current image captured by the vehicle-mounted camera, the lane area of ​​the target vehicle on which the vehicle-mounted camera is fixed in the current image is determined; then, based on the image attribute information of the lane area of ​​the target vehicle in the current image, the initial pitch angle corresponding to the vehicle-mounted camera when capturing the current image is determined; finally, based on the initial pitch angle and the historical pitch angle associated with the initial pitch angle, the pitch angle corresponding to the vehicle-mounted camera when capturing the current image is determined. In the present invention, the pitch angle of the vehicle-mounted camera when capturing the image is initially determined by the image attribute information of the lane area; and then, based on the initially determined pitch angle and the associated historical pitch angle, the pitch angle corresponding to the vehicle-mounted camera when capturing the current image is accurately determined, thereby improving the accuracy of distance measurement based on the image captured by the vehicle-mounted camera.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a method for determining a pitch angle of a vehicle-mounted camera provided by an embodiment of the present invention;

[0021] Figure 2A flowchart of another method for determining the pitch angle of a vehicle-mounted camera provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a vanishing point provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a pitch angle value provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of another pitch angle value provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the structure of a device for determining a pitch angle of a vehicle-mounted camera provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0028] Environmental perception system is one of the key technologies for autonomous driving. Environmental perception relies on a variety of sensors. Onboard cameras (also called "car cameras") are inexpensive and can be effectively used in autonomous driving environmental perception systems. The most critical part of visual perception is obtaining the distance information of obstacles ahead.

[0029] Currently, there are two types of visual ranging solutions: monocular vision ranging and stereoscopic vision ranging. Stereoscopic vision ranging requires large computational effort and relies on multiple cameras, making it difficult to apply to distance estimation for vehicle-mounted cameras. Monocular vision ranging is currently divided into ranging solutions based on fixed width information, ranging solutions based on inverse perspective transformation, and ranging solutions based on pitch angle (also called "pitch angle").

[0030] Monocular vision ranging based on pitch angle usually adopts the method of determining the distance information of obstacles around the current vehicle based on the pitch angle calibrated when the vehicle is stationary and the pictures taken by the on-board camera. However, when the vehicle is driving, the pitch angle of the on-board camera may change, resulting in inaccurate ranging results.

[0031] Based on this, the pitch angle determination method, device and electronic device of a vehicle-mounted camera provided by the embodiments of the present invention can be applied to the distance measurement process during the driving of various vehicles.

[0032] To facilitate understanding of this embodiment, a method for determining the pitch angle of a vehicle-mounted camera disclosed in an embodiment of the present invention is first introduced in detail.

[0033] The embodiment of the present invention provides a method for determining the pitch angle of a vehicle-mounted camera. Figure 1 As shown, the method comprises the following steps:

[0034] Step S100, based on the current image captured by the vehicle-mounted camera, determine the lane area of ​​the target vehicle in the current image; the vehicle-mounted camera is fixed to the target vehicle.

[0035] The lane area of ​​the target vehicle may be the area in front of the target vehicle where the vehicle-mounted camera taking the current picture is fixed, and the area includes lane lines. The lane lines in the area are usually the edge lines on both sides of the route where the target vehicle is located. The edge lines on both sides may be two solid lines, one dotted line and one solid line, or both may be two dotted lines.

[0036] In specific implementation, since the colors of the lane lines and the ground without lane lines are different, a better method can be selected from a variety of image recognition methods to perform image recognition on the current image taken by the vehicle camera to determine the lane lines in the current image, and the position coordinates of the lane lines in the current image can usually be obtained. The above-mentioned image recognition method can be performed by an image recognition model generated by a convolutional neural network, machine learning, etc., or it can be performed by performing image processing on the current image and then performing pixel statistics and analysis.

[0037] Since the images taken by the on-board camera have a certain shooting range, the scenery farther away from the on-board camera will be more blurred. In order to ensure the effect of lane line recognition, only the image area containing the lane line within the set range of the distance from the target vehicle (equivalent to the distance from the on-board camera) can be retained as the lane area. Among them, the above-mentioned set range can be determined based on experiments or experience. For example, the image area within 30 meters from the target vehicle can be retained, and the pixel coordinates outside this range can be deleted. Before determining the set range, the image from the perspective of the on-board camera can be converted into a bird's-eye view (i.e., a top view) through coordinate transformation and other processing.

[0038] Step S102: Determine the initial pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the image attribute information of the lane area of ​​the target vehicle in the current image.

[0039] The basic attributes of an image include pixels, resolution, size, color, etc. The above-mentioned image attribute information can usually be the pixel value of each pixel in the lane area of ​​the target vehicle. In order to facilitate image processing, the current image can also be converted into a grayscale image before processing.

[0040] In real life, the lanes of the target vehicle are considered as two parallel straight lines. In the pictures taken by the on-board camera, the lanes of the target vehicle will intersect in the distance due to the effect of perspective. The intersection point is also called the "vanishing point". The position of the vanishing point has a certain mathematical relationship with the pitch angle of the on-board camera. If you want to fit the straight line where the two lane lines of the lane area are located, you need to get the coordinates of several points on the lane line. In specific implementation, the lane area can be divided into several sub-areas, and the coordinates of a point on each lane line are determined based on each sub-area.

[0041] When there are dashed lane lines, the divided sub-areas may not have lane lines, and the sub-areas that do not include lane lines need to be filtered out based on the image attribute information of the sub-areas. For example, the lane line part in the lane area is lighter in color than the ground part, and the sub-areas can be filtered based on this feature. After filtering, the sub-areas containing lane lines can be processed as follows: the lane line part and other parts in the sub-area are distinguished by binarization or masking, and a more easily identifiable part is selected from the lane line part in the sub-area, such as the edge part, and the coordinates of each point on the lane line at the edge of the sub-area are determined. Since the lane line also has a certain width, the midpoint of the lane line width can be selected as the coordinates of the points on the lane line in the sub-area.

[0042] After determining the coordinates of multiple points on the lane line, the straight lines corresponding to the two lane lines can be fitted. Then the position coordinates of the vanishing point can be obtained, and based on the position coordinates of the vanishing point, the initial pitch angle corresponding to the vehicle camera when shooting the current image can be calculated.

[0043] Step S104: determining the pitch angle corresponding to the vehicle-mounted camera when capturing the current image according to the preliminary pitch angle and the historical pitch angle associated with the preliminary pitch angle.

[0044] After obtaining the preliminary pitch angle, it is usually first determined whether the preliminary pitch angle is within a preset pitch angle range, wherein the preset pitch angle range may be related to the pitch angle calibrated in a stationary state after the vehicle-mounted camera is installed on the target vehicle, such as the preset pitch angle range is within 1° to the left and right of the calibrated pitch angle. When the preliminary pitch angle is not within the preset pitch angle range, it can be determined that the preliminary pitch angle is invalid.

[0045] The above historical pitch angle may be a pitch angle calibrated in a stationary state after the vehicle-mounted camera is installed in the target vehicle, or may be a pitch angle of the vehicle-mounted camera determined based on historical images taken by the vehicle-mounted camera (the specific method may be determined by the method provided in the embodiment of the present invention). If the current image is the first image taken by the vehicle-mounted camera, the historical pitch angle may be a pitch angle calibrated in a stationary state. In other cases, the historical pitch angle is usually a pitch angle of a set number of vehicle-mounted cameras determined based on historical images.

[0046] If it is determined that the preliminary pitch angle is valid, the preliminary pitch angle and the historical pitch angle can be weighted averaged according to the set weight to reduce the error, thereby obtaining the pitch angle corresponding to the vehicle camera when shooting the current image. The obtained pitch angle can be determined as the historical pitch angle, and the pitch angle with the longest determination time in the historical pitch angles is deleted.

[0047] The present invention provides a method for determining the pitch angle of a vehicle-mounted camera. First, based on the current image captured by the vehicle-mounted camera, the lane area of ​​the target vehicle in the current image is determined; the vehicle-mounted camera is fixed to the target vehicle; then, according to the image attribute information of the lane area of ​​the target vehicle in the current image, the initial pitch angle corresponding to the vehicle-mounted camera when shooting the current image is determined; finally, according to the initial pitch angle and the historical pitch angle associated with the initial pitch angle, the pitch angle corresponding to the vehicle-mounted camera when shooting the current image is determined. In this method, the initial pitch angle of the image is determined by the image attribute information of the lane area; and then, according to the initial pitch angle and the associated historical pitch angle, the pitch angle corresponding to the vehicle-mounted camera when shooting the current image is determined. The accuracy of the distance measurement result of the vehicle-mounted camera is improved, thereby truly reflecting the distance information of obstacles around the current vehicle.

[0048] The present invention also provides another method for determining the pitch angle of a vehicle-mounted camera. Figure 1 This method is implemented based on the method shown. This method mainly describes the specific implementation process of determining the lane area of ​​the target vehicle in the current image based on the current image captured by the vehicle-mounted camera, and the specific implementation process of determining the initial pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the image attribute information of the lane area of ​​the target vehicle in the current image, and the specific implementation process of determining the pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the initial pitch angle and the historical pitch angle associated with the initial pitch angle. Figure 2 As shown, the method comprises the following steps:

[0049] Step S200, based on a preset image recognition method, the current image captured by the vehicle-mounted camera is recognized to obtain the vehicle lane position information of the target vehicle in the current image.

[0050] Specifically, the lane line is detected for the image taken by the vehicle camera. The lane line detection method can be a double lane line straight line detection, which is not specifically limited; after the lane line detection is completed, the ego lane line point coordinates of the current image taken by the vehicle camera will be output, and the position information of the ego lane will be determined according to the ego lane line (Ego Lane) point coordinates. During the detection, the graphic sequence including multiple frames of images taken by the vehicle camera can be input into the lane line detection network, and then each frame of the image and the ego lane line point coordinates corresponding to the image are output. If the ego lane cannot be distinguished in this step, it can be considered that the pitch angle calibration has failed, and the current method ends.

[0051] Step S202 , based on the vehicle lane position information and the preset distance, an image area of ​​a set portion of the vehicle lane including the target vehicle is obtained.

[0052] Specifically, the pixel coordinates of the self-vehicle lane are screened, and the lane line coordinates selected and output only retain the 30m mark point coordinates, which can determine the relationship between the pixel points in the current image and the actual distance through the pre-selected transformation matrix; further, the lane line coordinates selected and output only retain the 30m coordinates as the upper edge of the selected area, and the lower edge of the current image obtained by the on-board camera is used as the lower edge of the selected area, thereby obtaining an image area of ​​the set part of the self-vehicle lane containing the target vehicle.

[0053] Step S204: determining a coordinate set of a first lane and a coordinate set of a second lane in the lane area based on the image attribute information of the lane area of ​​the target vehicle.

[0054] In the specific implementation process, the above step S204 can be implemented in the following way:

[0055] (1) The lane area of ​​the target vehicle is divided into a set number of sub-areas (also called “sub-intervals”).

[0056] It can also be called "dividing the filtered pixel coordinates into sub-regions". Specifically, the number of sub-intervals can be set first, and the width range of each sub-interval is obtained by subtracting the upper edge from the lower edge of the selected interval and then dividing it by the number of intervals. In this way, the width of each interval is equal, and the width range is different. Then, in which sub-interval do the pixels of the vehicle lane that have been detected fall? At this point, the pixel coordinates of each sub-interval are obtained. The minimum and maximum values ​​of the horizontal coordinates of the pixels in each sub-interval are counted respectively, so that the upper left corner coordinates and the lower right corner coordinates of each sub-interval can be preliminarily determined. Since the image of the vehicle-mounted camera is small at a distance and large at a close distance. Therefore, the attenuation coefficient is set to ensure that the width of each sub-interval increases from far to near. At this point, the range of each sub-interval is delineated.

[0057] (2) Based on the image attribute information of the sub-regions, the divided sub-regions are screened to obtain screened sub-regions; the screened sub-regions include a portion of the first lane and a portion of the second lane.

[0058] The above-mentioned image attribute information may include the grayscale value of the pixel. In the screening process, for each sub-region, the grayscale value histogram of the sub-region is first obtained, and then the grayscale value histogram is filtered, and based on the preset grayscale threshold, the number of peaks and the peak positions in the filtered grayscale value histogram are determined; if the pixel value corresponding to the peak position is greater than the preset peak threshold, and the number of peaks is 2, the sub-region is determined as a screened sub-region.

[0059] In the process of determining the number of peaks and peak positions in the filtered gray value histogram, the peaks in the gray value histogram that are greater than or equal to a preset peak threshold can be determined as preliminary peaks; based on the preliminary peaks and the preset adjacent thresholds, the preliminary peaks are merged to obtain merged peaks; the number of merged peaks is determined as the number of peaks in the filtered gray value histogram, and the positions of the merged peaks are determined as the peak positions in the filtered gray value histogram.

[0060] Specifically, the grayscale histogram of each sub-region is first obtained, and the histogram is filtered to eliminate pseudo peaks. In addition, a low threshold (equivalent to the above-mentioned "peak threshold") is set to obtain each peak of the grayscale histogram. If the peak is less than the low threshold, it will not be included in the calculation of the number of peaks. For example, if the peak is less than 10, it will not be included in the calculation of the number of peaks.

[0061] Then set the proximity threshold. If the pixel value corresponding to each peak of the grayscale histogram is close to or lower than the proximity threshold, the peak with a large peak value is retained in the statistics of the number of peaks, and the peak with a lower peak value is not included in the statistics. That is, when the current peak value is 90, if the proximity threshold is set to 20, then several peaks near the current peak value, if they are within 70 to 110, are merged into one peak value, and the largest peak value is selected as the merged peak value, or when the distance between the two peaks is less than 30, only the peak value is retained. So far, if the number of peaks is 2, it is represented as a candidate area. Finally, it is necessary to filter the shadow scene. If the pixel values ​​corresponding to the two peaks are both lower than 150, the double peak value is invalid. So far, if it is still a bimodal histogram, it is the screened sub-area (also called "candidate area"). Otherwise, the sub-area is determined as an "invalid area" and removed.

[0062] (3) For each screened sub-region, a binarization process is performed on the sub-region to obtain a binary image corresponding to the sub-region; in the binary image, the pixel value of the pixel point in the area of ​​the partial lane is the first pixel value, and the pixel value of the pixel point in the area other than the area of ​​the partial lane is the second pixel value.

[0063] In the implementation process, since the image of each sub-region is different, it is impossible to filter the foreground and background by a fixed threshold. Therefore, the Otsu method is first used to find the segmentation threshold of the sub-region. After obtaining the threshold, the area greater than or equal to the threshold is set to a pixel value of 255, and the part less than the threshold is set to 0, that is, binarization processing. Then the binarization result is opened to eliminate the noise around the image. After that, the canny operation can be used to find the edge of the image.

[0064] (4) Determine the coordinates of the first lane and the coordinates of the second lane in the sub-area according to the first pixel value and the second pixel value of the pixel in the binary image.

[0065] In a specific implementation, the coordinates of the first lane and the coordinates of the second lane in the above-mentioned sub-area can be the midpoint coordinates of the lane line at the edge position of the area. According to the obtained image edge (also called "edge position"), the midpoint coordinates of the lane line are obtained. Traverse each row of the edge image in step 5. When the number of pixel values ​​​​of 255 is greater than 5 or the number is equal to 0, it is considered that the lane line information in the row is inaccurate, and the midpoint information is not obtained for this row. On the contrary, the maximum and minimum values ​​of the horizontal coordinates of each row with a pixel value of 255 are obtained, and the maximum value minus the minimum value is the midpoint coordinate. Finally, the midpoint coordinate plus the coordinates of the upper left corner of each sub-interval are the midpoint coordinates of the lane line of the area on the camera image.

[0066] (5) The coordinates of the first lane of all the screened sub-areas are determined as the coordinate set of the first lane; and the coordinates of the second lane of all the screened sub-areas are determined as the coordinate set of the second lane.

[0067] Step S206: Based on the coordinate set of the first lane, a first straight line corresponding to the first lane is generated by fitting.

[0068] Step S208: Based on the coordinate set of the second lane, a second straight line corresponding to the second lane is generated by fitting.

[0069] That is, according to the obtained midpoint coordinates, start fitting the two straight lines on the left and right of the lane, and obtain the coordinates of the intersection of the two straight lines. Usually, before the coordinates of the lane line are used for the straight line fitting corresponding to the lane, these precisely located coordinate points need to be dedistorted, and the latest distortion parameters and internal parameters need to be used here. The intersection coordinates obtained above, that is, the "vanishing point" coordinates, usually also need to be filtered.

[0070] Step S210, based on the intersection of the first straight line and the second straight line, calculate the initial pitch angle corresponding to the vehicle-mounted camera when shooting the current image. Specifically, the intersection of the first straight line and the second straight line (corresponding to the intersection of the two lane lines) is also called a vanishing point, and the pinhole imaging model of the vehicle-mounted camera is used to calculate the attitude angle of the target vehicle during driving. The pitch angle of the target vehicle during driving will cause a certain amount of offset in the horizontal and vertical directions of the road vanishing point, and the pitch angle of the target vehicle is calculated by reverse calculation based on the offset.

[0071] Step S212, determine whether the preliminary pitch angle is within the preset pitch angle range. If yes, execute step S214; if not, execute step S216. In specific implementation, if the pitch angle value is greater than 0.1 degrees of the static calibrated pitch angle or less than 1.5 degrees of the static calibrated pitch angle, it is considered that the currently obtained pitch angle is a valid value, that is, within the preset pitch angle range.

[0072] Step S214, performing weighted sum processing on the preliminary pitch angle and the historical pitch angle according to the set weight, to obtain the pitch angle corresponding to when the vehicle-mounted camera takes the current image, and then executing step S218.

[0073] Specifically, a container (equivalent to a storage space) may be set to store the pitch angles, and the container can accommodate a maximum of 5 pitch angles.

[0074] Step S216, based on the historical pitch angles, determining the pitch angle corresponding to the vehicle-mounted camera when capturing the current image.

[0075] Step S218: determining the preliminary pitch angle as the historical pitch angle.

[0076] Step S220: If the number of historical pitch angles is greater than a preset number threshold, the historical pitch angle corresponding to the historical image with the earliest shooting time among the multiple historical pitch angles is deleted.

[0077] The above steps S216 to S220 are also called pitch angle update strategies. Specifically, through the above steps, if the preliminary pitch angle is not obtained or the preliminary pitch angle is not within the preset pitch angle range (called "calibration failure"), if the current image is the first frame taken by the camera, the static calibration pitch angle is selected as the pitch angle to be obtained (that is, "the pitch angle corresponding to the vehicle-mounted camera when taking the current image"). If the calibration fails, the number of pitch angles in the container is less than 5, and the average of the pitch angles in the container is selected as the pitch angle to be obtained, and the current pitch angle to be obtained is stored in the container. If the calibration fails, the number of pitch angles in the container is greater than 5, and the first 4 pitch angles in the container are taken, and a certain weight is given to each of them, such as 0.4, 0.3, 0.2, and 0.1, and the weighted sum is the pitch angle to be obtained for the current frame. At the same time, the current pitch angle to be obtained is stored in the container, and the first pitch angle in the container is deleted.

[0078] After the above steps, if a preliminary pitch angle within the preset pitch angle range is obtained, if the current value is the first frame of the current image taken by the camera, the current pitch angle and the static pitch angle are selected according to the set weights, such as 0.8, 0.2, and the result of weighted summation is used as the final pitch angle (i.e., "the pitch angle corresponding to the vehicle-mounted camera when shooting the current image"), and the current pitch angle is stored in the container. If the pitch angle currently being obtained is less than 5, the current pitch angle and the average of the pitch angles in the container are selected according to the set weights, such as 0.8, 0.2, and the result of weighted summation is used as the final pitch angle, and the current pitch angle is stored in the container. If the number of pitch angles in the container is greater than 5, the current pitch angle and the first three pitch angles in the container are taken, and weighted 0.4, 0.3, 0.2, and 0.1 are given respectively, and the weighted sum is the pitch angle of the current frame. At the same time, the current pitch angle is stored in the container, and the first pitch angle in the container is deleted. At this point, the pitch angle update is complete, and the distance calculation can be performed based on the current pitch angle.

[0079] The method provided by the embodiment of the present invention is based on the screening of candidate areas, which can effectively solve the lane line scene blocked by shadows and improve the accuracy of lane positioning. At the same time, due to the precise positioning of the lane line, the pitch angle can be accurately obtained, and the pitch angle is updated on this basis, effectively solving the impact of sudden changes in the pitch angle during vehicle driving. This method has been implemented in large quantities, is simple and efficient, and can be applied in high-speed scenarios. As long as the lane line can be effectively detected, this method can work. Figure 3 The image captured by the onboard camera shown, and Figure 4 and Figure 5The schematic diagram of the pitch angle result determined based on the images taken by the vehicle-mounted camera is shown in the figure. The horizontal axis is the number of images taken, and the vertical axis is the angle value of the pitch angle, which are all obtained from actual applications. Corresponding to the above method embodiment, the embodiment of the present invention also provides a device for determining the pitch angle of a vehicle-mounted camera. Figure 6 As shown, the device comprises:

[0080] The lane area determination module 600 is used to determine the lane area of ​​the target vehicle in the current image based on the current image captured by the vehicle-mounted camera; the vehicle-mounted camera is fixed to the target vehicle;

[0081] The preliminary pitch angle determination module 602 is used to determine the preliminary pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the image attribute information of the lane area of ​​the target vehicle in the current image;

[0082] The pitch angle determination module 604 is used to determine the pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the preliminary pitch angle and the historical pitch angle associated with the preliminary pitch angle.

[0083] A pitch angle determination device for a vehicle-mounted camera provided in an embodiment of the present invention has the same technical features as a pitch angle determination method for a vehicle-mounted camera provided in the above embodiment, and therefore can also solve the same technical problems and achieve the same technical effects.

[0084] The embodiment of the present invention further provides an electronic device, see Figure 7 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned method for determining the pitch angle of the vehicle-mounted camera.

[0085] Further, Figure 7 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0086] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0087] The processor 130 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 130 or by instructions in the form of software.

[0088] The processor 130 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method in the aforementioned embodiment in combination with its hardware.

[0089] The pitch angle determination method and device of a vehicle-mounted camera and the computer program product of an electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0091] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0092] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0093] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0094] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for determining the pitch angle of a vehicle-mounted camera, characterized in that: include: Based on the acquired current image captured by the vehicle-mounted camera, determining a lane area of ​​the target vehicle in the current image; The vehicle-mounted camera is fixed to the target vehicle; The lane area includes a first lane and a second lane; According to the image attribute information of the lane area of ​​the target vehicle in the current image, determining the initial pitch angle corresponding to the on-board camera when shooting the current image, including: dividing the lane area of ​​the target vehicle into a set number of sub-areas; based on the image attribute information of the sub-areas, filtering the divided sub-areas to obtain filtered sub-areas; the filtered sub-areas include a portion of the first lane and a portion of the second lane; for each filtered sub-area, binarizing the sub-area to obtain a binary image corresponding to the sub-area; in the binary image, the pixel value of the pixel point in the area of ​​the partial lane is the first pixel value, except for the area of ​​the partial lane the pixel value of the pixel point in the area is the second pixel value; according to the first pixel value and the second pixel value of the pixel in the binary image, the coordinates of the first lane and the coordinates of the second lane in the sub-area are determined; the coordinates of the first lane of all the screened sub-areas are determined as the coordinate set of the first lane; the coordinates of the second lane of all the screened sub-areas are determined as the coordinate set of the second lane; based on the coordinate set of the first lane, a first straight line corresponding to the first lane is generated by fitting; based on the coordinate set of the second lane, a second straight line corresponding to the second lane is generated by fitting; based on the intersection of the first straight line and the second straight line, a preliminary pitch angle corresponding to the vehicle-mounted camera when shooting the current image is calculated; The pitch angle corresponding to the vehicle-mounted camera when shooting the current image is determined according to the preliminary pitch angle and the historical pitch angle associated with the preliminary pitch angle.

2. The method according to claim 1, characterized in that The lane area includes an image area including a set portion of the own vehicle lane of the target vehicle; the set portion of the own vehicle lane includes a portion of the own vehicle lane whose distance to the target vehicle is less than or equal to a preset distance; The step of determining the lane area of ​​the target vehicle in the current image based on the acquired current image captured by the vehicle-mounted camera comprises: Recognize the current image captured by the vehicle-mounted camera based on a preset image recognition method to obtain the vehicle lane position information of the target vehicle in the current image; Based on the own-vehicle lane position information and the preset distance, an image area of ​​a set portion of the own-vehicle lane including the target vehicle is obtained.

3. The method according to claim 1, characterized in that The image attribute information includes the grayscale value of the pixel; The step of screening the divided sub-regions based on the image attribute information of the sub-regions to obtain the screened sub-regions includes: Obtaining a grayscale value histogram of the sub-region; Performing filtering processing on the gray value histogram; Based on a preset grayscale threshold, determine the number and peak positions of the grayscale value histogram after filtering; If the pixel value corresponding to the peak position is greater than a preset peak threshold, and the number of peaks is 2, the sub-region is determined as a screened sub-region.

4. The method according to claim 3, characterized in that The grayscale threshold includes a peak threshold and a proximity threshold; The step of determining the number and position of peaks in the filtered gray value histogram based on a preset gray threshold comprises: Determine a peak value in the gray value histogram that is greater than or equal to a preset peak value threshold as a preliminary peak value; Based on the preliminary peak value and a preset proximity threshold, the preliminary peak value is merged to obtain a merged peak value; The number of the merged peaks is determined as the number of peaks in the filtered grayscale value histogram, and the positions of the merged peaks are determined as the peak positions in the filtered grayscale value histogram.

5. The method according to claim 1, characterized in that The step of determining the pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the preliminary pitch angle and the historical pitch angle associated with the preliminary pitch angle comprises: Determining whether the preliminary pitch angle is within a preset pitch angle range; If yes, performing weighted sum processing on the preliminary pitch angle and the historical pitch angle according to the set weight to obtain the pitch angle corresponding to the vehicle-mounted camera when shooting the current image; If not, determine the pitch angle corresponding to the vehicle-mounted camera when shooting the current image based on the historical pitch angle.

6. The method according to claim 1, characterized in that The historical pitch angles include multiple ones; the historical pitch angles are determined by lane area information of the target vehicle in the historical images taken by the vehicle-mounted camera; The method further comprises: If the preliminary pitch angle is within a preset pitch angle range, determining the preliminary pitch angle as a historical pitch angle; If the number of historical pitch angles is greater than a preset number threshold, the historical pitch angle corresponding to the historical image with the earliest shooting time among the plurality of historical pitch angles is deleted.

7. A device for determining the pitch angle of a vehicle-mounted camera, characterized in that: include: A lane area determination module, used to determine the lane area of ​​the target vehicle in the current image based on the current image captured by the vehicle-mounted camera; The vehicle-mounted camera is fixed to the target vehicle; The lane area includes a first lane and a second lane; The preliminary pitch angle determination module is used to determine the preliminary pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the image attribute information of the lane area of ​​the target vehicle in the current image, including: dividing the lane area of ​​the target vehicle into a set number of sub-areas; based on the image attribute information of the sub-areas, filtering the divided sub-areas to obtain filtered sub-areas; the filtered sub-areas include a part of the first lane and a part of the second lane; for each filtered sub-area, binarizing the sub-area to obtain a binary image corresponding to the sub-area; in the binary image, the pixel value of the pixel point in the area of ​​the partial lane is the first pixel value, except for the part of the vehicle the pixel value of the pixel point in the area outside the area of ​​the lane is the second pixel value; according to the first pixel value and the second pixel value of the pixel in the binary image, the coordinates of the first lane and the coordinates of the second lane in the sub-area are determined; the coordinates of the first lane of all the screened sub-areas are determined as the coordinate set of the first lane; the coordinates of the second lane of all the screened sub-areas are determined as the coordinate set of the second lane; based on the coordinate set of the first lane, a first straight line corresponding to the first lane is generated by fitting; based on the coordinate set of the second lane, a second straight line corresponding to the second lane is generated by fitting; based on the intersection of the first straight line and the second straight line, a preliminary pitch angle corresponding to the vehicle-mounted camera when shooting the current image is calculated; The pitch angle determination module is used to determine the pitch angle corresponding to the vehicle-mounted camera when shooting the current image according to the preliminary pitch angle and the historical pitch angle associated with the preliminary pitch angle.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method for determining the pitch angle of a vehicle-mounted camera as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the method for determining the pitch angle of a vehicle-mounted camera as described in any one of claims 1 to 6.

Images