A road segmentation method and device for actively adjusting a camera sight angle

Abstract

The embodiment of the specification provides a road segmentation method and device for actively adjusting a camera sight angle. In an initial state, the camera is not in the sight angle adjustment. The method comprises the following steps: S1: acquiring a road image in front of the camera; S2: inversely projecting the road image onto a road plane to obtain an overhead view in a road plane coordinate system; S3: performing road segmentation on the overhead view to obtain a segmentation image; S4: evaluating the segmentation image according to a pre-set evaluation standard to obtain an evaluation value of the road image; and S5: judging whether the evaluation value is greater than an evaluation threshold value, and according to the size comparison between the evaluation value and the evaluation threshold value, the sight angle of the camera is kept or the sight angle of the camera is actively adjusted to a target sight angle in a cycle of S1-S5. Through the method of the embodiment of the specification, the camera can be actively adjusted to achieve an angle, and the performance of road segmentation is improved.

Description

Technical Field

[0001] This specification relates to the field of intelligent vehicle driving, and in particular, to a road segmentation method and apparatus for actively adjusting the camera's line-of-sight angle. Background Technology

[0002] Road segmentation, also known as drivable area detection, aims to use onboard cameras to detect drivable areas of roads that a vehicle is currently traveling on or about to travel on, classifying each pixel of the image as either drivable or non-drivable. This is a key technology in the field of intelligent vehicles. Real-world urban traffic environments are exceptionally complex. Achieving stable road segmentation by intelligent vehicles requires overcoming various adverse factors. Current technologies, where cameras maintain a fixed orientation, cannot accurately segment roads in scenarios such as backlighting or U-turns. When a vehicle is traveling towards sunlight, the camera is highly likely to be overexposed due to direct sunlight or specular reflections from the road surface, affecting the accuracy of road segmentation. When a vehicle turns or makes a U-turn, the camera's limited field of view may cause the region of interest to move outside the camera's field of view. In such cases, no matter how high the segmentation accuracy, it is meaningless for the intelligent vehicle's driving task.

[0003] Therefore, there is an urgent need for a road segmentation method that can actively adjust the camera's viewing angle to improve the performance of road segmentation. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a road segmentation method and apparatus that actively adjusts the camera's line-of-sight angle, thereby improving the performance of road segmentation.

[0005] To achieve the above objectives, on the one hand, embodiments of this specification provide a road segmentation method for actively adjusting the camera's line-of-sight angle, wherein the camera is not initially adjusting its line-of-sight angle, including:

[0006] S1: Use a camera to capture images of the road ahead;

[0007] S2: Inversely project the road image onto the road plane to obtain a top view in the road plane coordinate system;

[0008] S3: Perform road segmentation on the top view to obtain a segmented image;

[0009] S4: Evaluate the segmented image using a pre-set evaluation standard to obtain the evaluation value of the road image;

[0010] S5: Determine whether the camera is adjusting its viewing angle;

[0011] S51: If not, determine whether the evaluation value of the road image is greater than the evaluation threshold;

[0012] S511: If so, maintain the camera's line of sight and repeat steps S1-S5.

[0013] S512: If not, then set the camera to be in the process of adjusting the viewing angle, actively adjust the viewing angle of the camera to a new candidate viewing angle, and repeat steps S1-S5.

[0014] S52: If so, then,

[0015] Actively adjust the camera's line of sight to a new candidate line of sight, repeat steps S1-S5 until there are no new candidate line of sight, and then set the camera's line of sight adjustment to be complete.

[0016] Once the camera's line of sight angle has been adjusted, the target line of sight angle is selected from all candidate line of sight angles based on the evaluation values ​​of the road images corresponding to all candidate line of sight angles.

[0017] Adjust the camera to the target line-of-sight angle and repeat steps S1-S5.

[0018] Preferably, the step of evaluating the segmented image using pre-set evaluation criteria to obtain the evaluation value of the road image further includes:

[0019] Extract features related to road segmentation accuracy from the segmented image;

[0020] The features are input into a pre-trained neural network model, and the first evaluation value is output to evaluate the accuracy of road segmentation.

[0021] Construct the region of interest for the current vehicle motion state in the vehicle coordinate system;

[0022] The region of interest is projected onto the camera coordinate system of the current camera to obtain the projected region of interest;

[0023] Based on the position and size of the projected region of interest, a second evaluation value is calculated to assess the usefulness of the segmented image for the current vehicle motion state.

[0024] The evaluation value is calculated based on the first evaluation value and the second evaluation value.

[0025] Preferably, the method for determining the features related to road segmentation accuracy includes:

[0026] By inversely projecting the known road image onto the road plane, a known top view in the road plane coordinate system is obtained;

[0027] The known top view is segmented into roads to obtain a known segmented image, where the segmentation accuracy of the known segmented image is a known quantity;

[0028] Obtain several original features related to the road shape from the known segmented image to form the original feature vector;

[0029] The original feature vector is subjected to three dimensionality reduction operations to obtain features related to road segmentation accuracy.

[0030] Preferably, the step of performing three dimensionality reductions on the original feature vector to obtain features related to road segmentation accuracy further includes:

[0031] Each original feature in the original feature vector is standardized to obtain a standardized feature vector.

[0032] The correlation coefficient is used to perform correlation analysis between every two features in the standardized feature vector. If the correlation coefficient between every two features is greater than a set coefficient, one of the features in every two features is removed to obtain the feature vector after the first dimensionality reduction. The correlation coefficient between any two features in the feature vector after the first dimensionality reduction is not greater than the set coefficient.

[0033] Construct the correlation coefficient matrix among the features in the feature vector after the first dimensionality reduction;

[0034] Using principal component analysis, the principal components are determined based on the eigenvectors of the correlation coefficient matrix and the eigenvectors after the first dimensionality reduction.

[0035] Based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal component, the feature vector after the first dimensionality reduction is subjected to a second dimensionality reduction to obtain the feature vector after the second dimensionality reduction.

[0036] The segmentation accuracy of the known segmented image is used as the accuracy feature. Correlation analysis is performed on each feature in the feature vector after the second dimensionality reduction and the accuracy feature using the correlation coefficient. If the correlation coefficient between the two is greater than a set coefficient, the feature is retained, and the feature vector after the third dimensionality reduction is obtained.

[0037] The features in the feature vector after the third dimensionality reduction are used as features related to road segmentation accuracy.

[0038] Preferably, the step of performing a second dimensionality reduction on the feature vector after the first dimensionality reduction, based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal components, to obtain the feature vector after the second dimensionality reduction, further includes:

[0039] After sorting the features in the feature vector after the first dimensionality reduction from largest to smallest according to their influence on the principal component, the sorting order of each feature is obtained. The first digit of the sorting order is 1, and the increment step of the sorting order is 1.

[0040] Set the initial number of statistics to 1;

[0041] The cumulative influence of all features whose sorting order is less than or equal to the number of features on the principal components is determined, and it is then determined whether the cumulative influence reaches a set level.

[0042] If so, then all features whose sorting order is less than or equal to the statistical number will be treated as new features, and the above statistical cumulative influence degree and judgment steps will be stopped;

[0043] If not, the count increments by 1, and the cumulative impact and judgment steps are repeated.

[0044] Using the new features, construct the feature vector after the second dimensionality reduction.

[0045] Preferably, the region of interest for constructing the current vehicle motion state in the vehicle coordinate system further includes:

[0046] Based on the coordinates of the three preview points (far, middle, and near) in the vehicle coordinate system corresponding to the current vehicle motion state, the far coordinate point, middle coordinate point, and near coordinate point are obtained.

[0047] The spline curve is obtained by fitting a cubic spline curve to the three coordinate points.

[0048] The spline curve is moved left and right along the tangent direction of the mid-coordinate point to obtain the left and right boundaries of the region of interest.

[0049] Connect the upper and lower vertices of the left and right boundaries with straight lines to obtain the upper and lower boundaries of the region of interest;

[0050] The closed shape enclosed by the left and right boundaries and the top and bottom boundaries is taken as the region of interest for the current vehicle motion state in the vehicle coordinate system.

[0051] Preferably, the step of calculating a second evaluation value for assessing the usefulness of the segmented image for the current vehicle motion state based on the position and size of the projected region of interest further includes:

[0052] The second evaluation value, used to assess the usefulness of the segmented image for the current vehicle motion state, is calculated using the following formula:

[0053]

[0054] Where κ2 is the second evaluation value, A1 is the pixel area of ​​the projected region of interest, A0 is the maximum pixel area obtained after projecting the region of interest onto the camera coordinate system corresponding to all different viewing angles, and W A d is the width of the road image captured by the camera. R α is the distance between the center point of the projected region of interest and the midpoint of the road image. C represents the completeness of the projected region of interest.

[0055] Preferably, the step of selecting the target line-of-sight angle from all candidate line-of-sight angles based on the evaluation values ​​of the road images corresponding to all candidate line-of-sight angles by the camera further includes:

[0056] Based on the threshold values ​​of the first and second evaluation values, the region where the viewing angle is maintained in the two-dimensional coordinate system is determined.

[0057] Using the first and second evaluation values ​​corresponding to each candidate viewing angle of the camera as the abscissa and ordinate respectively, the feature points of each candidate viewing angle on the two-dimensional coordinate system are plotted.

[0058] Candidate viewing angles of the feature points outside the retention area are eliminated to obtain the remaining candidate viewing angles;

[0059] The angle with the highest evaluation value among the remaining candidate line-of-sight angles is taken as the target line-of-sight angle.

[0060] Preferably, the step of inversely projecting the road image onto the road plane to obtain a top view in the road plane coordinate system further includes:

[0061] The road image is inversely projected onto the road plane to obtain an irregular shape;

[0062] Select a rectangular region of a set length and a set width from the irregular shape;

[0063] The rectangular region is set at a predetermined position and angle in the road plane coordinate system;

[0064] The irregular shapes within the rectangular area are pixelated into a top view according to a set ratio.

[0065] Preferably, the candidate line-of-sight angle includes: a candidate pitch angle and a candidate azimuth angle corresponding to the target pitch angle; or, a candidate azimuth angle and a candidate pitch angle corresponding to the target azimuth angle.

[0066] On the other hand, embodiments of this specification provide a road segmentation device that actively adjusts the camera's line-of-sight angle. Initially, the camera is not in the process of adjusting its line-of-sight angle. The device includes:

[0067] The acquisition module is used to acquire images of the road ahead using a camera;

[0068] The inverse projection module is used to inversely project the road image onto the road plane to obtain a top view in the road plane coordinate system.

[0069] A segmentation module is used to segment the top view into roads to obtain a segmented image;

[0070] The evaluation module is used to evaluate the segmented image according to a pre-set evaluation standard to obtain the evaluation value of the road image;

[0071] The determination module is used to determine whether the camera is adjusting its viewing angle.

[0072] S51: If not, determine whether the evaluation value of the road image is greater than the evaluation threshold;

[0073] S511: If so, maintain the camera's line-of-sight angle and repeatedly call the acquisition module until a decision is made.

[0074] Module;

[0075] S512: If not, then set the camera to be in the line-of-sight angle adjustment state, actively adjust the line-of-sight angle of the camera to a new candidate line-of-sight angle, and cyclically call the acquisition module to the judgment module;

[0076] S52: If so, then,

[0077] The camera's line of sight is actively adjusted to a new candidate line of sight angle. The acquisition module is repeatedly called to the judgment module until there are no new candidate line of sight angles, and then the camera's line of sight angle adjustment is completed.

[0078] Once the camera's line of sight angle has been adjusted, the target line of sight angle is selected from all candidate line of sight angles based on the evaluation values ​​of the road images corresponding to all candidate line of sight angles.

[0079] Adjust the camera to the target line-of-sight angle and repeatedly call the acquisition module to the judgment module.

[0080] In another aspect, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when executed by the processor, performs instructions according to any of the methods described above.

[0081] In another aspect, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of a computer device to perform instructions for any of the methods described above.

[0082] As can be seen from the technical solutions provided in the embodiments of this specification above, the method of this specification involves inversely projecting a road image onto a road plane coordinate system to obtain a top view, then segmenting the top view to obtain a segmented image, and then evaluating the segmented image to obtain an evaluation value for the road image. The evaluation value is used to evaluate the segmentation accuracy and usefulness of the segmented image for understanding vehicle motion. When the evaluation value is greater than an evaluation threshold, the camera's viewing angle can be maintained; otherwise, the camera's viewing angle needs to be adjusted. During adjustment, the evaluation value of the camera at candidate viewing angles is continuously obtained through a loop. Based on this, a suitable target viewing angle is selected to ensure the segmentation accuracy and usefulness of the segmented image for understanding vehicle motion, thereby improving the road segmentation performance under adverse factors such as backlighting and U-turns, and ensuring safe driving for intelligent vehicles.

[0083] To make the above and other objects, features and advantages of this specification more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 A schematic flowchart of a road segmentation method for actively adjusting the camera's line-of-sight angle, as provided in an embodiment of this specification, is shown.

[0086] Figure 2 This document illustrates a flowchart of an embodiment of the process of inversely projecting a road image onto a road plane to obtain a top view in the road plane coordinate system.

[0087] Figure 3 A schematic diagram of the coordinate system provided in the embodiments of this specification is shown;

[0088] Figure 4 This document illustrates a flowchart illustrating how a segmented image is evaluated using pre-set evaluation criteria to obtain an evaluation value for a road image, as provided in an embodiment of this specification.

[0089] Figure 5This specification illustrates a scanning diagram during camera line-of-sight angle adjustment, as provided in an embodiment.

[0090] Figure 6 A flowchart illustrating a method for determining features related to road segmentation accuracy provided in an embodiment of this specification is shown.

[0091] Figure 7 This document illustrates a flowchart of an embodiment of the process for performing three dimensionality reductions on the original feature vector to obtain features related to road segmentation accuracy.

[0092] Figure 8 This document illustrates a flowchart of a process for performing a second dimensionality reduction on a feature vector after the first dimensionality reduction, based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal components, according to an embodiment of this specification.

[0093] Figure 9 This document illustrates a flowchart of a process for constructing a region of interest in the current vehicle motion state in the vehicle coordinate system, as provided in an embodiment of this specification.

[0094] Figure 10 This specification provides a schematic diagram of the aiming point with the vehicle as a reference, as shown in the embodiments of this specification.

[0095] Figure 11 This document illustrates a flowchart of the process for selecting a target line of sight from all candidate line of sight angles, based on the evaluation values ​​of the road images corresponding to all candidate line of sight angles provided by the embodiments of this specification.

[0096] Figure 12 A schematic diagram of the holding area provided in the embodiments of this specification is shown;

[0097] Figure 13 This specification shows a schematic diagram of the module structure of a road segmentation device that actively adjusts the camera's line of sight angle, as provided in an embodiment of this specification.

[0098] Figure 14 A schematic diagram of the structure of a computer device provided in an embodiment of this specification is shown.

[0099] Explanation of symbols in the attached drawings:

[0100] 100. Data Acquisition Module;

[0101] 200. Reverse projection module;

[0102] 300. Segmentation module;

[0103] 400. Evaluation module;

[0104] 500. Judgment Module;

[0105] 1402. Computer equipment;

[0106] 1404, Processor;

[0107] 1406. Memory;

[0108] 1408. Drive mechanism;

[0109] 1410. Input / Output Module;

[0110] 1412. Input devices;

[0111] 1414. Output devices;

[0112] 1416. Presentation equipment;

[0113] 1418. Graphical User Interface;

[0114] 1420. Network interface;

[0115] 1422. Communication link;

[0116] 1424. Communication bus. Detailed Implementation

[0117] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the embodiments of this specification.

[0118] Road segmentation, also known as drivable area detection, aims to use onboard cameras to detect drivable areas of roads that a vehicle is currently traveling on or about to travel on, classifying each pixel of the image as either drivable or non-drivable. This is a key technology in the field of intelligent vehicles. Real-world urban traffic environments are exceptionally complex. Achieving stable road segmentation by intelligent vehicles requires overcoming various adverse factors. Current technologies, where cameras maintain a fixed orientation, cannot accurately segment roads in scenarios such as backlighting or U-turns. When a vehicle is traveling towards sunlight, the camera is highly likely to be overexposed due to direct sunlight or specular reflections from the road surface, affecting the accuracy of road segmentation. When a vehicle turns or makes a U-turn, the camera's limited field of view may cause the region of interest to move outside the camera's field of view. In such cases, no matter how high the segmentation accuracy, it is meaningless for the intelligent vehicle's driving task.

[0119] To address the aforementioned issues, this specification provides a road segmentation method that actively adjusts the camera's line-of-sight angle. Figure 1This is a flowchart illustrating a road segmentation method for actively adjusting the camera's line-of-sight angle, as provided in an embodiment of this specification. This specification provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel.

[0120] It should be noted that the terms "first," "second," etc., in the description, claims, and accompanying drawings of the embodiments in this specification are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0121] Reference Figure 1 This specification provides a road segmentation method that actively adjusts the camera's line-of-sight angle, wherein the camera is not initially adjusting its line-of-sight angle, including:

[0122] S1: Use a camera to capture images of the road ahead;

[0123] S2: Inversely project the road image onto the road plane to obtain a top view in the road plane coordinate system;

[0124] S3: Perform road segmentation on the top view to obtain a segmented image;

[0125] S4: Evaluate the segmented image using a pre-set evaluation standard to obtain the evaluation value of the road image;

[0126] S5: Determine whether the camera is adjusting its viewing angle;

[0127] S51: If not, determine whether the evaluation value of the road image is greater than the evaluation threshold;

[0128] S511: If so, maintain the camera's line of sight and repeat steps S1-S5.

[0129] S512: If not, then set the camera to be in the process of adjusting the viewing angle, actively adjust the viewing angle of the camera to a new candidate viewing angle, and repeat steps S1-S5.

[0130] S52: If so, then,

[0131] Actively adjust the camera's line of sight to a new candidate line of sight, repeat steps S1-S5 until there are no new candidate line of sight, and then set the camera's line of sight adjustment to be complete.

[0132] Once the camera's line of sight angle has been adjusted, the target line of sight angle is selected from all candidate line of sight angles based on the evaluation values ​​of the road images corresponding to all candidate line of sight angles.

[0133] Adjust the camera to the target line-of-sight angle and repeat steps S1-S5.

[0134] In the embodiments described in this specification, the camera is not in the initial state of line-of-sight angle adjustment, which can be understood as the camera maintaining a fixed line-of-sight angle.

[0135] The camera can be mounted on the top or front of the vehicle, or in a position with a wide field of view. As the vehicle moves, the camera will continuously collect images of the road ahead. The collected road images are in the camera coordinate system. However, in this embodiment, the camera's line of sight can be actively adjusted. This will cause the geometric features of the road images to change significantly for the same road surface object due to different camera line of sight, affecting the accuracy of subsequent road segmentation. To avoid the above situation, the road images collected by the camera can be back-projected onto the road plane through the camera's imaging model to obtain a top view of the road in the road plane coordinate system.

[0136] In the embodiments described in this specification, reference is made to Figure 2 The step of inversely projecting the road image onto the road plane to obtain a top view in the road plane coordinate system further includes:

[0137] S101: The road image is back-projected onto the road plane to obtain an irregular shape;

[0138] S102: Select a rectangular region of a set length and a set width from the irregular shape;

[0139] S103: Set the rectangular area at a set position and a set angle in the road plane coordinate system;

[0140] S104: Pixelate the irregular shapes within the rectangular area into a top view according to a set ratio.

[0141] Specifically, refer toFigure 3 The embodiments in this specification include a world coordinate system (CS). W Vehicle coordinate system (CS) V ), gimbal coordinate system (CS) T The pan-tilt head is the supporting device for mounting and fixing the camera, and the camera coordinate system (CS) is the key component. A ), road plane coordinate system (CS) R ). Among them, CS V At the center of the rear axle of the vehicle, CS R With CS V It is relatively fixed, and there is only translation along the negative Z-axis, the distance of which is the wheel radius r. w Assume CS R To CS V Transformed into T RV CS V To CS T The transformation to T VT CS T To CS A The transformation to T TA Then by CS A To CS R The transformation is:

[0142] T RA =T RV T VT T TA (1)

[0143] Among them, T RV =[1,0,0,0;0,1,0,0;0,0,1,r w ;0,0,0,1],r w T is the radius of the wheel. VT After a transformation that is calibrated to be known and fixed, T TA The value of T is calculated in real time based on the gimbal angle fed back by the gimbal, the gimbal kinematic model, and the hand-eye calibration results of the camera and gimbal. RA It can be calculated in real time according to the above formula. In addition, it is assumed that the intrinsic parameter matrix of the camera is C. C Having already obtained the image through camera calibration, the process of inversely projecting the road image onto the road plane is as follows:

[0144]

[0145] Where x1 is the image coordinate of a pixel in the road image, x1 = (r1, c1, 1), r1 is the row number of the pixel in the road image, c1 is the column number, and x R The z-coordinates of the next projected point in the road plane coordinate system after inverse projection onto the road plane. RLet z be the Z-axis of the road plane coordinate system. R =0 represents the road plane, and λ represents any non-zero natural number. For any pixel in the image, its intersection with the road plane can be calculated using the above formula. Each intersection is assigned a corresponding gray value, and the gray values ​​of other non-intersection points are calculated using bilinear interpolation. This yields the inverse projection of the image from the active camera onto the road plane.

[0146] The inverse projection of a road image onto the road plane results in a large, irregular shape, whereas a typical image is a regular rectangle. Therefore, the irregular shape needs to be converted into a top view first, and a suitable rectangular region is selected from this irregular shape to generate the top view. First, the length and width of the rectangle need to be determined. Then, the position and angle of the rectangle in the road plane coordinate system need to be determined. Finally, the projection within the rectangular region is pixelated according to a set ratio to form the top view. In the embodiment of this specification, the rectangle used has a length of 43.2m, a width of 21.6m, and an angle of T. RA The rotation component along the z-axis of the road plane coordinate system, with the starting position of the rectangle being CS. A The vertical projection of this rectangle onto the road plane. This rectangle is 0.054 × 0.054 m. 2 By pixelating the image at a ratio of 1 / pixel, you can obtain a top view with a size of 800×400 pixels.

[0147] After obtaining the top view, road segmentation can be performed on the top view to obtain a segmented image. The road segmentation algorithm can be the YOLOP algorithm.

[0148] Then, the segmented image can be evaluated using pre-set evaluation criteria to obtain the evaluation value of the road image, which can then be used as a reference. Figure 4 Specifically:

[0149] S201: Extract features related to road segmentation accuracy from the segmented image;

[0150] S202: Input the features into a pre-trained neural network model and output a first evaluation value for evaluating the accuracy of road segmentation;

[0151] S203: Construct the region of interest for the current vehicle motion state in the vehicle coordinate system;

[0152] S204: Project the region of interest onto the camera coordinate system of the current camera to obtain the projected region of interest;

[0153] S205: Calculate a second evaluation value for evaluating the usefulness of the segmented image for the current vehicle motion state based on the position and size of the projected region of interest;

[0154] S206: Calculate the evaluation value based on the first evaluation value and the second evaluation value.

[0155] Features related to road segmentation accuracy are input into a pre-trained neural network model to output a first evaluation value for road segmentation accuracy. When driving, drivers tend to anticipate the driving environment ahead by aiming a certain distance in the driving direction based on the current vehicle's movement, thus maximizing driving safety. Similarly, the vehicle's motion state determines the key areas for road segmentation, and the system should prioritize segmentation of these areas. This requires not only maximizing the first evaluation value but also meeting the detection requirements for key areas. Based on this, the embodiments in this specification propose constructing a Region of Interest (ROI) (i.e., the area requiring focus) based on the current vehicle motion state in the vehicle coordinate system. Since the ROI is in the vehicle coordinate system, it can be further projected onto the camera coordinate system of the current camera to obtain the projected ROI. Then, based on its position and size in the image, the current camera angle is evaluated to obtain a second evaluation value, ensuring effective detection of the ROI.

[0156] In this embodiment of the specification, it is necessary to further determine whether the camera is in the process of adjusting its viewing angle. Because initially, the camera is not in the process of adjusting its viewing angle, it will proceed to the S51 No branch, that is, to determine whether the evaluation value of the road image is greater than the evaluation threshold. The evaluation threshold can be set according to the actual situation. If the evaluation value is greater than the evaluation threshold, it means that the current viewing angle of the camera is appropriate and no adjustment is needed. The viewing angle is maintained, and the S1-S5 steps are repeated for the next cycle of judgment. If the evaluation value is not greater than the evaluation threshold, it means that the current viewing angle of the camera is inappropriate and needs to be adjusted. At this time, it is necessary to first set the camera to be in the process of adjusting its viewing angle, and then actively adjust the camera's viewing angle to a new candidate viewing angle. The S1-S5 steps are repeated for the next cycle of judgment. During the next cycle of judgment, since the camera is in the process of adjusting its viewing angle, it will proceed to the S52 Yes branch, actively adjust the camera's viewing angle to a new candidate viewing angle, and repeat the S1-S5 steps until there are no new candidate viewing angles. The viewing angle of the camera is then set to be adjusted. After the viewing angle adjustment is completed, the camera is no longer in the process of adjusting its viewing angle.

[0157] It should be noted that the candidate line-of-sight angle includes: a candidate pitch angle and a candidate azimuth angle corresponding to the target pitch angle; or, a candidate azimuth angle and a candidate pitch angle corresponding to the target azimuth angle. The target line-of-sight angle is composed of the target pitch angle and the target azimuth angle. In the embodiments of this specification, the rotation range of the pitch angle (along the y-axis) is [-π / 4, 0] rad, and the rotation range of the azimuth angle (along the z-axis) is [-π / 2, π / 2] rad. The frame rate of the active camera during scanning is 30 FPS. The pitch rotation speed is 15 RPM, and the azimuth rotation speed is 30 RPM.

[0158] Reference Figure 5 When adjusting the line of sight angle, you can first scan the pitch angle, and after determining a candidate pitch angle as the target pitch angle, then scan the azimuth angle corresponding to the target pitch angle to obtain the target azimuth angle. Alternatively, you can first rotate the azimuth angle, and after determining a candidate azimuth angle as the target azimuth angle, then rotate the azimuth angle corresponding to the target azimuth angle to obtain the target pitch angle.

[0159] Taking the following approach as an example: First, rotate the camera on the pitch angle. After determining a candidate pitch angle as the target pitch angle, rotate the camera on the azimuth angle corresponding to the target pitch angle to obtain the target azimuth angle. Assume there are a total of 5 new candidate pitch angles, α1 to α5. In the nth iteration, when actively adjusting the camera's line of sight angle, first adjust the line of sight angle to α1, repeat steps S1-S5, and obtain the evaluation value of the road image corresponding to α1. In the (n+1)th iteration, adjust the line of sight angle to α2, repeat steps S1-S5, and obtain the evaluation value of the road image corresponding to α2... After the (n+4)th iteration, obtain the evaluation values ​​of the road images corresponding to α1 to α5 respectively. Based on the evaluation values ​​corresponding to all candidate pitch angles (α1 to α5), select the target pitch angle from α1 to α5.

[0160] Maintaining the target pitch angle, a total of 5 new candidate azimuth angles, β1 to β5, are generated. Then, during the (n+5)th iteration, the camera's line-of-sight angle is actively adjusted to β1, and steps S1-S5 are repeated to obtain the evaluation value of the road image corresponding to β1. In the (n+6)th iteration, the line-of-sight angle is adjusted to β2, and steps S1-S5 are repeated to obtain the evaluation value of the road image corresponding to β2, and so on. After the (n+9)th iteration, the evaluation values ​​of the road images corresponding to β1 to β5 are obtained. Based on the evaluation values ​​corresponding to all candidate azimuth angles (β1 to β5) under the target pitch angle, the target azimuth angle is selected from β1 to β5. The target pitch angle and the target azimuth angle obtained at this time constitute the target line-of-sight angle.

[0161] It should be noted that in step S52, the camera's line-of-sight angle adjustment is not set to complete until no new candidate line-of-sight angles are found. Therefore, if only the target pitch angle or the target azimuth angle is determined, new candidate line-of-sight angles may still exist: the candidate azimuth angle corresponding to the target pitch angle, or the candidate pitch angle corresponding to the target azimuth angle. In this case, the camera's line-of-sight angle adjustment will not be set to complete.

[0162] After adjusting the camera to the target viewing angle, repeat steps S1-S5. At this time, the camera is not in the viewing angle adjustment stage, and a new round of image acquisition is performed.

[0163] In the embodiments described in this specification, reference is made to Figure 6 The method for determining the features related to road segmentation accuracy includes:

[0164] S301: Inversely project the known road image onto the road plane to obtain a known top view in the road plane coordinate system;

[0165] S302: Perform road segmentation on the known top view to obtain a known segmented image, wherein the segmentation accuracy of the known segmented image is a known quantity;

[0166] S303: Obtain several original features related to the road shape in the known segmented image to form the original feature vector;

[0167] S304: Perform dimensionality reduction three times on the original feature vector to obtain features related to road segmentation accuracy.

[0168] By analyzing several known samples, features related to road segmentation accuracy are determined. These known samples are known road images. For each known road image, it needs to be inversely projected onto the road plane to obtain a known top view in the road plane coordinate system. Then, road segmentation is performed on this known top view to obtain a known segmented image. The segmentation accuracy of this known segmented image can be calculated using the following formula:

[0169]

[0170] Among them, A CC T represents the segmentation accuracy of a known segmented image. P T represents a true positive, indicating the number of pixels within a correctly segmented lane. N True negatives represent the number of correctly segmented non-lane pixels, while false negatives (F) represent the number of correctly segmented non-lane pixels. N It is the number of pixels incorrectly segmented within the lane, false positives F P The error is that the number of pixels outside the lane is incorrectly classified as lane pixels.

[0171] Features in a segmented image can be categorized into shape-based features, grayscale-based features, and texture-based features. This specification's embodiments only collect shape-based features. Shape-based features primarily reflect the contour and region features of the known segmented image. Contour features are curves composed of a series of connected points, representing the basic shape of a road. The Halcon feature pool is selected; a feature pool refers to a collection of manually designed features. Sixty-four shape-related features were collected to form the feature pool, including features such as region area, rectangularity, and region central moment.

[0172] Not all features in the feature pool are related to the quality of road segmentation, and even among related features, the degree of correlation may vary. The embodiments in this specification select features from the feature pool to obtain relatively independent features that accurately reflect the quality of the segmentation result.

[0173] First, each feature in the feature pool is numbered to form an original feature vector: in Let di be the i-th original feature, and d0 be the dimension of the original feature, thus realizing the transformation from feature pool to feature vector. For several known samples, the original features... The sample vector is

[0174] Furthermore, refer to Figure 7 The original feature vector is subjected to three dimensionality reduction operations to obtain features related to road segmentation accuracy, specifically including:

[0175] S401: Standardize each original feature in the original feature vector to obtain a standardized feature vector;

[0176] S402: Use the correlation coefficient to perform correlation analysis between every two features in the standardized feature vector. If the correlation coefficient between every two features is greater than a set coefficient, then remove one of the two features to obtain the feature vector after the first dimensionality reduction. The correlation coefficient between any two features in the feature vector after the first dimensionality reduction is not greater than the set coefficient.

[0177] S403: Construct the correlation coefficient matrix between features in the feature vector after the first dimensionality reduction;

[0178] S404: Using principal component analysis, determine the principal components based on the eigenvectors of the correlation coefficient matrix and the eigenvectors after the first dimensionality reduction;

[0179] S405: Based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal component, perform a second dimensionality reduction on the feature vector after the first dimensionality reduction to obtain the feature vector after the second dimensionality reduction.

[0180] S406: Using the segmentation accuracy of the known segmented image as the accuracy feature, the correlation coefficient is used to perform correlation analysis between each feature in the feature vector after the second dimensionality reduction and the accuracy feature. If the correlation coefficient between the two is greater than a set coefficient, the feature is retained to obtain the feature vector after the third dimensionality reduction.

[0181] S407: Use the features in the feature vector after the third dimensionality reduction as features related to road segmentation accuracy.

[0182] The original features have different units and numerical ranges, making comparisons between features difficult. Standardization removes these unit restrictions, converting the data into dimensionless pure numerical values, facilitating comparisons and weighting of features with different units and magnitudes. The standardization formula for the sample vectors of the original features is as follows:

[0183]

[0184] In the formula, Representing original features The standardized sample vector, Represents sample vector The mean, Represents sample vector The standard deviation of . The standardized eigenvector is denoted as .

[0185] Correlation coefficients are used to analyze the correlation between any two features, and one of the strongly correlated features is removed. The correlation coefficient between any two features is:

[0186]

[0187] in, Represents the standardized sample vector Covariance between Represents the standardized sample vector standard deviation Represents the standardized sample vector The standard deviation.

[0188] Traverse from i=1 to i=d0 one by one. In other words, features that follow a given feature and have a correlation coefficient greater than a set coefficient are removed until the last feature is reached. The feature vector is generated by... Dimensional reduction The feature vectors with low correlation among the features were obtained after the first dimensionality reduction.

[0189] Using principal component analysis, based on Use smaller, independent, or unrelated features to explain the original feature vector Most of the variables in the model are used to establish a low-dimensional comprehensive index describing the eigenvectors. The principal components in principal component analysis are... The new features are formed by linear combinations or mixtures of the features in the original data. In the embodiments of this specification, the principal components are determined by the eigenvectors of the correlation coefficient matrix between the various features and the eigenvectors after the first dimensionality reduction. The corresponding formula is:

[0190]

[0191] Where Y M Represents the principal component vector of features. y i Let d represent the i-th principal component vector. y Indicates the number of principal components. This represents the matrix showing the influence of each feature on the principal components in the eigenvectors after the first dimensionality reduction. It is obtained based on the correlation coefficient matrix between each feature in the feature vector after the first dimensionality reduction.

[0192] Specifically, The construction process is as follows: the correlation coefficient matrix between features in the feature vector after the first dimensionality reduction is:

[0193]

[0194] in, Representation of features The correlation coefficient between them These are the feature vectors after the first dimensionality reduction. Then calculate any two features in the matrix. M eigenvalues and the corresponding feature vector in,

[0195] eigenvectors The vertical arrangement forms a matrix, which is the matrix representing the degree of influence of features on principal components.

[0196] In the embodiments described in this specification, reference is made toFigure 8 The step of performing a second dimensionality reduction on the feature vector after the first dimensionality reduction, based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal components, to obtain the feature vector after the second dimensionality reduction, further includes:

[0197] S501: After sorting the features in the feature vector after the first dimensionality reduction in descending order of their influence on the principal components, the sorting order of each feature is obtained. The first digit of the sorting order is 1, and the increment step of the sorting order is 1.

[0198] S502: Set the initial number of statistics to 1;

[0199] S503: Calculate the cumulative influence of all features whose sorting order is less than or equal to the number of features on the principal components, and determine whether the cumulative influence reaches the set influence level;

[0200] S504: If so, then all features whose sorting order is less than or equal to the statistical number will be treated as new features, and the above statistical cumulative influence degree and judgment steps will be stopped;

[0201] S505: If not, the count increments by 1, and the cumulative impact and judgment steps are repeated.

[0202] S506: Using the new features, construct the feature vector after the second dimensionality reduction.

[0203] Furthermore, the influence of each feature in the eigenvector after the first dimensionality reduction on the principal components can be described using the information contribution rate of the eigenvalues. The formula for calculating the information contribution rate is:

[0204]

[0205] Where, q p Let λ be the information contribution rate of the p-th feature in the feature vector after the first dimensionality reduction to the principal component. p The correlation coefficient matrix R M The p-th eigenvalue.

[0206] After calculating the influence of each feature on the principal components, the features are sorted from largest to smallest influence to obtain the sorting order. The first digit of the sorting is 1, and the increment step of the sorting order is 1, i.e., the sorting order is 1, 2, 3...

[0207] The initial number of statistics is set to 1. Since each feature has a corresponding ranking order, the cumulative influence of all features with a ranking order less than or equal to the number of statistics on the principal component is represented by the cumulative contribution rate in this embodiment of the specification. The calculation formula is as follows:

[0208]

[0209] When judging whether the cumulative influence degree reaches the set influence degree, that is, judging whether the cumulative contribution rate reaches the set contribution rate. The set contribution rate can be a value close to 1. In the embodiment of this specification, it is set to 0.95. If so, all features with the sorting order less than or equal to the statistical number are used as new features, and the loop of steps S503 - S505 is stopped. If not, the statistical number is incremented by 1, and the loop of steps S503 - S505 is repeated.

[0210] Construct a feature vector after the second dimensionality reduction using the new features

[0211] After obtaining the feature vector after the second dimensionality reduction, take the segmentation accuracy of the known segmented image as the accuracy feature X k (k = d0 + 1), select d3 (d3 < d2) features most relevant to the segmentation accuracy from to form a feature vector after the third dimensionality reduction Among them, the correlation analysis can be performed between each feature in the feature vector after the second dimensionality reduction and the accuracy feature using the correlation coefficient. Specifically, the correlation coefficient can be calculated according to formula (5).

[0212] Finally, take the features in the feature vector after the third dimensionality reduction as the features related to the road segmentation accuracy.

[0213] In the embodiment of this specification, referring to Figure 9 , the construction of the region of interest of the current vehicle motion state in the vehicle coordinate system further includes:

[0214] S601: Obtain the far coordinate point, the middle coordinate point, and the near coordinate point according to the coordinate points corresponding to the far, middle, and near three preview points of the current vehicle motion state in the vehicle coordinate system;

[0215] S602: Use a cubic spline curve to fit the three coordinate points to obtain a spline curve;

[0216] S603: Move the spline curve left and right along the tangent direction of the middle coordinate point to obtain the left and right boundaries of the region of interest;

[0217] S604: Connect the upper and lower vertices of the left and right boundaries with a straight line to obtain the upper and lower boundaries of the region of interest;

[0218] S605: Use the closed figure enclosed by the left and right boundaries and the upper and lower boundaries as the region of interest of the current vehicle motion state in the vehicle coordinate system.

[0219] Existing vehicle aiming technology generally only determines a single aiming point. However, in actual driving, the driver should pay attention not only to the information of a single aiming point, but also to an area centered on that aiming point. Therefore, in the embodiments of this specification, based on the intermediate aiming point, a far aiming point and a near aiming point are determined. After further constructing a spline curve, the spline curve is moved left and right to obtain the region of interest.

[0220] Assuming the time coefficient of the intermediate aiming point is c0, the time coefficients of the two aiming points, near and far, are:

[0221]

[0222] Where c1 is the pre-aiming time coefficient for the near pre-aiming point, c2 is the pre-aiming time coefficient for the far pre-aiming point, c0 is the pre-aiming time coefficient for the middle pre-aiming point, and Δ c For the aiming time coefficient interval, c0 and Δ c It can be set according to the actual situation.

[0223] Reference Figure 10 Where P1 (P1') is the far aiming point, P0 (P0') is the middle aiming point, and P2 (P2') is the near aiming point. All three aiming points are relative to the vehicle. By substituting the time coefficients of the three aiming points into the existing vehicle aiming technology, the far coordinate point, middle coordinate point, and near coordinate point of the three aiming points in the vehicle coordinate system can be obtained. The existing vehicle aiming technology can be found in the journal article "Path Following Control of Autonomous Four-Wheel-Independent-Drive Electric Vehicles via Second-OrderSliding Mode and Nonlinear Disturbance Observer Techniques".

[0224] After fitting a spline curve to the three coordinate points, the spline curve is shifted left by d / 2 along the tangent direction of the midpoint to obtain the left boundary of the region of interest (ROI), and shifted right by d / 2 to obtain the right boundary of the ROI. The upper vertices of the left and right boundaries are connected by straight lines to obtain the upper boundary of the ROI, and the lower vertices of the left and right boundaries are connected by straight lines to obtain the lower boundary of the ROI. Here, d represents the width of the vehicle. The closed shape enclosed by the left, right, upper, and lower boundaries constitutes the ROI of the current vehicle motion state in the vehicle coordinate system.

[0225] The region of interest (ROI) obtained above is in the vehicle coordinate system. This ROI can be projected onto the current camera's coordinate system to obtain the projected ROI. Specifically, since the vehicle coordinate system and the road coordinate system only have a Z-axis translation, the expression in the road coordinate system is the same as that in the vehicle coordinate system. Therefore, the ROI in the vehicle coordinate system can be projected onto the road coordinate system. V Projecting it along its Z-axis onto the road surface coordinate system yields the region of interest (ROI) in the road surface coordinate system. R Then, based on the camera's projection model, the ROI is... R Projecting onto the camera coordinate system yields the region of interest (ROI) in the camera coordinate system. I The process is as follows:

[0226]

[0227] in, For ROI R Points within the road surface coordinate system CS R The coordinates below, These are the coordinates obtained by projecting this point onto the camera coordinate system. in This represents the row number of the point in the road image captured by the camera. For column numbers.

[0228] Furthermore, based on the location and size of the projected region of interest (ROI), a second evaluation value is needed to assess the usefulness of the segmented image for the current vehicle motion state. Specifically, the ROI... I The more pixels a surface contains, the greater its ROI on the road surface. R The more detailed the region imaging, the greater the likelihood of accurate segmentation. ROI I The closer the center point is to the midpoint of the road image, the more stable the current line-of-sight angle is, and the more stable the ROI is when the vehicle turns. I The less likely it is to fall out of the camera's field of view while moving. When ROI I Getting too close to the edge of the road image can cause ROI to be affected. I Partial or even complete loss of information. In such cases, the segmented image obtained from road segmentation cannot be used to guide vehicle movement and therefore loses its reference value.

[0229] Based on the above analysis, the second evaluation value for assessing the usefulness of the segmented image for the current vehicle motion state is calculated using the following formula:

[0230]

[0231] Where κ2 is the second evaluation value, A1 is the pixel area of ​​the projected region of interest, A0 is the maximum pixel area obtained after projecting the region of interest onto the camera coordinate system corresponding to all different viewing angles, and W A d is the width of the road image captured by the camera. R α is the distance between the center point of the projected region of interest and the midpoint of the road image. C The completeness of the projected region of interest (ROI) is the percentage of the ROI after projection. I The ratio of the area of ​​the imaged portion to its total area in a road image.

[0232] Finally, based on the first and second evaluation values, the evaluation value is calculated using the following formula:

[0233]

[0234] Among them, κ A To evaluate the numerical value, κ 1,0 The minimum threshold of κ1, κ 2,0 The minimum threshold of κ2, where κ 1,0 and κ 2,0 It can be set according to the actual situation.

[0235] In the process of actively adjusting the camera's line of sight angle, you can first determine the target pitch angle, then determine the target azimuth angle under the target pitch angle, and then obtain the target line of sight angle. Alternatively, you can first determine the target azimuth angle, then determine the target pitch angle under the target azimuth angle, and then obtain the target line of sight angle. Both the target pitch angle and the target azimuth angle need to be determined based on the evaluation values.

[0236] Specifically, refer to Figure 11 The step of selecting the target line-of-sight angle from all candidate line-of-sight angles based on the evaluation values ​​of the road images corresponding to all candidate line-of-sight angles by the camera further includes:

[0237] S701: Determine the area where the viewing angle is maintained in the two-dimensional coordinate system based on the threshold of the first evaluation value and the threshold of the second evaluation value.

[0238] S702: Using the first evaluation value and the second evaluation value corresponding to each candidate viewing angle of the camera as the horizontal and vertical coordinates respectively, draw the feature points of each candidate viewing angle on the two-dimensional coordinate system.

[0239] S703: Eliminate the candidate viewing angles of the feature points outside the retention area to obtain the remaining candidate viewing angles;

[0240] S704: The angle with the largest evaluation value among the remaining candidate line-of-sight angles shall be taken as the target line-of-sight angle.

[0241] The threshold includes a minimum threshold and a maximum threshold, where κ 1,0 The minimum threshold of κ1, κ 2,0 The minimum threshold of κ2, as referenced Figure 12 , where κ 1,0 and k 2,0 The maximum thresholds for k1 and κ2 can be set according to the actual situation, with both being 1.0. The area where the viewing angle is maintained in the two-dimensional coordinate system is the region enclosed by the minimum thresholds of κ1 and k2, and the maximum thresholds of κ1 and κ2.

[0242] Each candidate viewing angle corresponds to a feature point in a two-dimensional coordinate system. Some candidate viewing angles may have their feature points within the preservation region, while others may have their feature points outside the preservation region. Figure 12 The trajectory in the image is the trajectory formed by the feature points. Candidate view angles outside the preservation region will be eliminated, and among the remaining candidate view angles, κ... A The largest is the target's line-of-sight angle.

[0243] It should be noted that the above steps S701-S704 are used to determine the target azimuth angle from the candidate azimuth angles corresponding to the target pitch angle after the target pitch angle has been determined, or to determine the target pitch angle from the candidate pitch angles corresponding to the target azimuth angle after the target azimuth angle has been determined. Taking the former as an example, the candidate line-of-sight angles involved in steps S701-S704 are the candidate azimuth angles corresponding to the target pitch angle. Taking the latter as an example, the candidate line-of-sight angles involved in steps S701-S704 are the candidate pitch angles corresponding to the target azimuth angle.

[0244] Of course, when determining the target pitch angle in the former case, it is also necessary to determine the target pitch angle from the candidate pitch angles, or when determining the target azimuth angle in the latter case, it is also necessary to determine the target azimuth angle from the candidate azimuth angles. This process can also be performed with reference to steps S701-S704, except that the candidate line-of-sight angles involved in steps S701-S704 are candidate pitch angles or candidate azimuth angles. Since the execution logic is similar, this embodiment of the specification will not be described in detail.

[0245] The method described in this specification involves inversely projecting a road image onto a road plane coordinate system to obtain a top view. This top view is then segmented to obtain segmented images, which are then evaluated to obtain an evaluation value. The evaluation value is used to assess the segmentation accuracy and usefulness of the segmented image for understanding vehicle motion. When the evaluation value exceeds an evaluation threshold, the camera's viewing angle can be maintained; otherwise, the viewing angle needs to be adjusted. During adjustment, the evaluation value of the camera at candidate viewing angles is continuously obtained through a loop. Based on this, a suitable target viewing angle is selected to ensure the segmentation accuracy and usefulness of the segmented image for understanding vehicle motion. This improves road segmentation performance under adverse conditions such as backlighting and U-turns, ensuring safe driving for intelligent vehicles.

[0246] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse.

[0247] This application provides users with access to relevant big data analysis (such as personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), allowing users to choose to agree to or reject automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0248] Based on the road segmentation method for actively adjusting the camera's line of sight as described above, this specification also provides a road segmentation device for actively adjusting the camera's line of sight. The device may include a system (including a distributed system), software (application), module, component, server, client, etc., using the method described in this specification, combined with necessary hardware implementation. Based on the same innovative concept, the devices in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the problem are similar, the implementation of specific devices in this specification can refer to the implementation of the aforementioned method, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0249] Specifically, Figure 13This is a schematic diagram of the module structure of one embodiment of a road segmentation device that actively adjusts the camera's line of sight, as provided in the embodiments of this specification. (Refer to...) Figure 13 As shown in the embodiment of this specification, a road segmentation device that actively adjusts the camera's line of sight angle is provided. In the initial state, the camera is not in the line of sight angle adjustment. The device includes: a data acquisition module 100, a reverse projection module 200, a segmentation module 300, an evaluation module 400, and a judgment module 500.

[0250] Acquisition module 100 is used to acquire images of the road ahead using a camera;

[0251] The inverse projection module 200 is used to inversely project the road image onto the road plane to obtain a top view in the road plane coordinate system.

[0252] The segmentation module 300 is used to segment the top view into roads to obtain a segmented image;

[0253] The evaluation module 400 is used to evaluate the segmented image according to a pre-set evaluation standard to obtain the evaluation value of the road image;

[0254] The judgment module 500 is used to determine whether the camera is adjusting its viewing angle.

[0255] S51: If not, determine whether the evaluation value of the road image is greater than the evaluation threshold;

[0256] S511: If so, maintain the camera's line-of-sight angle and repeatedly call the acquisition module until a decision is made.

[0257] Module;

[0258] S512: If not, then set the camera to be in the line-of-sight angle adjustment state, actively adjust the line-of-sight angle of the camera to a new candidate line-of-sight angle, and cyclically call the acquisition module to the judgment module;

[0259] S52: If so, then,

[0260] The camera's line of sight is actively adjusted to a new candidate line of sight angle. The acquisition module is repeatedly called to the judgment module until there are no new candidate line of sight angles, and then the camera's line of sight angle adjustment is completed.

[0261] Once the camera's line of sight angle has been adjusted, the target line of sight angle is selected from all candidate line of sight angles based on the evaluation values ​​of the road images corresponding to all candidate line of sight angles.

[0262] Adjust the camera to the target line-of-sight angle and repeatedly call the acquisition module to the judgment module.

[0263] Reference Figure 14 As shown, based on the road segmentation method for actively adjusting the camera's line of sight as described above, one embodiment of this specification also provides a computer device 1402, wherein the above method operates on the computer device 1402. The computer device 1402 may include one or more processors 1404, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each processing unit capable of implementing one or more hardware threads. The computer device 1402 may also include any memory 1406 for storing any kind of information such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 1406 and can run on the processor 1404. When the computer program is run by the processor 1404, it can execute instructions according to the above method. Non-limitingly, for example, the memory 1406 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 1402. In one scenario, when processor 1404 executes associated instructions stored in any memory or combination of memories, computer device 1402 can perform any operation of the associated instructions. Computer device 1402 also includes one or more drive mechanisms 1408 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0264] Computer device 1402 may also include an input / output module 1410 (I / O) for receiving various inputs (via input device 1412) and providing various outputs (via output device 1414). A specific output mechanism may include a presentation device 1416 and an associated graphical user interface 1418 (GUI). In other embodiments, the input / output module 1410 (I / O), input device 1412, and output device 1414 may be omitted, and the device may function solely as a computer device within a network. Computer device 1402 may also include one or more network interfaces 1420 for exchanging data with other devices via one or more communication links 1422. One or more communication buses 1424 couple the components described above together.

[0265] Communication link 1422 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1422 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0266] Corresponding to the method, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described method.

[0267] This specification also provides computer-readable instructions, wherein when a processor executes the instructions, the program therein causes the processor to perform the above-described method.

[0268] This specification also provides a computer program product that, when run by the processor of a computer device, executes the above-described method.

[0269] The computer program product described in this specification is a software product that mainly implements the methods described in this specification through a computer program.

[0270] It should be understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0271] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the embodiments of this specification, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0272] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments in this specification.

[0273] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0274] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0275] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described in this specification, depending on actual needs.

[0276] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0277] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this specification, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0278] This specification uses specific embodiments to illustrate the principles and implementation methods of the embodiments. The above description of the embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments in this specification. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments in this specification. Therefore, the content of this specification should not be construed as a limitation on the embodiments in this specification.

Claims

1. A road segmentation method that actively adjusts the camera's line-of-sight angle, characterized in that, Initially, the camera is not in the process of adjusting its viewing angle, including: S1: Use a camera to capture images of the road ahead; S2: Inversely project the road image onto the road plane to obtain a top view in the road plane coordinate system; S3: Perform road segmentation on the top view to obtain a segmented image; S4: Evaluate the segmented image using pre-set evaluation criteria to obtain the final evaluation value of the road image; S5: Determine whether the camera is adjusting its viewing angle; S51: If not, determine whether the final evaluation value of the road image is greater than the evaluation threshold; S511: If so, maintain the camera's line of sight and repeat steps S1-S5. S512: If not, then set the camera to be in the process of adjusting the viewing angle, actively adjust the viewing angle of the camera to a new candidate viewing angle, and repeat steps S1-S5. S52: If so, then, Actively adjust the camera's line of sight to a new candidate line of sight, repeat steps S1-S5 until there are no new candidate line of sight, and then set the camera's line of sight adjustment to be complete. Once the camera's line of sight angle has been adjusted, the target line of sight angle is selected from all candidate line of sight angles based on the final evaluation values ​​of the road images corresponding to all candidate line of sight angles. Adjust the camera to the target line-of-sight angle and repeat steps S1-S5.

2. The method according to claim 1, characterized in that, The step of evaluating the segmented image using pre-set evaluation criteria to obtain the final evaluation value of the road image further includes: Extract features related to road segmentation accuracy from the segmented image; The features are input into a pre-trained neural network model, and the first evaluation value is output to evaluate the accuracy of road segmentation. Construct the region of interest for the current vehicle motion state in the vehicle coordinate system; The region of interest is projected onto the camera coordinate system of the current camera to obtain the projected region of interest; Based on the position and size of the projected region of interest, a second evaluation value is calculated to assess the usefulness of the segmented image for the current vehicle motion state. The final evaluation value is calculated based on the first evaluation value and the second evaluation value.

3. The method according to claim 2, characterized in that, The method for determining the features related to road segmentation accuracy includes: By inversely projecting the known road image onto the road plane, a known top view in the road plane coordinate system is obtained; The known top view is segmented into roads to obtain a known segmented image, where the segmentation accuracy of the known segmented image is a known quantity; Obtain several original features related to the road shape from the known segmented image to form the original feature vector; The original feature vector is subjected to three dimensionality reduction operations to obtain features related to road segmentation accuracy.

4. The method according to claim 3, characterized in that, The step of performing three dimensionality reductions on the original feature vector to obtain features related to road segmentation accuracy further includes: Each original feature in the original feature vector is standardized to obtain a standardized feature vector. The correlation coefficient is used to perform correlation analysis between every two features in the standardized feature vector. If the correlation coefficient between every two features is greater than a set coefficient, one of the features in every two features is removed to obtain the feature vector after the first dimensionality reduction. The correlation coefficient between any two features in the feature vector after the first dimensionality reduction is not greater than the set coefficient. Construct the correlation coefficient matrix among the features in the feature vector after the first dimensionality reduction; Using principal component analysis, the principal components are determined based on the eigenvectors of the correlation coefficient matrix and the eigenvectors after the first dimensionality reduction. Based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal component, the feature vector after the first dimensionality reduction is subjected to a second dimensionality reduction to obtain the feature vector after the second dimensionality reduction. The segmentation accuracy of the known segmented image is used as the accuracy feature. Correlation analysis is performed on each feature in the feature vector after the second dimensionality reduction and the accuracy feature using the correlation coefficient. If the correlation coefficient between the two is greater than a set coefficient, the feature is retained, and the feature vector after the third dimensionality reduction is obtained. The features in the feature vector after the third dimensionality reduction are used as features related to road segmentation accuracy.

5. The method according to claim 4, characterized in that, The step of performing a second dimensionality reduction on the feature vector after the first dimensionality reduction, based on the degree of influence of each feature in the feature vector after the first dimensionality reduction on the principal components, to obtain the feature vector after the second dimensionality reduction, further includes: After sorting the features in the feature vector after the first dimensionality reduction from largest to smallest according to their influence on the principal component, the sorting order of each feature is obtained. The first digit of the sorting order is 1, and the increment step of the sorting order is 1. Set the initial number of statistics to 1; The cumulative influence of all features whose sorting order is less than or equal to the number of features on the principal components is determined, and it is then determined whether the cumulative influence reaches a set level. If so, then all features whose sorting order is less than or equal to the statistical number will be treated as new features, and the above statistical cumulative influence degree and judgment steps will be stopped; If not, the count increments by 1, and the cumulative impact and judgment steps are repeated. Using the new features, construct the feature vector after the second dimensionality reduction.

6. The method according to claim 2, characterized in that, The region of interest for constructing the current vehicle motion state in the vehicle coordinate system further includes: Based on the coordinates of the three preview points (far, middle, and near) in the vehicle coordinate system corresponding to the current vehicle motion state, the far coordinate point, middle coordinate point, and near coordinate point are obtained. The spline curve is obtained by fitting a cubic spline curve to the three coordinate points. The spline curve is moved left and right along the tangent direction of the mid-coordinate point to obtain the left and right boundaries of the region of interest. Connect the upper and lower vertices of the left and right boundaries with straight lines to obtain the upper and lower boundaries of the region of interest; The closed shape enclosed by the left and right boundaries and the top and bottom boundaries is taken as the region of interest for the current vehicle motion state in the vehicle coordinate system.

7. The method according to claim 2, characterized in that, The step of calculating a second evaluation value for assessing the usefulness of the segmented image for the current vehicle motion state based on the position and size of the projected region of interest further includes: The second evaluation value, used to assess the usefulness of the segmented image for the current vehicle motion state, is calculated using the following formula: Where κ2 is the second evaluation value, A1 is the pixel area of ​​the projected region of interest, A0 is the maximum pixel area obtained after projecting the region of interest onto the camera coordinate system corresponding to all different viewing angles, and W A d is the width of the road image captured by the camera. R α is the distance between the center point of the projected region of interest and the midpoint of the road image. C represents the completeness of the projected region of interest.

8. The method according to claim 2, characterized in that, The final evaluation value of the road images corresponding to all candidate line-of-sight angles by the camera, and the selection of the target line-of-sight angle from all candidate line-of-sight angles, further includes: Based on the threshold values ​​of the first and second evaluation values, the region where the viewing angle is maintained in the two-dimensional coordinate system is determined. Using the first and second evaluation values ​​corresponding to each candidate viewing angle of the camera as the abscissa and ordinate respectively, the feature points of each candidate viewing angle on the two-dimensional coordinate system are plotted. Candidate viewing angles of the feature points outside the retention area are eliminated to obtain the remaining candidate viewing angles; The angle with the highest final evaluation value among the remaining candidate line-of-sight angles is taken as the target line-of-sight angle.

9. The method according to claim 1, characterized in that, The step of inversely projecting the road image onto the road plane to obtain a top view in the road plane coordinate system further includes: The road image is inversely projected onto the road plane to obtain an irregular shape; Select a rectangular region of a set length and a set width from the irregular shape; The rectangular region is set at a predetermined position and angle in the road plane coordinate system; The irregular shapes within the rectangular area are pixelated into a top view according to a set ratio.

10. The method according to claim 1, characterized in that, The candidate line-of-sight angles include: candidate pitch angles and candidate azimuth angles corresponding to the target pitch angle; or, candidate azimuth angles and candidate pitch angles corresponding to the target azimuth angle.

11. A road segmentation device that actively adjusts the camera's line-of-sight angle, characterized in that, In its initial state, the camera is not in the process of adjusting its line of sight angle. The device includes: The acquisition module is used to acquire images of the road ahead using a camera; The inverse projection module is used to inversely project the road image onto the road plane to obtain a top view in the road plane coordinate system. A segmentation module is used to segment the top view into roads to obtain a segmented image; The evaluation module is used to evaluate the segmented image according to a pre-set evaluation standard to obtain the final evaluation value of the road image; The determination module is used to determine whether the camera is adjusting its viewing angle. S51: If not, determine whether the final evaluation value of the road image is greater than the evaluation threshold; S511: If so, maintain the camera's line of sight and repeatedly call the acquisition module to the judgment module; S512: If not, then set the camera to be in the line-of-sight angle adjustment state, actively adjust the line-of-sight angle of the camera to a new candidate line-of-sight angle, and cyclically call the acquisition module to the judgment module; S52: If so, then, The camera's line of sight is actively adjusted to a new candidate line of sight angle. The acquisition module is repeatedly called to the judgment module until there are no new candidate line of sight angles, and then the camera's line of sight angle adjustment is completed. Once the camera's line of sight angle has been adjusted, the target line of sight angle is selected from all candidate line of sight angles based on the final evaluation values ​​of the road images corresponding to all candidate line of sight angles. Adjust the camera to the target line-of-sight angle and repeatedly call the acquisition module to the judgment module.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1-10.

Images