Projection vehicle lamp illumination method and device based on image semantic segmentation target recognition

Abstract

The invention discloses a projection vehicle lamp illumination method and device based on image semantic segmentation target recognition. The method comprises the steps of 1, obtaining calibration parameters of a camera and a projection vehicle lamp on each side; step 2, obtaining structured data of the current image frame through a semantic segmentation model, obtaining related information of each target instance, and filling the related information of the target instance into the structured data of each corresponding target instance; 3, calculating the illuminance of each target instance; and step 4, converting the contours of the target instances from the image coordinate system of the current image frame to a projection vehicle lamp coordinate system, performing pixel-level brightness filling on a closed area formed by the contours of the target instances on the basis of the structured data of the target instances according to corresponding illuminance, generating a projection bitmap, and displaying the projection bitmap. The bitmap is used as a projection lighting scheme of a projection vehicle lamp array. According to the invention, the vehicle lighting system can be improved to realize the blind compensation and anti-dazzling effects.

Description

Technical Field

[0001] The present invention relates to the field of sensor perception and recognition technology, and in particular to a projection vehicle lamp lighting method and device based on image semantic segmentation target recognition. Background Art

[0002] The rapid development of semiconductor technology and artificial intelligence technology has promoted the upgrade of vehicle hardware from the sensor end to the execution control end, as well as software systems such as assisted driving functions. At the same time, it has also greatly improved the driving quality in terms of driving / driving safety and human-machine interaction friendliness. The vehicle lighting system is also evolving in the direction of intelligence. At present, the main lighting control systems on the market include two parts: the sensor end and the lighting control execution end. Among them, the sensor end uses optical sensors of different types and performances to capture surrounding environment information, and then transmits it to the downstream execution end after information extraction and identification. The lighting control execution end uses traditional high and low beam systems or point array lighting to achieve response actions such as blindness compensation and anti-glare at different granularities.

[0003] Today, mainstream sensor perception and recognition technologies mainly complete the location and classification recognition of the target level, with limited recognition resolution and feature classification for the target, and limited stability in distinguishing and recognizing complex multi-classification and occlusion (truncation) situations in actual traffic flow scenes. In terms of lighting execution control, traditional high and low beam and dot matrix lighting systems basically adopt longitudinal lighting control or lateral lighting control with insufficient control resolution, and cannot meet the lighting control strategy for individual targets.

[0004] In a control technology of a smart high beam disclosed in the prior art, a target image is acquired, and a three-dimensional detection frame of an obstacle in the target image is identified, and the anti-glare angle range corresponding to the target image is determined according to the three-dimensional detection frame, and the brightness of the smart high beam is adjusted according to the anti-glare angle range. However, this technical solution still has the following shortcomings, as listed below:

[0005] (1) In this technical solution, the identification information of the target obstacle is a three-dimensional detection frame, and there is no detailed identification of different obstacle features, such as pedestrian torso, vehicle body, windshield, rearview mirror and other areas. The target three-dimensional detection frame is transformed through projection and converted to the pointing range of the projection headlight. The accuracy is insufficient, resulting in the projection method that does not fit the target object. The obstructing obstacles behind the projection area are missed due to insufficient lighting. The resulting blind spots pose a great risk to the driving decision of the vehicle.

[0006] (2) The high beam control method in this technical solution is limited to the angle range for specific target adjustment, and the four-dimensional restrictions of the horizontal left and right boundaries and the vertical upper and lower boundaries. This leads to insufficient control resolution and is unable to adapt to targets with irregular contours (for example, parallel to the vertical or horizontal direction). There are areas of glare due to over-lighting or blind spots due to under-lighting, which increases the risk of driving safety.

[0007] (3) This technical solution lacks consideration of the vehicle's own and other different types of dynamic traffic participants' movement characteristics such as speed and direction, and lacks the ability to track and predict scene targets. The projection lamps have insufficient dynamic response to the lighting control of targets with large relative motion offsets from the vehicle, and are prone to shading or offset jitter of the target area, which may cause the oncoming target's vision to be affected by light flickering and other factors that affect the oncoming driving safety. Summary of the invention

[0008] The purpose of the present invention is to provide a method and device for controlling projection vehicle lighting by performing target recognition based on image semantic segmentation, which can improve the blindness compensation and anti-glare effects of the vehicle lighting system.

[0009] To achieve the above object, the present invention provides a projection vehicle lamp lighting method based on image semantic segmentation target recognition, which comprises:

[0010] Step 1: A camera and a projection lamp are set on each side of the vehicle body, and calibration parameters of the camera and the projection lamp on each side are obtained;

[0011] Step 2: According to the image frame acquired by the camera in real time, the structured data of the current image frame is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each target instance and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each target instance is obtained. The relevant information of the target instance includes the pixel coordinate distance between the target instance and the vehicle and the pixel-level two-dimensional speed. The relevant information of the target instance is filled into the structured data of each corresponding target instance;

[0012] Step 3, calculating the illumination of each target instance;

[0013] Step 4, through the calibration parameters of step 1, the contour of the target instance is converted from the image coordinate system of the current image frame to the projection headlight coordinate system, and according to the corresponding illuminance calculated in step 3, based on the structured data of the target instance obtained in step 2, the closed area formed by the contour of each target instance is filled with pixel-level brightness, and then a projection bitmap is generated, which is used as the projection lighting solution for the projection headlight array.

[0014] Furthermore, the method of “obtaining the contour of each target instance in the current image frame and the classification information corresponding to the contour by using the semantic segmentation model” in step 2 specifically includes:

[0015] Step 21a, according to the image frame acquired by the camera in real time, extract the pixel-level semantic contour and classification of the current image frame through the semantic segmentation model, output the contour point set, and the contour point set constitutes the contour;

[0016] Step 22a, iteratively calculating a plurality of continuous contour points in a loop to obtain a vector formed by two adjacent contour points, and determining whether the adjacent first vector and the second vector satisfy the vector collinearity theorem. If so, deleting the contour points shared by the first vector and the second vector, and updating the contour point set;

[0017] Step 23a, outputting structured data based on the contour point set updated in step 22a, combined with the classification information corresponding to the contour and the ID of the target instance.

[0018] Furthermore, step 3 specifically includes:

[0019] According to the classification information corresponding to the contour of the target instance, determine the category to which the pixel point (i, j) of the projection headlight belongs;

[0020] Combined with the category to which the pixel point (i, j) of the projected headlight belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, the contour point (x i ,y j ) illuminance g(x i ,y j ):

[0021]

[0022] Where i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively; the value of i is not less than 0 and not greater than the width of the current image frame; the value of j is not less than 0 and not greater than the height of the current image frame; α is the maximum intensity of the illumination of the projected vehicle light; δ is an attenuation coefficient based on the pixel coordinate distance d between the target instance and the vehicle, which is described by formula (2); θ is a measured constant attenuation coefficient within a limited distance, D max The maximum distance to enable the projection light to project illumination onto the subject of the target instance:

[0023]

[0024] Furthermore, step 3 specifically includes:

[0025] The following morphological techniques are used for expansion compensation to suppress the illumination value of the expansion area of ​​the key target. The direction and magnitude of the expansion refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively:

[0026] ①According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system;

[0027] ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, the contour point p (x,y) Replace with the point to be expanded q (x,y) ;

[0028] ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ①.

[0029] Furthermore, the method for obtaining the "semantic segmentation model" in step 2 includes:

[0030] Step 21b, sample data collection and labeling, specifically includes:

[0031] Step 211, combing the scene, outputting the core semantic elements of the driving traffic scene, as shown in the following table;

[0032]

[0033] Step 212, according to the road data collection action of driving the vehicle, output the collected and discretely extracted image set;

[0034] Step 213, annotating the image acquired in step 212, and outputting an image dataset with semantically annotated true values;

[0035] Step 22b, using the data set annotated in step 21b, perform transfer learning training and evaluation on the selected model to obtain a semantic segmentation model for projection headlights;

[0036] Step 23b, algorithm post-processing and result output.

[0037] Furthermore, the imaging surface of the camera on each side is parallel to the projection light source surface of the projection lamp, and the camera and the lateral side of the projection lamp are tightly fitted. The lateral distance between the optical center of the camera and the center point of the projection light source surface of the projection lamp is close to the sum of the lateral widths of the two, so as to maintain a relatively close viewing angle and stable projection transformation characteristics. The camera and projection lamp on each side are configured as an integrated module.

[0038] The present invention also provides a projection vehicle lamp lighting device based on image semantic segmentation target recognition, which comprises:

[0039] A parameter calibration unit, which is used for obtaining calibration parameters of the camera and projection lamp on each side of the vehicle body;

[0040] A target instance information processing unit is used to obtain structured data of the current image frame through a semantic segmentation model according to the image frame acquired by the camera in real time, wherein the structured data includes an ID corresponding to each target instance and semantic classification information and contour point sets corresponding to different setting levels, and then obtain relevant information of each target instance, wherein the relevant information of the target instance includes the pixel coordinate distance between the target instance and the vehicle and the pixel-level two-dimensional speed, and fill the relevant information of the target instance into the structured data of each corresponding target instance;

[0041] An illumination calculation unit, which is used to calculate the illumination of each target instance;

[0042] The projection lighting scheme generating unit is used to convert the contour of the target instance from the image coordinate system of the current image frame to the projection headlight coordinate system through the calibration parameters of the parameter calibration unit, and according to the corresponding illumination attribute calculated in step 3, on the basis of the structured data of the target instance obtained by the target instance information processing unit, perform pixel-level brightness filling on the closed area formed by the contour of each target instance, and then generate a projection bitmap, which serves as the projection lighting scheme of the projection headlight array.

[0043] Furthermore, the target instance information processing unit specifically includes:

[0044] The semantic segmentation model is used to extract the pixel-level semantic contour and classification of the current image frame according to the image frame acquired by the camera in real time, and output a set of contour points, which constitute the contour;

[0045] A contour point update model is used to iteratively calculate a plurality of continuous contour points through a loop, obtain a vector formed by two adjacent contour points, and determine whether the adjacent first vector and second vector satisfy the vector collinearity theorem. If so, the contour points shared by the first vector and the second vector are deleted, and the contour point set is updated;

[0046] The structured data output model is used to output structured data according to the contour point set updated by the contour point update model, combined with the classification information corresponding to the contour and the ID of the target instance.

[0047] Furthermore, the illumination calculation unit specifically includes:

[0048] A pixel point attribute determination model is used to determine the category to which the pixel point (i, j) of the projection headlight belongs according to the classification information corresponding to the contour of the target instance;

[0049] The contour point illumination calculation module is used to combine the category to which the pixel point (i, j) of the projection headlight belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, and calculate the contour point (x i ,y j ) illuminance g(x i ,y j ):

[0050]

[0051] Where i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively; the value of i is not less than 0 and not greater than the width of the current image frame; the value of j is not less than 0 and not greater than the height of the current image frame; α is the maximum intensity of the illumination of the projected vehicle light; δ is an attenuation coefficient based on the pixel coordinate distance d between the target instance and the vehicle, which is described by formula (2); θ is a measured constant attenuation coefficient within a limited distance, D max The maximum distance to enable the projection light to project illumination onto the subject of the target instance:

[0052]

[0053] Furthermore, the illumination calculation unit specifically includes:

[0054] The illumination modification module is used to use the following morphological techniques to perform expansion compensation and suppress the illumination value of the expansion area of ​​the key target. The direction and magnitude of the expansion refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively:

[0055] ①According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system;

[0056] ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, the contour point p (x,y) Replace with the point to be expanded q (x,y) ;

[0057] ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ①.

[0058] The present invention mainly optimizes and solves the following three problems:

[0059] 1. Aiming at the problem of inaccurate target area in traditional detection methods in anti-glare scenes, the present invention adopts a machine learning image semantic segmentation model method to obtain pixel-level contour information of the target, and converts it into the contour boundary of the projection light viewpoint through perspective projection transformation, providing refined and structured semantic information for the back-end projection light scene application.

[0060] 2. Aiming at the problem of insufficient control resolution of high and low beam systems and dot matrix lighting systems, the present invention adopts a high-resolution array projection lighting device, realizes a high-precision real-time lighting system through pixel-level projection point brightness and chromaticity control, and establishes a high-real-time data transmission channel with the detection system to realize a closed loop of sensing and control.

[0061] 3. Aiming at the relative spatiotemporal jitter problem caused by the movement characteristics of the vehicle itself and other dynamic traffic participants, the present invention combines semantic segmentation with morphological technology methods, adopts tracking and prediction methods to grade and smooth the illumination of the target boundary, and optimizes the application effect of back-end anti-glare and blind spots. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of a single-sided projection lamp and a camera as an integrated module according to an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of the arrangement of a single-sided projection lamp and a camera as an integrated module on a vehicle body according to an embodiment of the present invention.

[0064] Figure 3 FIG. 2 is a schematic diagram of a projection lamp according to an embodiment of the present invention.

[0065] Figure 4 Schematic diagram of thinning contour points according to an embodiment of the present invention.

[0066] Figure 5 Figures a, b, and c are schematic diagrams of blindness compensation and anti-glare lighting effects of projection headlights obtained after expansion compensation using morphological technology in an embodiment of the present invention.

[0067] Figure 6 The figure is a schematic diagram of the process of collecting and labeling sample data according to an embodiment of the present invention.

[0068] Figure 7 Schematic diagram of traffic semantic elements of a driving scene according to an embodiment of the present invention.

[0069] Figure 8 Schematic diagram of the iterative training and evaluation process of the semantic segmentation model of projection headlights according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In the drawings, the same or similar reference numerals are used to represent the same or similar elements or elements with the same or similar functions. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0071] In the description of the present invention, the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the scope of protection of the present invention.

[0072] The present invention relates to a projection vehicle lamp lighting method and system based on image semantic segmentation target recognition, which combines the high-resolution pixel-level target instance segmentation capability of the semantic segmentation model with the high-precision fine illumination control provided by the dense array light beam, thereby enhancing the scene response capability of the vehicle lighting system and improving the safety of assisted driving scenarios.

[0073] The projection vehicle lamp lighting method based on image semantic segmentation target recognition provided by an embodiment of the present invention includes:

[0074] Step 1: A camera and a projection lamp are set on each side of the vehicle body, and calibration parameters of the camera and the projection lamp on each side are obtained.

[0075] The camera and projection lights on each side of the vehicle body may be arranged in the following manner, for example: Figure 1 and Figure 2 As shown, the imaging plane of the camera on each side is parallel to the projection light source plane of the projection lamp, and the camera and the lateral side of the projection lamp are closely fitted, and the lateral distance between the optical center of the camera and the center point of the projection light source plane of the projection lamp is close to the sum of the lateral widths of the two, so as to maintain a relatively close viewing angle between the two and a stable projection transformation characteristic. Figure 1 As shown, the camera and projection lamp on each side are preferably configured as an integrated module. Of course, a separate configuration can also achieve the purpose of the present invention.

[0076] The calibration parameters of the camera include the camera's internal parameters, external parameters and distortion parameters, and the calibration parameters of the projection lamp include the projection lamp's external parameters and offset coordinates. The method for obtaining the camera's external parameters and the projection lamp's external parameters and offset coordinates specifically includes:

[0077] Step 11, calibrate the camera internal parameters. For example, a commonly used checkerboard calibration plate is used, the camera collects the calibration plate image, obtains a series of camera images of the calibration plate from different perspectives, and uses the Zhang Zhengyou calibration method to calibrate the camera internal parameters and distortion parameters.

[0078] Step 12, calibrate the external parameters of the camera and projection lamp. Figure 3 As shown in the figure, a vehicle equipped with projection lights on both sides is parked on the ground plane, and the four corner points of the vehicle are projected onto the ground plane to obtain the following Figure 3 The L-shaped projection corner points are shown. The image acquired by the camera is dedistorted using the calibrated camera internal parameters and distortion parameters, the pixel coordinates of the image coordinate system where the projection corner points are located are obtained, the coordinate difference between the projection corner points and the pixel coordinates is calculated as the offset coordinates, and finally, multiple groups of projection corner points are transformed to verify whether the calibration coordinate offset meets expectations.

[0079] Step 2, according to the image frame acquired by the camera in real time, the structured data of the current image frame is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each target instance and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each target instance is obtained. The relevant information of the target instance includes the pixel coordinate distance between the target instance and the vehicle and the pixel-level two-dimensional speed, and the relevant information of the target instance is filled into the structured data of each corresponding target instance. Among them, the ID corresponding to the target instance can be used as the target instance ID in the continuous buffered multi-frame contour cache queue according to the intersection and union ratio of the pixel coordinate distance between the target instance and the vehicle and the area of ​​the contour. The contour point set refers to the position coordinate set of the pixel point in the image coordinate system. The pixel coordinate distance between the target instance and the vehicle can be obtained by taking the pixel difference between the pixel coordinate of the centroid of the contour of each target instance and the pixel coordinate of the midpoint of the lower boundary of the current image frame in the horizontal x-axis and the vertical y-axis as the pixel coordinate distance between the corresponding target instance and the vehicle. The pixel-level two-dimensional velocity information can be obtained by the state estimation method based on the pixel coordinate distance of each target instance from the ego vehicle.

[0080] In one embodiment, Figure 4 As shown, the method of "obtaining the contours of each target instance in the current image frame and the classification information corresponding to the contours through the semantic segmentation model" in step 2 specifically includes:

[0081] Step 21a, based on the image frame acquired by the camera in real time, the pixel-level semantic contour and classification of the current image frame are extracted through the semantic segmentation model, and a contour point set is output, and the contour point set constitutes a contour.

[0082] Step 22a, iteratively calculates multiple consecutive contour points in a loop to obtain a vector composed of two adjacent contour points, and determines whether the adjacent first vector and second vector satisfy the vector collinearity theorem. If so, the contour points shared by the first vector and the second vector are deleted, and the contour point set is updated. In this way, the contour points are sparsed, thereby controlling the number of boundary points of the contour edge in the image coordinate system.

[0083] Step 23a, according to the contour point set updated in step 22a, combined with the classification information corresponding to the contour and the ID of the target instance, output structured data. As shown in Table 1 below, Table 1 is a predefined data structure list, which is filled with the contour information and classification information obtained in step 32, as the single-frame multi-instance semantic segmentation result data for storage and use, and the data structure column representation is shown in Table 1.

[0084] Table 1

[0085]

[0086] Step 3: Calculate the illumination of each target instance.

[0087] In one embodiment, step 3 can be implemented by using a common method, a solution of shading the foreground (people, cars) and filling the background (road), or by using the following method:

[0088] According to the classification information corresponding to the contour of the target instance, determine the category to which the pixel point (i, j) of the projection headlight belongs;

[0089] Combined with the category to which the pixel point (i, j) of the projected headlight belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, the contour point (x i ,y j ) illuminance g(x i ,y j ):

[0090]

[0091] Where i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively. The value of i is not less than 0 and not greater than the width of the current image frame, and the value of j is not less than 0 and not greater than the height of the current image frame. α is the maximum intensity of the illumination of the projected vehicle light. δ is an attenuation coefficient based on the pixel coordinate distance d between the target instance and the vehicle, which is described as formula (2). θ is a measured constant attenuation coefficient within a limited distance. For example, the value can be configured as 80% in the case of complex traffic flow in urban sections to limit the impact on other traffic participants, and 90% in the case of simple traffic flow on expressways. D max The maximum distance to enable the projection light to project illumination onto the subject of the target instance:

[0092]

[0093] In one embodiment, Figure 5 As shown in a, b, and c, step 3 specifically includes:

[0094] The following morphological techniques are used for expansion compensation to suppress the illumination value of the expansion area of ​​the key target. The direction and magnitude of the expansion refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively:

[0095] ①According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system;

[0096] ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, the contour point p (x,y) Replace with the point to be expanded q (x,y) ;

[0097] ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ①.

[0098] For target instances with speed, the correct instance contour is not included due to inaccurate semantic segmentation boundaries, and the projection lighting scheme lags due to factors such as system link delay. Morphological technology is used for expansion compensation to suppress the illumination value of the key target expansion area. The direction of expansion refers to the two-dimensional velocity vector direction of the current instance, and the expansion depth refers to the two-dimensional velocity scalar value.

[0099] Step 4: The contour of the target instance is converted from the image coordinate system of the current image frame to the projection headlight coordinate system through the calibration parameters of step 1, and the closed area formed by the contour of each target instance is filled with pixel-level brightness based on the structured data of the target instance obtained in step 2 according to the corresponding illumination attribute calculated in step 3, thereby generating a projection bitmap, which is used as the projection lighting scheme for the projection headlight array. The instance contour in the camera coordinate system is projected and transformed through the calibration parameters, and converted into the contour coordinates of the projection headlight coordinate system, and the closed area formed by the contour of each instance is filled with pixel-level brightness according to the corresponding illumination attribute, thereby generating a projection bitmap, which is transmitted to the projection lighting execution device for projection.

[0100] In one embodiment, the method for obtaining the “semantic segmentation model” in step 2 includes:

[0101] Step 21b, such as Figure 6 As shown in the figure, sample data collection and labeling include:

[0102] Step 211, scene combing, outputting the core semantic elements of the traffic scene, as shown in Table 1 below; Figure 7As shown in the figure, the traffic semantic elements for driving scenes mainly include roads and lighting environment, traffic participants and traffic signs. Among them, roads mainly cover urban and rural roads, highways, open sections and tunnels. Traffic participants are classified into motor vehicles, pedestrians and non-motor vehicles. Traffic signs include ground signs and road signs.

[0103] In addition, for the application of projection headlight lighting scenarios, more refined semantic segmentation and annotation of traffic participants is required. For pedestrians and people in non-enclosed passenger cabins such as non-motor vehicles, motorcycles, and convertible motor vehicles, the torso and head need to be further refined and annotated separately. For closed passenger cabin motor vehicles, the windshield and rearview mirror areas need to be annotated, and the secondary semantic classifications need to be circled separately. The semantic classifications can be evaluated and increased according to the actual scene conditions and needs, including but not limited to those described in Table 2.

[0104] Table 2

[0105]

[0106] In the table, "-" means "none", so the "-" in the primary and secondary semantic classifications of "road and lighting environment" means that no corresponding primary and secondary semantic classifications are made. The "-" in the secondary semantic classification of "traffic signs" means that no corresponding secondary semantic classification is made.

[0107] Step 212, according to the road data collection action of driving the vehicle, the collected and discretely extracted image set is output. For example, a vehicle equipped with a camera is used to plan and select areas that meet the combed scenes for driving, and image data is collected and stored during driving. Data collection ensures balanced coverage of different traffic scenes. The continuous image set that has been collected is obtained by equally spaced frame extraction to obtain a discrete data set, and similar images in the discrete set are eliminated through manual verification to reduce and optimize the representativeness of the quality of the data set to be annotated. The scale of the data set that is finally screened is maintained at, for example, more than 1,000 images. At this time, the total number of annotated target categories is multiplied by 1,000. For example, the number of annotated instances in this example includes 6 first-level categories and 4 second-level fine categories, totaling 10 categories.

[0108] Step 213, annotate the image acquired in step 212, and output an image dataset with semantic annotation truth values. For example, the dataset annotates the contours of semantic elements frame by frame, and after the annotation is completed, the annotation is verified through a visualization tool to check the consistency of the annotation boundaries, whether small targets, occluded or truncated targets are missed, and whether the labels are correctly matched. Among them, the boundary consistency can be, but is not limited to, the average error of boundary pixels.

[0109] Step 22b, using the data set annotated in step 21b, perform transfer learning training and evaluation on the selected model to obtain a semantic segmentation model of projection headlights.

[0110] like Figure 8 As shown, step 22b specifically includes the following four steps:

[0111] Step 221b, model framework selection.

[0112] Considering the model performance indicators, computing power and real-time requirements of the deployment platform, and referring to the benchmark segmentation evaluation of public data sets in the traffic neighborhood, we select a suitable model framework, such as U-Net, from the existing model implementations commonly used in the industry.

[0113] Step 222b, model pre-training.

[0114] The model framework's effect on feature learning in specific fields depends on feature training of large-scale data sets. The scale of private data in actual projects is difficult to meet the model's internalization of generalized features, so the model needs to be pre-trained using public large-scale traffic data sets, such as Cityscapes and KITTI. The pre-trained model is evaluated through indicators on the test set.

[0115] Step 223b, model transfer learning.

[0116] The pre-trained model is trained and feature transferred on the labeled sample data set. The work that needs to be done before training includes feature semantic label adaptation (10 labels), sample set division, and training hyperparameter configuration (number of iterations, learning rate, etc.). The sample set division refers to the training set, validation set, and test set, and is randomly divided according to 7:2:1 or a similar ratio. After completing the above parameter configuration, start the model migration training, and monitor whether the iterative change convergence trend of the model's core indicators meets expectations. If not, it is necessary to stop and locate the problem scene, supplement the scene data through step 1, or optimize the parameter configuration in this step to restart the training. The technology in this field is mature and can be completed after iteration. And if it is found that there are scenes with elements that do not meet the requirements during the test and verification process, the scene image can be supplemented again through step 1 and steps 1 and 2 can be completed to achieve the expected effect.

[0117] Step 224b, sample testing and indicator evaluation.

[0118] The model that has completed transfer training is tested and evaluated on the divided test set, focusing on the accuracy set recall rate indicators, and checking whether the model generalization requirements are met by sampling visualization samples and annotating model prediction results. When the proportion of samples or scenarios that do not meet expectations found through sample inspection exceeds expectations, it is necessary to perform additional annotations on such scenes and re-perform transfer learning training to improve the model effect.

[0119] Step 23b, algorithm post-processing and result output.

[0120] The embodiment of the present invention further provides a projection vehicle lamp lighting device based on image semantic segmentation target recognition, which includes a parameter calibration unit, a target instance information processing unit, a light intensity calculation unit and a projection lighting scheme generation unit, wherein:

[0121] The parameter calibration unit is used for obtaining calibration parameters of the camera and projection lamp on each side of the vehicle body.

[0122] The target instance information processing unit is used to obtain the structured data of the current image frame through the semantic segmentation model according to the image frame acquired by the camera in real time. The structured data includes the ID corresponding to each target instance and the semantic classification information and contour point sets corresponding to different setting levels, and then obtain the relevant information of each target instance. The target instance related information includes the pixel coordinate distance and pixel-level two-dimensional speed between the target instance and the vehicle, and fills the relevant information of the target instance into the structured data of each corresponding target instance.

[0123] The illumination calculation unit is used to calculate the illumination of each target instance.

[0124] The projection lighting scheme generating unit is used to convert the contour of the target instance from the image coordinate system of the current image frame to the projection headlight coordinate system through the calibration parameters of the parameter calibration unit, and according to the corresponding illumination attributes calculated in step 3, on the basis of the structured data of the target instance obtained by the target instance information processing unit, perform pixel-level brightness filling on the closed area formed by the contour of each target instance, and then generate a projection bitmap, which serves as the projection lighting scheme of the projection headlight array.

[0125] In one embodiment, the target instance information processing unit specifically includes a semantic segmentation model, a contour point update model and a structured data output model, wherein:

[0126] The semantic segmentation model is used to extract the pixel-level semantic contours and classification of the current image frame based on the image frame acquired by the camera in real time, and output a set of contour points, which constitute the contour.

[0127] The contour point update model is used to iteratively calculate multiple consecutive contour points through a loop to obtain a vector composed of two adjacent contour points, and to determine whether the adjacent first vector and second vector satisfy the vector collinearity theorem. If so, the contour points shared by the first vector and the second vector are deleted and the contour point set is updated.

[0128] The structured data output model is used to output structured data based on the contour point set updated by the contour point update model, combined with the classification information corresponding to the contour and the ID of the target instance.

[0129] In one embodiment, the illumination calculation unit specifically includes a pixel point attribute determination model and a contour point illumination calculation module, wherein:

[0130] The pixel attribute determination model is used to determine the category to which the pixel point (i, j) of the projection headlight belongs according to the classification information corresponding to the contour of the target instance;

[0131] The contour point illumination calculation module is used to combine the category to which the pixel point (i, j) of the projection headlight belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, and calculate the contour point (x i ,y j ) illuminance g(x i ,y j ).

[0132] In one embodiment, the illumination calculation unit further includes an illumination modification module, which is used to use the following morphological technology to perform expansion compensation to suppress the illumination value of the expansion area of ​​the key target, and the direction and magnitude of the expansion refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively:

[0133] ①According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system;

[0134] ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, the contour point p (x,y) Replace with the point to be expanded q (x,y) ;

[0135] ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ①.

[0136] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A projection vehicle lighting method based on image semantic segmentation and target recognition, characterized in that: include: Step 1: A camera and a projection lamp are set on each side of the vehicle body, and calibration parameters of the camera and the projection lamp on each side are obtained; Step 2: According to the image frame acquired by the camera in real time, the structured data of the current image frame is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each target instance and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each target instance is obtained. The relevant information of the target instance includes the pixel coordinate distance between the target instance and the vehicle and the pixel-level two-dimensional speed. The relevant information of the target instance is filled into the structured data of each corresponding target instance; Step 3, calculating the illumination of each target instance; Step 4, through the calibration parameters of step 1, the contour of the target instance is converted from the image coordinate system of the current image frame to the projection headlight coordinate system, and according to the corresponding illuminance calculated in step 3, based on the structured data of the target instance obtained in step 2, the closed area formed by the contour of each target instance is filled with pixel-level brightness, and then a projection bitmap is generated, which is used as the projection lighting solution for the projection headlight array.

2. The projection vehicle lighting method based on image semantic segmentation target recognition according to claim 1, characterized in that: The method of "obtaining the contour of each target instance in the current image frame and the classification information corresponding to the contour through the semantic segmentation model" in step 2 specifically includes: Step 21a, according to the image frame acquired by the camera in real time, extract the pixel-level semantic contour and classification of the current image frame through the semantic segmentation model, output the contour point set, and the contour point set constitutes the contour; Step 22a, iteratively calculating a plurality of continuous contour points in a loop to obtain a vector formed by two adjacent contour points, and determining whether the adjacent first vector and the second vector satisfy the vector collinearity theorem. If so, deleting the contour points shared by the first vector and the second vector, and updating the contour point set; Step 23a, outputting structured data based on the contour point set updated in step 22a, combined with the classification information corresponding to the contour and the ID of the target instance.

3. The projection vehicle lamp lighting method based on image semantic segmentation target recognition according to claim 1 or 2, characterized in that: Step 3 specifically includes: According to the classification information corresponding to the contour of the target instance, determine the category to which the pixel point (i, j) of the projection headlight belongs; Combined with the category to which the pixel point (i, j) of the projected headlight belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, the contour point (x i ,y j ) illuminance g(x i ,y j ): Where i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively; the value of i is not less than 0 and not greater than the width of the current image frame; the value of j is not less than 0 and not greater than the height of the current image frame; α is the maximum intensity of the illumination of the projected vehicle light; δ is an attenuation coefficient based on the pixel coordinate distance d between the target instance and the vehicle, which is described by formula (2); θ is a measured constant attenuation coefficient within a limited distance, D max The maximum distance to enable the projection light to project illumination onto the subject of the target instance:

4. The projection vehicle lighting method based on image semantic segmentation target recognition as claimed in claim 3, characterized in that: Step 3 specifically also includes: The following morphological techniques are used for expansion compensation to suppress the illumination value of the expansion area of ​​the key target. The direction and magnitude of the expansion refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively: ①According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system; ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, the contour point p (x,y) Replace with the point to be expanded q (x,y) ; ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ①.

5. The projection vehicle lamp lighting method based on image semantic segmentation target recognition according to claim 1 or 2, characterized in that: The method for obtaining the "semantic segmentation model" in step 2 includes: Step 21b, sample data collection and labeling, specifically includes: Step 211, combing the scene, outputting the core semantic elements of the driving traffic scene, as shown in the following table; Step 212, according to the road data collection action of driving the vehicle, output the collected and discretely extracted image set; Step 213, annotating the image acquired in step 212, and outputting an image dataset with semantically annotated true values; Step 22b, using the data set annotated in step 21b, perform transfer learning training and evaluation on the selected model to obtain a semantic segmentation model for projection headlights; Step 23b, algorithm post-processing and result output.

6. The projection vehicle lighting method based on image semantic segmentation target recognition as claimed in claim 2, characterized in that: The imaging surface of the camera on each side is parallel to the projection light source surface of the projection lamp, and the camera and the lateral side of the projection lamp are closely fitted. The lateral distance between the optical center of the camera and the center point of the projection light source surface of the projection lamp is close to the sum of the lateral widths of the two, so as to maintain a relatively close viewing angle and stable projection transformation characteristics. The camera and projection lamp on each side are configured as an integrated module.

7. A projection vehicle lamp lighting device based on image semantic segmentation target recognition, characterized in that: include: A parameter calibration unit, which is used for obtaining calibration parameters of the camera and projection lamp on each side of the vehicle body; A target instance information processing unit is used to obtain structured data of the current image frame through a semantic segmentation model according to the image frame acquired by the camera in real time, wherein the structured data includes an ID corresponding to each target instance and semantic classification information and contour point sets corresponding to different setting levels, and then obtain relevant information of each target instance, wherein the relevant information of the target instance includes the pixel coordinate distance between the target instance and the vehicle and the pixel-level two-dimensional speed, and fill the relevant information of the target instance into the structured data of each corresponding target instance; An illumination calculation unit, which is used to calculate the illumination of each target instance; The projection lighting scheme generating unit is used to convert the contour of the target instance from the image coordinate system of the current image frame to the projection headlight coordinate system through the calibration parameters of the parameter calibration unit, and according to the corresponding illumination attribute calculated in step 3, on the basis of the structured data of the target instance obtained by the target instance information processing unit, perform pixel-level brightness filling on the closed area formed by the contour of each target instance, and then generate a projection bitmap, which serves as the projection lighting scheme of the projection headlight array.

8. The projection vehicle lamp lighting device based on image semantic segmentation target recognition according to claim 7, characterized in that: The target instance information processing unit specifically includes: The semantic segmentation model is used to extract the pixel-level semantic contour and classification of the current image frame according to the image frame acquired by the camera in real time, and output a set of contour points, which constitute the contour; A contour point update model is used to iteratively calculate a plurality of continuous contour points through a loop, obtain a vector formed by two adjacent contour points, and determine whether the adjacent first vector and second vector satisfy the vector collinearity theorem. If so, the contour points shared by the first vector and the second vector are deleted, and the contour point set is updated; The structured data output model is used to output structured data according to the contour point set updated by the contour point update model, combined with the classification information corresponding to the contour and the ID of the target instance.

9. The projection vehicle lamp lighting device based on image semantic segmentation target recognition according to claim 7 or 8, characterized in that: The illumination calculation unit specifically includes: A pixel point attribute determination model is used to determine the category to which the pixel point (i, j) of the projection headlight belongs according to the classification information corresponding to the contour of the target instance; The contour point illumination calculation module is used to combine the category to which the pixel point (i, j) of the projection headlight belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, and calculate the contour point (x i ,y j ) illuminance g(x i ,y j ): Where i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively; the value of i is not less than 0 and not greater than the width of the current image frame; the value of j is not less than 0 and not greater than the height of the current image frame; α is the maximum intensity of the illumination of the projected vehicle light; δ is an attenuation coefficient based on the pixel coordinate distance d between the target instance and the vehicle, which is described by formula (2); θ is a measured constant attenuation coefficient within a limited distance, D max The maximum distance to enable the projection light to project illumination onto the subject of the target instance:

10. The projection vehicle lamp lighting device based on image semantic segmentation target recognition according to claim 9, characterized in that: The illumination calculation unit specifically includes: The illumination modification module is used to use the following morphological techniques to perform expansion compensation and suppress the illumination value of the expansion area of ​​the key target. The direction and magnitude of the expansion refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively: ①According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system; ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, the contour point p (x,y) Replace with the point to be expanded q (x,y) ; ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ①.

Images