Projection car light illumination method and device based on image semantic segmentation target recognition

Abstract

The application discloses a projection car lamp lighting method and device based on image semantic segmentation target recognition, which comprises the following steps: step 1, obtaining the calibration parameters of each side camera and the projection car lamp; step 2, obtaining the structured data of the current image frame through a semantic segmentation model, obtaining the related information of each target instance, and filling the related information of the target instance into the structured data of each corresponding target instance; step 3, calculating the illumination of each target instance; step 4, converting the contour of the target instance from the image coordinate system of the current image frame to the projection car lamp coordinate system, performing pixel-level brightness filling on the closed area formed by the contour of each target instance according to the corresponding illumination and on the basis of the structured data of the target instance, and generating a projection bitmap, which is used as the projection car lamp array projection lighting scheme. The application can improve the effect of realizing blind filling and anti-dazzling of the vehicle lighting system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensor perception recognition, in particular to a projection vehicle lamp lighting method and device based on image semantic segmentation target recognition. BACKGROUND

[0002] With the rapid development of semiconductor technology and artificial intelligence technology, the hardware from the sensor end to the execution control end and the software system of the auxiliary driving function of the vehicle are upgraded, and at the same time, the driving / riding safety and the man-machine interaction friendliness are greatly improved. The vehicle light system is also evolving towards intelligence. The main light control system on the market currently includes two parts of the sensor end and the lighting control execution end. Among them, different types and performance optical sensors are used in the sensor end to capture the surrounding environment information, which is extracted, recognized, and then transmitted to the downstream execution end. The lighting control execution end uses traditional high / low beam systems or dot array lighting to realize different granularities for blind compensation, anti-dazzling and other response actions.

[0003] At present, the mainstream sensor perception recognition technology mainly completes the position and classification recognition of the target level, and the recognition resolution and feature classification of the target are limited, and the recognition stability is limited in the complex multi-classification and occlusion (truncation) of the actual traffic flow scene. In terms of lighting execution control, the traditional high / low beam and dot array lighting system basically uses longitudinal lighting control or transverse lighting control, which has insufficient resolution and cannot meet the lighting control strategy for individual targets.

[0004] In a control technology of an intelligent high beam disclosed by the prior art, a target image is acquired, a three-dimensional detection frame of an obstacle in the target image is recognized, a glare prevention angle range corresponding to the target image is determined according to the three-dimensional detection frame, and the brightness of the intelligent high beam is adjusted according to the glare prevention angle range. However, this technical solution still has many deficiencies, such as the following points:

[0005] (1) In this technical solution, the recognition information of the target obstacle is a three-dimensional detection frame, and different obstacle features are not identified in detail, such as the body trunk of a pedestrian, the vehicle body of a vehicle, the windshield, the rearview mirror and other regions. The target three-dimensional detection frame is converted to the projection vehicle lamp view point pointing range through projection transformation, and the accuracy is insufficient, which causes the projection mode not to fit the target object, and the occlusion obstacle behind the projection area is missed due to insufficient lighting. The blind area caused thereby poses a great risk to the driving decision of the ego vehicle.

[0006] (2) The technical scheme limits the adjustment of the high beam control mode for specific targets to only the angle range, horizontal left and right boundaries, and vertical up and down boundaries in four dimensions, resulting in insufficient control resolution, inability to adapt to irregular target profiles (e.g., parallel to the vertical or horizontal direction), and the existence of glare or blind area cases of over-illumination, increasing the risk of driving safety.

[0007] (3) The technical scheme lacks consideration of the speed, direction, and other moving characteristics of the vehicle itself and other different categories of dynamic traffic participants, lacks scene target tracking and prediction capabilities, and the dynamic response of the projection car light to the illumination control of targets with a large relative motion offset from the vehicle is insufficient, which can cause light blocking or offset jitter in the blind area of the target, and may cause the opposite target driver's vision to be affected by light flickering, which can affect the safety of the opposite driving. SUMMARY

[0008] The purpose of the present application is to provide a method and device for controlling the illumination of a projection car light based on image semantic segmentation for target recognition, which can improve the effect of the vehicle lighting system in achieving blind area compensation and anti-glare.

[0009] To achieve the above-mentioned purpose, the present application provides a projection car light illumination method based on image semantic segmentation target recognition, which comprises:

[0010] Step 1: A camera and a projection car light are arranged on each side of the vehicle body, and the calibration parameters of each side camera and projection car light are obtained;

[0011] Step 2: According to the image frame obtained by the camera in real time, the structured data of the current image frame is obtained through a semantic segmentation model, the structured data includes the ID corresponding to each target instance, the semantic classification information corresponding to different setting levels, and the contour point set, the contour point set constitutes a contour, and then the related information of each target instance is obtained, the related information of the target instance includes the pixel coordinate distance and the pixel level two-dimensional speed of the target instance and the vehicle, and the related information of the target instance is filled into the structured data of each corresponding target instance;

[0012] Step 3: Calculate the illumination of each target instance;

[0013] Step 4: According to the calibration parameters of step 1, the contour of the target instance is converted from the image coordinate system of the current image frame to the coordinate system of the projection car light, and the closed area formed by the contour of each target instance is filled with pixel-level brightness based on the structured data of the target instance obtained in step 2 according to the corresponding illumination calculated in step 3, and then a projection bitmap is generated, which is used as a projection illumination scheme for the projection car light array.

[0014] Further, the method of "obtaining the contour of each target instance of the current image frame and the classification information corresponding to the contour by the semantic segmentation model" in step 2 specifically comprises:

[0015] Step 21a, according to the image frame acquired by the camera in real time, the pixel-level semantic contour and classification of the current image frame are extracted by the semantic segmentation model, and a contour point set is outputted, which constitutes a contour;

[0016] Step 22a, the adjacent two contour points are obtained by iterative calculation through a loop, and it is judged whether the adjacent first vector and second vector satisfy the vector collinearity theorem, if yes, the contour point shared by the first vector and the second vector is deleted, and the contour point set is updated;

[0017] Step 23a, according to the contour point set updated in step 22a, the structured data is outputted by combining the classification information corresponding to the contour and the ID of the target instance.

[0018] Further, step 3 specifically comprises:

[0019] According to the classification information corresponding to the contour of the target instance, the classification to which the pixel point (i, j) of the projection car lamp belongs is determined;

[0020] According to the classification to which the pixel point (i, j) of the projection car lamp belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the ego vehicle, the illumination g(x i ,y j ) of each contour point (x max ,y (x,y) ) of the target instance is calculated according to formula (1):

[0021]

[0022] In the formula, i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively, the value of i is not less than 0 and not greater than the width of the current image frame, and the value of j is not less than 0 and not greater than the height of the current image frame; α is the maximum intensity of the projection car lamp illumination; δ is a decay coefficient according to the pixel coordinate distance d between the target instance and the ego vehicle, which is described as formula (2); θ is a measured constant decay coefficient within a limited distance, D (x,y) is the farthest distance of the projection car lamp to the main body of the target instance:

[0023]

[0024] Further, step 3 specifically further comprises:

[0025] The morphological technique is used for inflation compensation, and the illumination value of the key target inflation area is suppressed, and the direction and size of the inflation are respectively referred to the vector direction and numerical value of the pixel-level two-dimensional speed of the target instance:

[0026] ① According to the selected target instance contour point p (x,y) And the pixel-level two-dimensional speed, calculate the to-be-inflated point q (x,y) Along the horizontal and vertical directions of the pixel-level two-dimensional speed of the target instance;

[0027] ②Judge whether the to-be-inflated point q (x,y) Is in the contour of the selected target instance in the current image frame, if yes, go to step ③, otherwise replace the contour point p (x,y) With the to-be-inflated point q (x,y) ;

[0028] ③Judge whether all contour points of the selected target instance are polled once, and end if the polling is completed, otherwise jump to ①.

[0029] Further, the method for obtaining the "semantic segmentation model" in step 2 comprises:

[0030] Step 21b, sample data acquisition and labeling, specifically comprising:

[0031] Step 211, scene sorting, outputting the core semantic elements of the driving traffic scene, as shown in the following table;

[0032]

[0033] Step 212, according to the driving vehicle, the road collection data action is outputted, and the collected and discrete extracted image set is outputted;

[0034] Step 213, labeling the target of the image collected in step 212, and outputting the image dataset with semantic annotation true value;

[0035] Step 22b, using the dataset labeled in step 21b, performing transfer learning training and evaluation on the selected model to obtain the projection car lamp semantic segmentation model;

[0036] Step 23b, algorithm post-processing and result output.

[0037] Further, the imaging surface of each side camera is parallel to the projection light source surface of the projection car lamp, and the camera and the projection car lamp are closely attached to the lateral side, the optical center of the camera and the center point of the projection light source surface of the projection car lamp are transversely close to the sum of the transverse widths of the two, so as to maintain the relatively close visual angle and stable projection transformation characteristics of the two, and each side camera and projection car lamp are set as an integrated module.

[0038] The application also provides a projection vehicle lamp lighting device based on image semantic segmentation target recognition, which comprises:

[0039] A parameter calibration unit is configured to set a camera and a projection vehicle lamp on each side of a vehicle body, and acquire calibration parameters of the camera and the projection vehicle lamp on each side;

[0040] A target instance information processing unit is configured to acquire structured data of a current image frame by a semantic segmentation model according to an image frame acquired in real time by the camera, the structured data comprising an ID corresponding to each target instance, semantic classification information corresponding to different setting levels and a contour point set, and further acquire target instance related information, the target instance related information comprising a pixel coordinate distance of the target instance from the vehicle and a pixel level two-dimensional velocity, and fill the target instance related information into the structured data of each corresponding target instance;

[0041] An illumination intensity calculation unit is configured to calculate the illumination intensity of each target instance;

[0042] A projection lighting scheme generation unit is configured to convert the contour of the target instance from an image coordinate system of the current image frame to a projection vehicle lamp coordinate system according to the calibration parameters of the parameter calibration unit, perform pixel level brightness filling on a closed region formed by the contour of each target instance according to the corresponding illumination intensity attribute calculated in step 3 and the structured data of the target instance obtained by the target instance information processing unit, and further generate a projection bitmap, which is used as a projection vehicle lamp array projection lighting scheme.

[0043] Further, the target instance information processing unit specifically comprises:

[0044] A semantic segmentation model is configured to extract pixel level semantic contours and classifications of a current image frame according to an image frame acquired in real time by the camera, and output a contour point set, the contour point set forming a contour;

[0045] A contour point updating model is configured to iteratively calculate a plurality of contour points by a loop, obtain a vector formed by two adjacent contour points, and determine whether the first vector and the second vector meet a vector collinearity theorem, if yes, delete a contour point shared by the first vector and the second vector, and update the contour point set;

[0046] A structured data output model is configured to output structured data according to the contour point set updated by the contour point updating model, in combination with the classification information corresponding to the contour and the ID of the target instance.

[0047] Further, the illumination intensity calculation unit specifically comprises:

[0048] A pixel point attribute determination model is configured to determine a classification to which a pixel point (i, j) of the projection vehicle lamp belongs according to the classification information corresponding to the contour of the target instance.

[0049] a contour point illumination calculation module for calculating the illumination g(x i ,y j ) of each contour point (x i ,y j ) of the target instance according to formula (1) in combination with the classification to which the pixel point (i,j) of the projected car light belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the ego vehicle:

[0050]

[0051] wherein i and j are the horizontal and vertical pixel coordinates of the pixel point (i,j), respectively, the value of i is not less than 0 and not greater than the width of the current image frame, the value of j is not less than 0 and not greater than the height of the current image frame, α is the maximum intensity of the projected car light illumination, δ is a decay coefficient according to the pixel coordinate distance d between the target instance and the ego vehicle, which is described as formula (2), θ is a measured constant decay coefficient within a limited distance, D max is the farthest distance at which the projected car light can project the main illumination of the target instance:

[0052]

[0053] Further, the illumination calculation unit specifically further comprises:

[0054] an illumination modification module for performing inflation compensation by using the following morphological technique to suppress the illumination value of the key target inflation area, and the direction and size of inflation are respectively referred to the vector direction and value of the pixel-level two-dimensional velocity of the target instance:

[0055] ① according to the contour point p (x,y) and the pixel-level two-dimensional velocity of the selected target instance, calculating the to-be-inflated point q (x,y) in the horizontal and vertical directions of the pixel-level two-dimensional velocity of the target instance; wherein (x,y) is the position coordinate of the contour point in the image coordinate system;

[0056] ② judging whether the to-be-inflated point q (x,y) is within the contour of the selected target instance in the current image frame, if yes, proceeding to step ③, otherwise replacing the contour point p (x,y) with the to-be-inflated point q (x,y) ;

[0057] ③ judging whether all contour points of the selected target instance have been polled once, if not, jumping to step ①.

[0058] The present application mainly optimizes and solves the following three aspects:

[0059] 1. In view of the problem of inaccurate target area in traditional detection methods under anti-dazzling scenes, the image semantic segmentation model method of machine learning is adopted to obtain the pixel-level contour information of the target, and the contour boundary of the projection headlight view point is converted through perspective projection transformation, so as to provide fine and structured semantic information for the rear-end projection headlight scene application.

[0060] 2. In view of the problem of insufficient control resolution of high beam and low beam systems and dot matrix lighting systems, the high-resolution area array projection lighting device is adopted to realize high-precision lighting real-time system through pixel-level projection point brightness and chroma control, and a high real-time data transmission channel is established with the detection system to realize the closed loop of sensing and control.

[0061] 3. In view of the problem of relative space-time jitter caused by the moving characteristics of the vehicle itself and other dynamic traffic participants, the semantic segmentation and morphological technique method are combined, the tracking and prediction mode is adopted to grade and smooth the detection target boundary illumination, and the application effect of the rear-end anti-dazzling and blind area is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 It is a schematic diagram of a one-sided projection headlight and camera integrated module of an embodiment of the present application.

[0063] Figure 2 It is a schematic diagram of the arrangement of a one-sided projection headlight and camera integrated module on a vehicle body of an embodiment of the present application.

[0064] Figure 3 It is a schematic diagram of a projection headlight of an embodiment of the present application.

[0065] Figure 4 It is a schematic diagram of a sparse contour point of an embodiment of the present application.

[0066] Figure 5 a, b and c in the figure are schematic diagrams of the projection headlight blind area compensation and anti-dazzling lighting effect obtained after the morphological technique is used for expansion compensation in an embodiment of the present application.

[0067] Figure 6 It is a schematic diagram of the process of sample data collection and labeling of an embodiment of the present application.

[0068] Figure 7 It is a schematic diagram of traffic semantic elements of a driving scene of an embodiment of the present application.

[0069] Figure 8 It is a schematic diagram of the process of iterative training and evaluation of a projection headlight semantic segmentation model of an embodiment of the present application. DETAILED DESCRIPTION

[0070] In the drawings, the same or similar notations used in the same or similar elements or elements having the same or similar functions. The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0071] In the description of the present application, the terms "center", "longitudinal", "transverse", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore cannot be understood as a limitation on the scope of protection of the present application.

[0072] The present application relates to a projection car lamp lighting method and system based on image semantic segmentation target recognition, which combines the high-resolution pixel-level target instance segmentation capability of the semantic segmentation model and the high-precision fine illumination control provided by the dense area array light beam, enhances the scene response capability of the vehicle lighting system, and improves the safety of the auxiliary driving scene.

[0073] The projection car lamp lighting method based on image semantic segmentation target recognition provided by the embodiments of the present application comprises:

[0074] Step 1, a camera and a projection car lamp are arranged on each side of the vehicle body, and the calibration parameters of the camera and the projection car lamp on each side are obtained.

[0075] Among them, the camera and the projection car lamp arranged on each side of the vehicle body can be arranged in the following manner, for example: Figure 1 and Figure 2 As shown in the drawings, the imaging surface of the camera on each side is parallel to the projection light source surface of the projection car lamp, and the camera and the lateral side of the projection car lamp are closely attached, the transverse distance between the optical center of the camera and the center point of the projection light source surface of the projection car lamp is close to the sum of the transverse widths of the two, so as to maintain the close visual angle and stable projection transformation characteristics of the two. As shown in the drawings, the camera and the projection car lamp on each side are preferably arranged as an integrated module, of course, separate arrangement can also achieve the purpose of the present application. Figure 1

[0076] The calibration parameters of the camera include the intrinsic parameters, extrinsic parameters and distortion parameters of the camera, and the calibration parameters of the projection car lamp include the extrinsic parameters and offset coordinates of the projection car lamp. Among them, the method for obtaining the extrinsic parameters of the camera and the extrinsic parameters and offset coordinates of the projection car lamp comprises:

[0077] Step 11, calibrate the intrinsic parameters of the camera. For example, a commonly used checkerboard calibration board is used, the camera captures the image of the calibration board, a series of different angle images of the camera to the calibration board are obtained, and Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters and distortion parameters of the camera.

[0078] ​Step 12, calibrate the camera and the projection car light external parameters. For example, as shown in Figure 3 , the vehicle equipped with the projection car light on both sides is parked on the ground plane, and the four corner points of the vehicle are projected onto the ground plane to obtain the L-shaped projection corner points as shown in Figure 3 . The obtained image coordinates of the projection corner points are processed by using the internal parameters and distortion parameters of the calibrated camera to obtain the pixel coordinates of the image coordinate system, and the coordinate difference between the projection corner points and the pixel coordinates is calculated as the offset coordinates. Finally, the offset coordinates are checked by transforming multiple sets of projection corner points to verify whether the offset coordinates meet the expected value.

[0079] Step 2, according to the image frame obtained by the camera in real time, the structured data of the current image frame is obtained by the semantic segmentation model, which includes the ID corresponding to each target instance, the semantic classification information corresponding to different setting levels, and the contour point set. The contour point set constitutes a contour, and then the relevant information of each target instance is obtained, including the pixel coordinate distance between the target instance and the ego vehicle and the pixel-level two-dimensional speed. The relevant information of the target instance is filled into the structured data of each corresponding target instance. The ID corresponding to the target instance can be used as the target instance ID in the continuous buffer multi-frame contour cache queue according to the pixel coordinate distance between the target instance and the ego vehicle and the intersection-over-union value of the area of the contour. The contour point set refers to the position coordinate set of the pixel points in the image coordinate system. The pixel coordinate distance between the target instance and the ego vehicle can be obtained by calculating the pixel difference between the pixel coordinates of the centroid of the contour of each target instance and the midpoint of the lower boundary of the current image frame in the horizontal x-axis and the vertical y-axis, respectively. The pixel-level two-dimensional speed information can be obtained by using the state estimation method according to the pixel coordinate distance between each target instance and the ego vehicle.

[0080] In one embodiment, as shown in Figure 4 , the method of "obtaining the contour of each target instance and the classification information corresponding to the contour in the current image frame by the semantic segmentation model" in step 2 specifically includes:

[0081] Step 21a, according to the image frame obtained by the camera in real time, the pixel-level semantic contour and classification of the current image frame are extracted by the semantic segmentation model, and the contour point set is outputted, which constitutes a contour.

[0082] Step 22a, the adjacent two contour points are obtained by iteratively calculating a plurality of contour points in a loop, and it is determined whether the adjacent first vector and second vector satisfy the vector collinearity theorem. If yes, the contour point shared by the first vector and the second vector is deleted, and the contour point set is updated. In this way, the contour points are thinned out, and the number of boundary points of the contour edge in the image coordinate system is controlled.

[0083] Step 23a, according to the updated contour point set of step 22a, combining the classification information corresponding to the contour and the ID of the target instance, output structured data. As shown in Table 1, Table 1 is a list of pre-defined data structures, filling the contour information obtained in step 32, classification information, as a single-frame multi-instance semantic segmentation result data for storage and use, the data structure list is shown in Table 1.

[0084] Table 1

[0085]

[0086] Step 3, calculate the illumination of each target instance.

[0087] In one embodiment, step 3 can use the commonly used method and the scheme of shading the foreground (person, vehicle) and supplementing the light of the background (road), or the following method can be used to realize:

[0088] According to the classification information corresponding to the contour of the target instance, determine the classification to which the pixel point (i, j) of the projected car light belongs;

[0089] According to the classification to which the pixel point (i, j) of the projected car light belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the ego vehicle, calculate the illumination g(x i ,y j ) of each contour point (x i ,y j ) of the target instance according to formula (1):

[0090]

[0091] In the formula, i and j are the horizontal and vertical pixel coordinates of the pixel point (i, j), respectively, the value of i is not less than 0 and not greater than the width of the current image frame, and the value of j is not less than 0 and not greater than the height of the current image frame; alpha is the maximum intensity of the projected car light illumination; delta is a decay coefficient according to the pixel coordinate distance d between the target instance and the ego vehicle, which is described as formula (2); theta is a measured constant decay coefficient within a limited distance, such as 80% in a complex urban road traffic flow, to limit the influence on other traffic participants, and 90% in a simple expressway traffic flow, D max In order to enable the farthest distance of the main body projection illumination of the projected car light to the target instance:

[0092]

[0093] In one embodiment, as shown in a, b, and c in Figure 5 , step 3 further includes:

[0094] The following morphological techniques are used for dilation compensation to suppress the illumination value of the key target dilation area. The direction and magnitude of the dilation refer to the vector direction and value of the pixel-level two-dimensional velocity of the target instance respectively:

[0095] ① According to the contour point p of the selected target instance (x,y) and pixel-level two-dimensional velocity, calculate the horizontal and vertical expansion points q along the pixel-level two-dimensional velocity of the target instance (x,y) ; Where (x, y) is the position coordinate of the contour point in the image coordinate system;

[0096] ② Determine the expansion point q (x,y) Is it within the contour of the selected target instance in the current image frame? If so, proceed to step ③, otherwise, set the contour point p (x,y) Replace with the point to be expanded q (x,y) ;

[0097] ③ Determine whether all contour points of the selected target instance have been polled once. If polling is completed, jump to ① otherwise.

[0098] For target instances with speed, where the correct instance contours are not included due to inaccurate semantic segmentation boundaries, and where projection lighting schemes lag due to factors such as system link delay, morphological technology is used for expansion compensation to suppress the illumination value of the key target expansion area. The direction of expansion refers to the two-dimensional velocity vector direction of the current instance, and the expansion depth refers to the two-dimensional velocity scalar value.

[0099] In step 4, the target instance's outline is transformed from the image coordinate system of the current image frame to the projection headlight coordinate system using the calibration parameters from step 1. Based on the structured data of the target instance obtained in step 2, the enclosed area formed by the outline of each target instance is filled with pixel-level brightness according to the corresponding illumination properties calculated in step 3. This generates a projection bitmap, which serves as the projection lighting solution for the projection headlight array. The instance outline in the camera coordinate system is projected and transformed using the calibration parameters into the outline coordinates of the projection headlight coordinate system. The enclosed area formed by each instance outline is filled with pixel-level brightness according to the corresponding illumination properties. This generates a projection bitmap, which is transmitted to the projection lighting execution device for projection.

[0100] In one embodiment, the method for obtaining the “semantic segmentation model” in step 2 includes:

[0101] Step 21b, such as Figure 6 As shown in the figure, sample data collection and annotation specifically include:

[0102] Step 211, scene sorting, outputting the core semantic elements of the traffic scene, as shown in Table 1 below; Figure 7As shown, the traffic semantic elements for the driving scene mainly include roads and lighting environments, traffic participants, and traffic signs. Among them, the roads mainly cover urban and rural roads, expressways, and cover open road sections and tunnels. The traffic participants are classified into motor vehicles, pedestrians, and non-motor vehicles. The traffic signs include ground signs and indicating signs.

[0103] In addition, for the projection car lamp lighting scene application, more detailed semantic segmentation annotation is required for the traffic participants. For pedestrians and non-closed passenger cabin vehicles such as non-motor vehicles, motorcycles, and open-top motor vehicles, the torso and head need to be further separately annotated. For closed passenger cabin motor vehicles, the windshield, rearview mirror area, and secondary semantic classification need to be annotated and circled. The semantic classification can be evaluated and increased according to the actual scene situation and needs, including but not limited to those described in Table 2.

[0104] Table 2

[0105]

[0106] In the table, “-” means “none”, so “-” in the primary semantic classification and the secondary semantic classification of “road and lighting environment” means that the corresponding primary and secondary semantic classification is not made. “-” in the secondary semantic classification of “traffic signs” means that the corresponding secondary semantic classification is not made.

[0107] Step 212, according to the road collection data action of the driving vehicle, output the collected and discretely extracted image set. For example, using a vehicle equipped with a camera, respectively planning and selecting the area to drive in accordance with the sorted scene, collecting image data and storing during driving, and ensuring balanced coverage of different traffic scenes. The continuous image set completed by the collection is obtained by the equal interval frame extraction method to obtain the discrete data set, and the similar images in the discrete set are removed by manual checking to reduce and optimize the quality of the representative of the to-be-labeled data set. The size of the final screened data set is maintained at, for example, 1000 or more. At this time, the total number of labeling targets is classified into 1000 times the size of the number of categories, such as the number of labeling instances in this example, which includes 6 categories of primary classification and 4 categories of secondary fine classification, a total of 10 categories.

[0108] Step 213, label the targets of the images collected in step 212, and output the image data set with semantic label true value. For example: the data set is labeled with semantic element contour for each frame of image, and after labeling, the labeling is checked by a visualization tool, and the boundary coincidence degree is checked, as well as whether the small target, occlusion or truncated target is missed, the label is correctly matched, and the like. Among them, the boundary coincidence degree can be but not limited to using the average error of boundary pixels.

[0109] Step 22b, using the data set labeled in step 21b, the selected model is trained and evaluated to obtain the projection headlight semantic segmentation model.

[0110] As shown in Figure 8 Step 22b specifically includes the following four links:

[0111] Step 221b, model framework selection.

[0112] Integrating model performance indicators with deployment platform computing power and real-time demand, in the existing industry widely used model implementation, referring to the traffic neighborhood public data set benchmark subdivision evaluation, select the appropriate model framework, such as U-Net, etc.

[0113] Step 222b, model pre-training.

[0114] The model framework relies on the feature learning effect of the specific field on the large-scale data set feature training, and the actual project private data size is difficult to meet the internalization of the model for generalization features, and the model needs to be pre-trained, using public large-scale traffic data sets, such as Cityscapes, KITTI, etc. The pre-trained model is evaluated by the test set.

[0115] Step 223b, model transfer learning.

[0116] The pre-trained model is trained and features are migrated on the labeled sample data set. The work needed before training includes three aspects of feature semantic label adaptation (10 labels), sample set division, and training hyperparameter configuration (iteration number, learning rate, etc.). Among them, the sample set is divided according to the training set, validation set, test set according to the ratio of 7:2:1 or similar. Complete the above parameter configuration to start model migration training, and monitor whether the iterative change convergence trend of the core indicators of the model meets the expectation. If not, stop and locate the problem scene, supplement the scene data through step 1, or optimize the parameter configuration in this step to restart the training. This field is mature, and through iteration, it can be completed. And if there are elements of the scene that do not meet the requirements in the test verification process, the scene image can be supplemented through step 1, and steps 1 and 2 can be completed to achieve the expected effect.

[0117] Step 224b, sample test and index evaluation.

[0118] The model completed the migration training is tested and evaluated on the divided test set, attention is paid to the accuracy set recall rate index, and whether the model generalization requirement is met is checked by sampling and visualizing the sample and the labeled model prediction result, when the proportion of the samples or scenes found by the sample check that do not meet the expectation exceeds the expectation, the original supplementary labeling is needed for the scenes, and the migration learning training is re-performed to improve the model effect.

[0119] Step 23b, algorithm post-processing and result output.

[0120] The embodiment of the application also provides a projection car lamp lighting device based on image semantic segmentation target recognition, which comprises a parameter calibration unit, a target instance information processing unit, an illumination calculation unit and a projection lighting scheme generation unit, wherein:

[0121] The parameter calibration unit is used for setting a camera and a projection car lamp on each side of a vehicle body, and acquiring calibration parameters of the camera and the projection car lamp on each side.

[0122] The target instance information processing unit is used for acquiring structured data of a current image frame through a semantic segmentation model according to an image frame acquired by the camera in real time, the structured data comprising an ID corresponding to each target instance, semantic classification information corresponding to different setting levels and a contour point set, and further acquiring target instance related information, the target instance related information comprising a pixel coordinate distance of the target instance from the vehicle and a pixel level two-dimensional speed, and filling the target instance related information into the structured data of each corresponding target instance.

[0123] The illumination calculation unit is used for calculating the illumination of each target instance.

[0124] The projection lighting scheme generation unit is used for converting the contour of the target instance from an image coordinate system of the current image frame to a projection car lamp coordinate system through the calibration parameters of the parameter calibration unit, performing pixel level brightness filling on a closed region formed by the contour of each target instance according to the corresponding illumination attribute calculated in step 3 and the structured data of the target instance obtained by the target instance information processing unit, and further generating a projection bitmap, the bitmap serving as a projection car lamp array projection lighting scheme.

[0125] In one embodiment, the target instance information processing unit specifically comprises a semantic segmentation model, a contour point updating model and a structured data output model, wherein:

[0126] The semantic segmentation model is used for extracting a pixel level semantic contour and classification of a current image frame according to an image frame acquired by the camera in real time, and outputting a contour point set, the contour point set forming a contour.

[0127] The contour point updating model is used to iteratively calculate a plurality of contour points in succession by looping, obtain a vector formed by two adjacent contour points, determine whether the first vector and the second vector adjacent to the first vector satisfy the vector collinearity theorem, and if so, delete the contour point shared by the first vector and the second vector, and update the contour point set.

[0128] The structured data output model is used to output structured data according to the contour point set updated by the contour point updating model, in combination with the classification information corresponding to the contour and the ID of the target instance.

[0129] In an embodiment, the illumination calculation unit specifically comprises a pixel point attribute determination model and a contour point illumination calculation module, wherein:

[0130] The pixel point attribute determination model is used to determine the classification to which the pixel point (i, j) of the projected car lamp belongs according to the classification information corresponding to the contour of the target instance.

[0131] The contour point illumination calculation module is used to calculate the illumination g(x i ,y j ) of each contour point (x i ,y j ) of the target instance according to formula (1) in combination with the classification to which the pixel point (i, j) of the projected car lamp belongs in the image coordinate system and the pixel coordinate distance d between the target instance and the ego vehicle.

[0132] In an embodiment, the illumination calculation unit specifically further comprises an illumination modification module, which is used to perform inflation compensation by adopting the following morphological technique to suppress the illumination value of the inflation area of the key target, and the direction and size of inflation refer to the vector direction and numerical value of the pixel-level two-dimensional velocity of the target instance respectively:

[0133] ① According to the selected contour point p (x,y) and the pixel-level two-dimensional velocity of the target instance, calculate the to-be-inflated point q (x,y) in the horizontal and vertical directions of the pixel-level two-dimensional velocity of the target instance; wherein (x, y) is the position coordinate of the contour point in the image coordinate system.

[0134] ② Determine whether the to-be-inflated point q (x,y) is within the contour of the selected target instance in the current image frame, if so, proceed to step ③, otherwise replace the contour point p (x,y) with the to-be-inflated point q (x,y) .

[0135] ③ Determine whether all contour points of the selected target instance have been polled once, if not, jump to step ①.

[0136] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Those skilled in the art should understand that the technical solutions described in the foregoing embodiments can be modified, or some technical features thereof can be replaced by equivalent ones; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A projection headlamp illumination method based on image semantic segmentation target recognition, characterized in that, The method comprises the following steps: Step 1, a camera and a projection car lamp are arranged on each side of the vehicle body, and the calibration parameters of the camera and the projection car lamp on each side are obtained; Step 2, according to the image frames obtained by the camera in real time, the structured data of the current image frame is obtained through a semantic segmentation model, the structured data comprising the ID corresponding to each target instance, the semantic classification information corresponding to different setting levels and the contour point set, the contour point set forming a contour, and the relevant information of each target instance is obtained, the relevant information of the target instance comprising the pixel coordinate distance between the target instance and the ego vehicle and the pixel-level two-dimensional velocity, and the relevant information of the target instance is filled into the structured data of each corresponding target instance; Step 3, the illuminance of each target instance is calculated; Step 4, according to the calibration parameters in step 1, the contour of the target instance is converted from the image coordinate system of the current image frame to the coordinate system of the projection car lamp, and the closed area formed by the contour of each target instance is filled with pixel-level brightness according to the corresponding illuminance calculated in step 3 and the structured data of the target instance obtained in step 2, and a projection bitmap is generated, which is used as a projection car lamp array projection lighting scheme; The method for obtaining the contour of each target instance and the classification information corresponding to the contour through the semantic segmentation model in step 2 comprises the following steps: Step 21a, according to the image frames obtained by the camera in real time, the pixel-level semantic contour and classification of the current image frame are extracted through the semantic segmentation model, and a contour point set is output, the contour point set forming a contour; Step 22a, a plurality of contour points are iteratively calculated through a loop to obtain a vector formed by two adjacent contour points, and it is determined whether the first vector and the second vector adjacent to each other satisfy the vector collinearity theorem, if yes, the contour point shared by the first vector and the second vector is deleted, and the contour point set is updated; Step 23a, according to the updated contour point set in step 22a, the structured data is output in combination with the classification information corresponding to the contour and the ID of the target instance; Step 3 specifically comprises the following steps: According to the classification information corresponding to the contour of the target instance, the classification to which the pixel point (i, j) of the projection car lamp belongs is determined; Combined with the category of the pixel point (i, j) of the projected headlight in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, the contour point (x i ,y j ) illuminance g(x i ,y j ): In the formula, i and j are respectively the horizontal and vertical pixel coordinates of the pixel point (i, j), the value of i is not less than 0 and not greater than the width of the current image frame, the value of h is not less than 0 and not greater than the height of the current image frame; a is the maximum intensity of the projection headlight illumination; δ is a decay coefficient according to the pixel coordinate distance d between the target instance and the ego vehicle, which is described as formula (2); θ is a measured constant decay coefficient within a limited distance, D max The farthest distance of the projection headlight to the main body projection illumination of the target instance: The following morphological technique is used for inflation compensation to suppress the illuminance value of the key target inflation area, and the direction and size of inflation are respectively referred to the vector direction and value of the pixel-level two-dimensional velocity of the target instance: ① According to the contour point p of the selected target instance (x,y) And the pixel-level two-dimensional velocity, calculate the to-be-inflated point q in the horizontal and vertical directions of the pixel-level two-dimensional velocity along the target instance (x,y) ; wherein (x, y) is the position coordinates of the contour point in the image coordinate system ② determine whether the point q to be inflated is inside the contour of the selected target instance of the current image frame, if yes, go to step ③, otherwise replace the contour point p (x,y) with the point q to be inflated (x,y) ; and (x,y) ; Step 2, the structured data of the current image frame is obtained through a semantic segmentation model, the structured data comprising the ID corresponding to each target instance, the semantic classification information corresponding to different setting levels and the contour point set, the contour point set forming a contour, and the relevant information of each target instance is obtained, the relevant information of the target instance comprising the pixel coordinate distance between the target instance and the ego vehicle and the pixel-level two-dimensional velocity, and the relevant information of the target instance is filled into the structured data of each corresponding target instance; 2. The projection headlamp illumination method based on image semantic segmentation object recognition of claim 1, wherein, Step 21b, sample data acquisition and labeling, specifically comprising the following steps: Step 211, scene sorting, outputting core semantic elements of a driving traffic scene, the traffic elements comprising roads and lighting environments, traffic participants and traffic signs, the first-level semantic classification of the traffic participants comprising motor vehicles, pedestrians and non-motor vehicles, the second-level semantic classification of the traffic participants comprising windshields, rearview mirrors and torsos, heads, and the first-level semantic classification of the traffic signs comprising ground signs and indicating signs; Step 212, collecting data on roads according to the driving vehicle, and outputting a collected and discretely extracted image set; ​ Step 213, labeling the completed image collected in step 212, and outputting an image dataset with semantic label true value; Step 22b, using the dataset labeled in step 21b, performing transfer learning training and evaluation on the selected model to obtain a projection car lamp semantic segmentation model; Step 23b, algorithm post-processing and result output.

3. The projection headlamp illumination method based on image semantic segmentation object recognition of claim 1, wherein, The imaging surface of each side camera is parallel to the projection light source surface of the projection car lamp, and the camera closely adheres to the lateral side of the projection car lamp, the transverse distance between the optical center of the camera and the center point of the projection light source surface of the projection car lamp is close to the sum of the transverse widths of the two, so as to maintain the close visual angle and stable projection transformation characteristics of the two, and each side camera and projection car lamp are arranged as an integrated module.

4. A projection car lamp lighting device based on image semantic segmentation target recognition, characterized in that, Comprise: A parameter calibration unit is arranged on each side of the vehicle body with a camera and a projection car lamp, and calibration parameters of each side camera and projection car lamp are obtained; A target instance information processing unit is used to obtain structured data of the current image frame through the semantic segmentation model according to the image frame obtained by the camera in real time, the structured data includes the ID corresponding to each target instance, the semantic classification information corresponding to different setting levels and the contour point set, and then the related information of each target instance is obtained, the related information of the target instance includes the pixel coordinate distance and the pixel level two-dimensional speed of the target instance and the ego vehicle, and the related information of the target instance is filled into the structured data of each corresponding target instance; An illumination calculation unit is used to calculate the illumination of each target instance; A projection lighting scheme generation unit is used to convert the contour of the target instance from the image coordinate system of the current image frame to the projection car lamp coordinate system through the calibration parameters of the parameter calibration unit, and perform pixel level brightness filling on the closed area formed by the contour of each target instance according to the corresponding illumination attribute calculated in step 3 and the structured data of the target instance obtained by the target instance information processing unit, and then generate a projection bitmap, which is used as a projection car lamp array projection lighting scheme; The target instance information processing unit specifically comprises: A semantic segmentation model is used to extract the pixel level semantic contour and classification of the current image frame according to the image frame obtained by the camera in real time, and output a contour point set, which constitutes a contour; A contour point updating model is used to iteratively calculate a plurality of contour points through a loop to obtain a vector formed by two adjacent contour points, and determine whether the first vector and the second vector adjacent to the first vector satisfy the vector collinearity theorem, if yes, the contour point shared by the first vector and the second vector is deleted, and the contour point set is updated; A structured data output model is used to output structured data according to the contour point set updated by the contour point updating model, in combination with the classification information corresponding to the contour and the ID of the target instance; The illumination calculation unit specifically comprises: A pixel point attribute determination model is used to determine the classification to which the pixel point (i, j) of the projection car lamp belongs according to the classification information corresponding to the contour of the target instance; The contour point illumination calculation module is used to combine the classification of the pixel point (i, j) of the projected headlight in the image coordinate system and the pixel coordinate distance d between the target instance and the vehicle, and calculate the contour point (x i ,y j ) illuminance g(x i ,y j ): In the formula, i and j are respectively the horizontal and vertical pixel coordinates of the pixel point (i, j), the value of i is not less than 0 and not greater than the width of the current image frame, the value of j is not less than 0 and not greater than the height of the current image frame; a is the maximum intensity of the projection headlight illumination; δ is a decay coefficient according to the pixel coordinate distance d between the target instance and the ego vehicle, which is described as formula (2); θ is a measured constant decay coefficient within a limited distance, D max The farthest distance of the projection headlight to the main body of the target instance The illumination modification module is used for expansion compensation by adopting a morphological technique, and the illumination value of the key target expansion area is suppressed, and the direction and size of expansion are respectively referred to the vector direction and numerical value of the pixel-level two-dimensional velocity of the target instance: ① According to the contour point p of the selected target instance (x,y) And the pixel-level two-dimensional velocity, calculate the to-be-inflated point q in the horizontal and vertical directions of the pixel-level two-dimensional velocity along the target instance (x,y) ; Wherein (x, y) is the position coordinates of the contour point in the image coordinate system; ② determine whether the point q to be inflated is inside the contour of the selected target instance of the current image frame, if yes, go to step ③, otherwise replace the contour point p (x,y) with the point q to be inflated (x,y) ; and (x,y) ; ③Judge whether all contour points of the selected target instance are polled once, and the polling is completed, otherwise jump to ①.

5. The projection headlamp illumination device based on image semantic segmentation object recognition of claim 4, wherein, The method for obtaining the semantic segmentation model comprises: Step 21b, sample data acquisition and labeling, specifically comprising: Step 211, scene carding, outputting core semantic elements of driving traffic scenes, traffic elements including roads and lighting environments, traffic participants and traffic signs, the first-level semantic classification of traffic participants including motor vehicles, pedestrians and non-motor vehicles, the second-level semantic classification of traffic participants including windshields, rearview mirrors and torsos, heads, and the first-level semantic classification of traffic signs including ground signs and indicating signs; Step 212, collecting road data according to the driving vehicle, and outputting the collected and discretely extracted image set; Step 213, labeling the target of the image collected in step 212, and outputting the image data set with semantic label true value; Step 22b, using the data set labeled in step 21b, performing transfer learning training and evaluation on the selected model to obtain the projection car lamp semantic segmentation model; Step 23b, algorithm post-processing and result output.

6. The projection headlamp illumination device based on image semantic segmentation object recognition of claim 4, wherein, The imaging surface of each side camera is parallel to the projection light source surface of the projection car lamp, and the camera and the projection car lamp are closely attached to the lateral side, the transverse distance between the optical center of the camera and the center point of the projection light source surface of the projection car lamp is close to the sum of the transverse widths of the two, so as to maintain the relatively close visual angle and stable projection transformation characteristics of the two, and the camera and the projection car lamp on each side are arranged as an integrated module.

Images