Method and device for enhancing traffic sign prompting effect through projection of projection vehicle lamp

Abstract

The invention discloses a road sign effect enhancement method and device based on a projection vehicle lamp. The method comprises the following steps: step 1, acquiring traffic sign information; step 2, acquiring information to be projected; step 3, judging whether the total number of the information to be projected is greater than 1, if so, entering step 4; 4, according to the grade classification of the traffic signs in the projection information, the speed difference between the current vehicle speed and the maximum limiting speed of the current road section, the distance difference between the current position and the target position, the weather difference between the current weather and the ideal weather, the preset grade classification weight, and the dynamically adjusted set speed weight, the traffic signs in the projection information are obtained. Calculating the priority of each piece of to-be-projected information according to the distance weight and the weather weight, and sorting the to-be-projected information from large to small according to the priorities to form a sequence. The traffic sign information acquired by the intelligent vehicle can be projected to the front of the vehicle after being processed, so that the prompt effect is enhanced, a driver is prevented from missing the traffic sign, and the driving efficiency and safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of road traffic sign recognition, and in particular to a method and device for enhancing the prompt effect of traffic signs by projection of projection vehicle lights. Background Art

[0002] Road traffic signs form a "traffic language" with their shapes, colors, patterns, and texts, providing information to all road users, guiding and regulating road users to use roads in an orderly manner, reducing traffic congestion, giving full play to the standardized governance capabilities, and improving the overall efficiency of road network operations. For example, prohibition signs can help drivers predict danger in advance; warning signs can help drivers avoid obstacles in advance in certain situations; and instruction signs can help drivers perform control preprocessing to ensure that driving follows road instructions.

[0003] For smart cars, the correct identification and precise application of traffic signs are particularly important. Nowadays, with the increasingly detailed and strict traffic regulations, many drivers need to pay attention not only to the driving conditions on the road, but also to the traffic signs on the roadside when facing special driving environments such as highways. When the speed is fast, the driver often cannot see all the traffic signs. In addition, on roads with complex traffic information, if the installation location of some traffic signs is not humane enough, it will also make it difficult for the driver to fully notice them. This is especially true when driving at night, and the limited visual conditions will also increase the possibility of missing traffic signs. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for enhancing the warning effect of traffic signs by projection of projection lights, which can project the traffic sign information obtained by the smart car to the front of the vehicle after processing, so as to enhance the warning effect, prevent the driver from missing the traffic signs, and improve driving efficiency and safety.

[0005] To achieve the above object, the present invention provides a method for enhancing the effect of road signs based on projection lamps, which comprises:

[0006] Step 1, obtaining traffic sign information;

[0007] Step 2, filtering traffic sign information according to the type of current road user, and then obtaining information to be projected according to the filtered traffic sign information and current state information;

[0008] Step 3, determining whether the total amount of information to be projected is greater than 1, if so, proceeding to step 4, otherwise, projecting the information to be projected to the front of the vehicle;

[0009] Step 4, according to the classification of traffic signs in the projection information, the speed difference Δv between the current speed of the vehicle and the maximum speed limit of the road section where the vehicle is currently located, the distance difference Δd between the current position of the vehicle and the target position, the weather difference Δw between the current weather and the ideal weather, the pre-set classification weight k0, and the dynamically adjusted speed weight k1, distance weight k2 and weather weight k3, the priority P of each information to be projected is calculated using formula (1), and then the information is sorted from large to small according to the priority to form a sequence, and the information to be projected is projected in front of the vehicle according to the sequence through the projection lamp; wherein the classification of traffic signs is divided into three categories: prohibition, warning and indication;

[0010] P=k0+k1*Δv+k2*Δd+k3*Δw (1).

[0011] Furthermore, the value of k1 is set by any of the following methods:

[0012] Method 1: Set k1 to V×v, where V is the speed class label. When V corresponds to the high-speed limit category, the value is 1; when V corresponds to the low-speed limit category, the value is -1; and when it corresponds to other categories, the value is 0. v is the speed limit gain. When Δv is a positive number, the value of v is 1; when Δv is less than or equal to 0, the value of v is -1.

[0013] Method 2: Set k1 to V*|Δv|, Δv=V c -V l ;

[0014] Method 3: Set k1 to V*Δv, where V is the speed class label, a Boolean value, 1 if it is a speed class, and 0 otherwise. When limiting high speed, Δv=V c -V l , when limiting low speed, Δv=V l -V c .

[0015] Furthermore, the value of k2 is set by any of the following methods:

[0016] Method 1: k2 is 1 when Δd is distance information related to the target point, and k2 is 0 when Δd is distance information unrelated to the target point;

[0017] Method 2: First calculate the value of d0-Δd. When the value of d0-Δd is the distance information related to the target point, the value of k2 is 1. When the value of d0-Δd is the distance information unrelated to the target point, the value of d0-Δd is 0. Where d0 is the target distance constant.

[0018] Method 3: Where d0 is the target distance constant.

[0019] Furthermore, the value of k3 is set by any of the following methods:

[0020] Method 1: k3 is a weather label, which is a Boolean value, and w is the difference between the current weather and the ideal weather;

[0021] Method 2: k3 is a weather label, which is a Boolean value. Its value is the matching result between the traffic sign type and the current weather system warning information. w is the warning information of the weather system of the current road section, and its value gradually increases according to the warning category.

[0022] Furthermore, step 1 specifically includes:

[0023] Step 11, obtaining an image in front of the vehicle through a vehicle-mounted camera;

[0024] Step 12, identifying traffic sign information in the image, specifically includes:

[0025] According to the image acquired in real time by the on-board camera, the structured data of the current image is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each traffic sign, the semantic classification information corresponding to different setting levels and the contour point set. The contour point set constitutes the contour, and then the relevant information of each traffic sign is obtained. The traffic sign related information includes the pixel coordinate distance between the traffic sign and the vehicle, and the relevant information of the traffic sign is filled into the structured data of each corresponding traffic sign; according to the calibration parameters of the on-board camera and the projection lamp, the contour of the traffic sign is converted from the image coordinate system of the current image to the projection lamp coordinate system, and according to the illumination of each traffic sign, on the basis of the structured data of the traffic sign, the closed area formed by the contour of each traffic sign is filled with pixel-level brightness, and then the traffic sign information is obtained.

[0026] Furthermore, before projecting the information to be projected in front of the vehicle, step 4 further includes:

[0027] The angle α1 between the line connecting the center of the detection frame of the traffic sign in the image and the center position of the vehicle-mounted camera and the horizontal plane is calculated, so as to obtain the angle α2 between the line connecting the geometric center of the detection frame of the traffic sign in the image and the center position of the projection camera and the horizontal plane.

[0028] The present invention also provides a road sign effect enhancement device based on a projection lamp, which comprises:

[0029] A recognition unit, which is used to obtain traffic sign information;

[0030] An information acquisition unit to be projected, which is used to filter traffic sign information according to the type of current road users, and then obtain the information to be projected according to the filtered traffic sign information and current state information;

[0031] a projection information quantity determination unit, which is used to determine whether the total amount of the to-be-projected information is greater than 1, and if the projection information quantity determination unit determines that the total amount of the to-be-projected information is greater than 1, projecting the to-be-projected information in front of the vehicle;

[0032] The projection decision unit is used to calculate the priority P of each information to be projected using formula (1) when the projection information quantity judgment unit determines that the projection information quantity judgment unit is yes, according to the classification of traffic signs in the projection information, the speed difference Δv between the current speed of the vehicle and the maximum speed limit of the road section where the vehicle is currently located, the distance difference Δd between the current position of the vehicle and the target position, the weather difference Δw between the current weather and the ideal weather, the pre-set classification weight k0, and the dynamically adjusted speed weight k1, distance weight k2 and weather weight k3, and then sort the priorities from large to small to form a sequence, and project the information to be projected to the front of the vehicle according to the sequence through the projection lamp; wherein the classification of traffic signs is divided into three categories: prohibition, warning and indication;

[0033] P=k0+k1*Δv+k2*Δd+k3*Δw (1).

[0034] Furthermore, the value of k1 is set by any of the following methods:

[0035] Method 1: Set k1 to V×v, where V is the speed class label. When V corresponds to the high-speed limit category, the value is 1; when V corresponds to the low-speed limit category, the value is -1; and when it corresponds to other categories, the value is 0. v is the speed limit gain. When Δv is a positive number, the value of v is 1; when Δv is less than or equal to 0, the value of v is -1.

[0036] Method 2: Set k1 to V*|Δv|, Δv=V c -V l ;

[0037] Method 3: Set k1 to V*Δv, where V is the speed class label, a Boolean value, 1 if it is a speed class, and 0 otherwise. When limiting high speed, Δv=V c -V l , when limiting low speed, Δv=V l -V c ;

[0038] The value of k2 can be set in any of the following ways:

[0039] Method 1: k2 is 1 when Δd is distance information related to the target point, and k2 is 0 when Δd is distance information unrelated to the target point.

[0040] Method 2: First calculate the value of d0-Δd. When the value of d0-Δd is the distance information related to the target point, the value of k2 is 1. When the value of d0-Δd is the distance information unrelated to the target point, the value of d0-Δd is 0. Where d0 is the target distance constant.

[0041] Method 3: Where d0 is the target distance constant;

[0042] The value of k3 can be set in any of the following ways:

[0043] Method 1: k3 is a weather label, which is a Boolean value, and w is the difference between the current weather and the ideal weather;

[0044] Method 2: k3 is a weather label, which is a Boolean value. Its value is the matching result between the traffic sign type and the current weather system warning information. w is the warning information of the weather system of the current road section, and its value gradually increases according to the warning category.

[0045] Furthermore, the identification unit specifically includes:

[0046] A data acquisition subunit, which is used to obtain an image in front of the vehicle through an on-board camera;

[0047] The sign recognition subunit is used to recognize traffic sign information in the image, and specifically includes:

[0048] According to the image acquired in real time by the on-board camera, the structured data of the current image is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each traffic sign and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each traffic sign is obtained. The traffic sign related information includes the pixel coordinate distance between the traffic sign and the vehicle, and the relevant information of the traffic sign is filled into the structured data of each corresponding traffic sign; according to the calibration parameters of the on-board camera and the projection lamp, the contour of the traffic sign is converted from the image coordinate system of the current image to the projection lamp coordinate system, and according to the illumination of each traffic sign, on the basis of the structured data of the traffic sign, the closed area formed by the contour of each traffic sign is filled with pixel-level brightness, and then the traffic sign information is obtained.

[0049] Furthermore, the projection decision unit is used to calculate the angle α1 between the horizontal plane and the line connecting the center of the detection frame of the traffic sign in the image and the center position of the vehicle-mounted camera before projecting the information to be projected in front of the vehicle, thereby obtaining the angle α2 between the horizontal plane and the line connecting the geometric center of the detection frame of the traffic sign in the image and the center position of the projection headlight.

[0050] Since the present invention adopts the above technical solution, it has at least the following beneficial effects:

[0051] 1. The method for detecting and identifying traffic signs of the present invention ensures the real-time nature of information obtained from high-precision maps, thereby compensating for the possible recognition failure of the perception system.

[0052] 2. When multiple traffic sign information appears at the same time, the priority determination mechanism of the present invention can arrange the received information by priority according to the driving conditions of the vehicle, thereby reducing the energy consumed by the driver on the traffic signs.

[0053] 3. The present invention utilizes projection lights to project traffic sign information in front of the car, which not only meets the lighting requirements but also efficiently presents the traffic sign information to achieve the effect of assisting driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of a single speed limit traffic sign scenario according to an embodiment of the present invention.

[0055] Figure 2 Schematic diagram of a multi-traffic sign scenario according to an embodiment of the present invention.

[0056] Figure 3 The figure is a schematic diagram of determining the projection angle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In the drawings, the same or similar reference numerals are used to represent the same or similar elements or elements with the same or similar functions. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0058] In the description of the present invention, the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the scope of protection of the present invention.

[0059] The method for enhancing the road sign effect based on the projection lamp provided by the embodiment of the present invention includes:

[0060] Step 1: Obtain traffic sign information.

[0061] In one embodiment, the traffic sign information may be obtained through traffic sign recognition TSR (Traffic Sign Recognition), such as shown in the following steps 11 and 12:

[0062] Step 11, detection: obtain the image in front of the vehicle through the vehicle-mounted camera.

[0063] Detection is used to remove the interference of invalid information on recognition, and only processes the region of interest (ROI) to reduce the amount of calculation required for computer image processing. Since the functions of traffic signs are to be indicative, suggestive and warning, they are all set to be eye-catching, brightly colored, simple in graphics, clear in meaning, and fixed in shape. Therefore, ROI is usually divided by its color and shape. This embodiment divides ROI according to color. The color design of traffic signs generally uses red for prohibition, yellow for warning, and blue for instruction. RGB is the basic color of all colors, but the RGB color space also contains other information such as brightness and saturation, so HSI is used for normalization. H stands for Hue, which means hue, S stands for Saturation, which means color saturation, and I stands for Intensity, which means brightness:

[0064]

[0065]

[0066]

[0067] Then, a threshold is set in the HSI color space, and the image is binarized by the threshold method, that is, the global contrast greater than the threshold is set to 1, and less than the threshold is set to 0. Then, in such a binary image, the contour of the salient area corresponding to the value of 1 is extracted, and the ROI is divided.

[0068] Step 12, recognition: Recognition is to further determine the meaning of the segmented traffic sign. After detecting the traffic sign and setting the non-traffic sign area as the background, the segmented image must also undergo normalization and preprocessing operations, such as eliminating the influence of noise, shooting jitter, lighting and other factors, and then recognize its kernel content. Construct a neural network, stack the convolution layer and the pooling layer, then pass through the fully connected layer, and finally use softmax to map to the probability of each category. The one with the largest probability is the recognition result. This method can be directly implemented using existing open source projects.

[0069] In another embodiment, the recognition can also be achieved through a YOLO (You Only Look Once) deep neural network model. The image of the road ahead of the vehicle captured by the camera in the intelligent vehicle perception system is used as the input of the YOLO (You Only Look Once) deep neural network model, and all traffic signs in the image can be detected and their meanings can be recognized.

[0070] Considering the low light conditions during nighttime driving, the recognition efficiency of traffic signs by the perception system is reduced. A lighting strategy for the projection lamp-assisted perception system to recognize traffic signs is proposed. The calculation unit completes the coordinate transformation according to the position of the candidate frame (detection result) returned by the perception system in the image, as well as the relative position of the camera and the lamp, to obtain the relative angle between the detection result and the lamp. The projection lamp control unit controls the lamp to separate most of the light beams to accurately illuminate the area to be recognized, so as to improve the accuracy of detection and recognition; and its surrounding areas, to prevent possible missed detection.

[0071] In one embodiment, identification can also be obtained by the following method:

[0072] According to the image acquired in real time by the on-board camera, the structured data of the current image is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each traffic sign and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each traffic sign is obtained. The relevant information of the traffic sign includes the pixel coordinate distance between the traffic sign and the vehicle. The relevant information of the traffic sign is filled into the structured data of each corresponding traffic sign; according to the calibration parameters of the on-board camera and the projection lamp, the contour of the traffic sign is converted from the image coordinate system of the current image to the projection lamp coordinate system, and according to the illumination of each traffic sign, the closed area formed by the contour of each traffic sign is filled with pixel-level brightness based on the structured data of the traffic sign, and then the traffic sign information is obtained. Among them, the ID corresponding to the traffic sign can be used as the traffic sign ID in the multi-frame contour cache queue of continuous buffering according to the intersection-and-union ratio of the pixel coordinate distance between the traffic sign and the vehicle and the contour area. The contour point set refers to the position coordinate set of the pixel point in the image coordinate system. The pixel coordinate distance between the traffic sign and the vehicle can be obtained by taking the pixel difference between the pixel coordinate of the centroid of the outline of each traffic sign and the pixel coordinate of the midpoint of the lower boundary of the current image frame on the horizontal x-axis and the vertical y-axis as the pixel coordinate distance between the corresponding traffic sign and the vehicle. The illumination of each traffic sign can be achieved by using the common method and the scheme of foreground (people, vehicles) shading and background (road) fill light.

[0073] In one embodiment, the method of “obtaining the contour of each traffic sign in the current image frame and the classification information corresponding to the contour by using a semantic segmentation model” specifically includes:

[0074] Step 121a, based on the image frame acquired by the camera in real time, the pixel-level semantic contour and classification of the current image frame are extracted through the semantic segmentation model, and a contour point set is output, and the contour point set constitutes a contour.

[0075] Step 122a, iteratively calculate multiple consecutive contour points through a loop to obtain a vector composed of two adjacent contour points, and determine whether the adjacent first vector and second vector satisfy the vector collinearity theorem. If so, delete the contour points shared by the first vector and the second vector, and update the contour point set.

[0076] Step 123a, outputting structured data according to the contour point set updated in step 122a, combined with the classification information corresponding to the contour and the ID of the traffic sign. The structured data may be a predefined data structure, consisting of multi-level semantic classification.

[0077] In one embodiment, a method for obtaining a “semantic segmentation model” includes:

[0078] Step 121b, sample data collection and labeling, specifically includes:

[0079] Step 1211, scene combing, outputting the core semantic elements of driving traffic scenes, as shown in Table 1 below; the traffic semantic elements for driving scenes mainly include roads and lighting environment, traffic participants and traffic signs. Among them, roads mainly cover urban and rural roads, highways, and cover open sections and tunnels. Traffic participants are classified into motor vehicles, pedestrians and non-motor vehicles. Traffic signs include ground signs and signposts.

[0080] In addition, for the application of projection headlight lighting scenarios, more refined semantic segmentation and annotation of traffic participants is required. For pedestrians and people in non-enclosed passenger cabins such as non-motor vehicles, motorcycles, and convertible motor vehicles, the torso and head need to be further refined and annotated separately. For closed passenger cabin motor vehicles, the windshield and rearview mirror areas need to be annotated, and the secondary semantic classifications need to be circled separately. The semantic classifications can be evaluated and increased according to the actual scene conditions and needs, including but not limited to those described in Table 2.

[0081] Table 1

[0082]

[0083] In the table, "-" means "none", so the "-" in the primary and secondary semantic classifications of "road and lighting environment" means that no corresponding primary and secondary semantic classifications are made. The "-" in the secondary semantic classification of "traffic signs" means that no corresponding secondary semantic classification is made.

[0084] Step 1212, according to the road data collection action of driving the vehicle, the collected and discretely extracted image set is output. For example, a vehicle equipped with a camera is used to plan and select areas that meet the combed scenes for driving, and image data is collected and stored during driving. Data collection ensures balanced coverage of different traffic scenes. The continuous image set that has been collected is obtained by equally spaced frame extraction to obtain a discrete data set, and similar images in the discrete set are eliminated through manual verification to reduce and optimize the representativeness of the quality of the data set to be annotated. The scale of the data set that is finally screened is maintained at, for example, more than 1,000 images. At this time, the total number of annotated target categories is multiplied by 1,000. For example, the number of annotated instances in this example includes 6 first-level categories and 4 second-level fine categories, totaling 10 categories.

[0085] Step 2113, annotate the image acquired in step 1212, and output an image dataset with semantic annotation truth values. For example, the dataset annotates the semantic element contours frame by frame, and after the annotation is completed, the annotation is verified through a visualization tool to check the consistency of the annotation boundary, whether small targets, occluded or truncated targets are missed, and whether the labels are correctly matched. Among them, the boundary consistency can be, but is not limited to, the average error of the boundary pixels.

[0086] Step 122b, using the data set annotated in step 121b, perform transfer learning training and evaluation on the selected model to obtain a semantic segmentation model of projection headlights.

[0087] Step 122b specifically includes the following four steps:

[0088] Step 1221b, model framework selection.

[0089] The comprehensive model performance indicators are combined with the computing power and real-time requirements of the deployment platform. Among the existing model implementations commonly used in the industry, refer to the benchmark segmentation evaluation of public data sets in the traffic neighborhood and select a suitable model framework, such as U-Net.

[0090] Step 1222b, model pre-training.

[0091] The model framework's effect on feature learning in specific fields depends on feature training of large-scale data sets. The scale of private data in actual projects is difficult to meet the model's internalization of generalized features, so the model needs to be pre-trained using public large-scale traffic data sets, such as Cityscapes and KITTI. The pre-trained model is evaluated through indicators on the test set.

[0092] Step 1223b, model transfer learning.

[0093] The pre-trained model is trained and feature transferred on the labeled sample data set. The work that needs to be done before training includes feature semantic label adaptation (10 labels), sample set division, and training hyperparameter configuration (number of iterations, learning rate, etc.). The sample set division refers to the training set, validation set, and test set, and is randomly divided according to 7:2:1 or a similar ratio. After completing the above parameter configuration, start the model migration training, and monitor whether the iterative change convergence trend of the model's core indicators meets expectations. If not, it is necessary to stop and locate the problem scene, supplement the scene data through step 1, or optimize the parameter configuration in this step to restart the training. The technology in this field is mature and can be completed after iteration. And if it is found that there are scenes with elements that do not meet the requirements during the test and verification process, the scene image can be supplemented again through step 1 and steps 1 and 2 can be completed to achieve the expected effect.

[0094] Step 1224b, sample testing and indicator evaluation.

[0095] The model that has completed transfer training is tested and evaluated on the divided test set, focusing on the accuracy set recall rate indicators, and checking whether the model generalization requirements are met by sampling visualization samples and annotating model prediction results. When the proportion of samples or scenarios that do not meet expectations found through sample inspection exceeds expectations, it is necessary to perform additional annotations on such scenes and re-perform transfer learning training to improve the model effect.

[0096] Step 123b, algorithm post-processing and result output.

[0097] In another embodiment, traffic sign information can also be obtained from high-precision maps. Since the recognition effect of the perception system is affected by many factors, such as weather: rain, fog, dust and other weather conditions will affect the working state of the sensor. Damaged or blocked traffic signs will also interfere with the recognition effect. However, the traffic sign information of the high-precision map is entered in advance, which provides another way to obtain traffic sign information. According to the location information of the vehicle, the nearby traffic sign information can be directly obtained from the high-precision map.

[0098] Step 2: filter the traffic sign information according to the type of the current road user, and then obtain the information to be projected according to the filtered traffic sign information and the current state information.

[0099] In this embodiment, the acquired traffic sign information is screened, useful information is extracted, and the vehicle information is integrated to form more efficient information to be projected that is more conducive to the driver's understanding.

[0100] In one embodiment, step 2 specifically includes:

[0101] Step 21, information screening. When the vehicle equipped with projection lights belongs to one of the categories, the information provided by the road signs related to this type of road user is useful information, otherwise it is useless information, and the useless information will be screened and filtered. For example, ordinary passenger car drivers only need to pay attention to information related to ordinary passenger cars, and do not need to consider information such as "no tractors allowed", "weight limit 10t" and "height limit 3.5m". The classification of these road users can be determined when the projection lights are deployed, such as ordinary passenger cars, buses and trucks.

[0102] Step 22, information integration. The information acquisition unit to be projected processes the acquired traffic sign information into more intuitive information and provides it to the driver. In general, after seeing a road traffic sign, the driver often needs to judge the driving behavior in combination with the current driving situation of the vehicle. For example, when seeing a speed limit sign, it is necessary to check the current vehicle speed. This embodiment integrates the filtered traffic sign information with other vehicle information fed back by the smart car, and then provides the processed information to the driver, which is more convenient for the driver to drive. For example, the information acquisition unit to be projected receives traffic sign information of a speed limit of 50, and the current vehicle speed fed back by the chassis is 60, then the information to be projected of speeding 10 is generated. For another example, if the information acquisition unit to be projected receives road guidance information that is irrelevant to the destination fed back by the navigation system, then the information will not be generated.

[0103] Step 3, considering that multiple traffic signs may appear at the same time, it is necessary to determine whether the total number of information to be projected is greater than 1. If so, proceed to step 4, otherwise, project the information to be projected in front of the vehicle. For example, three traffic signs are obtained: "Speed ​​limit 90", "No U-turn", and "XScenic Area 1km". Combined with the navigation information of the vehicle, it is determined that the XScenic Area has nothing to do with the destination, and the current vehicle speed is 100. Then the information to be projected is "Speeding 10" and "No U-turn". That is, the number of information to be projected is 2.

[0104] Step 4, according to the classification of traffic signs in the projection information, the speed difference Δv between the current speed of the vehicle and the maximum speed limit of the road section where the vehicle is currently located, the distance difference Δd between the current position of the vehicle and the target position, the weather difference Δw between the current weather and the ideal weather, the pre-set classification weight k0, and the dynamically adjusted speed weight k1, distance weight k2 and weather weight k3, the priority P of each information to be projected is calculated using formula (1), and then the information is sorted from large to small according to the priority to form a sequence, and the information to be projected is projected in front of the vehicle according to the sequence through the projection lamp; wherein the classification of traffic signs is divided into three categories: prohibition, warning and indication;

[0105] P=k0+k1*Δv+k2*Δd+k3*Δw (1).

[0106] This embodiment introduces a priority determination mechanism to sort the filtered and integrated information and generate a priority queue of information to be projected.

[0107] The value of k1 can be set by any of the following methods:

[0108] Method 1: Set k1 to V×v, where V is the speed class label. When V corresponds to the high-speed limit category, the value is 1; when V corresponds to the low-speed limit category, the value is -1; and when it corresponds to other categories, the value is 0. v is the speed limit gain. When Δv is a positive number, the value of v is 1; when Δv is less than or equal to 0, the value of v is -1.

[0109] Method 2: Set k1 to V*|Δv|, Δv=V c -V l ;

[0110] Method 3: Set k1 to V*Δv, where V is the speed class label, a Boolean value, 1 if it is a speed class, and 0 otherwise. When limiting high speed, Δv=V c -V l , when limiting low speed, Δv=V l -V c .

[0111] The value of k2 can be set in any of the following ways:

[0112] Method 1: k2 is 1 when Δd is distance information related to the target point, and k2 is 0 when Δd is distance information unrelated to the target point.

[0113] Method 2: First calculate the value of d0-Δd. When the value of d0-Δd is the distance information related to the target point, the value of k2 is 1. When the value of d0-Δd is the distance information unrelated to the target point, the value of d0-Δd is 0. Where d0 is the target distance constant.

[0114] Method 3: Where d0 is the target distance constant.

[0115] The value of k3 can be set in any of the following ways:

[0116] Method 1: k3 is a weather label, which is a Boolean value, and w is the difference between the current weather and the ideal weather; for example, in rainy and snowy weather, w is proportional to the precipitation, such as w = precipitation / 100.

[0117] Method 2: k3 is a weather label, a Boolean value, whose value is the match between the traffic sign type and the current weather system warning information, and w is the warning information of the weather system of the current road section, and its value gradually increases according to the warning category (blue, yellow, orange and red). For example, if the traffic sign information is "Watch out for fog", and the weather warning of the current road section is a yellow warning for heavy fog, then k3 = 1, w = 60.

[0118] In one embodiment, the priority P of each piece of information to be projected can also be calculated using formula (2):

[0119] P=L+V(v(V c -V l ))+D(5-d)+Ww (2)

[0120] Among them, L represents the level label, which is reclassified into three categories based on the existing classification of traffic signs: prohibition, warning, and direction, with values ​​of 90, 60, and 30 respectively; V is the speed label, with high speed limit as 1, low speed limit as -1, and otherwise 0; when the vehicle's current speed V c Greater than the speed limit V l , the speed limit gain v is 1, otherwise the speed limit gain v is -1. For example, the current speed limit is 60, and the vehicle speed is 90, then the priority of the speed limit information will produce a gain of 30, and when the current speed V c Much less than the speed limit V l , the priority of the speed limit information will be reduced accordingly; D is a distance label, which is a Boolean value, 1 if it is a distance type, otherwise 0. d (in km) is the distance to the target location. For example, the closer the distance to the exit to be driven, the higher the priority. W is a weather label, which is a Boolean value, including weather-related signs such as "easy to slip" and "beware of crosswinds"; w is the difference between the current weather and the ideal weather (clear and windless). The worse the weather, the greater the value. This embodiment also means that a traffic sign can have different labels.

[0121] The priority determination mechanism adopted in this embodiment arranges the received information by priority according to the driving conditions of the vehicle when multiple traffic sign information appears at the same time, thereby reducing the energy consumed by the driver on the traffic signs.

[0122] In one embodiment, in step 4, when the information to be projected is projected in sequence in front of the vehicle, the traffic sign information with the top three priorities is preferably selected and projected in a manner of gradually weakening the effect or reducing the projection size.

[0123] This embodiment utilizes projection lights to project traffic sign information in front of the car, which not only meets the lighting requirements but also efficiently presents the traffic sign information to achieve the effect of assisted driving.

[0124] Figure 1 This is a single speed limit traffic sign scene. When the current traffic standard is a speed limit of 50, the speed indicated by a in the figure is 40, and the information to be projected is "10 below the speed limit"; the speed indicated by b in the figure is 60, and the information to be projected is "10 over the speed limit".

[0125] Figure 2 This is a multi-traffic sign scenario. In this scenario, the system receives multiple traffic sign information. Among them, "No Overtaking" and "Speed ​​Limit" are prohibition messages, L is 90, and "Slippery" is a warning message, L is 60. However, due to the speed difference of speed messages, the priority of "Speed ​​Limit" is reduced by 40. Therefore, the final priority queue is "No Overtaking", "Slippery" and "Speed ​​Limit".

[0126] In one embodiment, Figure 3 As shown, step 4, before projecting the information to be projected to the front of the vehicle, also includes:

[0127] The angle α1 between the line connecting the geometric center of the detection frame of the traffic sign in the image and the center of the vehicle camera and the horizontal plane is calculated, so as to obtain the angle α2 between the line connecting the center of the detection frame of the traffic sign in the image and the center of the projection headlight and the horizontal plane. In this way, the projection headlight can be controlled to separate multiple beams to accurately illuminate the detection frame area, so as to improve the accuracy of detection and recognition and its surrounding areas, so as to prevent possible missed detection. Among them, the "center" can generally be regarded as the "geometric center".

[0128] The embodiment of the present invention also includes a road sign effect enhancement device based on projection lamps, which includes a recognition unit, a unit for acquiring information to be projected, a unit for determining the amount of projection information, and a projection decision unit, wherein:

[0129] The recognition unit is used to obtain traffic sign information.

[0130] The information to be projected acquiring unit is used for filtering traffic sign information according to the type of current road users, and then acquiring the information to be projected according to the filtered traffic sign information and current state information.

[0131] The projection information quantity determination unit is used to determine whether the total amount of the information to be projected is greater than 1. If the projection information quantity determination unit determines that the total amount is no, the information to be projected is projected in front of the vehicle.

[0132] The projection decision unit is used to calculate the priority P of each information to be projected using formula (1) when the projection information quantity judgment unit determines that the projection information quantity judgment unit is yes, based on the level classification of traffic signs in the projection information, the speed difference Δv between the current speed of the vehicle and the maximum speed limit of the road section where the vehicle is currently located, the distance difference Δd between the current position of the vehicle and the target position, the weather difference Δw between the current weather and the ideal weather, the pre-set level classification weight k0, and the dynamically adjusted speed weight k1, distance weight k2 and weather weight k3, and then sort them from the highest priority to the lowest to form a sequence, and project the information to be projected to the front of the vehicle according to the sequence through the projection lamp; wherein the level classification of traffic signs is divided into three categories: prohibition, warning and indication.

[0133] In one embodiment, the identification unit specifically includes a data acquisition subunit and a sign identification subunit, wherein:

[0134] The data acquisition subunit is used to obtain the image in front of the vehicle through the on-board camera.

[0135] The sign recognition subunit is used to recognize traffic sign information in the image, which specifically includes:

[0136] According to the image acquired in real time by the on-board camera, the structured data of the current image is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each traffic sign and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each traffic sign is obtained. The traffic sign related information includes the pixel coordinate distance between the traffic sign and the vehicle, and the relevant information of the traffic sign is filled into the structured data of each corresponding traffic sign; according to the calibration parameters of the on-board camera and the projection lamp, the contour of the traffic sign is converted from the image coordinate system of the current image to the projection lamp coordinate system, and according to the illumination of each traffic sign, on the basis of the structured data of the traffic sign, the closed area formed by the contour of each traffic sign is filled with pixel-level brightness, and then the traffic sign information is obtained.

[0137] In one embodiment, the projection decision unit is used to calculate the angle α1 between the horizontal plane and the line connecting the center of the detection frame of the traffic sign in the image and the center of the vehicle-mounted camera before projecting the information to be projected in front of the vehicle, thereby obtaining the angle α2 between the horizontal plane and the line connecting the geometric center of the detection frame of the traffic sign in the image and the center of the projection headlight.

[0138] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing road sign effects based on projection vehicle lights, characterized in that: include: Step 1, obtaining traffic sign information; Step 2, filtering traffic sign information according to the type of current road user, and then obtaining information to be projected according to the filtered traffic sign information and current state information; Step 3, determining whether the total amount of information to be projected is greater than 1, if so, proceeding to step 4, otherwise, projecting the information to be projected to the front of the vehicle; Step 4, according to the classification of traffic signs in the projection information, the speed difference Δv between the current speed of the vehicle and the maximum speed limit of the road section where the vehicle is currently located, the distance difference Δd between the current position of the vehicle and the target position, the weather difference Δw between the current weather and the ideal weather, the pre-set classification weight k0, and the dynamically adjusted speed weight k1, distance weight k2 and weather weight k3, the priority P of each information to be projected is calculated using formula (1), and then the information is sorted from large to small according to the priority to form a sequence, and the information to be projected is projected in front of the vehicle according to the sequence through the projection lamp; wherein the classification of traffic signs is divided into three categories: prohibition, warning and indication; P=k0+k1*Δv+k2*Δd+k3*Δw (1).

2. The method for enhancing the road sign effect based on projection lamps according to claim 1, characterized in that: The value of k1 can be set in any of the following ways: Method 1: Set k1 to V×v, where V is the speed class label. When V corresponds to the high-speed limit category, the value is 1; when V corresponds to the low-speed limit category, the value is -1; and when it corresponds to other categories, the value is 0. v is the speed limit gain. When Δv is a positive number, the value of v is 1; when Δv is less than or equal to 0, the value of v is -1. Method 2: Set k1 to V*|Δv|, Δv=V c -V l ; Method 3: Set k1 to V*Δv, where V is the speed class label, a Boolean value, 1 if it is a speed class, and 0 otherwise. When limiting high speed, Δv=V c -V l , when limiting low speed, Δv=V l -V c .

3. The method for enhancing road sign effects based on projection lamps according to claim 1 or 2, characterized in that: The value of k2 can be set in any of the following ways: Method-: k2 is 1 when Δd is distance information related to the target point, and k2 is 0 when Δd is distance information unrelated to the target point; Method 2: First calculate the value of d0-Δd. When the value of d0-Δd is the distance information related to the target point, the value of k2 is 1. When the value of d0-Δd is the distance information unrelated to the target point, the value of d0-Δd is 0. Where d0 is the target distance constant. Method 3: Where d0 is the target distance constant.

4. The method for enhancing the road sign effect based on projection lamps according to claim 3, characterized in that: The value of k3 can be set in any of the following ways: Method 1: k3 is a weather label, which is a Boolean value, and w is the difference between the current weather and the ideal weather; Method 2: k3 is a weather label, which is a Boolean value. Its value is the matching result between the traffic sign type and the current weather system warning information. w is the warning information of the weather system of the current road section, and its value gradually increases according to the warning category.

5. The method for enhancing the road sign effect based on projection lamps according to any one of claims 1 to 4, characterized in that: Step 1 specifically includes: Step 11, obtaining an image in front of the vehicle through a vehicle-mounted camera; Step 12, identifying traffic sign information in the image, specifically includes: According to the image acquired in real time by the on-board camera, the structured data of the current image is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each traffic sign, the semantic classification information and the contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each traffic sign is obtained. The traffic sign related information includes the pixel coordinate distance between the traffic sign and the vehicle, and the relevant information of the traffic sign is filled into the structured data of each corresponding traffic sign; according to the calibration parameters of the on-board camera and the projection lamp, the contour of the traffic sign is converted from the image coordinate system of the current image to the projection lamp coordinate system, and according to the illumination of each traffic sign, on the basis of the structured data of the traffic sign, the closed area formed by the contour of each traffic sign is filled with pixel-level brightness, and then the traffic sign information is obtained.

6. The method for enhancing the road sign effect based on projection lamps according to claim 5, characterized in that: Step 4, before projecting the information to be projected in front of the vehicle, further includes: The angle α1 between the line connecting the center of the detection frame of the traffic sign in the image and the center position of the vehicle-mounted camera and the horizontal plane is calculated, so as to obtain the angle α2 between the line connecting the geometric center of the detection frame of the traffic sign in the image and the center position of the projection camera and the horizontal plane.

7. A road sign effect enhancement device based on projection lamps, characterized in that: include: A recognition unit, which is used to obtain traffic sign information; An information acquisition unit to be projected, which is used to filter traffic sign information according to the type of current road users, and then obtain the information to be projected according to the filtered traffic sign information and current state information; a projection information quantity determination unit, which is used to determine whether the total amount of the to-be-projected information is greater than 1, and if the projection information quantity determination unit determines that the total amount of the to-be-projected information is greater than 1, projecting the to-be-projected information in front of the vehicle; The projection decision unit is used to calculate the priority P of each information to be projected using formula (1) when the projection information quantity judgment unit determines that the projection information quantity judgment unit is yes, according to the classification of traffic signs in the projection information, the speed difference Δv between the current speed of the vehicle and the maximum speed limit of the road section where the vehicle is currently located, the distance difference Δd between the current position of the vehicle and the target position, the weather difference Δw between the current weather and the ideal weather, the pre-set classification weight k0, and the dynamically adjusted speed weight k1, distance weight k2 and weather weight k3, and then sort the priorities from large to small to form a sequence, and project the information to be projected to the front of the vehicle according to the sequence through the projection lamp; wherein the classification of traffic signs is divided into three categories: prohibition, warning and indication; P=k0+k1*Δv+k2*Δd+k3*Δw (1).

8. The road sign effect enhancement device based on projection lamps as claimed in claim 1, characterized in that: The value of k1 can be set in any of the following ways: Method 1: Set k1 to V×v, where V is the speed class label. When V corresponds to the high-speed limit category, the value is 1; when V corresponds to the low-speed limit category, the value is -1; and when it corresponds to other categories, the value is 0. v is the speed limit gain. When Δv is a positive number, the value of v is 1; when Δv is less than or equal to 0, the value of v is -1. Method 2: Set k1 to V*|Δv|, Δv=V c -V l ; Method 3: Set k1 to V*Δv, where V is the speed class label, a Boolean value, 1 if it is a speed class, and 0 otherwise. When limiting high speed, Δv=V c -V l , when limiting low speed, Δv=V l -V c ; The value of k2 can be set in any of the following ways: Method-: k2 is 1 when Δd is distance information related to the target point, and k2 is 0 when Δd is distance information unrelated to the target point; Method 2: First calculate the value of d0-Δd. When the value of d0-Δd is the distance information related to the target point, the value of k2 is 1. When the value of d0-Δd is the distance information unrelated to the target point, the value of d0-Δd is 0. Where d0 is the target distance constant. Method 3: Where d0 is the target distance constant; The value of k3 can be set in any of the following ways: Method 1: k3 is a weather label, which is a Boolean value, and w is the difference between the current weather and the ideal weather; Method 2: k3 is a weather label, which is a Boolean value. Its value is the matching result between the traffic sign type and the current weather system warning information. w is the warning information of the weather system of the current road section, and its value gradually increases according to the warning category.

9. The road sign effect enhancement device based on projection lamps according to claim 7 or 8, characterized in that: The identification unit specifically includes: A data acquisition subunit, which is used to obtain an image in front of the vehicle through an on-board camera; The sign recognition subunit is used to recognize traffic sign information in the image, and specifically includes: According to the image acquired in real time by the on-board camera, the structured data of the current image is obtained through the semantic segmentation model. The structured data includes the ID corresponding to each traffic sign and the semantic classification information and contour point set corresponding to different setting levels. The contour point set constitutes the contour, and then the relevant information of each traffic sign is obtained. The traffic sign related information includes the pixel coordinate distance between the traffic sign and the vehicle, and the relevant information of the traffic sign is filled into the structured data of each corresponding traffic sign; according to the calibration parameters of the on-board camera and the projection lamp, the contour of the traffic sign is converted from the image coordinate system of the current image to the projection lamp coordinate system, and according to the illumination of each traffic sign, on the basis of the structured data of the traffic sign, the closed area formed by the contour of each traffic sign is filled with pixel-level brightness, and then the traffic sign information is obtained.

10. The road sign effect enhancement device based on projection lamps as claimed in claim 9, characterized in that: The projection decision unit is used to calculate the angle α1 between the line connecting the center of the detection frame of the traffic sign in the image and the center position of the vehicle-mounted camera and the horizontal plane before projecting the information to be projected in front of the vehicle, so as to obtain the angle α2 between the line connecting the geometric center of the detection frame of the traffic sign in the image and the center position of the projection headlight and the horizontal plane.

Images