Variable traffic information board identification method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides a variable traffic information board identification method and device, equipment, a medium and a program product, and particularly relates to the technical field of traffic information processing. The method comprises the following steps: carrying out variable traffic information board detection on a collected original image to obtain a variable traffic information board detection result; the variable traffic information board detection result is mapped to a coordinate system of a target point cloud set from an original image coordinate system, so that an interested space is determined in a point cloud space of the target point cloud set, and the target point cloud set is a point cloud set matched with the original image in multiple point cloud sets; and identifying target point cloud data in the interested space, wherein the target point cloud data is point cloud data representing the variable traffic information board. The method is used for achieving the effect of improving the recognition accuracy of the variable traffic information board.

Description

Technical Field

[0001] The present application relates to the technical field of traffic information processing, and in particular to a method, device, equipment, medium and program product for identifying a variable traffic information sign. Background Art

[0002] Variable traffic information signs, also known as changeable-message signs, are traffic signs that can change their display content according to changes in traffic, road, weather and other conditions. They store a variety of information, and the controller can manually or automatically display the corresponding information on the sign through a remote control device in real time according to changes in the driving environment, informing and warning the driver to take corresponding safe and reasonable driving measures. Variable traffic information signs can generally be used to display content such as speed control, lane control, road conditions, and weather conditions. When there is no need to display variable information according to changes in traffic, road, weather and other conditions, the display screen of the variable traffic information sign should be turned off.

[0003] The recognition and extraction of variable traffic information signs is an important topic in intelligent transportation systems. The recognition of traditional traffic information signs relies on reflective materials, which have high reflection intensity in the point cloud data of LiDAR and are easy to be recognized. However, since variable traffic information signs use LED display screens, their reflection intensity is usually low, resulting in low recognition accuracy of variable traffic information signs. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment, medium and program product for identifying a variable traffic information sign, so as to achieve the effect of improving the recognition accuracy of the variable traffic information sign.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying a variable traffic information sign, comprising:

[0006] Perform variable traffic information sign detection on the collected original image to obtain a variable traffic information sign detection result;

[0007] Mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set to determine a space of interest in the point cloud space of the target point cloud set, wherein the target point cloud set is a point cloud set among multiple point cloud sets that matches the original image;

[0008] Target point cloud data is identified in the space of interest, where the target point cloud data is point cloud data representing a variable traffic information sign.

[0009] In a possible implementation manner, mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set to determine the space of interest in the point cloud space of the target point cloud set includes:

[0010] Extracting bounding box corner points from the variable traffic information sign detection result;

[0011] Based on the conversion parameters between the original image coordinate system and the coordinate system of the target point cloud set, mapping the bounding box corner points to rays in the point cloud space of the target point cloud set;

[0012] The space of interest is defined in the point cloud space of the target point cloud set based on the ray.

[0013] In a possible implementation manner, the performing variable traffic information sign detection on the collected original image to obtain the variable traffic information sign detection result includes:

[0014] Inputting the collected original image into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection, and obtaining a variable traffic information sign detection result;

[0015] The variable traffic information sign detection model is trained in the following way:

[0016] Extracting the location of the variable traffic information sign from the acquired map data;

[0017] Determining the scope of the search area based on the location of the variable traffic information sign;

[0018] Querying image shooting information, determining images shot within the search area, and obtaining image samples for training, wherein the image samples are images containing variable traffic information signs;

[0019] The image samples are used to perform model training to obtain the variable traffic information sign detection model.

[0020] In a possible implementation manner, the using the image samples to perform model training to obtain the variable traffic information sign detection model includes:

[0021] Annotating the image sample to obtain an image sample containing an annotation frame;

[0022] Identifying an image sample to be enhanced from the image sample including the marked frame, the image sample to be enhanced being an image sample whose image area selected by the marked frame meets a preset image area requirement;

[0023] For the image sample to be enhanced, a cropping region range is determined based on the location of the marked box thereof, and the image sample to be enhanced is cropped based on the cropping region range to obtain a cropped image;

[0024] Select a target image from the image data set, and overlay the cropped image onto the target image to generate a new image sample containing a marked box;

[0025] The variable traffic information sign detection model is obtained by using other image samples containing marked frames except the image samples to be enhanced and the new image samples containing marked frames to perform model training.

[0026] In a possible implementation, the variable traffic information sign detection model includes a backbone network, a neck network and a detection head, and the collected original image is input into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection, and obtain a variable traffic information sign detection result, including:

[0027] Inputting the original image into the backbone network to extract image features and obtain image features;

[0028] Inputting the image features into the neck network to extract features of different scales and fusing the features of different scales to obtain fused features;

[0029] The fused features are input into the detection head to perform variable traffic information sign detection to obtain the variable traffic information sign detection result.

[0030] In a possible implementation manner, identifying target point cloud data in the space of interest includes:

[0031] Extracting point cloud data in the space of interest from the target point cloud set as candidate point cloud data;

[0032] Performing clustering processing on the candidate point cloud data to obtain point cloud clusters;

[0033] Extracting cluster features of the point cloud cluster, and determining a target point cloud cluster based on the extracted cluster features, wherein the cluster features include one or more of a surface feature and a density feature;

[0034] The point cloud data in the target point cloud cluster is determined as the target point cloud data.

[0035] In a possible implementation manner, before mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set, the method further includes:

[0036] Calculating a time difference between an image timestamp of the original image and acquisition timestamps of the plurality of point cloud sets;

[0037] Based on the calculated time difference, a target point cloud set matching the original image is determined.

[0038] In a second aspect, an embodiment of the present application provides a variable traffic information sign recognition device, comprising:

[0039] A detection module is used to detect variable traffic information signs on the collected original image and obtain a detection result of the variable traffic information sign;

[0040] A mapping module, used for mapping the detection result of the variable traffic information sign from the original image coordinate system to the coordinate system of the target point cloud set, so as to determine the space of interest in the point cloud space of the target point cloud set, wherein the target point cloud set is a point cloud set among multiple point cloud sets that matches the original image;

[0041] The identification module is used to identify target point cloud data in the space of interest, wherein the target point cloud data is point cloud data representing a variable traffic information sign.

[0042] In a possible implementation manner, the mapping module is specifically used to:

[0043] Extracting bounding box corner points from the variable traffic information sign detection result;

[0044] Based on the conversion parameters between the original image coordinate system and the coordinate system of the target point cloud set, mapping the bounding box corner points to rays in the point cloud space of the target point cloud set;

[0045] The space of interest is defined in the point cloud space of the target point cloud set based on the ray.

[0046] In a possible implementation manner, the detection module is specifically used to:

[0047] Inputting the collected original image into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection, and obtaining a variable traffic information sign detection result;

[0048] The variable traffic information sign detection model is trained in the following way:

[0049] Extracting the location of the variable traffic information sign from the acquired map data;

[0050] Determining the scope of the search area based on the location of the variable traffic information sign;

[0051] Querying image shooting information, determining images shot within the search area, and obtaining image samples for training, wherein the image samples are images containing variable traffic information signs;

[0052] The image samples are used to perform model training to obtain the variable traffic information sign detection model.

[0053] In a possible implementation manner, the using the image samples to perform model training to obtain the variable traffic information sign detection model includes:

[0054] Annotating the image sample to obtain an image sample containing an annotation frame;

[0055] Identifying an image sample to be enhanced from the image sample including the marked frame, the image sample to be enhanced being an image sample whose image area selected by the marked frame meets a preset image area requirement;

[0056] For the image sample to be enhanced, a cropping region range is determined based on the location of the marked box thereof, and the image sample to be enhanced is cropped based on the cropping region range to obtain a cropped image;

[0057] Select a target image from the image data set, and overlay the cropped image onto the target image to generate a new image sample containing a marked box;

[0058] The variable traffic information sign detection model is obtained by using other image samples containing marked frames except the image samples to be enhanced and the new image samples containing marked frames to perform model training.

[0059] In a possible implementation, the variable traffic information sign detection model includes a backbone network, a neck network and a detection head, and the detection module is specifically used for:

[0060] Inputting the original image into the backbone network to extract image features and obtain image features;

[0061] Inputting the image features into the neck network to extract features of different scales and fusing the features of different scales to obtain fused features;

[0062] The fused features are input into the detection head to perform variable traffic information sign detection to obtain the variable traffic information sign detection result.

[0063] In a possible implementation manner, the identification module is specifically used to:

[0064] Extracting point cloud data in the space of interest from the target point cloud set as candidate point cloud data;

[0065] Performing clustering processing on the candidate point cloud data to obtain point cloud clusters;

[0066] Extracting cluster features of the point cloud cluster, and determining a target point cloud cluster based on the extracted cluster features, wherein the cluster features include one or more of a surface feature and a density feature;

[0067] The point cloud data in the target point cloud cluster is determined as the target point cloud data.

[0068] In a possible implementation manner, the variable traffic information sign recognition device is further used for:

[0069] Calculating a time difference between an image timestamp of the original image and acquisition timestamps of the plurality of point cloud sets;

[0070] Based on the calculated time difference, a target point cloud set matching the original image is determined.

[0071] In a third aspect, an embodiment of the present application provides a variable traffic information sign recognition device, including: a memory, a processor;

[0072] The memory stores computer-executable instructions;

[0073] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0075] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0076] The variable traffic information sign recognition method, device, equipment, medium and program product provided in the embodiment of the present application can obtain the variable traffic information sign detection result by performing variable traffic information sign detection on the collected original image. Then, the variable traffic information sign detection result can be mapped from the original image coordinate system to the coordinate system of the target point cloud set that matches the original image, so that the space of interest can be determined in the point cloud space of the target point cloud set, and then the target point cloud data representing the variable traffic information sign can be identified in the space of interest. The technical solution of the present application applies the image detection result to the point cloud segmentation, which makes up for the disadvantage that the point cloud data does not have the color and texture information of the object, and maps the image detection result to the three-dimensional space of the point cloud data through coordinate transformation, so as to provide prior information for extracting the variable traffic information sign in the point cloud data, so as to achieve the effect of improving the recognition accuracy of the variable traffic information sign. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] Figure 1 A schematic diagram of a scenario of a variable traffic information sign recognition method provided in this application;

[0079] Figure 2 Schematic diagram of the process of the variable traffic information sign recognition method provided in this application Figure 1 ;

[0080] Figure 3 Schematic diagram of the process of the variable traffic information sign recognition method provided in this application Figure 2 ;

[0081] Figure 4 A schematic diagram of the image enhancement process provided by an embodiment of the present application;

[0082] Figure 5 A schematic diagram of mapping detected bounding box corner points to a point cloud space provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of the process of the variable traffic information sign recognition method provided in this application;

[0084] Figure 7 A schematic diagram of the structure of the variable traffic information sign recognition device provided in this application;

[0085] Figure 8 This is a schematic diagram of the structure of the variable traffic information sign recognition device provided in this application.

[0086] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0087] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0088] The application scenario of the present application is specifically the scenario of identifying variable traffic information signs. A variable traffic information sign is an electronic traffic sign that displays traffic information in real time through an LED display or other display technology. This information includes, but is not limited to, traffic congestion conditions, accident notifications, special event reminders, special area signs, and speed limits. Their main function is to help drivers understand the road conditions ahead in a timely manner, so as to make reasonable driving decisions, avoid traffic congestion, improve road traffic efficiency, and enhance traffic safety. Variable traffic information signs are usually installed above the road (such as on a gantry on a highway) or on the side of the road to ensure that the driver can clearly see the displayed information while driving.

[0089] Variable traffic information signs are similar in shape and size to ordinary traffic signs. Therefore, in the process of automated high-precision map production, previous recognition algorithms found it difficult to distinguish between ordinary traffic signs and variable traffic information signs.

[0090] There are two methods to identify variable traffic information signs: using point cloud data for identification and using image data for identification.

[0091] The recognition of variable traffic information signs using point cloud data mainly depends on the intensity information and geometric features of the point cloud. The point cloud data representing the variable traffic information sign is identified by combining the intensity information and geometric features. The identified variable traffic information sign point cloud data is vectorized and the vectorization result is stored in vector data format. These data can be used for applications such as high-precision map construction and traffic management systems.

[0092] Using image data to identify variable traffic information signs can be to use an image detection model to identify variable traffic information signs in an image and obtain image detection results. By using the internal and external parameters of the camera, the image detection results can be converted into three-dimensional space. Coordinate system conversion is required to convert image coordinates into camera coordinates and then into world coordinates. Then, the image detection results in three-dimensional space are vectorized.

[0093] In the process of identifying variable traffic information signs, using point cloud data and image data each has its own advantages, disadvantages and challenges:

[0094] Since point cloud data mainly relies on geometric and intensity information, there are similarities in the geometric shapes of ordinary traffic signs and variable traffic information signs. This similarity can easily lead to misidentification, that is, misidentifying ordinary traffic signs as variable traffic information signs.

[0095] Although image data provides rich color and texture information that can help distinguish between ordinary traffic signs and variable traffic information signs, in the process of converting image detection results into three-dimensional space, due to camera pose errors, image distortion and depth estimation errors, the data accuracy of the extracted variable traffic information signs is difficult to meet the requirements of high-precision maps.

[0096] The variable traffic information sign recognition method provided by the present application can obtain the variable traffic information sign detection result by performing variable traffic information sign detection on the collected original image. Then, the variable traffic information sign detection result can be mapped from the original image coordinate system to the coordinate system of the target point cloud set that matches the original image, so that the space of interest can be determined in the point cloud space of the target point cloud set, and then the target point cloud data representing the variable traffic information sign can be identified in the space of interest. The technical solution of the present application applies the image detection result to the point cloud segmentation, which makes up for the disadvantage that the point cloud data does not have the color and texture information of the object, and maps the image detection result to the three-dimensional space of the point cloud data through coordinate transformation, which provides prior information for extracting the variable traffic information sign in the point cloud data, and solves the current technical problem of low recognition accuracy of the variable traffic information sign.

[0097] Figure 1 A schematic diagram of a scene of a variable traffic information sign recognition method provided in this application, such as Figure 1 As shown, the camera and the laser radar can be used to collect environmental data at the same time to generate image data and point cloud data. In the terminal device 101, the variable traffic information sign is recognized in combination with the image data and the point cloud data to improve the recognition accuracy of the variable traffic information sign. The terminal device can be a computing unit in an intelligent transportation system, such as a vehicle.

[0098] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0099] Figure 2 Schematic diagram of the process of the variable traffic information sign recognition method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0100] S201, performing variable traffic information sign detection on the collected original image to obtain a variable traffic information sign detection result.

[0101] In an embodiment of the present application, image processing technology or a deep learning model can be used to detect variable traffic information signs in the original image to obtain variable traffic information sign detection results.

[0102] S202, mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set to determine the space of interest in the point cloud space of the target point cloud set, where the target point cloud set is a point cloud set in multiple point cloud sets that matches the original image.

[0103] Based on the conversion parameters between the image coordinate system of the original image and the coordinate system of the target point cloud set, the variable traffic information sign detection result can be mapped to the point cloud space of the target point cloud set, so that the space of interest can be determined in the point cloud space. The determination of the space of interest is based on the mapped variable traffic information sign detection result. The space of interest corresponds to the position of the variable traffic information sign detected in the original image, is a three-dimensional space area that needs to be paid attention to in the point cloud space, and contains point cloud data related to the variable traffic information sign.

[0104] S203: Identify target point cloud data in the space of interest, where the target point cloud data is point cloud data representing a variable traffic information sign.

[0105] The geometric features and intensity information of the point cloud data are analyzed in the space of interest, and the point cloud data representing the variable traffic information sign is identified.

[0106] This application projects the image detection results into the point cloud space, which makes up for the lack of object color and texture information in the point cloud data. It can improve the recognition effect of variable traffic information signs while ensuring that the requirements of high-precision maps are met.

[0107] The variable traffic information sign recognition method provided in the embodiment of the present application can obtain the variable traffic information sign detection result by performing variable traffic information sign detection on the collected original image. Then, the variable traffic information sign detection result can be mapped from the original image coordinate system to the coordinate system of the target point cloud set that matches the original image, so that the space of interest can be determined in the point cloud space of the target point cloud set, and then the target point cloud data representing the variable traffic information sign can be identified in the space of interest. The technical solution of the present application applies the image detection result to the point cloud segmentation, which makes up for the disadvantage that the point cloud data does not have the color and texture information of the object, and maps the image detection result to the three-dimensional space of the point cloud data through coordinate transformation, so as to provide prior information for extracting the variable traffic information sign in the point cloud data, so as to achieve the effect of improving the recognition accuracy of the variable traffic information sign.

[0108] Figure 3 Schematic diagram of the process of the variable traffic information sign recognition method provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, a variable traffic information sign recognition method is described in detail, and the method includes:

[0109] S301, performing variable traffic information sign detection on the collected original image to obtain a variable traffic information sign detection result.

[0110] In a possible implementation, performing variable traffic information sign detection on the collected original image to obtain a variable traffic information sign detection result may specifically include:

[0111] The collected original image is input into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection and obtain the variable traffic information sign detection result.

[0112] Among them, the variable traffic information sign detection model is trained in the following way:

[0113] Extracting the location of the variable traffic information sign from the acquired map data;

[0114] Determine the scope of the search area based on the location of the variable traffic information sign;

[0115] Querying image shooting information, determining images shot within the search area, and obtaining image samples for training, where the image samples are images containing variable traffic information signs;

[0116] Image samples are used for model training to obtain a variable traffic information sign detection model.

[0117] Map data can be obtained from a variety of sources, such as open source map platforms, geographic information public service platforms, or professional map data providers. When obtaining map data, relevant laws, regulations, and privacy policies must be followed to ensure the legality and security of the data.

[0118] In this embodiment, map data can be obtained, and the location of the variable traffic information sign can be extracted from the obtained map data, assuming it is location P. Based on the location of the variable traffic information sign, a reasonable search area range is determined. For example, a circular area with location P as the center and R as the radius can be used as the search area range. Images captured within the search area range are obtained. Specifically, image capture information can be queried. The image capture information records the image identifier and the image capture location point. Based on the image capture information, the target capture location point that falls within the search area range can be determined, and then the image captured at the target capture location point can be obtained. Images containing the variable traffic information sign are screened out from these images as image samples. Model training is performed using the collected image samples. Among them, images can be captured by autonomous driving vehicles.

[0119] In a possible implementation, the variable traffic information sign detection model includes a backbone network, a neck network, and a detection head. The collected original image is input into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection, and the variable traffic information sign detection result is obtained, which may specifically include:

[0120] Input the original image into the backbone network to extract image features and obtain image features;

[0121] Input the image features into the neck network to extract features of different scales and fuse the features of different scales to obtain fused features;

[0122] The fused features are input into the detection head to detect the variable traffic information sign and obtain the variable traffic information sign detection result.

[0123] In this embodiment, the original image can be input into the backbone network, and after multiple layers of convolution and pooling operations, basic image features can be extracted. The neck network can further process the image features extracted by the backbone network, extract features of different scales and fuse them. The purpose of extracting features of different scales is to enhance the model's detection ability for targets of different sizes. Feature maps of different scales can help the model capture key information of large objects, medium objects and small objects respectively. The detection head is responsible for the final target detection task and generates variable traffic information sign detection results based on the fused features.

[0124] In a possible implementation, image samples are used to perform model training to obtain a variable traffic information sign detection model, including:

[0125] Annotate the image sample to obtain the image sample containing the annotation box;

[0126] Identify an image sample to be enhanced from the image sample containing the annotation frame, the image sample to be enhanced being an image sample whose image area selected by the annotation frame meets the preset image area requirement;

[0127] For the image sample to be enhanced, a cropping region range is determined based on the location of the marked box thereof, and the image sample to be enhanced is cropped based on the cropping region range to obtain a cropped image;

[0128] Select a target image from the image dataset, overlay the cropped image onto the target image to generate a new image sample containing a labeled box;

[0129] The model is trained by using other image samples containing annotated frames except the image samples to be enhanced, and new image samples containing annotated frames to obtain a variable traffic information sign detection model.

[0130] In this embodiment, an annotation tool can be used to annotate the image sample, and an annotation box can be drawn on the image sample to mark the variable traffic information sign therein. The annotation box can be represented by two corner points, the upper left corner and the lower right corner. Each image sample can correspond to a text file with the same name, and the text file is used to store the annotation information. By annotating the image sample, an image sample containing the annotation box can be obtained.

[0131] Before model training, preprocessing can be performed. In the case of poor small target detection results, some labeled image samples can be enhanced. By migrating small targets to other images, the diversity and number of small targets in the training data can be increased, thereby improving the model's ability to recognize small targets.

[0132] Reference Figure 4 As shown, it is a schematic diagram of the image enhancement process provided in an embodiment of the present application, by cutting out a cropped image containing a small target from an image sample containing a small target of a variable traffic information sign, that is, an image sample to be enhanced (a small target can be defined as a target whose pixel area of ​​the image area selected by the annotation box is less than 20×20), and overlaying the cropped image to the same position in other images, which means that the original content will be overwritten by the cropped image, and a new image sample containing the small target is generated to participate in model training.

[0133] In a possible implementation manner, before mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set, the variable traffic information sign recognition method of the embodiment of the present application may further specifically include:

[0134] Calculating the time difference between the image timestamp of the original image and the acquisition timestamps of the plurality of point cloud sets;

[0135] Based on the calculated time difference, a target point cloud set matching the original image is determined.

[0136] Each original image and point cloud set usually carries a timestamp when it is collected. These timestamps are used to record the specific time of data collection. The image timestamp of the original image can be extracted. Traverse each point cloud set and extract its acquisition timestamp. Calculate the time difference between the image timestamp of the original image and the acquisition timestamp of each point cloud set. The point cloud set whose time difference meets the preset difference condition can be selected as the matching target point cloud set. For example, the point cloud set closest to the image timestamp of the original image can be selected as the matching target point cloud set. The point cloud set with the smallest time difference is selected because it is most likely to be collected at the same time or almost at the same time as the image, ensuring the consistency of the two in space and time.

[0137] S302: extracting the corner points of the bounding box from the detection result of the variable traffic information sign.

[0138] The variable traffic sign detection model that has been pre-trained is used to detect variable traffic sign on the input original image. The variable traffic sign detection model can output a series of bounding boxes to represent the detected targets. The variable traffic sign detection result can include the pixel coordinate values ​​of the corner points of the bounding box and the confidence of the bounding box.

[0139] S303 . Based on the conversion parameters between the original image coordinate system and the coordinate system of the target point cloud set, map the corner points of the bounding box to rays in the point cloud space of the target point cloud set, where the target point cloud set is a point cloud set among the multiple point cloud sets that matches the original image.

[0140] In the embodiment of the present application, the camera internal and external parameters can be used to map the bounding box corner points in the variable traffic information sign detection result to three-dimensional rays starting from the camera center in the point cloud space of the target point cloud set.

[0141] S304 : Define a space of interest in the point cloud space of the target point cloud set based on the ray.

[0142] Reference Figure 5 As shown, it is a schematic diagram of mapping the detected bounding box corner points to the point cloud space provided by an embodiment of the present application. The four corner points of the bounding box are mapped to four three-dimensional rays with the same starting point in the point cloud space. By specifying the length of the ray, a tetrahedron can be obtained. This tetrahedron defines a spatial region, namely, the space of interest.

[0143] The variable traffic information sign detection results are mapped to the point cloud space, and in the point cloud space, the space of interest is defined based on the mapped variable traffic information sign detection results. In the above scheme, the corner points of the bounding box are mapped. In another embodiment, the center point and size information of the bounding box can be determined first, and then this information can be mapped to the point cloud space to define the space of interest. For example, in the point cloud space, the mapped center point can be used as the center of the space of interest, and the spatial size and shape of the space of interest can be defined according to the mapped size information.

[0144] In addition, variable traffic information sign detection can be performed on the collected original image, and feature points of the variable traffic information sign in the original image can be extracted. These feature points provide key geometric information about the variable traffic information sign, and then the extracted feature points can be mapped to the point cloud space to define the space of interest.

[0145] S305 . Identify target point cloud data in the space of interest, where the target point cloud data is point cloud data representing a variable traffic information sign.

[0146] In a possible implementation, identifying target point cloud data in the space of interest may specifically include:

[0147] Extracting point cloud data in the space of interest from the target point cloud set as candidate point cloud data;

[0148] Perform clustering on the candidate point cloud data to obtain point cloud clusters;

[0149] Extracting cluster features of the point cloud cluster, and determining the target point cloud cluster based on the extracted cluster features, wherein the cluster features include one or more of surface features and density features;

[0150] The point cloud data in the target point cloud cluster is determined as the target point cloud data.

[0151] Among them, the nearest neighbor clustering algorithm can be used to cluster the candidate point cloud data to obtain point cloud clusters. Based on the extracted cluster features, the target point cloud clusters can be screened out and processed according to the following steps:

[0152] 1. Use the SOR filtering algorithm to denoise each point cloud cluster.

[0153] 2. Use the region growing algorithm to fit the point cloud cluster to a plane and obtain the normal vector of the fitting plane. Calculate the angle between the normal vector and the GPS trajectory vector adjacent to the point cloud cluster. If the angle is greater than 40°, filter out the point cloud cluster.

[0154] 3. Project the point cloud cluster to the XOY plane, perform principal component analysis on the projected point cloud cluster, calculate the major axis and minor axis (thickness), and filter out point cloud clusters with a major axis less than 0.2 meters or a thickness greater than 0.5 meters.

[0155] 4. Calculate the distance from the center point of the point cloud cluster to the road surface, and filter out point cloud clusters with a distance less than 1.5 meters.

[0156] 5. Calculate the density of the point cloud cluster: divide the total number of point cloud clusters by the major axis to obtain the density in the main direction of the point cloud cluster (that is, the number of point clouds contained in 1 meter), and filter out the point cloud clusters with a density less than 20.

[0157] After extracting the point cloud data representing the variable traffic information sign, the point cloud data representing the variable traffic information sign can be vectorized, the least squares method can be used to perform plane fitting on these target point cloud data, the α-shape algorithm can be used to extract the contour points of the target point cloud data, and the contour point coordinates and the normal vector of the fitting plane can be output as vectorization results.

[0158] Reference Figure 6 As shown, it is a schematic diagram of the process of the variable traffic information sign recognition method provided by the present application, collecting image samples for training, then annotating the variable traffic information sign in the image samples, and using the annotated image samples to train the variable traffic information sign detection model. Using the variable traffic information sign detection model, the input original image is subjected to variable traffic information sign detection to obtain the variable traffic information sign detection result. From the point cloud data set, the target point cloud set matching the input original image is searched, and based on the conversion parameters between the image coordinate system of the original image and the coordinate system of the target point cloud set, the bounding box corner points in the variable traffic information sign detection result are mapped to rays in the point cloud space of the target point cloud set, and the ray length can be limited. Four rays are used to form a tetrahedron, and this tetrahedron defines a spatial region, namely, the space of interest. The point cloud data in the tetrahedron space is extracted as the candidate point cloud data of the variable traffic information sign. Then, the candidate point cloud data is clustered, the cluster features of the cluster clusters are extracted, and then the point cloud clusters are screened according to the cluster features, and the point cloud data in the target point cloud cluster is determined as the point cloud data representing the variable traffic information sign. Finally, the extracted point cloud data representing the variable traffic information sign is vectorized.

[0159] In order to solve the problem of misidentifying ordinary traffic signs as variable traffic information signs, this application introduces image processing technology to analyze the texture and color information in the image, which can more accurately distinguish ordinary traffic signs from variable traffic information signs. In addition, by using the internal and external parameters of the camera, the image detection results are mapped into the point cloud space, which not only enhances the semantic information of the point cloud data, but also significantly improves the accuracy of distinguishing ordinary traffic signs from variable traffic information signs.

[0160] In view of the fact that vectorization processing based solely on image detection results will result in low extraction accuracy, this application adopts the strategy of mapping image detection results to point cloud space. This method not only retains the accuracy of image detection, but also combines the high-precision characteristics of point cloud data, effectively avoiding the accuracy loss that may occur during the vectorization process, thereby achieving more accurate recognition of variable traffic information signs.

[0161] The variable traffic information sign recognition method provided in the embodiment of the present application can obtain the variable traffic information sign detection result by performing variable traffic information sign detection on the collected original image. Then, the variable traffic information sign detection result can be mapped from the original image coordinate system to the coordinate system of the target point cloud set that matches the original image, so that the space of interest can be determined in the point cloud space of the target point cloud set, and then the target point cloud data representing the variable traffic information sign can be identified in the space of interest. The technical solution of the present application applies the image detection result to the point cloud segmentation, which makes up for the disadvantage that the point cloud data does not have the color and texture information of the object, and maps the image detection result to the three-dimensional space of the point cloud data through coordinate transformation, so as to provide prior information for extracting the variable traffic information sign in the point cloud data, so as to achieve the effect of improving the recognition accuracy of the variable traffic information sign.

[0162] Figure 7 The schematic diagram of the structure of the variable traffic information sign recognition device provided in this application is as follows: Figure 7 As shown, the variable traffic information sign recognition device 70 provided in this embodiment includes:

[0163] The detection module 701 is used to detect the variable traffic information sign on the collected original image and obtain the variable traffic information sign detection result;

[0164] A mapping module 702 is used to map the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set to determine the space of interest in the point cloud space of the target point cloud set, where the target point cloud set is a point cloud set in the plurality of point cloud sets that matches the original image;

[0165] The identification module 703 is used to identify target point cloud data in the space of interest, where the target point cloud data is point cloud data representing a variable traffic information sign.

[0166] In a possible implementation, the mapping module is specifically used for:

[0167] Extract bounding box corner points from the variable traffic information sign detection results;

[0168] Based on the transformation parameters between the original image coordinate system and the coordinate system of the target point cloud set, the bounding box corner points are mapped to rays in the point cloud space of the target point cloud set;

[0169] A space of interest is defined in the point cloud space of the target point cloud set based on the ray.

[0170] In a possible implementation, the detection module is specifically used to:

[0171] Inputting the collected original image into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection and obtain a variable traffic information sign detection result;

[0172] Among them, the variable traffic information sign detection model is trained in the following way:

[0173] Extracting the location of the variable traffic information sign from the acquired map data;

[0174] Determine the scope of the search area based on the location of the variable traffic information sign;

[0175] Querying image shooting information, determining images shot within the search area, and obtaining image samples for training, where the image samples are images containing variable traffic information signs;

[0176] Image samples are used for model training to obtain a variable traffic information sign detection model.

[0177] In a possible implementation, image samples are used to perform model training to obtain a variable traffic information sign detection model, including:

[0178] Annotate the image sample to obtain the image sample containing the annotation box;

[0179] Identify an image sample to be enhanced from the image sample containing the annotation frame, the image sample to be enhanced being an image sample whose image area selected by the annotation frame meets the preset image area requirement;

[0180] For the image sample to be enhanced, a cropping region range is determined based on the location of the marked box thereof, and the image sample to be enhanced is cropped based on the cropping region range to obtain a cropped image;

[0181] Select a target image from the image dataset, overlay the cropped image onto the target image to generate a new image sample containing a labeled box;

[0182] The model is trained by using other image samples containing annotated frames except the image samples to be enhanced, and new image samples containing annotated frames to obtain a variable traffic information sign detection model.

[0183] In a possible implementation, the variable traffic information sign detection model includes a backbone network, a neck network and a detection head, and the detection module is specifically used for:

[0184] Input the original image into the backbone network to extract image features and obtain image features;

[0185] Input the image features into the neck network to extract features of different scales and fuse the features of different scales to obtain fused features;

[0186] The fused features are input into the detection head to detect the variable traffic information sign and obtain the variable traffic information sign detection result.

[0187] In a possible implementation, the identification module is specifically used to:

[0188] Extracting point cloud data in the space of interest from the target point cloud set as candidate point cloud data;

[0189] Perform clustering on the candidate point cloud data to obtain point cloud clusters;

[0190] Extracting cluster features of the point cloud cluster, and determining the target point cloud cluster based on the extracted cluster features, wherein the cluster features include one or more of surface features and density features;

[0191] The point cloud data in the target point cloud cluster is determined as the target point cloud data.

[0192] In a possible implementation, the variable traffic information sign recognition device is further used for:

[0193] Calculating the time difference between the image timestamp of the original image and the acquisition timestamps of the plurality of point cloud sets;

[0194] Based on the calculated time difference, a target point cloud set matching the original image is determined.

[0195] The variable traffic information sign recognition device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be described in detail here.

[0196] Figure 8 This is a schematic diagram of the structure of the variable traffic information sign recognition device provided in this application. Figure 8 As shown, the variable traffic information sign recognition device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus.

[0197] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0198] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0199] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0200] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0201] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0202] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0203] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0204] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0205] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0206] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0209] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0210] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0211] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for identifying a variable traffic information sign, characterized in that: include: Perform variable traffic information sign detection on the collected original image to obtain a variable traffic information sign detection result; Mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of a target point cloud set to determine a space of interest in the point cloud space of the target point cloud set, wherein the target point cloud set is a point cloud set among multiple point cloud sets that matches the original image; Target point cloud data is identified in the space of interest, where the target point cloud data is point cloud data representing a variable traffic information sign.

2. The method according to claim 1, characterized in that Mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set to determine the space of interest in the point cloud space of the target point cloud set includes: Extracting bounding box corner points from the variable traffic information sign detection result; Based on the conversion parameters between the original image coordinate system and the coordinate system of the target point cloud set, mapping the bounding box corner points to rays in the point cloud space of the target point cloud set; The space of interest is defined in the point cloud space of the target point cloud set based on the ray.

3. The method according to claim 1, characterized in that The detecting of variable traffic information signs on the collected original image to obtain the detection result of variable traffic information signs includes: Inputting the collected original image into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection, and obtaining a variable traffic information sign detection result; The variable traffic information sign detection model is trained in the following way: Extracting the location of the variable traffic information sign from the acquired map data; Determining the scope of the search area based on the location of the variable traffic information sign; Querying image shooting information, determining images shot within the search area, and obtaining image samples for training, wherein the image samples are images containing variable traffic information signs; The image samples are used to perform model training to obtain the variable traffic information sign detection model.

4. The method according to claim 3, characterized in that The method of using the image samples to perform model training to obtain the variable traffic information sign detection model includes: Annotating the image sample to obtain an image sample containing an annotation frame; Identifying an image sample to be enhanced from the image sample including the marked frame, the image sample to be enhanced being an image sample whose image area selected by the marked frame meets a preset image area requirement; For the image sample to be enhanced, a cropping region range is determined based on the location of the marked box thereof, and the image sample to be enhanced is cropped based on the cropping region range to obtain a cropped image; Select a target image from the image data set, and overlay the cropped image onto the target image to generate a new image sample containing a marked box; The variable traffic information sign detection model is obtained by using other image samples containing marked frames except the image samples to be enhanced and the new image samples containing marked frames to perform model training.

5. The method according to any one of claims 3-4, characterized in that: The variable traffic information sign detection model comprises a backbone network, a neck network and a detection head. The collected original image is input into a pre-trained variable traffic information sign detection model to perform variable traffic information sign detection, and a variable traffic information sign detection result is obtained, including: Inputting the original image into the backbone network to extract image features and obtain image features; Inputting the image features into the neck network to extract features of different scales and fusing the features of different scales to obtain fused features; The fused features are input into the detection head to perform variable traffic information sign detection to obtain the variable traffic information sign detection result.

6. The method according to any one of claims 1 to 4, characterized in that: The identifying target point cloud data in the space of interest includes: Extracting point cloud data in the space of interest from the target point cloud set as candidate point cloud data; Performing clustering processing on the candidate point cloud data to obtain point cloud clusters; Extracting cluster features of the point cloud cluster, and determining a target point cloud cluster based on the extracted cluster features, wherein the cluster features include one or more of a surface feature and a density feature; The point cloud data in the target point cloud cluster is determined as the target point cloud data.

7. The method according to any one of claims 1 to 4, characterized in that: Before mapping the variable traffic information sign detection result from the original image coordinate system to the coordinate system of the target point cloud set, the method further includes: Calculating a time difference between an image timestamp of the original image and acquisition timestamps of the plurality of point cloud sets; Based on the calculated time difference, a target point cloud set matching the original image is determined.

8. A variable traffic information sign recognition device, characterized in that: include: A detection module is used to detect variable traffic information signs on the collected original image and obtain a detection result of the variable traffic information sign; A mapping module, used for mapping the detection result of the variable traffic information sign from the original image coordinate system to the coordinate system of the target point cloud set, so as to determine the space of interest in the point cloud space of the target point cloud set, wherein the target point cloud set is a point cloud set among multiple point cloud sets that matches the original image; The identification module is used to identify target point cloud data in the space of interest, wherein the target point cloud data is point cloud data representing a variable traffic information sign.

9. A variable traffic information sign recognition device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor; and / or, The computer program product comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images