An unmanned container truck road sign recognition positioning method and system and a storage medium

Abstract

The present application relates to a kind of unmanned truck road sign identification positioning method, system and storage medium, the method comprises: S1. Vehicle travels in port, based on vehicle camera, obtain port image data information, the port image data information is preprocessed, based on vehicle laser radar, obtain port point cloud data information, the port point cloud data information is preprocessed, output processed port image data information and processed port point cloud data information;S2. The processed port image data information is input to detection identification network and carries out the detection and identification of sign, and outputs sign label information and sign image position information.The present application not only has high sign identification precision, and identification speed is fast, and positioning error is small, facilitate the transportation and driving safety of unmanned truck.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned container trucks, in particular to a road sign recognition and positioning method and system for unmanned container trucks and a storage medium. BACKGROUND

[0002] Road signs play a key role in traffic safety and are an important factor in guiding autonomous driving of unmanned vehicles. When an unmanned vehicle is driving on a road, it needs to consider the information of road signs on the road at the decision and planning level, travel according to the information provided by the road signs, and correct the position using related road sign information to improve the robustness and accuracy of positioning.

[0003] The first prior art provides a road sign recognition system, which comprises an industrial computer, an edge computing box and an industrial camera installed on an unmanned vehicle; the industrial computer is in communication connection with the edge computing box and the industrial camera; the industrial camera is used to capture a front road image of the unmanned vehicle; the front road image includes a road sign; the industrial computer is used to forward the front road image to the edge computing box; the edge computing box comprises a target detection module; the target detection module is used to detect the front road image by using a target detection model to obtain the category of the road sign and the position area of the road sign in the front road image.

[0004] A laser radar installed on the unmanned vehicle and in communication connection with the industrial computer; the laser radar is used to scan to obtain laser radar point cloud data in front of the unmanned vehicle; the edge computing box further comprises a target positioning module, which can identify the sign, but the identification accuracy is not high, the identification speed is slow, and the positioning error is too large. SUMMARY

[0005] In view of the above deficiencies of the prior art, the present application provides a road sign recognition and positioning method and system for unmanned container trucks, which has high sign recognition accuracy, fast recognition speed, small positioning error and facilitates transportation and driving safety of the unmanned container truck.

[0006] To achieve the above objects and other related objects, the technical solutions provided by the present application are as follows:

[0007] A road sign recognition and positioning method for unmanned container trucks, the method comprising:

[0008] S1. The vehicle drives in the port, obtains port image data information based on a vehicle-mounted camera, pre-processes the port image data information, obtains port point cloud data information based on a vehicle-mounted laser radar, pre-processes the port point cloud data information, and outputs the processed port image data information and the processed port point cloud data information;

[0009] S2. The processed port image data information is input into a detection and recognition network for detection and recognition of labels, and label name information and label image position information are output;

[0010] S3. Based on the label image position information and the label name information, a frustum filtering algorithm is used to screen the processed port point cloud data information, and screened port point cloud data information is output;

[0011] S4. The screened port point cloud data information is input into a point cloud label extraction algorithm for label name and label position extraction, and port label position information and port label name information are output.

[0012] Further, in step S2, the input of the processed port image data information into the detection and recognition network for detection and recognition of labels includes:

[0013] S21. The processed port image data information is input into a mobilenet network for detection, and a label region quadruple is output;

[0014] S22. Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the label region quadruple is inversely perspective transformed to obtain a label region image, the label region image is binarized using an OSTU threshold, and a label binarized image is output;

[0015] S23. A 3x3, 5x5 disk erosion operator and a disk expansion operator are respectively applied to the label binarized image, and classification image data information is output;

[0016] S24. The classification image data information is input into a shufflenet network for recognition, and label name information and label image position information are output.

[0017] Further, the pre-processing of the port image data information is to remove duplicate images, perform image channel equalization processing, and perform image equal proportion envelope processing.

[0018] Further, the pre-processing of the port point cloud data information is denoising processing, isolated point removal processing, and intensity filtering processing.

[0019] Further, the outlier point elimination process is to adopt radius filtering to search the number of adjacent point clouds within a preset radius, and if the number of adjacent point clouds is less than a preset threshold, it is determined as an outlier point, and the outlier point of the point cloud is removed.

[0020] Further, the intensity filtering process is to set a preset illumination threshold, and if it is greater than the preset illumination threshold, it is retained, otherwise it is eliminated.

[0021] Further, in step S3, the frustum filtering algorithm includes:

[0022] S31. Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the processed port point cloud data information is projected into the camera coordinate system,

[0023] Pc i =T c v×Pv i ,

[0024] wherein T c v is a coordinate conversion matrix, is a point cloud coordinate matrix, is a point cloud coordinate matrix in the camera coordinate system;

[0025] S32. Based on the point cloud coordinate matrix in the camera coordinate system filtering near points and far points in the point cloud coordinate matrix using an inverse depth function z,

[0026]

[0027] wherein θ min is a near point threshold, θ max is a far point threshold, z(x) is an inverse depth function, and the filtered point cloud coordinate matrix

[0028] S33. Project the filtered point cloud coordinate matrix into the image coordinate system,

[0029]

[0030] wherein v i is the point cloud coordinate in the image coordinate system, T is a conversion function, and K is a conversion factor;

[0031] S34. If v i is in the signboard image position, then is in the frustum, which meets the requirements.

[0032] Further, in step S4, the point cloud signboard extraction algorithm includes:

[0033] S41. Based on the screened port point cloud data information, a Kmeans algorithm accelerated by Kdtree is used to cluster the screened port point cloud data information, and port point cloud clustering matrix data information is output;

[0034] S42. Based on the port point cloud clustering data information, a maximum cluster filtering function is used,

[0035] C im = argmaxC i |, wherein C im is the maximum cluster filtering function, C i is the clustering matrix, and label information and label position information are output.

[0036] In order to achieve the above-mentioned purpose and other related purposes, the present application also provides an unmanned truck road sign identification and positioning system, comprising a computer device programmed or configured to perform the steps of any one of the unmanned truck road sign identification and positioning methods.

[0037] In order to achieve the above-mentioned purpose and other related purposes, the present application also provides a computer readable storage medium having a computer program programmed or configured to perform any one of the unmanned truck road sign identification and positioning methods stored thereon.

[0038] The present application has the following positive effects:

[0039] 1. The present application identifies and locates the port sign by using the vehicle-mounted camera and vehicle-mounted laser radar, which not only improves the identification accuracy, but also reduces the installation cost and industrial cost.

[0040] 2. The present application projects the point cloud data of the port sign into the camera coordinate system for data filtering and judgment, which not only improves the system response speed, but also reduces the positioning error, and facilitates the positioning and driving safety of the unmanned truck. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a schematic diagram of the method flow of the present application;

[0042] Figure 2 is a schematic diagram of the label network detection accuracy of the present application;

[0043] Figure 3 is a schematic diagram of the label recognition accuracy of the present application;

[0044] Figure 4 is a schematic diagram of the identification and positioning of the present application. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present disclosure are described herein below with reference to the accompanying drawings, in which various specific details are set forth to assist in a thorough understanding of these embodiments. It should be understood, however, that these embodiments are only exemplary and that various changes and modifications can be made thereto without departing from the scope and spirit of the present disclosure. Also, the present disclosure is not intended to be limited to the described embodiments but rather includes all possible alternatives and modifications.

[0046] Embodiment 1: As shown in the figure, an unmanned container truck road sign recognition positioning method, the method comprises: Figure 1

[0047] S1. The vehicle travels in the port, based on the vehicle-mounted camera, obtains the port image data information, pre-processes the port image data information, based on the vehicle-mounted laser radar, obtains the port point cloud data information, pre-processes the port point cloud data information, and outputs the processed port image data information and the processed port point cloud data information;

[0048] S2. The processed port image data information is input into a detection recognition network for detection and recognition of the sign, and the sign label information and the sign image position information are output;

[0049] S3. Based on the sign image position information and the sign label information, a frustum filtering algorithm is used to screen the processed port point cloud data information, and the screened port point cloud data information is output;

[0050] S4. The screened port point cloud data information is input into a point cloud sign extraction algorithm for sign label and sign position extraction, and the port sign position information and the port sign label information are output.

[0051] In this embodiment, in step S2, the processed port image data information is input into a detection recognition network for detection and recognition of the sign, which comprises:

[0052] S21. The processed port image data information is input into a mobilenet network for detection, and the region four-tuple of the sign is output;

[0053] S22. Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the region four-tuple of the sign is inversely perspective transformed to obtain a sign region image, the sign region image is binarized using an OSTU threshold value, and a sign binarized image is output;

[0054] S23. A 3x3, 5x5 disk erosion operator and a disk expansion operator are respectively applied to the sign binarized image, and classification image data information is output;

[0055] ​S24. inputting the classified image data information into a shufflenet network for identification, and outputting label name information and label image position information.

[0056] In the embodiment, the port image data information is preprocessed by removing duplicate images, image channel equalization processing, and image equal proportion envelope processing.

[0057] In the embodiment, the port point cloud data information is preprocessed by noise reduction processing, isolated point removal processing, and intensity filtering processing.

[0058] Embodiment 2: Based on the unmanned container truck landmark identification positioning method of embodiment 1, the present application is further described and explained as follows.

[0059] The present application adopts a two-stage identification method to solve the problem of low label accuracy, first, two lightweight mobilenet and shufflenet networks are used to realize detection and id identification functions respectively, ensuring that the total execution time on the CPU is < 150ms, second, the image is scaled using the equal proportion envelope method to reduce the distortion caused by different proportion samples, and the external data is more effectively used for training.

[0060] Due to the use of a double network design, the algorithm inference speed requirement is high, the signboard recognition adopts fp16 quantization, constant equivalent merging, cascade conv merging and other network simplification processing, and uses the Opencv DNN module to deploy the onnx format neural network, the shufflenet used in the present scheme uses the Identity layer which is not supported by Opencv DNN, therefore, the Identity layer is added in the OpenCV 4.6 library published in the present software package.

[0061] In the embodiment, the isolated point removal processing is to use radius filtering to search the number of adjacent point clouds within a preset radius, if the number of adjacent point clouds is less than a preset threshold, it is determined as an isolated point, and the isolated point of the point cloud is removed.

[0062] In the embodiment, the intensity filtering processing is to set a preset illumination threshold, if it is greater than the preset illumination threshold, it is retained, otherwise it is removed.

[0063] As shown in Figure 2 or Figure 3 In the embodiment, in step S3, the frustum filtering algorithm includes:

[0064] S31. based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, projecting the processed port point cloud data information into a camera coordinate system,

[0065] Pc i = T cv x Pv i ,

[0066] where T c v is a coordinate transformation matrix, Pv i is a point cloud coordinate matrix, Pc i is a point cloud coordinate matrix in a camera coordinate system;

[0067] S32. Based on the point cloud coordinate matrix Pc i in the camera coordinate system, filter the near and far points in the point cloud coordinate matrix using an inverse depth function z,

[0068]

[0069] where θ min is a near point threshold, θ max is a far point threshold, z(x) is an inverse depth function, and the filtered point cloud coordinate matrix

[0070] S33. Project the filtered point cloud coordinate matrix into an image coordinate system,

[0071]

[0072] where v i is a point cloud coordinate in the image coordinate system, T is a transformation function, and K is a transformation factor;

[0073] S34. If v i is in the signboard image position, then is within the view cone, meeting the requirements.

[0074] In this embodiment, in step S4, the point cloud signboard extraction algorithm includes:

[0075] S41. Based on the filtered port point cloud data information, use a Kdtree accelerated Kmeans algorithm to cluster the filtered port point cloud data information, and output port point cloud clustering matrix data information;

[0076] S42. Based on the port point cloud clustering data information, use a maximum cluster filtering function,

[0077] C im = argmax C i |, where C im is a maximum cluster filtering function, C i is a clustering matrix, and output signboard label information and signboard position information.

[0078] As Figure 4As shown, the filtering method based on coverage and the center point calculation of the sign: due to the deviation of the installation position of the laser radar and the camera, there may be point clouds extracted by the clustering algorithm above that are not signs, and the system uses a coverage filtering method to ignore the sign when the minimum rectangular bounding box of the point cloud in the image coordinates and the bounding box identified by the image differ too much.

[0079] Finally, by using the feature that the sign is axisymmetric, the center point of the point cloud is calculated as the coordinate of the sign using an average method.

[0080] To achieve the above object and other related objects, the present application further provides an unmanned truck road sign identification and positioning system, comprising a computer device programmed or configured to perform the steps of any one of the unmanned truck road sign identification and positioning methods.

[0081] To achieve the above object and other related objects, the present application further provides a computer readable storage medium having stored thereon a computer program programmed or configured to perform any one of the unmanned truck road sign identification and positioning methods.

[0082] Any reference to memory, storage, database, or other medium used in the embodiments provided herein can include non-volatile and / or volatile memory. Non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] In summary, the present application not only has high sign identification accuracy and fast identification speed, but also has small positioning error, facilitating the transportation of unmanned trucks and driving safety.

[0084] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. An unmanned container truck road sign recognition positioning method, characterized in that, The method comprises: S1. The vehicle travels in the port, obtains port image data information based on the vehicle-mounted camera, pre-processes the port image data information, obtains port point cloud data information based on the vehicle-mounted laser radar, pre-processes the port point cloud data information, and outputs the processed port image data information and the processed port point cloud data information; S2. The processed port image data information is input into a detection and recognition network to detect and recognize the label, and label name information and label image position information are output; S3. Based on the label image position information and the label name information, a frustum filtering algorithm is used to screen the processed port point cloud data information, and screened port point cloud data information is output; S4. The screened port point cloud data information is input into a point cloud label extraction algorithm to extract label name and label position, and port label position information and port label name information are output; In step S2, inputting the processed port image data information into the detection and recognition network to detect and recognize the label comprises: S21. The processed port image data information is input into a mobilenet network for detection, and a label region quadruple is output; S22. Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the label region quadruple is inversely perspective transformed to obtain a label region image, the label region image is binarized by using an OSTU threshold value, and a label binarized image is output; S23. A 3x3 disk erosion operator and a 5x5 disk expansion operator are respectively applied to the label binarized image, and classification image data information is output; S24. The classification image data information is input into a shufflenet network for recognition, and label name information and label image position information are output.

2. The unmanned straddle carrier road sign identification positioning method of claim 1, wherein: The pre-processing of the port image data information is to remove duplicate images, perform image channel equalization processing, and perform image equal proportion envelope processing.

3. The unmanned straddle carrier road sign identification positioning method of claim 1, wherein: The pre-processing of the port point cloud data information is noise reduction processing, isolated point removal processing, and intensity filtering processing.

4. The unmanned straddle carrier road sign identification positioning method of claim 3, wherein: The isolated point removal processing is to use radius filtering to search for the number of adjacent point clouds within a preset radius, and if the number of adjacent point clouds is less than a preset threshold, it is determined that the point cloud is an isolated point, and the isolated point is removed.

5. The unmanned straddle carrier road sign identification positioning method of claim 3, wherein: The intensity filtering processing is to set a preset illumination threshold, and if it is greater than the preset illumination threshold, it is retained, otherwise it is removed.

6. The unmanned straddle carrier road sign identification positioning method of claim 1, wherein, In step S3, the frustum filtering algorithm comprises: S31. Based on the intrinsic and extrinsic parameters of the vehicle-mounted camera, the processed port point cloud data information is projected into a camera coordinate system, Pc i = T c v x Pv i , where T c v is a coordinate transformation matrix, Pv i is a point cloud coordinate matrix, Pc i is a point cloud coordinate matrix in the camera coordinate system; S32. filter the point cloud coordinate matrix Pc in the camera coordinate system based on the inverse depth function z i , filter the near and far points in the point cloud coordinate matrix using the inverse depth function z , Wherein, θ min is a near-point threshold, θ max is a far-point threshold, z(x) is an inverse depth function, and the filtered point cloud coordinate matrix is output. S33. projecting the filtered point cloud coordinate matrix into the image coordinate system, ， wherein v i is the point cloud coordinate in the image coordinate system, T is a conversion function, and K is a conversion factor. S34. If v i In the sign image position, then In the view frustum, the requirements are met.

7. The unmanned straddle carrier road sign identification positioning method of claim 1, wherein, In step S4, the point cloud label extraction algorithm comprises: S41. Based on the screened port point cloud data information, a Kdtree accelerated Kmeans algorithm is used to cluster the screened port point cloud data information, and port point cloud clustering matrix data information is output; S42. Based on the port point cloud clustering matrix data information, a maximum cluster filtering function C im = argmax C i , wherein C im is the maximum cluster filtering function, C i is the clustering matrix, the output label tag information and the label position information.

8. An unmanned straddle carrier road sign identification positioning system comprising a computer device, characterized by, The computer device is programmed or configured to perform the steps of the unmanned container truck road sign recognition positioning method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program programmed or configured to perform the unmanned container truck road sign identification positioning method of any one of claims 1-7.

Images