Signboard identification method and device, equipment and storage medium

Abstract

The invention discloses a signboard identification method, device and equipment and a storage medium, and relates to the technical field of vehicle-mounted binocular vision, and the method comprises the steps: obtaining a to-be-identified signboard; the signboard to be recognized is processed through a target detection algorithm, information of the signboard to be recognized is determined, and the information comprises category information and pixel block semantic information; processing the semantic information of the pixel block to obtain a sub-pixel contour of the signboard to be identified; establishing a first coordinate system and a second coordinate system, and processing the sub-pixel contour of the to-be-identified signboard to obtain a three-dimensional coordinate of a contour point set of the to-be-identified signboard; and processing the three-dimensional coordinates of the contour point set of the to-be-identified signboard to obtain the signboard pose of the to-be-identified signboard, so that all types of traffic signs can be accurately identified, and the method is suitable for spatial pose estimation of signboards in various shapes.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted binocular vision technology, and in particular to a sign recognition method, device, equipment and storage medium. Background Art

[0002] Traffic sign recognition (TSR) plays an important role in autonomous driving systems. The information carried by traffic signs is an important element in the construction of driving maps and the basis for vehicle motion control decisions, such as ensuring that vehicles are driving legally and in compliance with regulations by identifying speed limit signs and controlling the speed in a timely manner.

[0003] In reality, the road space where vehicles travel is equipped with traffic signs of different directions, angles, front and back, and of different types. Accurately identifying the spatial position of the sign is also a major challenge.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a sign recognition method, device, equipment and storage medium, aiming to solve the technical problem of traffic sign recognition and posture estimation during vehicle driving.

[0006] To achieve the above object, the present invention provides a sign recognition method, which comprises the following steps:

[0007] Obtain the sign to be identified;

[0008] Processing the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information;

[0009] Processing the semantic information of the pixel block to obtain the sub-pixel outline of the sign to be identified;

[0010] Establishing a first coordinate system and a second coordinate system, and processing the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified;

[0011] The three-dimensional coordinates of the outline point set of the sign to be identified are processed to obtain the sign position and posture of the sign to be identified.

[0012] In one embodiment, the step of processing the pixel block semantic information to obtain the sub-pixel contour of the sign to be identified includes:

[0013] Performing connected domain processing on the semantic information of the pixel block and circumscribing a convex polygon to obtain a closed contour of the sign to be identified;

[0014] Acquire each edge point in the closed contour, and process along the normal direction of the closed contour in a local window centered at the edge point to obtain a gradient value of each edge point in the closed contour;

[0015] The gradient value is fitted using a Gaussian curve to obtain a sub-pixel profile of the sign to be identified.

[0016] In one embodiment, the step of establishing the first coordinate system and the second coordinate system, and processing the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified includes:

[0017] Establishing a first coordinate system according to a first camera, and establishing a second coordinate system according to a second camera, wherein the first camera and the second camera are arranged on both sides of the vehicle;

[0018] Acquire a first image of the sub-pixel contour in the first coordinate system and a second image of the sub-pixel contour in the second coordinate system;

[0019] Performing contour matching processing on the first image and the second image to obtain matching contours in the first image and the second image;

[0020] The matching contour is processed to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0021] In one embodiment, the step of performing contour matching processing on the first image and the second image to obtain matching contours in the first image and the second image includes:

[0022] Acquire a first preset value and a second preset value of the sub-pixel profile in the first image and a third preset value and a fourth preset value in the second image, wherein the first preset value is greater than the second preset value, and the third preset value is greater than the fourth preset value;

[0023] Contour matching is performed on the contours of the sub-pixel contour in the first image and the second image using the first preset value, the second preset value, the third preset value, and the fourth preset value to obtain matching contours in the first image and the second image.

[0024] In one embodiment, the step of processing the matching contour to obtain the three-dimensional coordinates of the contour point set of the sign to be identified includes:

[0025] Establishing a binocular system coordinate system based on the first coordinate system and the second coordinate system;

[0026] The matching contour is processed based on the binocular system coordinate system to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0027] In one embodiment, the step of processing the three-dimensional coordinates of the contour point set of the sign to be identified to obtain the sign posture of the sign to be identified includes:

[0028] Processing the three-dimensional coordinates of the outline point set of the sign to be identified by principal component analysis to obtain the normal vector of the outline point set of the sign to be identified and the main direction and secondary direction of the outline point set of the sign to be identified;

[0029] Based on the normal vector, the main direction and the secondary direction, a coordinate system of the sign to be identified in the binocular system coordinate system is obtained;

[0030] The coordinate system of the sign to be identified is processed by a conversion matrix to obtain the sign position and posture of the sign to be identified in the vehicle coordinate system.

[0031] In one embodiment, the step of processing the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information, comprises:

[0032] The target classification unit of the target detection algorithm performs target classification on the sign to be identified to obtain category information of the sign to be identified;

[0033] The semantic segmentation unit of the target detection algorithm performs semantic segmentation on the sign to be identified to obtain pixel block semantic information.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a sign recognition device, the device comprising:

[0035] A deep learning module is used to obtain a sign to be identified; process the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information;

[0036] A first processing module is used to process the semantic information of the pixel block to obtain the sub-pixel outline of the sign to be identified;

[0037] A second processing module is used to establish a first coordinate system and a second coordinate system, and process the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified;

[0038] The third processing module is used to process the three-dimensional coordinates of the outline point set of the sign to be identified to obtain the sign position and posture of the sign to be identified.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also proposes a sign recognition device, which includes: a memory, a processor, and a sign recognition program stored in the memory and executable on the processor, wherein the sign recognition program is configured to implement the steps of the sign recognition method described above.

[0040] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a sign recognition program is stored, and when the sign recognition program is executed by a processor, the steps of the sign recognition method described above are implemented.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] Acquire the sign to be identified; process the sign to be identified through the target detection algorithm to determine the information of the sign to be identified, wherein the information includes category information and pixel block semantic information; process the pixel block semantic information to obtain the sub-pixel contour of the sign to be identified; establish a first coordinate system and a second coordinate system, and process the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified; process the three-dimensional coordinates of the contour point set of the sign to be identified to obtain the sign pose of the sign to be identified, which can accurately identify all types of traffic signs and is suitable for the spatial pose estimation of signs of various shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of the first embodiment of the method for identifying a sign in the present application is provided;

[0046] Figure 2 A schematic diagram of a binocular vision system device provided in Example 1 of the sign recognition method of the present application;

[0047] Figure 3A schematic diagram of the detected signboard posture in the vehicle coordinate system provided in the first embodiment of the signboard recognition method of the present application;

[0048] Figure 4 A schematic diagram of a flow chart provided for the second embodiment of the method for identifying a signboard of the present application;

[0049] Figure 5 A flowchart of the third embodiment of the method for identifying a sign in the present application is provided;

[0050] Figure 6 A schematic diagram of an epipolar-corrected image contour provided in Embodiment 3 of the sign recognition method of the present application;

[0051] Figure 7 A schematic diagram of the binocular principle provided in Example 3 of the sign recognition method of the present application;

[0052] Figure 8 This is a schematic diagram of the module structure of the sign recognition device according to an embodiment of the present application;

[0053] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the sign recognition method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a sign recognition device, etc. The following takes the sign recognition device as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the present application embodiment provides a sign recognition method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the sign recognition method of the present application.

[0059] In this embodiment, the sign recognition method includes steps S10 to S30:

[0060] Step S10, obtaining a sign to be identified;

[0061] It should be noted that this strategy only requires two monocular cameras or binocular modules in the vehicle hardware configuration. The binocular sensors can be installed in front of the vehicle or on both sides of the vehicle to identify the sign to be identified, such as Figure 2 As shown, Figure 2 Schematic diagram of binocular vision system device.

[0062] It should be understood that the sign to be recognized generally refers to a traffic sign that has not been recognized or is being recognized in the traffic sign recognition system, providing the driver with necessary traffic information to reduce the probability of traffic accidents and improve road safety.

[0063] Step S20, processing the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information;

[0064] It should be noted that the current vehicle system uses navigation data or combines with the front camera to identify some common traffic signs, which is to classify traffic signs. This is a classification task, and the strategy uses a semantic segmentation-based method to process the signboards.

[0065] It should be understood that the target detection algorithm used in this strategy can be a one-stage target detection algorithm, such as Yolo (You Only Look Once), which means that the class category and bounding box of the object in the image can be identified by "looking" only once, and has shown excellent performance in many classification tasks.

[0066] Category information refers to the classification information of objects in the image, such as no left turn, speed limit 60, etc.

[0067] Pixel block semantic information refers to the semantic label of each pixel block (or region) in the image, indicating what kind of object or part of the scene the region belongs to.

[0068] Furthermore, the step S20 includes:

[0069] The target classification unit of the target detection algorithm classifies the target sign to be identified, and obtains the category information of the target sign to be identified;

[0070] The semantic segmentation unit of the target detection algorithm performs semantic segmentation on the sign to be identified to obtain the semantic information of the pixel block.

[0071] It should be noted that the target classification unit is responsible for determining which category the target in the image belongs to. In the scenario of traffic sign recognition, the target classification unit will analyze the sign to be recognized and determine whether it represents a specific category such as a speed limit sign, a stop sign, or a give way sign. This process involves the classification layer in the deep learning model, which classifies the objects in the input image based on the features learned in the training phase to obtain category information.

[0072] The semantic segmentation unit focuses on each pixel in the image and distinguishes all target objects and background in the image by identifying which category each pixel belongs to. In traffic sign recognition, the semantic segmentation unit will process the sign more carefully and obtain pixel-level classification results, which helps to understand the boundaries and shapes of the sign more accurately.

[0073] YOLACT (You Only Look At CoefficienTs) adopted in this strategy adds a mask branch on the basis of Yolov5, realizing two tasks: target classification and semantic segmentation. Through the vehicle data taken in different weather conditions, scenes and times, YOLACT can accurately obtain the category of the sign and the semantic information of the pixel block after training.

[0074] Step S30, processing the semantic information of the pixel block to obtain the sub-pixel outline of the sign to be identified;

[0075] It should be noted that sub-pixel contour refers to the use of algorithms in image processing to more precisely locate the edges of objects below the pixel level.

[0076] In the specific implementation, the connected domain and the circumscribed convex polygon are calculated for the semantic pixels to obtain the complete and closed outline of the sign, and the sub-pixel outline of the sign to be identified is obtained through sub-pixel edge calculation.

[0077] Since traditional image processing techniques can usually only identify edges at the pixel level, which may limit the accuracy of edge detection, sub-pixel contour technology can provide higher edge positioning accuracy than traditional methods by interpolating between pixels.

[0078] Step S40, establishing a first coordinate system and a second coordinate system, and processing the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified;

[0079] It should be noted that the first coordinate system and the second coordinate system are established according to each camera in the two monocular cameras or binocular modules adopted by this strategy, and the three-dimensional coordinates of the signboard contour point set are obtained by obtaining the sub-pixel contour point set of the signboard and then converting these points from the image space to the three-dimensional space.

[0080] In the specific implementation, the maximum and minimum values ​​of the contour rows in the left and right images are used as screening conditions, and a threshold is set to screen the contours in the left and right images. The contours that meet the threshold conditions are used as a pair of candidate matching pairs, and then the matched contours are processed by the dual-target positioning method to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0081] Step S50, processing the three-dimensional coordinates of the contour point set of the sign to be identified to obtain the sign position and posture of the sign to be identified.

[0082] It should be noted that the signboard posture of the signboard to be identified refers to the position and direction of the signboard to be identified relative to the vehicle coordinate system.

[0083] In the specific implementation, the three-dimensional coordinates of the signboard contour point set to be identified are processed by principal component analysis to obtain the signboard coordinate system in the binocular system coordinate system. The signboard is then transformed according to the conversion matrix from the binocular vision system coordinate system to the vehicle coordinate system (rear axle center) to obtain the signboard pose in the vehicle coordinate system.

[0084] Furthermore, the step S50 includes:

[0085] The three-dimensional coordinates of the contour point set of the sign to be identified are processed by the principal component analysis method to obtain the normal vector of the contour point set of the sign to be identified and the main direction and the secondary direction of the contour point set of the sign to be identified;

[0086] Based on the normal vector, the main direction and the secondary direction, the coordinate system of the sign to be identified in the binocular system coordinate system is obtained;

[0087] The coordinate system of the sign to be identified is processed by the transformation matrix to obtain the position and posture of the sign to be identified in the vehicle coordinate system.

[0088] It should be noted that principal component analysis, or PCA, is a statistical method whose main goal is to reduce the number of features of a data set while retaining the variability of the original data as much as possible. It converts a set of possibly related variables into a set of linearly unrelated variables through orthogonal transformation. These unrelated variables are called principal components.

[0089] Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and select the eigenvectors corresponding to the three largest eigenvalues ​​according to the size of the eigenvalues. These eigenvectors correspond to the three directions in which the point set changes the most.

[0090] The eigenvector corresponding to the smallest eigenvalue is selected as the normal vector of the signboard, and the eigenvectors corresponding to the two largest eigenvalues ​​represent the two main directions of the signboard, namely major and minor.

[0091] The center of mass of the sign is taken as the origin, the normal vector determines the direction of the Z axis, the major and minor directions determine the X axis and the Y axis respectively, and the sign coordinate system Ot-XYZ is established in the binocular system coordinate system.

[0092] The point set in the signboard coordinate system is transformed into the vehicle coordinate system using the transformation matrix Ts-v to obtain the position and direction of the signboard in the vehicle coordinate system.

[0093] In the specific implementation, the PCA method is used to calculate the point set E, and the normal vector n and the two directions of the perpendicular normal, major and minor, can be obtained. At this time, the signboard coordinate system Ot-XYZ in the binocular system coordinate system is obtained. According to the conversion matrix Ts-v between the binocular vision system coordinate system and the vehicle coordinate system (rear axle center), the signboard pose in the vehicle coordinate system can be obtained, as shown in Figure 3 shown.

[0094] It should be noted that the final posture information can be used for navigation, obstacle avoidance and decision making of autonomous vehicles.

[0095] The present embodiment provides a sign recognition method, which obtains a sign to be recognized; processes the sign to be recognized through a target detection algorithm to determine information of the sign to be recognized, wherein the information includes category information and pixel block semantic information; processes the pixel block semantic information to obtain a sub-pixel contour of the sign to be recognized; establishes a first coordinate system and a second coordinate system, and processes the sub-pixel contour of the sign to be recognized to obtain a three-dimensional coordinate of a contour point set of the sign to be recognized; processes the three-dimensional coordinates of the contour point set of the sign to be recognized to obtain a sign pose of the sign to be recognized, which can accurately recognize all types of traffic signs and is suitable for spatial pose estimation of signs of various shapes.

[0096] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S30, the sign recognition method includes steps S301 to S303:

[0097] Step S301, performing connected domain processing on the semantic information of the pixel block and circumscribing a convex polygon to obtain a closed contour of the sign to be identified;

[0098] It should be noted that the semantic pixels of the signboard are identified by finding the pixel blocks corresponding to the signboard in the image. These pixel blocks form one or more connected domains. For each connected domain, its circumscribed convex polygon can be calculated. A convex polygon is the smallest convex polygon that encloses the connected domain.

[0099] In a specific implementation, after obtaining the semantic information of the signboard, the connected domain and the circumscribed convex polygon are calculated for the semantic pixels to obtain a complete and closed outline of the signboard.

[0100] Step S302, obtaining each edge point in the closed contour, processing along the normal direction of the closed contour in a local window centered at the edge point, and obtaining a gradient value of each edge point in the closed contour;

[0101] It should be noted that the gradient value calculation is to calculate the derivative of the image brightness change along the normal direction within the local window, which reflects the steepness of the image edge.

[0102] In the specific implementation, for a point p(x,y) on the edge, its contour normal is defined as d(dx,dy), dx=I(x+1,y)-I(x,y), dy=I(x,y+1)-I(x,y), d=d / ||d||.

[0103] Step S303: Use a Gaussian curve to fit the gradient value to obtain a sub-pixel outline of the sign to be identified.

[0104] It should be noted that the sub-pixel edge calculation method can be described as: for a point p(x, y) on the edge, its contour normal is defined as d(dx, dy), dx = I(x+1, y) - I(x, y), dy = I(x, y+1) - I(x, y), d = d / ||d||.

[0105] Calculate the absolute value of the gradient g(t) at p+td, t∈[-n,-n+1,…,n], and use (t,g(t)) to fit the Gaussian curve:

[0106] ln G(t)=at 2 +bt+c

[0107] in,

[0108] Then all point pairs (t, g(t)) can construct the equation system:

[0109] Mv=l

[0110] in,

[0111] Solving for v, we get The sub-pixel edge point position can be obtained as p+μd.

[0112] The above process is repeated to obtain the sub-pixel edge point set of the entire contour and then obtain the sub-pixel contour of the sign to be identified.

[0113] Sub-pixel edge information can provide richer features, thereby improving recognition accuracy.

[0114] In a specific implementation, for each point in the contour, a curve is used to fit its gradient value along the normal direction of the contour in a local window centered at the point, and the maximum point of the fitting curve is used as the sub-pixel edge point of the hole contour.

[0115] This embodiment provides a method for recognizing a sign. After obtaining the semantic information of the sign, the connected domain and the circumscribed convex polygon are calculated for the semantic pixels to obtain a complete and closed outline of the sign. The sub-pixel outline of the sign to be identified is then calculated using a sub-pixel edge calculation method, which can more accurately express the outline and shape of the object in the image, and is helpful for subsequent image analysis and processing.

[0116] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 5 , step S40, the sign recognition method further includes steps S401 to S404:

[0117] Step S401, establishing a first coordinate system according to a first camera, and establishing a second coordinate system according to a second camera, wherein the first camera and the second camera are arranged on both sides of the vehicle;

[0118] It should be noted that the binocular sensor can be installed in front of the vehicle or on both sides of the vehicle to identify the sign to be identified, wherein the binocular sensor has two image acquisition devices, namely a first camera and a second camera, and respective coordinate systems, namely a first coordinate system and a second coordinate system, are established based on the first camera and the second camera to process the images in subsequent steps.

[0119] Step S402, acquiring a first image of the sub-pixel contour in the first coordinate system and a second image of the sub-pixel contour in the second coordinate system;

[0120] It should be noted that the two images obtained by the binocular sensor are the basis of the stereo vision system. They simulate the way humans observe objects with both eyes, and capture images of the same scene from different angles through two parallel or baseline cameras. This configuration enables the system to calculate the depth information of objects in the scene based on the matching of feature points between the two images. The first image can be understood as the left image, and the second image can be understood as the right image.

[0121] Step S403, performing contour matching processing on the first image and the second image to obtain matching contours in the first image and the second image;

[0122] It should be noted that the purpose of performing contour matching on the first image and the second image is mainly to find the contours of the same object in the two images and determine the corresponding relationship between them.

[0123] In a specific implementation, a threshold is set to screen contours in the first image and the second image, and contours that meet the threshold condition are taken as a pair of candidate matching pairs.

[0124] Furthermore, the step S403 includes:

[0125] Obtaining a first preset value and a second preset value of the sub-pixel profile in the first image and a third preset value and a fourth preset value in the second image, wherein the first preset value is greater than the second preset value, and the third preset value is greater than the fourth preset value;

[0126] Contour matching is performed on the contours of the sub-pixel contour in the first image and the second image using the first preset value, the second preset value, the third preset value, and the fourth preset value to obtain matching contours in the first image and the second image.

[0127] It should be noted that the maximum and minimum values ​​of the sub-pixel contour of the sign in the image are calculated. Since the maximum and minimum values ​​of the same edge of the binocular vision image in the left and right images are equal after the epipolar correction, the maximum and minimum values ​​of the contour rows in the left and right images are used as screening conditions, and a threshold is set to screen the contours in the left and right images, and the contours that meet the threshold conditions are taken as a pair of candidate matching pairs.

[0128] It should be understood that the first preset value, the second preset value, the third preset value and the fourth preset value are respectively the maximum value of the left image contour row, the minimum value of the left image contour row, the maximum value of the right image contour row and the minimum value of the right image contour row.

[0129] In the specific implementation, it is assumed that the left and right image contour threshold conditions can be set as:

[0130] Maximum value of the left image contour row – maximum value of the right image contour row | < row maximum threshold.

[0131] Minimum value of the left image contour row – Minimum value of the right image contour row | < minimum value threshold of row.

[0132] (maximum value of the left image contour row – minimum value of the left image contour row) - (maximum value of the right image contour row – minimum value of the right image contour row) | < height difference threshold.

[0133] Step S404: Process the matching contour to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0134] In a specific implementation, the matching contours of the first image and the second image are processed by a binocular positioning method, so that the three-dimensional coordinates of each point in the signboard contour point set can be obtained.

[0135] like Figure 6 As shown in , the corresponding feature points in the epipolar corrected image are on the same horizontal line, that is, the matched contours in the left and right images correspond one to one to each contour point in the row. Since the obtained signboard contour is complete and closed, the left points in the same row of the contour correspond to the left points, and the right points correspond to the right points, as shown in Figure 6 In the example, P1l corresponds to P1r, and P2l corresponds to P2r.

[0136] Furthermore, the step S404 includes:

[0137] Establishing a binocular system coordinate system based on the first coordinate system and the second coordinate system;

[0138] The matching contour is processed based on the binocular system coordinate system to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0139] It should be noted that, assuming that the corresponding image points of a point P(X, Y, Z) in the space on the left and right images are pl(x l ,y l ) and pr(x r ,y r ), which is located in the same row of the left and right images. Assume that the left camera is the principal point (x 0l ,y 0l ), the right camera image principal point (x 0r ,y 0r ) are located in the same row, with a main distance f l and f r Equal, such as Figure 7 Schematic diagram of binocular principle shown.

[0140] In the specific implementation, it can be known from the calibration that (x 0l ,y 0l ),(x 0r ,y 0r ), f l =f r =f, corresponding point pair (x l ,y l ), (x r ,y r ) is a known quantity, and the binocular imaging principle can be:

[0141]

[0142] In this way, the three-dimensional coordinates of the signboard contour point set are obtained, denoted as edge E = {P1, P2,…, Pm}, where m is the number of points.

[0143] This embodiment provides a method for sign recognition. This strategy uses a binocular vision system and can provide rich image information in the perception of the autonomous driving environment, including the distance, color, texture and other information of the object. The system adopted by this strategy is simple, highly robust, and has a small amount of calculation. It can be run on a vehicle chip with CPU computing and does not require the configuration of a specific parallel computing chip. It is not only applicable to the field of vehicle autonomous driving, but also to the field of real-life map construction. The method proposed by this strategy can solve the construction of traffic sign elements in driving maps and provide a decision-making basis for vehicle motion control and planning.

[0144] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the sign recognition method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0145] This application also provides a sign recognition device, please refer to Figure 8 , the sign recognition device comprises:

[0146] The deep learning module 10 is used to obtain the sign to be identified; process the sign to be identified by using a target detection algorithm to determine the information of the sign to be identified, wherein the information includes category information and pixel block semantic information.

[0147] The first processing module 20 is used to process the semantic information of the pixel block to obtain the sub-pixel outline of the sign to be identified.

[0148] The second processing module 30 is used to establish a first coordinate system and a second coordinate system, and process the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0149] The third processing module 40 is used to process the three-dimensional coordinates of the contour point set of the sign to be identified to obtain the sign position and posture of the sign to be identified.

[0150] The sign recognition device provided by the present application adopts the sign recognition method in the above embodiment, which can solve the technical problem of needing to recognize and estimate the posture of traffic signs during vehicle driving. Compared with the prior art, the beneficial effects of the sign recognition device provided by the present application are the same as the beneficial effects of the sign recognition method provided by the above embodiment, and the other technical features in the sign recognition device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0151] In one embodiment, the deep learning module 10 is also used to perform target classification on the sign to be identified through the target classification unit of the target detection algorithm to obtain category information of the sign to be identified; and perform semantic segmentation on the sign to be identified through the semantic segmentation unit of the target detection algorithm to obtain pixel block semantic information.

[0152] In one embodiment, the first processing module 20 is further used to perform connected domain processing on the semantic information of the pixel block and circumscribe a convex polygon to obtain a closed contour of the sign to be identified; obtain each edge point in the closed contour, and process along the normal direction of the closed contour in a local window centered on the edge point to obtain a gradient value of each edge point in the closed contour;

[0153] The gradient value is fitted using a Gaussian curve to obtain the sub-pixel outline of the sign to be identified.

[0154] In one embodiment, the second processing module 30 is further used to establish a first coordinate system based on the first camera and a second coordinate system based on the second camera, wherein the first camera and the second camera are arranged on both sides of the vehicle; obtain a first image of the sub-pixel contour in the first coordinate system and a second image of the sub-pixel contour in the second coordinate system; perform contour matching processing on the first image and the second image to obtain matching contours in the first image and the second image; process the matching contours to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0155] In one embodiment, the second processing module 30 is further used to obtain a first preset value and a second preset value of the sub-pixel contour in the first image and a third preset value and a fourth preset value in the second image, wherein the first preset value is greater than the second preset value, and the third preset value is greater than the fourth preset value; and the contours of the sub-pixel contour in the first image and the second image are matched by the first preset value, the second preset value, the third preset value and the fourth preset value to obtain matching contours in the first image and the second image.

[0156] In one embodiment, the second processing module 30 is further used to establish a binocular system coordinate system based on the first coordinate system and the second coordinate system; and process the matching contour based on the binocular system coordinate system to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

[0157] In one embodiment, the third processing module 40 is also used to process the three-dimensional coordinates of the contour point set of the sign to be identified by principal component analysis to obtain the normal vector of the contour point set of the sign to be identified and the main direction and secondary direction of the contour point set of the sign to be identified; based on the normal vector and the main direction and the secondary direction, obtain the coordinate system of the sign to be identified in the binocular system coordinate system; process the coordinate system of the sign to be identified by means of a transformation matrix to obtain the sign pose of the sign to be identified in the vehicle coordinate system.

[0158] The present application provides a sign recognition device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the sign recognition method in the above-mentioned embodiment one.

[0159] Reference below Fig. 9 , which shows a schematic diagram of the structure of a sign recognition device suitable for implementing the embodiment of the present application. The sign recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The sign recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0160] like Fig. 9As shown, the sign recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. In RAM 1004, various programs and data required for the operation of the sign recognition device are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. Communication device 1009 can allow the sign recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a sign recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0161] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0162] The sign recognition device provided by the present application adopts the sign recognition method in the above embodiment, which can solve the technical problem of needing to recognize and estimate the posture of traffic signs during vehicle driving. Compared with the prior art, the beneficial effects of the sign recognition device provided by the present application are the same as the beneficial effects of the sign recognition method provided by the above embodiment, and the other technical features in the sign recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0163] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0164] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0165] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the sign recognition method in the above-mentioned embodiment.

[0166] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0167] The computer-readable storage medium may be included in the sign recognition device; or may exist independently without being assembled into the sign recognition device.

[0168] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the sign recognition device, the sign recognition device enables the following: to obtain the sign to be recognized; to process the sign to be recognized through the target detection algorithm to determine the information of the sign to be recognized, wherein the information includes category information and pixel block semantic information; to process the pixel block semantic information to obtain the sub-pixel contour of the sign to be recognized; to establish a first coordinate system and a second coordinate system, and to process the sub-pixel contour of the sign to be recognized to obtain the three-dimensional coordinates of the contour point set of the sign to be recognized; to process the three-dimensional coordinates of the contour point set of the sign to be recognized to obtain the sign posture of the sign to be recognized.

[0169] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0171] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0172] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sign recognition method, and can solve the technical problem of needing to recognize and estimate the posture of traffic signs during vehicle driving. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the sign recognition method provided in the above-mentioned embodiment, and will not be elaborated here.

[0173] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned sign recognition method are implemented.

[0174] The computer program product provided by the present application can solve the technical problem of identifying traffic signs and estimating their posture during vehicle driving. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the sign recognition method provided by the above embodiment, and will not be elaborated here.

[0175] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for identifying a sign, characterized in that: The sign recognition method comprises the following steps: Obtain the sign to be identified; Processing the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information; Processing the semantic information of the pixel block to obtain the sub-pixel outline of the sign to be identified; Establishing a first coordinate system and a second coordinate system, and processing the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified; The three-dimensional coordinates of the outline point set of the sign to be identified are processed to obtain the sign position and posture of the sign to be identified.

2. The sign recognition method according to claim 1, characterized in that: The step of processing the pixel block semantic information to obtain the sub-pixel contour of the sign to be identified includes: Performing connected domain processing on the semantic information of the pixel block and circumscribing a convex polygon to obtain a closed contour of the sign to be identified; Acquire each edge point in the closed contour, and process along the normal direction of the closed contour in a local window centered at the edge point to obtain a gradient value of each edge point in the closed contour; The gradient value is fitted using a Gaussian curve to obtain a sub-pixel profile of the sign to be identified.

3. The sign recognition method according to claim 1, characterized in that: The step of establishing the first coordinate system and the second coordinate system, and processing the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified includes: Establishing a first coordinate system according to a first camera, and establishing a second coordinate system according to a second camera, wherein the first camera and the second camera are arranged on both sides of the vehicle; Acquire a first image of the sub-pixel contour in the first coordinate system and a second image of the sub-pixel contour in the second coordinate system; Performing contour matching processing on the first image and the second image to obtain matching contours in the first image and the second image; The matching contour is processed to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

4. The sign recognition method according to claim 3, characterized in that: The step of performing contour matching processing on the first image and the second image to obtain matching contours in the first image and the second image includes: Acquire a first preset value and a second preset value of the sub-pixel profile in the first image and a third preset value and a fourth preset value in the second image, wherein the first preset value is greater than the second preset value, and the third preset value is greater than the fourth preset value; The first preset value, the second preset value, the third preset value and the fourth preset value are used to perform contour matching on the contours of the sub-pixel contour in the first image and the second image to obtain matching contours in the first image and the second image.

5. The sign recognition method according to claim 3, characterized in that: The step of processing the matching contour to obtain the three-dimensional coordinates of the contour point set of the sign to be identified includes: Establishing a binocular system coordinate system based on the first coordinate system and the second coordinate system; The matching contour is processed based on the binocular system coordinate system to obtain the three-dimensional coordinates of the contour point set of the sign to be identified.

6. The sign recognition method according to claim 1, characterized in that: The step of processing the three-dimensional coordinates of the outline point set of the sign to be identified to obtain the sign posture of the sign to be identified includes: Processing the three-dimensional coordinates of the outline point set of the sign to be identified by principal component analysis to obtain the normal vector of the outline point set of the sign to be identified and the main direction and secondary direction of the outline point set of the sign to be identified; Based on the normal vector, the main direction and the secondary direction, a coordinate system of the sign to be identified in the binocular system coordinate system is obtained; The coordinate system of the sign to be identified is processed by a conversion matrix to obtain the sign position and posture of the sign to be identified in the vehicle coordinate system.

7. The sign recognition method according to any one of claims 1 to 6, characterized in that: The step of processing the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information, comprises: The target classification unit of the target detection algorithm performs target classification on the sign to be identified to obtain category information of the sign to be identified; The semantic segmentation unit of the target detection algorithm performs semantic segmentation on the sign to be identified to obtain pixel block semantic information.

8. A sign recognition device, characterized in that: The device comprises: A deep learning module is used to obtain a sign to be identified; process the sign to be identified by a target detection algorithm to determine information of the sign to be identified, wherein the information includes category information and pixel block semantic information; A first processing module is used to process the semantic information of the pixel block to obtain the sub-pixel outline of the sign to be identified; A second processing module is used to establish a first coordinate system and a second coordinate system, and process the sub-pixel contour of the sign to be identified to obtain the three-dimensional coordinates of the contour point set of the sign to be identified; The third processing module is used to process the three-dimensional coordinates of the outline point set of the sign to be identified to obtain the sign position and posture of the sign to be identified.

9. A sign recognition device, characterized in that: The device comprises: a memory, a processor, and a sign recognition program stored in the memory and executable on the processor, wherein the sign recognition program is configured to implement the steps of the sign recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a sign recognition program, and when the sign recognition program is executed by the processor, the steps of the sign recognition method according to any one of claims 1 to 7 are implemented.

Images