Signboard corner point recognition method and device based on center point constraint

Abstract

The application relates to a center point constraint-based road sign corner point recognition method and device. The method comprises the following steps: acquiring a picture, the picture carrying a square road sign element, recognizing the picture by using a trained convolutional neural network model, recognizing a center point corresponding to the square road sign element and two road sign corner points in the picture in the picture, establishing a two-dimensional coordinate system based on the picture, and calculating the remaining two road sign corner points corresponding to the square road sign element based on the constraint relationship between the center point and the two road sign corner points of the square road sign element in the two-dimensional coordinate system. Since the remaining two road sign corner points are calculated based on the constraint relationship between the center point and the two road sign corner points in the two-dimensional coordinate system established by the picture, the square road sign element is partially blocked by an occlusion, and all road sign corner points of the square road sign can still be accurately recognized.

Description

Technical Field

[0001] This application relates to the field of road sign recognition technology, and in particular to a method, apparatus, device and storage medium for road sign corner point recognition based on center point constraints. Background Technology

[0002] Road signs are an indispensable element in building high-precision maps. Most road signs are square in shape. The core premise of building square road signs is to identify all the corner points (i.e., the four corner points) of the square road sign in the image, and then construct the road sign based on the corner points.

[0003] However, due to obstructions such as trees, some corners of the square road signs cannot be identified, which is a problem that researchers urgently need to solve. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a method and apparatus for road sign corner point recognition based on center point constraints, which can accurately identify all corner points of square road signs.

[0005] The first aspect of this application provides a method for road sign corner point recognition based on center point constraints, including:

[0006] Obtain an image containing square road sign elements;

[0007] The trained convolutional neural network model is used to identify the center point and two corner points of the square road sign element in the image.

[0008] Establish a two-dimensional coordinate system based on the image;

[0009] In the two-dimensional coordinate system, the constraint relationship between the center point corresponding to the square road sign element and two of the road sign corner points is used to calculate the other two road sign corner points corresponding to the square road sign element.

[0010] Preferably, two of the road sign corner points are diagonally opposite each other.

[0011] Preferably, the step of using a trained convolutional neural network model to recognize the image and identify the center point and two corner points of the square road sign element in the image includes:

[0012] The image is convolved along multiple branches to obtain multiple convolutional feature maps;

[0013] Set the fusion ranking order of the multiple convolutional feature maps;

[0014] Based on the fusion ranking order, multiple convolutional feature maps are fused sequentially to obtain a fused feature map. Each feature fusion is followed by at least two ordinary convolution calculations before the next feature fusion.

[0015] Perform ordinary convolution on the fused feature map to obtain the target convolutional feature map;

[0016] Based on the target convolutional feature map, the center point and two corner points of the square road sign element are identified.

[0017] Preferably, when performing convolution processing on the image along multiple branches:

[0018] Each branch performs dilated convolution and regular convolution on the image sequentially; or

[0019] Each branch performs ordinary convolution calculations on the image multiple times consecutively.

[0020] Preferably, the dilated convolution calculation includes:

[0021] Initialize the convolution kernel P*P and set the dilation rate of the convolution kernel P*P, where P is a positive odd number in the interval [1,7].

[0022] Set the number of dilated convolution channels;

[0023] Dilated convolution is performed on the image based on the convolution kernel P*P and the number of dilated convolution channels.

[0024] Preferably, the ordinary convolution calculation includes:

[0025] Initialize the convolution kernel to N*N, where N is a positive odd number in the interval [1,7].

[0026] Set the number of channels for a normal convolution;

[0027] Perform ordinary convolution calculations on the image or the fused feature map based on the convolution kernel N*N and the number of ordinary convolution channels.

[0028] Preferably, the number of ordinary convolution channels used in the ordinary convolution calculation of the fused feature map is equal to 3;

[0029] The step of identifying the center point and two corner points of the square road sign element based on the target convolutional feature map includes:

[0030] The road sign corner points of the target convolution feature map with ordinary convolution channel number = 1 and the road sign corner points of the target convolution feature map with ordinary convolution channel number = 2 are selected as two road sign corner points.

[0031] The corner point of the road sign in the target convolution feature map with channel number 3 of the ordinary convolution is taken as the center point.

[0032] A second aspect of this application provides a road sign corner recognition device based on center point constraints, comprising:

[0033] The acquisition module is used to acquire images, which carry square road sign elements;

[0034] The recognition module is used to recognize the image using a trained convolutional neural network model, and to identify the center point and two corner points of the square road sign element in the image.

[0035] The module is used to establish a two-dimensional coordinate system based on the image;

[0036] The calculation module is used to calculate the other two corner points of the square road sign element based on the constraint relationship between the center point and two corner points of the square road sign element in the two-dimensional coordinate system.

[0037] A third aspect of this application provides an electronic device, comprising:

[0038] Processor; and

[0039] The memory stores executable code, which, when executed by the processor, causes the processor to perform the road sign corner recognition method based on center point constraints as described above.

[0040] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the road sign corner recognition method based on center point constraints as described above.

[0041] The technical solution provided in this application may include the following beneficial effects:

[0042] The technical solution of this application involves acquiring an image containing square road sign elements, and using a trained convolutional neural network model to identify the center point and two corner points of the square road sign elements. A two-dimensional coordinate system is established based on the image. Within this system, the constraint relationship between the center point and the two corner points is used to calculate the remaining two corner points of the square road sign elements. Because the remaining two corner points are calculated using the constraint relationship between the center point and the two corner points within the two-dimensional coordinate system established from the image, even when the square road sign elements are partially obscured, all corner points of the square road sign can still be accurately identified.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0044] The above and other objects, features and advantages of this application will become more apparent from the following description of exemplary embodiments of this application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components.

[0045] Figure 1 This is a flowchart illustrating the road sign corner point recognition method based on center point constraints, as shown in the embodiments of this application.

[0046] Figure 2 This is a schematic diagram of the application framework structure of a convolutional neural network shown in the embodiments of this application;

[0047] Figure 3 This is a simulated schematic diagram illustrating the identification of the center point and two corner points of a road sign based on the target convolutional feature map, as shown in the embodiments of this application.

[0048] Figure 4 This is a schematic diagram illustrating a two-dimensional coordinate system established from an image, as shown in an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of the structure of a road sign corner recognition device based on center point constraints, as shown in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0051] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0052] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0053] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0054] Road signs are an indispensable element in building high-precision maps. For the construction of square road signs, the core prerequisite is to identify all the corner points of the road signs (i.e., the four corner points of the road signs). However, due to the obstruction of trees and other objects, some corner points of square road signs may not be identified.

[0055] To address the aforementioned problems, this application provides a method and apparatus for road sign corner point recognition based on center point constraints, capable of accurately identifying all corner points of square road signs. To facilitate understanding of the technical solution of this application, the following detailed description, in conjunction with the accompanying drawings, illustrates the technical solution of this application.

[0056] Figure 1 A flowchart illustrating a road sign corner point recognition method based on center point constraints is shown in an embodiment of this application.

[0057] Please see Figure 1 A method for identifying road sign corner points based on center point constraints includes the following steps:

[0058] Step 11: Obtain the image, which contains square road sign elements.

[0059] The image is a photograph taken by a vehicle-mounted camera. The image contains square road sign elements. The number of road signs in the image is not limited to one; there may be multiple signs. When there are multiple square road sign elements, their sizes may vary. Based on specifications, square road sign elements can be categorized into small, medium, and large square road sign elements.

[0060] Step 12: Use the trained convolutional neural network model to recognize the image, identifying the center point of the square road sign element and two corner points of the road sign.

[0061] After obtaining the image, a trained convolutional neural network model is used to identify the center point and two corner points of the square road sign element. Preferably, the two corner points are diagonally opposite, meaning that when the two corner points are lifted, they form one diagonal line corresponding to the square road sign element.

[0062] The specific process by which the convolutional neural network model identifies the center point and two corner points of the square road sign element in the image is as follows:

[0063] Step S121: Perform convolution processing on the image along multiple branches to obtain multiple convolutional feature maps.

[0064] The number of branches is not specifically limited, but considering the overall computational load of the convolutional neural network model, three branches are preferred. Each branch can perform convolutional processing on the image by sequentially performing dilated convolution and regular convolution, or by performing multiple consecutive regular convolutions to extract features from the square road sign elements in the image.

[0065] If we perform dilated convolution and regular convolution on the image sequentially, the dilated convolution has a larger receptive field than the regular convolution. By combining the two convolution calculations, they complement each other, allowing us to extract more effective information from the image and thus enabling the feature extraction of square road sign elements from the image.

[0066] The advantages of using multiple consecutive ordinary convolution operations on the image are: 1. Multiple ordinary convolution operations can enrich the model's representational ability to a greater extent; 2. Spatially, translation is not deformed, ensuring that the main information is not lost; 3. Multiple ordinary convolution operations are conducive to information filtering and avoid the influence of various noises.

[0067] The dilated convolution calculation is processed as follows: Initialize the convolution kernel P*P and set the dilation rate of the convolution kernel P*P, where P is a positive odd number in the interval [1,7]; set the number of dilated convolution channels; perform dilated convolution calculation on the image based on the convolution kernel P*P and the number of dilated convolution channels.

[0068] The processing method for ordinary convolution calculation is as follows: initialize the convolution kernel N*N, where N is a positive odd number in the interval [1,7]; set the number of ordinary convolution channels; perform ordinary convolution calculation on the image or fused feature map according to the convolution kernel N*N and the number of ordinary convolution channels.

[0069] Step S122: Set the fusion ranking order of multiple convolutional feature maps.

[0070] After performing convolution processing on the image along different branches to obtain multiple convolutional feature maps, a fusion ranking order for these feature maps is set. Subsequent feature fusion calculations will be performed based on this fusion ranking order.

[0071] Step S123: Based on the fusion ranking order, multiple convolutional feature maps are fused sequentially to obtain a fused feature map. Each feature fusion is followed by at least one ordinary convolution calculation before the next feature fusion.

[0072] For example, assuming we have three convolutional feature maps, we first fuse the first and second convolutional feature maps (the type of the fused convolutional feature map is still a convolutional feature map). Then, we perform at least two ordinary convolution calculations on the convolutional feature map obtained after fusing the first and second convolutional feature maps. Finally, we fuse it with the third convolutional feature map to obtain a fused feature map.

[0073] Step S124: Perform ordinary convolution calculation on the fused feature map to obtain the target convolution feature map.

[0074] Step S125: Identify the center point and two corner points of the square road sign element based on the target convolutional feature map.

[0075] The number of channels in a regular convolution operation performed on the fused feature map is equal to 3, such as... Figure 3 As shown, the road sign corner points of the target convolutional feature map with ordinary convolutional channel number = 1 and the road sign corner points of the target convolutional feature map with ordinary convolutional channel number = 2 are taken as two road sign corner points; the road sign corner point of the target convolutional feature map with ordinary convolutional channel number = 3 is taken as the center point.

[0076] To better understand the process by which convolutional neural networks identify the center point and two corner points of square road sign elements in an image.

[0077] Please see Figure 2 Assume there are 3 branches. Figure 2 The diagram shows branches ①, ②, and ③. Branch ① performs dilated convolution and regular convolution on the image sequentially, while branches ② and ③ perform regular convolution on the image multiple times consecutively.

[0078] After performing convolution processing on the image along the three branches, three convolutional feature maps are obtained. The fusion order of these three convolutional feature maps is set: first, the convolutional feature map corresponding to branch ① is fused with the convolutional feature map corresponding to branch ②. After fusion, at least two ordinary convolution calculations are performed before fusion with the convolutional feature map corresponding to branch ③, resulting in a fused convolutional feature map.

[0079] Perform ordinary convolution on the fused convolutional feature map to obtain the target convolutional feature map. Based on the target convolutional feature map, identify the center point and two corner points of the square road sign elements. Then, use the target convolutional feature map with channel number 1 (i.e., the road sign corner points) to perform ordinary convolution on the target convolutional feature map. Figure 3 Point A shown) and the road sign corner points of the target convolution feature map with ordinary convolution channel number = 2 are used as two of the road sign corner points (i.e. Figure 3 Point C shown); the corner point of the road sign in the target convolution feature map with ordinary convolution channel number = 3 is taken as the center point (i.e. Figure 3 Point E (as shown).

[0080] It should be noted that the first dilated convolution calculation performed on the image in branch ① uses a 3x3 dilated convolution kernel with a dilation rate of 2; the first ordinary convolution calculation performed on the image in branch ② uses a 3x3 ordinary convolution kernel; and the first ordinary convolution calculation performed on the image in branch ③ uses a 1x1 ordinary convolution kernel. The 3x3 dilated convolution kernel with a dilation rate of 2 is equivalent to a 5x5 ordinary convolution kernel and is mainly used to extract large square road sign elements. The 3x3 ordinary convolution kernel is mainly used to extract medium-sized square road sign elements. The 1x1 ordinary convolution kernel is mainly used to extract small square road sign elements. Feature extraction of square road sign elements of different sizes is achieved through the above methods.

[0081] Step 13: Establish a two-dimensional coordinate system based on the image.

[0082] like Figure 4 The diagram shows a two-dimensional coordinate system XOY established based on an image.

[0083] Step 14: In a two-dimensional coordinate system, determine the constraint relationship between the center point of the square road sign element and two of its corner points, and calculate the remaining two corner points of the square road sign element.

[0084] like Figure 4 As shown, after establishing a two-dimensional coordinate system XOY based on the image, since the coordinates of the center point (i.e., point E) and the coordinates of two road sign corner points (i.e., points A and C) are known, and the center point and two road sign corner points have a constraint relationship in the two-dimensional coordinate system XOY, the constraint relationship among the three can be represented by the following formula:

[0085] x1 + x2 = 2x5 (1)

[0086] x³ + x⁴ = 2x⁵ (2)

[0087] y l +y4=2y5 (3)

[0088] y2+y3=2y5 (4)

[0089] The coordinates of point A are (x1, y1), point B is (x2, y2), point C is (x3, y3), point D is (x4, y4), and point E is (x5, y5). Since the coordinates of points A, C, and E are known through the recognition by the convolutional neural network model, the coordinates of point B and point D can be solved by combining equations (1), (2), (3), and (4). This completes the recognition of all corner points of the square road sign element (i.e., the recognition of the four road sign corner points).

[0090] The method in this embodiment acquires an image containing square road sign elements, uses a trained convolutional neural network model to recognize the image, identifies the center point and two corner points of the square road sign elements, establishes a two-dimensional coordinate system based on the image, and calculates the remaining two corner points of the square road sign elements using the constraint relationship between the center point and the two corner points within this two-dimensional coordinate system. Because the remaining two corner points are calculated using the constraint relationship between the center point and the two corner points within the two-dimensional coordinate system established from the image, even if part of the square road sign element is obscured, all corner points of the square road sign can still be accurately identified.

[0091] Furthermore, two additional road sign corner points are calculated by using the constraint relationship between the center point and two road sign corner points that are diagonally related. Compared to identifying four road sign corner points for a square road sign element, the method in this embodiment can reduce the amount of identification data by 25%.

[0092] Corresponding to the aforementioned application function method embodiments, this application also provides a corresponding embodiment of an automatic identification device for road sign corner points.

[0093] Figure 5 A schematic diagram of a road sign corner recognition device 50 based on center point constraints is shown in an embodiment of this application.

[0094] Please see Figure 5 A road sign corner point recognition device based on center point constraint includes: a data acquisition module 510, a recognition module 520, a data establishment module 530, and a calculation module 540.

[0095] The acquisition module 510 is used to acquire images, which contain square road sign elements;

[0096] The recognition module 520 is used to recognize the image using a trained convolutional neural network model, and to identify the center point and two corner points of the square road sign element in the image.

[0097] Module 530 is used to establish a two-dimensional coordinate system based on an image;

[0098] The calculation module 540 is used to calculate the other two corner points of the square road sign element by considering the constraint relationship between the center point of the square road sign element and two corner points of the road sign in a two-dimensional coordinate system.

[0099] In this embodiment, the acquisition module 510 acquires an image containing square road sign elements. The recognition module 520 uses a trained convolutional neural network model to recognize the image, identifying the center point and two corner points of the square road sign elements. The establishment module 530 establishes a two-dimensional coordinate system based on the image. In this two-dimensional coordinate system, the constraint relationship between the center point and the two corner points of the square road sign elements is used to calculate the remaining two corner points. Because the remaining two corner points are calculated using the constraint relationship between the center point and the two corner points in the two-dimensional coordinate system established based on the image, even if part of the square road sign element is obscured, all the corner points of the square road sign can still be accurately identified.

[0100] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operation has been described in detail in the method embodiments corresponding to the apparatus, and will not be elaborated further here.

[0101] Please see Figure 6 The electronic device 600 includes a processor 610 and a memory 620.

[0102] The processor 610 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0103] Memory 620 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 610 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 620 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. The memory 620 stores executable code, which, when processed by the processor 610, can cause the processor 610 to execute part or all of the methods described above.

[0104] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0105] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0106] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying corner points of road signs based on center point constraints, characterized in that, include: Acquire images captured by a vehicle-mounted camera, the images containing square road sign elements; The trained convolutional neural network model is used to identify the center point and two corner points of the square road sign element in the image. Establish a two-dimensional coordinate system based on the image; In the two-dimensional coordinate system, the constraint relationship between the center point corresponding to the square road sign element and two of the road sign corner points is used to calculate the other two road sign corner points corresponding to the square road sign element; The step of using a trained convolutional neural network model to recognize the image, identifying the center point and two corner points of the square road sign element in the image, includes: The image is convolved along multiple branches to obtain multiple convolutional feature maps; Set the fusion ranking order of the multiple convolutional feature maps; Based on the fusion ranking order, multiple convolutional feature maps are fused sequentially to obtain a fused feature map. Each feature fusion is followed by at least two ordinary convolution calculations before the next feature fusion. Perform ordinary convolution on the fused feature map to obtain the target convolutional feature map; Based on the target convolutional feature map, the center point and two corner points of the square road sign element are identified.

2. The road sign corner point recognition method based on center point constraint according to claim 1, characterized in that, Two of the road signs are diagonally opposite each other.

3. The road sign corner point recognition method based on center point constraints according to claim 1, characterized in that, When performing convolution processing on the image along multiple branches: Each branch performs dilated convolution and regular convolution on the image sequentially; or Each branch performs ordinary convolution calculations on the image multiple times consecutively.

4. The road sign corner point recognition method based on center point constraint according to claim 3, characterized in that, The dilated convolution calculation includes: Initialize the convolution kernel P*P and set the dilation rate of the convolution kernel P*P, where P is a positive odd number in the interval [1,7]. Set the number of dilated convolution channels; Dilated convolution is performed on the image based on the convolution kernel P*P and the number of dilated convolution channels.

5. The road sign corner point recognition method based on center point constraint according to claim 3, characterized in that, The ordinary convolution calculation includes: Initialize the convolution kernel to N*N, where N is a positive odd number in the interval [1,7]. Set the number of channels for a normal convolution; Perform ordinary convolution calculations on the image or the fused feature map based on the convolution kernel N*N and the number of ordinary convolution channels.

6. The road sign corner point recognition method based on center point constraint according to claim 1, characterized in that, The number of ordinary convolution channels used in the ordinary convolution calculation of the fused feature map is equal to 3; The step of identifying the center point and two corner points of the square road sign element based on the target convolutional feature map includes: The road sign corner points of the target convolution feature map with ordinary convolution channel number = 1 and the road sign corner points of the target convolution feature map with ordinary convolution channel number = 2 are selected as two road sign corner points. The corner point of the road sign in the target convolution feature map with channel number 3 of the ordinary convolution is taken as the center point.

7. A road sign corner recognition device based on center point constraints, characterized in that, include: The acquisition module is used to acquire images captured by the vehicle-mounted camera, the images of which contain square road sign elements; The recognition module is used to recognize the image using a trained convolutional neural network model, identifying the center point and two corner points of the square road sign element in the image; it performs convolution processing on the image along multiple branches to obtain multiple convolutional feature maps; it sets the fusion ranking order of the multiple convolutional feature maps; it performs feature fusion on the multiple convolutional feature maps sequentially based on the fusion ranking order to obtain a fused feature map, wherein at least two ordinary convolution calculations are performed after each feature fusion before the next feature fusion; it performs ordinary convolution calculations on the fused feature map to obtain a target convolutional feature map; and it identifies the center point and two corner points of the square road sign element based on the target convolutional feature map. The module is used to establish a two-dimensional coordinate system based on the image; The calculation module is used to calculate the other two corner points of the square road sign element based on the constraint relationship between the center point and two corner points of the square road sign element in the two-dimensional coordinate system.

8. An electronic device, characterized in that, include: processor; as well as A memory storing executable code, which, when executed by the processor, causes the processor to perform the road sign corner recognition method based on center point constraints as described in any one of claims 1 to 6.

9. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the road sign corner point recognition method based on center point constraints as described in any one of claims 1 to 6.

Images