Key point based traffic sign text online detection method and system

Abstract

The application relates to the technical field of target detection, and provides a traffic sign text online detection method and system based on key points. The method comprises the following steps: acquiring a road image based on a vehicle-mounted image acquisition device; inputting the road image into a detection network to obtain traffic sign text key points; determining a text image according to the key points, and identifying the text image to obtain an online detection result of the traffic sign text; and the detection network is obtained based on sample and label training. The application outputs traffic sign text key points based on the detection network, effectively reduces the complexity of the detection network, increases the identification speed, and is more suitable for the low calculation resource requirement and real-time requirement of vehicle-mounted online traffic sign identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a traffic sign text online detection method and system based on key points. BACKGROUND

[0002] Road traffic signs and markings guide road users to use the road in an orderly manner to promote road safety, and the recognition of traffic signs in a driving assistance system can provide corresponding assistance for vehicle control without interruption. For example, prohibition signs can help the system to make early danger prediction; warning signs can help the system to make early obstacle avoidance processing in certain situations; and indication signs can help the system to make control preprocessing to ensure that the vehicle follows the road indication. Therefore, correct recognition and accurate application of traffic signs can provide more perfect assistance for driving assistance systems and even autonomous driving.

[0003] However, due to the factors of hardware cost, size, environment, etc. of the vehicle terminal, the hardware resources of the car are limited in the automatic driving or assisted driving scene, and the computing resources that can be allocated to the traffic sign recognition task are usually limited.

[0004] In the prior art, for the detection task of traffic sign text, a general text recognition model is often directly used. Although such a model has good recognition effect on text, especially complex text (such as baseline curved text, complex layout text, rarely seen font text, and large amount of text, etc.), due to the limitation of model structure and computational demand, it is not suitable for deployment on a vehicle terminal.

[0005] Therefore, how to provide a traffic sign text detection method and system with low computing resource demand and strong pertinence has become a technical problem that needs to be solved in the industry. SUMMARY

[0006] The present application provides a traffic sign text online detection method and system based on key points, which solves the defects of complex model structure and high computational demand in the prior art, and realizes more targeted traffic sign text detection.

[0007] The present application provides a traffic sign text online detection method based on key points, applied to a vehicle terminal, comprising:

[0008] acquiring a road image based on a vehicle-mounted image acquisition device;

[0009] inputting the road image into a detection network to obtain traffic sign text key points;

[0010] determining a text image according to the key points, and identifying the text image to obtain an online detection result of the traffic sign text;

[0011] The detection network is trained based on samples and labels.

[0012] According to the online detection method for traffic sign text based on key points provided by the application, the key points are four vertices of a rectangular text box; the detection network comprises a single-channel key point decoder; and the channel of the key point decoder can output at least four key point coordinates with confidence.

[0013] According to the online detection method for traffic sign text based on key points provided by the application, the key points are four vertices of a rectangular text box; the detection network comprises a four-channel key point decoder; and the channel of the key point decoder can output at least one key point coordinate with confidence.

[0014] According to the online detection method for traffic sign text based on key points provided by the application, the step of determining a text image according to the key points comprises:

[0015] The key points are sorted based on confidence;

[0016] The fourth key point after sorting is judged;

[0017] If the confidence of the fourth key point is greater than a set threshold, the first key point, the second key point, the third key point and the fourth key point are determined as vertices of a text image;

[0018] If the confidence of the fourth key point is not greater than the set threshold, a supplementary key point of a rectangular text box is fitted based on geometric relations according to the first key point, the second key point and the third key point, and the first key point, the second key point, the third key point and the supplementary key point are determined as vertices of a text image;

[0019] The road image is segmented according to the vertices of the text image, and a text image is obtained.

[0020] According to the online detection method for traffic sign text based on key points provided by the application, the detection network comprises a single-channel multi-text region decoder; and the channel of the multi-text region decoder can output probability maps of at least two text regions.

[0021] According to the online detection method for traffic sign text based on key points provided by the application, the label comprises a first sub-label, or the label comprises a first sub-label and a second sub-label.

[0022] The first sub-label refers to:

[0023] a single probability map comprising four key point probability distributions; or,

[0024] four probability maps comprising a single keypoint probability distribution;

[0025] The second sub-label refers to a probability map of at least two text regions.

[0026] The application further provides a keypoint-based online traffic sign text detection system, applied to a vehicle terminal, comprising:

[0027] An acquisition module is configured to acquire a road image based on a vehicle image acquisition device.

[0028] A detection module is configured to input the road image into a detection network to obtain traffic sign text key points.

[0029] A text module is configured to determine a text image according to the key points and identify the text image to obtain an online detection result of the traffic sign text.

[0030] The detection network is trained based on samples and labels.

[0031] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the keypoint-based online traffic sign text detection method according to any one of the above.

[0032] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the keypoint-based online traffic sign text detection method according to any one of the above.

[0033] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the steps of the keypoint-based online traffic sign text detection method according to any one of the above.

[0034] The keypoint-based online traffic sign text detection method and system provided by the application effectively reduce the complexity of the detection network, increase the recognition speed, and are more suitable for the low computing resource requirement and real-time requirement of vehicle online traffic sign recognition. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0036] Figure 1 This is a flowchart illustrating the online traffic sign text detection method based on key points provided by the present invention.

[0037] Figure 2 This is a schematic diagram of the structure of the online traffic sign text detection system based on key points provided by the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0039] Figure label:

[0040] 1: Acquisition module; 2: Detection module; 3: Text module;

[0041] 310: Processor; 320: Communication interface; 330: Memory;

[0042] 340: Communication bus. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] The following is combined with Figure 1 This invention describes an online traffic sign text detection method based on key points.

[0045] like Figure 1 As shown, this embodiment of the invention provides an online traffic sign text detection method based on key points, applied to an in-vehicle terminal, including:

[0046] Step 102: Acquire road images using vehicle-mounted image acquisition equipment;

[0047] Step 104: Input the road image into the detection network to obtain key points of traffic sign text;

[0048] Step 106: Determine the text image based on the key points, and identify the text image to obtain the online detection result of the traffic sign text;

[0049] The detection network is trained based on samples and labels.

[0050] The embodiment is based on detecting network output traffic sign text key points, effectively reducing the complexity of the detection network, increasing the recognition speed, and being more suitable for the low computing resource requirements and real-time requirements of vehicle-mounted online traffic sign recognition.

[0051] The following will illustrate the computational amount and parameter amount advantages of the embodiment relative to the traditional segmented text detection method.

[0052] Due to the flexible setting scheme of the network feature extraction part, and the difference between the traditional method and the embodiment method in the feature extraction part is not the main reason for the difference in computing resources and speed between the two, only the part of the last layer feature map after convolution to get the output value is considered.

[0053] In the embodiment method, since the output is a key point, only a single (k x k x c) x 1 convolution kernel is needed to convolve the last layer feature map, so as to obtain a key point output result with a dimension of 1 x h x w. The computational amount required by this process is M = 1 x h x w x (k x k x c) x 2, and the parameter amount involved is P = (k x k x c) x 1.

[0054] In the traditional method, since the output is segmented text, the output dimension is usually high, for example, three-dimensional output of text position, text width, and text height. In a similar three-dimensional output scenario, three (k x k x c) x 1 convolution kernels are needed to convolve the last layer feature map to get the output of each dimension. The computational amount required by this process is M' = 3 x M, and the designed parameter amount is P' = 3 x P.

[0055] It can be seen that the embodiment method effectively reduces the number of network parameters and the amount of calculation, and can effectively improve the efficiency of vehicle-mounted traffic sign text detection.

[0056] According to the above embodiment, in the embodiment:

[0057] The key points are the 4 vertices of a rectangular text box; the detection network includes a single-channel key point decoder; the channel of the key point decoder can output at least 4 key point coordinates with confidence.

[0058] The step of determining a text image according to the key points includes:

[0059] Sorting the key points based on confidence;

[0060] Judging the fourth key point after sorting:

[0061] If the confidence of the fourth key point is greater than a set threshold, the first key point, the second key point, the third key point, and the fourth key point are determined as the text image vertices.

[0062] If the confidence of the fourth key point is not greater than a set threshold, then according to the first key point, the second key point, and the third key point, a supplementary key point of a rectangular text box is fitted based on a geometric relationship, and the first key point, the second key point, the third key point, and the supplementary key point are determined as vertices of a text image.

[0063] According to the vertices of the text image, the road image is segmented to obtain a text image.

[0064] Another preferred embodiment of the present embodiment is that the key points are four vertices of a rectangular text box, the detection network includes a four-channel key point decoder, and the channel of the key point decoder can output at least one key point coordinate with a confidence.

[0065] The two schemes in the present embodiment can be set based on actual computing power of a vehicle terminal and requirements, the single-channel scheme has a lower requirement for computing resources, and the four-channel scheme can obtain a text image with higher quality.

[0066] According to any of the above embodiments, in the present embodiment:

[0067] The detection network includes a single-channel multi-text region decoder, and the channel of the multi-text region decoder can output a probability map of at least two text regions.

[0068] The probability map refers to a set of probabilities of a text region center point (or other specified point) at each coordinate.

[0069] The probability map output of the above multi-text region decoder corresponds to a probability map label, that is:

[0070] The label includes a first sub-label, or the label includes the first sub-label and a second sub-label.

[0071] The first sub-label refers to:

[0072] a single probability map including four key point probability distributions; or

[0073] four probability maps including a single key point probability distribution.

[0074] The second sub-label refers to a probability map of at least two text regions.

[0075] By setting the label as a probability map (and the corresponding key point decoder and multi-text region decoder are also output in the form of a probability map), the ratio of positive and negative samples in the training process can be more balanced, thereby avoiding poor effects of a neural network training process in which key points are used as labels and outputs.

[0076] If the label and output in the form of probability graph are not used, there is only one positive sample in the training process of the neural network, and the number of negative samples is much larger than that of the positive samples (for example, for pixel-level detection accuracy, assuming that there are 10 pixels in the image, the number of positive samples is 1, and the number of negative samples is 9, and in actual use, the ratio will be more uneven), which further leads to extremely low efficiency of the training process of the neural network, and even cannot converge to obtain a usable network.

[0077] The key point-based traffic sign text online detection device provided by the present application is described below. The key point-based traffic sign text online detection device described below can be referred to in correspondence with the key point-based traffic sign text online detection method described above.

[0078] As shown in Figure 2 The present application also provides a key point-based traffic sign text online detection system applied to a vehicle terminal, which comprises:

[0079] The acquisition module 1 is configured to acquire a road image based on a vehicle image acquisition device.

[0080] The detection module 2 is configured to input the road image into a detection network to obtain traffic sign text key points.

[0081] The text module 3 is configured to determine a text image based on the key points and identify the text image to obtain an online detection result of the traffic sign text.

[0082] The detection network is trained based on samples and labels.

[0083] Further, the text module 3 comprises:

[0084] The sorting unit is configured to sort the key points based on confidence.

[0085] The judgment unit is configured to judge the fourth key point after sorting:

[0086] If the confidence of the fourth key point is greater than a set threshold, the first key point, the second key point, the third key point and the fourth key point are determined as text image vertices.

[0087] If the confidence of the fourth key point is not greater than the set threshold, the supplementary key points of the rectangular text box are fitted based on the geometric relationship according to the first key point, the second key point and the third key point, and the first key point, the second key point, the third key point and the supplementary key points are determined as text image vertices.

[0088] The segmentation unit is configured to segment the road image based on the text image vertices to obtain a text image.

[0089] Figure 3 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 3 As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a key point-based online detection method for traffic sign text, which includes: obtaining a road image based on a vehicle-mounted image acquisition device; inputting the road image into a detection network to obtain traffic sign text key points; determining a text image according to the key points and identifying the text image to obtain an online detection result of the traffic sign text; and the detection network is obtained based on sample and label training.

[0090] In addition, the logical instruction in the memory 330 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0091] On the other hand, the present application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the key point-based online detection method for traffic sign text provided by the above-mentioned methods, which includes: obtaining a road image based on a vehicle-mounted image acquisition device; inputting the road image into a detection network to obtain traffic sign text key points; determining a text image according to the key points and identifying the text image to obtain an online detection result of the traffic sign text; and the detection network is obtained based on sample and label training.

[0092] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a key point based online detection method of traffic sign text provided by the above method, the method comprising: acquiring a road image based on a vehicle-mounted image acquisition device; inputting the road image into a detection network to obtain traffic sign text key points; determining a text image according to the key points, and identifying the text image to obtain an online detection result of the traffic sign text; and the detection network is obtained based on training of samples and labels.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0094] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0095] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for online detection of traffic sign text based on key points, characterized in that, Applications in vehicle-mounted terminals include: Road images are acquired using vehicle-mounted image acquisition equipment; The road image is input into the detection network to obtain traffic sign text key points. The key points are the four vertices of a rectangular text box. The detection network includes a single-channel key point decoder. The channel of the key point decoder can output at least four key point coordinates with confidence scores. Based on the key points, the text image is determined, and the text image is recognized to obtain the online detection result of the traffic sign text; The detection network is trained based on samples and labels. The labels include a first sub-label, or the labels include a first sub-label and a second sub-label. The first sub-label refers to a single probability map including the probability distribution of four key points, or four probability maps including the probability distribution of a single key point. The second sub-label refers to a probability map of at least two text regions.

2. The online traffic sign text detection method based on key points according to claim 1, characterized in that, The key points are the four vertices of the rectangular text box; the detection network includes a four-channel key point decoder; the channels of the key point decoder can output at least one key point coordinate with confidence.

3. The online traffic sign text detection method based on key points according to claim 1, characterized in that, The step of determining the text image based on the key points includes: The key points are ranked based on confidence level; Make a judgment on the fourth key point after sorting: If the confidence level of the fourth key point is greater than the set threshold, then the first key point, the second key point, the third key point, and the fourth key point are determined as text image vertices; If the confidence level of the fourth key point is not greater than the set threshold, then based on the first, second and third key points, the supplementary key points of the rectangular text box are obtained by fitting based on the geometric relationship, and the first key point, the second key point, the third key point and the supplementary key points are determined as the vertices of the text image. The road image is segmented based on the vertices of the text image to obtain the text image.

4. The online traffic sign text detection method based on key points according to claim 1, characterized in that, The detection network includes a single-channel multi-text region decoder; the channels of the multi-text region decoder can output probability maps of at least two text regions.

5. A traffic sign text online detection system based on key points, characterized in that, Applications in vehicle-mounted terminals include: The acquisition module is used to acquire road images based on the vehicle-mounted image acquisition device; The detection module is used to input the road image into the detection network to obtain traffic sign text key points. The key points are the four vertices of a rectangular text box. The detection network includes a single-channel key point decoder. The channel of the key point decoder can output at least four key point coordinates with confidence scores. The text module is used to determine the text image based on the key points and to identify the text image to obtain the online detection result of the traffic sign text; The detection network is trained based on samples and labels. The labels include a first sub-label, or the labels include a first sub-label and a second sub-label. The first sub-label refers to a single probability map including the probability distribution of four key points; or, four probability maps including the probability distribution of a single key point. The second sub-label refers to a probability map of at least two text regions.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the online traffic sign text detection method based on key points as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the online traffic sign text detection method based on key points as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the online traffic sign text detection method based on key points as described in any one of claims 1 to 4.

Images