Method and system for detecting an occluded sign, electronic device, storage medium

Abstract

A method and system for detecting whether a signboard is blocked, an electronic device, and a storage medium, the method comprising: acquiring an image of a signboard to be detected; performing feature extraction on the image based on a feature extraction network to obtain a feature map; predicting a bounding box of the signboard in the image based on the feature map using a bounding box prediction network; predicting an unblocked area of the signboard in the image based on the feature map using an unblocked area prediction network; and determining whether the signboard is blocked based on the bounding box and the unblocked area. The detection of whether a signboard is blocked can reduce false positives and solve the problem of uneven samples and difficulty in training a prediction model due to a lack of blocked signboard samples in practice.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method, system, electronic device, and storage medium for detecting obscured signs. Background Technology

[0002] Obscuring road signs is detrimental to road users and also hinders autonomous driving. Traffic management departments typically remove obstructions from road signs regularly. Current technologies for determining whether a sign is obscured rely on either the sign's own sensor design or traffic imagery. The latter method commonly uses object detection techniques to detect sign obstruction.

[0003] Obscuring signs are usually caused by overgrown vegetation, which is very common in traffic images. Therefore, the detection of obscuring signs by object detection methods is easily affected by vegetation, leading to false detections, such as when leaves are close to the sign. In addition, there are relatively few actual samples of obscuring signs, resulting in an imbalanced sample, which is not conducive to model training. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention provides a method, system, electronic device, and storage medium for detecting obstructed signs.

[0005] This invention provides a method for detecting obstructed signs, the method comprising:

[0006] Acquire an image showing whether the sign to be detected is obscured;

[0007] The image is used to extract features based on a feature extraction network to obtain a feature map.

[0008] Based on the feature map, a bounding box prediction network is used to predict the bounding box of the sign in the image;

[0009] Based on the feature map, an unoccluded region prediction network is used to predict the unoccluded region of the sign in the image.

[0010] Based on the surrounding box and the unobstructed area, it is determined whether the sign is obstructed.

[0011] According to a method for detecting obstructed signs provided by the present invention, determining whether the sign is obstructed includes:

[0012] Calculate the first perimeter and / or first area of ​​the bounding box;

[0013] Calculate the second perimeter and / or second area of ​​the unobstructed area;

[0014] Calculate a first ratio of the first perimeter to the second perimeter, and / or calculate a second ratio of the first area to the second area;

[0015] If the first ratio is less than the first threshold or the second ratio is less than the second threshold, then the sign is determined to be obscured; otherwise, the sign is determined to be unobscured.

[0016] According to a method for detecting occluded signs provided by the present invention, predicting the unoccluded area of ​​the sign in the image includes:

[0017] Based on the feature map, a contour region prediction network is used to predict the contour region of the unoccluded area.

[0018] Based on the outline region, update the unoccluded region.

[0019] According to a method for detecting obstructed signs provided by the present invention, the method further includes:

[0020] Calculate a first percentage of the width of the bounding box relative to the width of the image, and / or calculate a second percentage of the height of the bounding box relative to the height of the image;

[0021] Take the larger of the first percentage, the second percentage, or both, as the intermediate coefficient;

[0022] Based on a preset piecewise function, calculate the first threshold and the second threshold;

[0023] The piecewise function is based on the value of the intermediate coefficient to divide the data into segments, and the piecewise function is a linear function of the intermediate coefficient.

[0024] According to the present invention, a method for detecting occluded signs includes the following steps for predicting the bounding box dependency of the sign in the image:

[0025] Step a: Obtain the probability map of sign categories. Based on the probability map, take the image points in the probability map whose probability exceeds the third threshold as the center points of the bounding box.

[0026] Step b: Predict the width and height of the bounding box.

[0027] According to a method for detecting occluded signs provided by the present invention, predicting the unoccluded area of ​​the sign in the image includes:

[0028] Obtain a probability map of sign categories, and based on the probability map, determine whether each image point belongs to a sign category;

[0029] The unobstructed area is formed based on all image points belonging to the sign category.

[0030] According to a method for detecting obscured signs provided by the present invention, predicting the outline region of the unobscured area includes:

[0031] Obtain the binary mask;

[0032] The binary mask is used to predict the contour region.

[0033] This invention also provides a detection system for obstructed signs, the system comprising:

[0034] The acquisition module acquires an image of the sign to be detected;

[0035] The feature extraction module extracts features from the image based on a feature extraction network to obtain a feature map;

[0036] The first prediction module, based on the feature map, uses a bounding box prediction network to predict the bounding box of the sign in the image;

[0037] The second prediction module, based on the feature map, uses an unoccluded region prediction network to predict the unoccluded region of the sign in the image;

[0038] The judgment module determines whether the sign is obscured based on the bounding box and the unobstructed area.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for detecting obstructed signs as described in any of the preceding claims.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for detecting obstructed signs as described in any of the preceding claims.

[0041] The present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the method for detecting obstructed signs as described in any of the preceding claims.

[0042] The method, system, electronic device, and storage medium for detecting occluded signs provided by this invention can reduce false detections and solve the problem of difficulty in training prediction models due to uneven sample size caused by the small number of occluded sign samples in practice. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a method for detecting obstructed signs provided by the present invention;

[0045] Figure 2 This is a schematic diagram of a target detection network structure based on feature maps provided by the present invention;

[0046] Figure 3 A schematic diagram of a detection system for obstructed signs provided by the present invention;

[0047] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] The method for detecting obstructed signs provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0050] Figure 1 A flowchart illustrating a method for detecting obstructed signs provided by the present invention is shown below. Figure 1 As shown, the present invention provides a method for detecting obstructed signs, which may include the following steps.

[0051] S100: Obtain the image of the sign to be detected.

[0052] S200. Extract features from the image using a feature extraction network to obtain a feature map;

[0053] S300: Based on feature maps, a bounding box prediction network is used to predict the bounding boxes of signs in the image.

[0054] S400: Based on feature maps, an unoccluded region prediction network is used to predict the unoccluded regions of signs in the image.

[0055] S500: Based on the bounding box and the unobstructed area, determine whether the sign is obstructed.

[0056] Optionally, the image of whether the sign to be detected is obscured is captured by the vehicle's camera during autonomous driving.

[0057] Optionally, both the bounding box of the sign in the predicted image and the unoccluded area of ​​the sign in the predicted image are implemented using a neural network.

[0058] Optionally, determining whether the sign is obscured includes:

[0059] Calculate the first perimeter and / or first area of ​​the bounding box;

[0060] Calculate the second perimeter and / or second area of ​​the unobstructed area;

[0061] Calculate a first ratio of the first perimeter to the second perimeter, and / or calculate a second ratio of the first area to the second area;

[0062] If the first ratio is less than the first threshold or the second ratio is less than the second threshold, then the sign is determined to be obscured; otherwise, the sign is determined to be unobscured.

[0063] Optionally, the method further includes:

[0064] Calculate a first percentage of the width of the bounding box relative to the width of the image, and / or calculate a second percentage of the height of the bounding box relative to the height of the image;

[0065] Take the larger of the first percentage, the second percentage, or both, as the intermediate coefficient;

[0066] Calculate the first threshold and the second threshold based on the preset piecewise function;

[0067] The piecewise function is a linear function of the intermediate coefficients, where the segments are based on the magnitude of the intermediate coefficients.

[0068] Preferably, the linear function has the same slope in different segments.

[0069] Preferably, the first threshold and the second threshold are denoted as T1 and T2, respectively, and the calculation methods for T1 and T2 include:

[0070] Let w and h be the width and height of the bounding box, respectively, and let W and H be the width and height of the image, respectively.

[0071] Calculate the intermediate coefficient r, and the calculation formula of r is as follows:

[0072] r = max(w / W, h / H)

[0073] If r > 1 / 8, the calculation formulas of T1 and T2 are as follows:

[0074] T1 = 0.9 + (r - 1 / 8) * 0.1

[0075] T2 = 0.85 + (r - 1 / 8) * 0.1

[0076] If 1 / 32 < r < 1 / 8, the calculation formulas of T1 and T2 are as follows:

[0077] T1 = 0.8 + (r - 1 / 32) * 0.1

[0078] T2 = 0.75 + (r - 1 / 32) * 0.1

[0079] If r < 1 / 32, the calculation formulas of T1 and T2 are as follows:

[0080] T1 = 0.75 + (r - 1 / 8) * 0.1

[0081] T2 = 0.7 + (r - 1 / 8) * 0.1

[0082] By calculating and obtaining the values of T1 and T2 in segments as described above, the false detection rate of the signboard can be reduced.

[0083] Preferably, Figure 2 is a schematic structural diagram of a detection network provided by the present invention. As Figure 2 shown, the feature extraction network includes a backbone network and FPN. After obtaining the feature map based on this feature extraction network, the bounding box of the signboard in the image is predicted through the bounding box prediction network (bounding box detection head), the unoccluded area of the signboard in the image is predicted through the unoccluded area prediction network (unoccluded area detection head), and the contour area of the signboard in the image is predicted through the contour area prediction network (contour area detection head).

[0084] Optionally, predicting the bounding box of the signboard in the image depends on the following steps:

[0085] Step a: Obtain the probability map of the signboard category. Based on this probability map, the image points in the probability map with a probability exceeding the third threshold are used as the center points of the bounding box;

[0086] Step b: Predict the width and height of the bounding box.

[0087] Optionally, predicting the unoccluded area of the signboard in the image includes:

[0088] Obtain a probability map of sign categories, and based on the probability map, determine whether each image point belongs to a sign category;

[0089] The unobstructed area is formed based on all image points belonging to the sign category.

[0090] Preferably, the acquisition of the probability map includes: adding a Gaussian probability region near the center of the bounding box in the image. The center of this region is the center point of the bounding box, the radius is 1 / 4 of the smaller value of the width and height of the bounding box, the probability value is 1 at the center point and decreases to 0 at the edge.

[0091] Optionally, predict the unobstructed area of ​​the sign in the image, including:

[0092] Based on feature maps, a contour region prediction network is used to predict the contour regions of unoccluded areas.

[0093] Update the unoccluded area based on the outline region.

[0094] Optionally, the outline region of the unoccluded area is predicted, including:

[0095] Obtain the binary mask;

[0096] The binary mask is used to predict the contour region.

[0097] Preferably, predicting the contour region of the unoccluded area includes: first, obtaining the contour boundary region of the unoccluded area, which extends outward from the contour line as the center line to the unoccluded side of the sign, with an extension width of max(10, 0.25*min(h, w)), where w and h are the predicted width and height of the bounding box of the sign; second, based on the probability map of the contour boundary region, after comparison with a fourth threshold, generating a binary mask of the contour region; finally, multiplying the binary mask by the original image to obtain the predicted contour region. Further, the fourth threshold is 0.6. By specifically predicting the boundary region, the final updated unoccluded area is closer to the real area.

[0098] This embodiment detects occluded signs by using the method of calculating the perimeter and / or area ratio, which reduces false detections and solves the problem of difficulty in training the prediction model due to the uneven sample size caused by the small number of actual occluded signs.

[0099] Furthermore, based on the foregoing embodiments, in another embodiment, this embodiment provides a method for detecting occluded signs, wherein the prediction model for predicting the bounding box of the sign in the image includes at least one of the following:

[0100] Faster R-CNN model, YOLO model, CenterNet model.

[0101] This embodiment discloses a method for predicting bounding boxes of signs in an image using a specific object detection model.

[0102] The detection system for obstructed signs provided by the present invention will be described below. The detection system for obstructed signs described below can be referred to in correspondence with the detection method for obstructed signs described above.

[0103] Figure 3 A schematic diagram of a detection system for obstructed signs provided by the present invention is shown below. Figure 3 As shown, the present invention also provides a detection system for obstructed signs, the system comprising:

[0104] The acquisition module acquires the image of the sign to be detected;

[0105] The feature extraction module extracts features from the image based on a feature extraction network to obtain a feature map.

[0106] The first prediction module is based on the feature map and uses a bounding box prediction network to predict the bounding box of the sign in the image.

[0107] The second prediction module is based on the feature map and uses an unoccluded region prediction network to predict the unoccluded region of the sign in the image.

[0108] The judgment module determines whether the sign is obscured based on the bounding box and the unobstructed area.

[0109] This embodiment can reduce false detections for the detection of occluded signs and solves the problem of difficulty in training the prediction model due to the small number of actual samples of occluded signs.

[0110] Figure 4 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a method for detecting obstructed signs, the method including:

[0111] Acquire an image showing whether the sign to be detected is obscured;

[0112] The image is used to extract features based on a feature extraction network to obtain a feature map.

[0113] Based on the feature map, a bounding box prediction network is used to predict the bounding box of the sign in the image;

[0114] Based on the feature map, an unoccluded region prediction network is used to predict the unoccluded region of the sign in the image.

[0115] Based on the bounding box and the unobstructed area, determine whether the sign is obstructed.

[0116] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the obstruction sign detection method provided by the above methods, the method comprising:

[0118] Acquire an image showing whether the sign to be detected is obscured;

[0119] The image is used to extract features based on a feature extraction network to obtain a feature map.

[0120] Based on the feature map, a bounding box prediction network is used to predict the bounding box of the sign in the image;

[0121] Based on the feature map, an unoccluded region prediction network is used to predict the unoccluded region of the sign in the image.

[0122] Based on the bounding box and the unobstructed area, determine whether the sign is obstructed.

[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods for detecting obstructed signs provided above, the methods comprising:

[0124] Acquire an image showing whether the sign to be detected is obscured;

[0125] The image is used to extract features based on a feature extraction network to obtain a feature map.

[0126] Based on the feature map, a bounding box prediction network is used to predict the bounding box of the sign in the image;

[0127] Based on the feature map, an unoccluded region prediction network is used to predict the unoccluded region of the sign in the image.

[0128] Based on the bounding box and the unobstructed area, determine whether the sign is obstructed.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting obstructed signs, characterized in that, The method includes: Acquire an image showing whether the sign to be detected is obscured; The image is used to extract features based on a feature extraction network to obtain a feature map. Based on the feature map, a bounding box prediction network is used to predict the bounding box of the sign in the image; Based on the feature map, an unoccluded region prediction network is used to predict the unoccluded region of the sign in the image. Based on the surrounding box and the unobstructed area, determine whether the sign is obstructed; Determining whether the sign is obscured includes: Calculate the first perimeter and / or first area of ​​the bounding box; Calculate the second perimeter and / or second area of ​​the unobstructed area; Calculate a first ratio of the first perimeter to the second perimeter, and / or calculate a second ratio of the first area to the second area; If the first ratio is less than the first threshold or the second ratio is less than the second threshold, then the sign is determined to be obscured; otherwise, the sign is determined to be unobscured.

2. The method for detecting obstructed signs according to claim 1, characterized in that, Predicting the unobstructed area of ​​the sign in the image includes: Based on the feature map, a contour region prediction network is used to predict the contour region of the unoccluded area. Based on the outline region, update the unoccluded region.

3. The method for detecting obstructed signs according to claim 1, characterized in that, The method further includes: Calculate a first percentage of the width of the bounding box relative to the width of the image, and / or calculate a second percentage of the height of the bounding box relative to the height of the image; Take the larger of the first percentage, the second percentage, or both, as the intermediate coefficient; Based on a preset piecewise function, calculate the first threshold and the second threshold; The piecewise function is based on the value of the intermediate coefficient to divide the data into segments, and the piecewise function is a linear function of the intermediate coefficient.

4. The method for detecting obstructed signs according to claim 1, characterized in that, Predicting the bounding box dependency of the sign in the image includes the following steps: Step a: Obtain the probability map of sign categories. Based on the probability map, take the image points in the probability map whose probability exceeds the third threshold as the center points of the bounding box. Step b: Predict the width and height of the bounding box.

5. The method for detecting obstructed signs according to claim 1, characterized in that, Predicting the unobstructed area of ​​the sign in the image includes: Obtain a probability map of sign categories, and based on the probability map, determine whether each image point belongs to a sign category; The unobstructed area is formed based on all image points belonging to the sign category.

6. The method for detecting obstructed signs according to claim 2, characterized in that, Predicting the contour region of the unoccluded area includes: Obtain the binary mask; The binary mask is used to predict the contour region.

7. A detection system for obstructed signs, characterized in that, For implementing the steps of the method for detecting obstructed signs as described in any one of claims 1-6, the system comprises: The acquisition module acquires an image of the sign to be detected; The feature extraction module extracts features from the image based on a feature extraction network to obtain a feature map; The first prediction module, based on the feature map, uses a bounding box prediction network to predict the bounding box of the sign in the image; The second prediction module, based on the feature map, uses an unoccluded region prediction network to predict the unoccluded region of the sign in the image; The judgment module determines whether the sign is obscured based on the bounding box and the unobstructed area.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for detecting obstructed signs as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting obstructed signs as described in any one of claims 1-6.

Images