Logo detection method and device, nonvolatile storage medium and computer device

Abstract

The application discloses a kind of sign detection method, device, nonvolatile storage medium and computer equipment.Therein, the method includes: obtaining historical frame, target frame and the motion information of vehicle, wherein historical frame and target frame are the image obtained by the camera of vehicle shooting;According to motion information and historical frame in historical detection frame, determine target area, wherein historical detection frame is used to mark traffic sign in historical frame;According to target area and target frame, generate target detection frame, wherein target detection frame is used to mark target traffic sign in target area in target frame.The application solves the technical problem that a large amount of computing power is required to identify traffic sign from video images in real time while driving.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method, apparatus, non-volatile storage medium, and computer device for detecting a mark. Background Technology

[0002] With the development of artificial intelligence and the increasing demand for autonomous driving, traffic sign detection has become an essential capability for autonomous vehicles. The detection results not only provide road regulation input for autonomous driving functions but also serve as a key input element for localization algorithms, making traffic sign detection of great significance. Currently, some teams have proposed traffic sign detection schemes, but these schemes still suffer from problems such as poor robustness, low accuracy, and insufficient information. For example, the proposed schemes either have high computational requirements, making them unable to meet the real-time requirements of sign recognition, or are prone to missed detections and false detections.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a sign detection method, apparatus, non-volatile storage medium, and computer equipment to at least solve the technical problem that a solution for timely identification of traffic signs from video images during traffic flow requires a large amount of computing power.

[0005] According to one aspect of the present invention, a sign detection method is provided, comprising: acquiring historical frames, a target frame, and vehicle motion information, wherein the historical frames and the target frame are images captured by a camera of the vehicle; determining a target region based on the motion information and historical detection boxes in the historical frames, wherein the historical detection boxes are used to mark traffic signs in the historical frames; and generating a target detection box based on the target region and the target frame, wherein the target detection box is used to mark target traffic signs located within the target region in the target frame.

[0006] Optionally, the method further includes: identifying the target traffic sign and obtaining a classification result of the target traffic sign; identifying traffic auxiliary sign information corresponding to the target traffic sign based on the target area and the target detection box; and outputting a sign detection result corresponding to the target traffic sign based on the classification result and the traffic auxiliary sign information.

[0007] Optionally, determining the target region based on the motion information and the historical detection boxes in the historical frames includes: determining the first coordinates of the traffic sign marked by the historical detection box in the world coordinate system based on the historical frames; predicting the second coordinates of the traffic sign marked by the historical detection box in the world coordinate system at the time corresponding to the target frame based on the motion information; generating a prediction box corresponding to the historical detection box in the target frame based on the second coordinates; and generating the target region including the prediction box based on the prediction box.

[0008] Optionally, generating a target detection box based on the target region and the target frame includes: cropping the image corresponding to the target region from the target frame to obtain a target image; and performing image recognition on the target image to obtain the target detection box.

[0009] Optionally, the step of performing image recognition on the target image to obtain the target detection box includes: performing image recognition on the target image to obtain an initial detection box; establishing an MHT tree structure based on the historical detection boxes and the initial detection box, wherein the MHT tree structure is used to characterize the weak matching relationship between the historical detection boxes and the initial detection box; and selecting the target detection box from the initial detection box based on the MHT tree structure.

[0010] Optionally, the step of filtering the target detection box from the initial detection box based on the MHT tree structure includes: filtering the target detection box from the initial detection box based on the Hungarian algorithm, wherein the Hungarian algorithm is used to correct the weak matching relationship in the MHT tree structure into a one-to-one correspondence between the historical detection boxes and the target detection box.

[0011] Optionally, the step of filtering the target detection box from the initial detection box based on the Hungarian algorithm includes: obtaining the first texture corresponding to each of the historical detection boxes and the second texture corresponding to each of the initial detection boxes; matching the globally optimal detection box in the initial detection box with the historical detection boxes one-to-one according to the first texture, the second texture and the cost function constructed based on the Hungarian algorithm to obtain the target detection box, wherein the cost function is constructed based on the texture features and feature similarity of the textures.

[0012] According to another aspect of the present invention, a sign detection device is also provided, comprising: an acquisition module, configured to acquire historical frames, a target frame, and vehicle motion information, wherein the historical frames and the target frame are images captured by a camera of the vehicle; a determination module, configured to determine a target region based on the motion information and historical detection boxes in the historical frames, wherein the historical detection boxes are used to mark traffic signs in the historical frames; and a generation module, configured to generate a target detection box based on the target region and the target frame, wherein the target detection box is used to mark traffic signs located within the target region in the target frame.

[0013] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is executed, the device where the non-volatile storage medium is located is controlled to perform any of the above-described flag detection methods.

[0014] According to another aspect of the present invention, a computer device is also provided, the computer device including a memory and a processor, the memory being used to store a program, and the processor being used to run the program stored in the memory, wherein the program, when running, executes any of the above-described flag detection methods.

[0015] In this embodiment of the invention, a method is adopted to establish the relationship between traffic signs between multiple frames of images before and after the capture. By connecting the relationship between historical frames and target frames based on vehicle motion information, the purpose of predicting the target detection box for marking traffic signs in the target frame based on the historical detection box in the historical frame is achieved. This realizes the technical effect of efficiently and accurately identifying traffic signs from real-time video images of vehicles, and solves the technical problem that the scheme of timely identifying traffic signs from video images during traffic requires a lot of computing power. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal for implementing a marker detection method is shown.

[0018] Figure 2 This is a schematic flowchart of a mark detection method provided according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of coordinate system transformation relationships provided by an optional embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram illustrating the principle of detecting the position of a detection box in a target frame according to an optional embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of traffic sign detection results provided by an optional embodiment of the present invention;

[0022] Figure 6 This is a structural block diagram of a mark detection device provided according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] According to an embodiment of the present invention, a marker detection method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a marker detection method is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the flag detection method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the flag detection method of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0030] To address the aforementioned issues, existing traffic sign detection schemes suffer from poor robustness, low accuracy, and insufficient information. For example: 1) Converting the image color gamut to HSV and then filtering by pass range, followed by connecting component algorithms to determine the location and features of the traffic sign. However, this method heavily relies on the uniformity of traffic sign features; furthermore, if elements with similar features appear in the image, false detections are highly likely. 2) Using artificial intelligence methods, deep learning models are used to directly detect traffic signs in the image, thus improving accuracy. While this improves precision, it doesn't correlate traffic signs between frames, making it impossible to determine whether a detected sign is a newly appearing target or one that has existed for some time. Furthermore, large-scale deep learning networks require significant computational power, often failing to meet the real-time requirements of vehicle-side domain control.

[0031] Figure 2 This is a flowchart illustrating the mark detection method provided according to an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0032] Step S202: Obtain historical frames, target frames, and vehicle motion information, wherein the historical frames and target frames are images captured by the vehicle's camera.

[0033] The target frame can be an image captured by the vehicle's camera at the current moment, while the historical frame can be an image captured by the same camera at a previous moment (i.e., a moment before the current moment). For example, the historical frame can be the image preceding the target frame. The vehicle's motion information can be its kinematic information, which includes the vehicle's motion state from the moment the historical frame was captured to the moment the target frame was captured.

[0034] Step S204: Determine the target region based on motion information and historical detection boxes in historical frames, wherein the historical detection boxes are used to mark traffic signs in historical frames.

[0035] To quickly detect traffic signs from the target frame, this embodiment can utilize the detection results of traffic signs in historical frames to assist in the detection of traffic signs in the target frame. Optionally, this embodiment can acquire the image information of the current frame, the kinematic information of the vehicle, and the final detection result of the traffic sign in the previous frame (i.e., the historical detection boxes in the historical frames), and filter the historical detection boxes based on a preset confidence threshold, directly discarding detection results that do not meet the conditions, such as discarding detection boxes with too low a confidence threshold.

[0036] As an optional embodiment, determining the target region based on motion information and historical detection boxes in historical frames may include the following steps: determining the first coordinates of the traffic sign marked by the historical detection box in the world coordinate system based on the historical frame; predicting the second coordinates of the traffic sign marked by the historical detection box in the world coordinate system at the time corresponding to the target frame based on the motion information; generating a prediction box corresponding to the historical detection box in the target frame based on the second coordinates; and generating a target region including the prediction box based on the prediction box.

[0037] Subsequently, based on the principle of similar triangles, the possible positions of the detection boxes in the next frame (i.e., the target frame) of the historical frame can be predicted. The vehicle's movement distance and orientation angle changes between consecutive frames are predicted using vehicle kinematic information (e.g., vehicle speed, yaw angle change rate), and the coordinates are transformed to the camera coordinate system using camera extrinsic parameters. Figure 3 This is a schematic diagram of coordinate system transformation relationships provided by an optional embodiment of the present invention. The relationship between the world coordinate system, camera coordinate system, and image coordinate system is as follows: Figure 3 As shown: In the figure, O is the origin of the camera coordinate system, and p is a point in the image coordinate system. After obtaining the vehicle's position change in the camera coordinate system based on the vehicle's kinematic information, since driving cameras often use a pinhole model, the coordinate systems have a pinhole imaging relationship. The approximate position of the detection box in the current frame can be estimated based on the detection results in the previous frame using the similar triangle method.

[0038] Figure 4 This is a schematic diagram illustrating the principle of inferring the position of a detection box in a target frame according to an optional embodiment of the present invention, as shown below. Figure 4 As shown, the vehicle's movement distance and orientation angle change between the historical frame and the target frame can be calculated first using vehicle kinematics information. Then, based on the principle of similar triangles, the distance between the traffic sign detected in the historical frame and the vehicle body in the VCS coordinate system (vehicle coordinate system) can be estimated.

[0039]

[0040] Where x vcs f represents the distance between the traffic sign and the vehicle body in the VCS coordinate system. v s represents the image distance of the camera used. H Let s be the height of the traffic sign frame in the VCS coordinate system. imH This represents the height of the traffic sign frame in the image coordinate system.

[0041] Similarly, the lateral position of a traffic sign in the VCS coordinate system can be calculated by acquiring information such as the vanishing point of the camera, the yaw rate of the vehicle, and the lateral position of the traffic sign in the image.

[0042] After obtaining the position information of traffic signs in the VCS coordinate system from historical frames, the possible position of the same traffic sign bounding box in the image coordinate system in the target frame can be predicted again using the vehicle's kinematic information and the principle of similar triangles. The specific implementation scheme is as follows:

[0043]

[0044] Among them, y im Let y be the ordinate of the center point of the target traffic sign bounding box in the predicted target frame in the image coordinate system. vp The vanishing point of the camera. This represents the ordinate of the center point of the traffic sign detection box in the historical frame within the image coordinate system. Let yaw be the position of the traffic sign on the x-axis in the VCS coordinate system for a historical frame, L be the distance the vehicle travels between two frames, and yaw be the distance the vehicle travels between the two frames. rate This represents the rate of change of the vehicle's yaw angle between two frames.

[0045] Similarly, the horizontal coordinate of the center point of the traffic sign box in the current frame can be calculated in the image coordinate system, thus obtaining the possible location of the detection box in the target frame predicted by the historical detection boxes in the historical frames, that is, the target region in the target frame is predicted.

[0046] Step S206: Generate a target detection box based on the target region and the target frame, wherein the target detection box is used to mark the target traffic signs located within the target region in the target frame.

[0047] Through the above steps, a relationship between traffic signs is established between multiple frames of images captured before and after the capture. By connecting the relationship between historical frames and target frames based on vehicle motion information, the goal of predicting the target detection box for marking traffic signs in the target frame based on the historical detection box in the historical frame is achieved. This realizes the technical effect of efficiently and accurately identifying traffic signs from real-time video images of vehicles, and solves the technical problem that the scheme of timely identification of traffic signs from video images during traffic movement requires a large amount of computing power.

[0048] As an optional embodiment, the above method further includes the following steps: identifying the target traffic sign and obtaining the classification result of the target traffic sign; identifying the traffic auxiliary sign information corresponding to the target traffic sign based on the target area and the target detection box; and outputting the sign detection result corresponding to the target traffic sign based on the classification result and the traffic auxiliary sign information. This optional embodiment provides a method for refining the identified target traffic sign. Figure 5 This is a schematic diagram of traffic sign detection results provided by an optional embodiment of the present invention, such as... Figure 5 As shown, traffic signs typically consist of traffic graphic symbols and auxiliary signs. Figure 5 The circular sign enclosed in the upper right square, indicating a speed limit of 40 km / h, is the traffic sign graphic, while the portion enclosed in the lower right square, indicating a "ramp," is the auxiliary sign. In this optional embodiment, the classification result of the target traffic sign can be obtained through image recognition based on the traffic sign graphic, and the traffic auxiliary sign information can be obtained through recognition based on the traffic auxiliary sign. When identifying the type of traffic sign, combining the traffic sign graphic and the auxiliary sign information can output an accurate traffic sign detection result, improving the accuracy of the detection result.

[0049] As an optional implementation, four deep learning models can be used to detect traffic signs: a full-image detection model, a fast detection model, an auxiliary sign detection model, and a classification model. The four models are briefly described below:

[0050] Full-image detection model: The model takes a complete image frame as input, such as a historical frame or a target frame. The model detects traffic signs within the entire image region of a frame, but does not classify the content. This model can determine whether to use an inter-frame call scheme based on the design and computing power of the operating platform. If inter-frame call is used, a fast detection model needs to be called to supplement the image frames where the full-image detection model was not called. For example, if the vehicle platform system has sufficient computing power, every frame captured by the vehicle's camera can be input into the full-image detection model, which will detect traffic signs sequentially from each frame. If the vehicle platform system's computing power is insufficient, the full-image detection model can be called every one frame or any number of frames, while a fast detection model is called in the intervening frames to detect traffic signs in the region.

[0051] Fast Detection Model: The input to the model is a Region of Interest (ROI) image. The ROI's range is selected based on the predicted position of the detection box in the previous frame within the current frame, and traffic signs are detected within this range. This model also does not classify content. Its function is essentially the same as the full-image detection model, but due to the smaller input, a model with fewer parameters can be used to significantly improve computational efficiency. Optionally, the input to the fast detection model can be the target frame described in the above embodiments and optional embodiments. The detection box in the previous frame can be a historical detection box from a previous frame. The fast detection model can delineate a target region in the target frame based on the historical detection boxes and vehicle motion information, generate a target detection box based on the target region, and detect the target traffic sign within the image range of the target frame corresponding to the target detection box. That is, if the target traffic sign is located within a historical detection box in a previous frame, it is highly likely to be located within a target detection box in the target frame. Clearly, the fast detection model does not require complex full-image detection of the target frame, thus greatly saving computational power and speeding up the detection time of traffic signs.

[0052] Classification Model: The input of this model is the image within the bounded area of ​​the above full-image detection model or fast detection model. It classifies the content within the image and outputs the sign detection results corresponding to the target traffic sign, such as speed limit and specific value, speed limit cancellation and specific value, no parking, etc.

[0053] Traffic sign auxiliary detection model: The input of this model is a ROI image, which is obtained based on the location of the traffic sign detected in the current target frame, and it detects traffic sign auxiliary signs within this range, such as detecting... Figure 5 The auxiliary card in the middle, that is, the one that is detected Figure 5 The area enclosed in the lower box represents the "ramp". This model is mainly used to expand traffic sign detection information and determine the lane relationships corresponding to the traffic sign content.

[0054] In practical applications, after obtaining the location of the predicted bounding box, the system determines whether to use a full-image detection model or a fast detection model based on the preset scheme selected in the application. If the full-image detection model is used, the detection bounding box is obtained directly, and distortion correction is performed before further processing. If the fast detection model is used, the location of the predicted bounding box is expanded to obtain the Region of Interest (ROI), and detection is performed within this region. After obtaining the target detection bounding box, distortion correction is performed. The fast detection model can use any lightweight deep learning model that meets the requirements of accuracy and real-time performance.

[0055] Using a fast detection model can effectively reduce the false negative rate. Furthermore, to improve efficiency and suppress false positives when using a fast detection model, the following processing can be performed:

[0056] The multiple bounding boxes in the image input to the fast detection model are sorted by size, with larger boxes prioritized for recognition. This is because larger bounding boxes in the image indicate that the traffic sign within that bounding box is closer to the vehicle, thus receiving higher detection priority. Conversely, if a smaller bounding box is located within a larger bounding box, the smaller bounding box is not detected.

[0057] Obtain the detection results of all fast detections of traffic signs in the image, use the NMS (Non-Maximum Suppression) algorithm to obtain non-overlapping detection boxes, and return the confidence results of each detection box.

[0058] After the fast detection model completes its detection, all detection boxes are initially screened based on the confidence scores of the output detection boxes, with those below the confidence threshold being directly deleted. Subsequently, an MHT tree structure is constructed between the detection boxes and the predicted boxes to initially establish matching relationships. A predicted box is a parent node, and the detection boxes within its ROI are its child nodes. It should be noted that the predicted boxes are based on historical detection boxes in historical frames, predicting the detection boxes in the target frame. For each predicted box, it can be reasonably expanded to obtain the corresponding ROI region. Since the ROI region can be large or small, it may contain multiple initial detection boxes (i.e., candidate boxes within the ROI region). It is necessary to determine which of these initial detection boxes corresponds one-to-one with a historical detection box, thus identifying the target detection box (i.e., the target detection box and the historical detection box both define the same traffic sign in reality). For each predicted box, detection boxes with a distance less than a threshold can be stored as candidate boxes to be matched, achieving a preliminary one-to-many weak matching relationship, where one historical detection box corresponds to multiple candidate boxes.

[0059] As an optional embodiment, image recognition is performed on the target image to obtain target detection boxes, including the following steps: image recognition is performed on the target image to obtain initial detection boxes; based on historical detection boxes and initial detection boxes, an MHT tree structure is established, wherein the MHT tree structure is used to characterize the weak matching relationship between historical detection boxes and initial detection boxes; based on the MHT tree structure, target detection boxes are obtained by filtering from the initial detection boxes.

[0060] Furthermore, the candidate boxes output by the model can be further filtered based on preset width and height thresholds. If a candidate box is located at the edge of the image, considering that image edge distortion is large and can easily lead to false detections, and that traffic signs located at the image edge mean that they are about to be overtaken by vehicles, their information is often no longer important, the candidate box can also be discarded. In addition, if a candidate box is located below the vanishing point of the image, it will also be discarded, because according to common sense, traffic signs will not appear below the horizon. After the above filtering, candidate boxes with a weak correlation with the predicted box are retained and added to the unclassified array. If the number of elements in the unclassified array is less than the threshold, it is determined in a subsequent step whether to add detection box results to the current frame. Finally, all unidentified detection boxes are returned, completing the detection process.

[0061] As an optional embodiment, generating a target detection box based on the target region and the target frame includes: cropping the image corresponding to the target region from the target frame to obtain a target image; and performing image recognition on the target image to obtain the target detection box.

[0062] As an optional embodiment, the target detection box is obtained by filtering from the initial detection box based on the MHT tree structure, including the following steps: the target detection box is obtained by filtering from the initial detection box based on the Hungarian algorithm, wherein the Hungarian algorithm is used to correct the weak matching relationship in the MHT tree structure into a one-to-one correspondence between the historical detection box and the target detection box.

[0063] As an optional embodiment, the target detection box is obtained by filtering from the initial detection box based on the Hungarian algorithm, including: obtaining the first texture corresponding to the historical detection boxes and the second texture corresponding to the initial detection box; matching the globally optimal detection box in the initial detection box with the historical detection boxes one-to-one according to the first texture, the second texture and the cost function constructed based on the Hungarian algorithm to obtain the target detection box, wherein the cost function is constructed based on the texture features and feature similarity of the texture.

[0064] The above optional embodiments include a method for determining the target detection box. Specifically, the following optional embodiments can be used for processing:

[0065] A post-processing module is used, whose main function is to achieve one-to-one matching between the predicted bounding box and the detection bounding box. In this process, traffic signs are classified and auxiliary signs are detected. The detection results are then adjusted in size using a filter, and the final detection results are output.

[0066] First, a classification model can be called on all remaining detection box results (candidate boxes) to obtain the attributes of traffic signs in each box. Then, the Hungarian algorithm is used to correct the one-to-many association relationship into a one-to-one association relationship. The Hungarian algorithm is a combinatorial optimization algorithm that solves the task allocation problem in polynomial time. Using this algorithm and a defined cost function, a globally optimal one-to-one matching result between the predicted box and the target detection box in the initial detection box can be generated. The cost function can be used to determine the distance, aspect ratio, area difference, and feature difference between any two boxes in the predicted and candidate boxes, helping to determine which of the two currently detected boxes matches a traffic sign in a historical detection box in a historical frame. Specifically, matching can be performed based on texture features in CNNs, using the differences in texture between different targets to distinguish different objects. Texture features are obtained and further feature vectors are derived. The difference between feature vectors is calculated using Euclidean distance, cosine distance, etc. The more similar two objects are, the smaller their feature distance, and vice versa. Therefore, the feature distance can be used to determine whether objects in consecutive frames have a high degree of similarity, thereby associating objects between two frames. In this way, the Hungarian algorithm is initially used to optimize one-to-many matching in the MHT tree structure. For detection results that cannot be matched using CNN features, a cost function can be further constructed based on the IOU (Intersection of Union) with the predicted bounding box and the center distance, and the Hungarian algorithm can be used for secondary matching to finally obtain the globally optimal one-to-one matching relationship between the historical detection box and the target detection box.

[0067] Based on the matching results, the information of successfully matched target detection boxes can be updated in the structure of successfully matched prediction boxes to ensure the transitivity of inter-frame relationships. If a target detection box fails to match and is not located at the edge of the image, the candidate box is added to the prediction box array and enters the processing sequence of the next frame. This indicates that the traffic sign in the candidate box is appearing for the first time in the target frame and is used to predict traffic signs in images after the target frame.

[0068] By establishing a one-to-one matching relationship, the attribute information of the historical detection box in the historical frame is obtained, and the current attribute and the historical attribute are used to vote to select the detection result attribute with the most votes. The detection result attribute is then compared with the threshold to determine whether it meets the output standard.

[0069] The voting queue can be designed as a fixed-length sliding window to reduce false detections and improve the accuracy of attribute detection results. For example, if a voting queue of length 5 is created, all traffic sign objects require at least 5 consecutive frames with matching detection results. The final attribute output will be the attribute with the most votes from these 5 votes. For instance, if the voting results for a detected traffic sign in 5 consecutive frames are: No Stopping, Speed ​​Limit 70, Speed ​​Limit 60, Speed ​​Limit 60, Speed ​​Limit 60, then the final output will be "Speed ​​Limit 60" as the detection result.

[0070] After classification, all larger speed limit traffic signs are retrieved, and the auxiliary sign recognition model is called on them to verify whether the recognition results contain auxiliary information such as "ramp" to determine the lane associated with the traffic sign. This model can be configured to be called every other frame to save computing power, and if the total number of votes for an attribute exceeds a threshold, the model can be stopped. The logic of the auxiliary sign recognition model is as follows:

[0071] like Figure 5 As shown, the upper frame (the frame that encloses the "40" sign) is the detection frame in the current frame. Based on this detection frame, it is expanded to obtain the outermost frame that encloses the entire sign (i.e., the ROI region frame). Within this ROI region, the auxiliary sign detection model is called to obtain the lower frame (the frame that encloses the "ramp" sign) and the auxiliary sign information within it.

[0072] Subsequently, the predicted bounding box array is maintained, and objects that have not established a matching relationship across multiple consecutive frames are deleted. Simultaneously, predicted bounding boxes with high overlap are merged. Finally, the Kalman filter algorithm is applied to all objects in the array. Using the predicted width and height of the current bounding box as input and the width and height of the detected bounding box in the current frame as observations, the two are combined using Kalman gain to obtain the final detection box size information. Specifically, applying the Kalman filter algorithm makes the width and height of the detection boxes more stable, and a weighted average of the predicted box size and the current detection box size is used to fine-tune the boundaries of the current detection box.

[0073] The specific Kalman filter process is as follows:

[0074] (Predicting state changes)

[0075] (Predicting and estimating the covariance matrix)

[0076] (Measurement residuals)

[0077] (Measurement of residual covariance)

[0078] (Optimal Kalman gain)

[0079] (Updated state estimate)

[0080] P k|k =(IK k H h )P k|k-1 (Updated covariance estimate)

[0081] This completes the entire traffic sign processing procedure and outputs the final traffic sign detection results.

[0082] The overall testing process is shown in the following diagram:

[0083] Step S1: Predict the position of the historical detection box in the historical frame and predict its position in the current target frame to obtain the ROI region in the target frame.

[0084] Step S2: Expand the ROI region, call the detection model, and remove some invalid detection boxes based on prior information, such as detection boxes that are already located at the image edge of the target frame;

[0085] Step S3: Use the MHT tree structure to establish a rough match between the predicted bounding box and the detected bounding box;

[0086] Step S4: Select high-priority detection boxes as recognition candidate boxes;

[0087] Step S5: Call the classification model to identify the content in the detection box;

[0088] Step S6: Establish a one-to-one matching relationship between the predicted bounding box and the detected bounding box using the Hungarian algorithm;

[0089] Step S7: Establish the detection box that did not achieve a match as a new prediction box target, indicating that the traffic sign in the prediction box may be a traffic sign that has not appeared in the historical frame.

[0090] Step S8: Call the auxiliary card model for detection;

[0091] Step S9: Correct the position and size of the traffic sign detection box using Kalman filtering;

[0092] Step S10: Output the position of the traffic sign detection box.

[0093] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the mark detection method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0095] According to embodiments of the present invention, a sign detection apparatus for implementing the above-described sign detection method is also provided. Figure 6 This is a structural block diagram of a mark detection device provided according to an embodiment of the present invention, such as... Figure 6 As shown, the sign detection device includes: an acquisition module 62, a determination module 64, and a generation module 66. The sign detection device will be described below.

[0096] The acquisition module 62 is used to acquire historical frames, target frames, and vehicle motion information, wherein the historical frames and target frames are images captured by the vehicle's camera;

[0097] The determination module 64 is used to determine the target region based on motion information and historical detection boxes in historical frames, wherein the historical detection boxes are used to mark traffic signs in historical frames;

[0098] The generation module 66 is used to generate target detection boxes based on the target region and the target frame, wherein the target detection boxes are used to mark traffic signs located within the target region in the target frame.

[0099] It should be noted that the acquisition module 62, the determination module 64, and the generation module 66 mentioned above correspond to steps S202 to S206 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in the embodiments.

[0100] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0101] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the mark detection method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned mark detection method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0102] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring historical frames, target frames, and vehicle motion information, wherein the historical frames and target frames are images captured by the vehicle's camera; determining a target area based on the motion information and historical detection boxes in the historical frames, wherein the historical detection boxes are used to mark traffic signs in the historical frames; and generating target detection boxes based on the target area and the target frame, wherein the target detection boxes are used to mark target traffic signs located within the target area in the target frame.

[0103] Optionally, the processor may also execute program code for the following steps: identifying the target traffic sign and obtaining the classification result of the target traffic sign; identifying the traffic auxiliary sign information corresponding to the target traffic sign based on the target area and the target detection box; and outputting the sign detection result corresponding to the target traffic sign based on the classification result and the traffic auxiliary sign information.

[0104] Optionally, the processor may also execute program code for the following steps: determining the target region based on motion information and historical detection boxes in historical frames, including: determining the first coordinates of the traffic sign marked by the historical detection box in the world coordinate system based on the historical frame; predicting the second coordinates of the traffic sign marked by the historical detection box in the world coordinate system at the time corresponding to the target frame based on the motion information; generating a prediction box corresponding to the historical detection box in the target frame based on the second coordinates; and generating a target region including the prediction box based on the prediction box.

[0105] Optionally, the processor may also execute program code for the following steps: generating a target detection box based on the target region and the target frame, including: cropping the image corresponding to the target region in the target frame to obtain a target image; and performing image recognition on the target image to obtain the target detection box.

[0106] Optionally, the processor may also execute program code for the following steps: performing image recognition on the target image to obtain target detection boxes, including: performing image recognition on the target image to obtain initial detection boxes; establishing an MHT tree structure based on historical detection boxes and initial detection boxes, wherein the MHT tree structure is used to characterize the weak matching relationship between historical detection boxes and initial detection boxes; and selecting target detection boxes from the initial detection boxes based on the MHT tree structure.

[0107] Optionally, the processor may also execute program code for the following steps: obtaining target detection boxes from the initial detection boxes based on the MHT tree structure, including: obtaining target detection boxes from the initial detection boxes based on the Hungarian algorithm, wherein the Hungarian algorithm is used to correct the weak matching relationships in the MHT tree structure into a one-to-one correspondence between historical detection boxes and target detection boxes.

[0108] Optionally, the processor may also execute program code for the following steps: obtaining target detection boxes from the initial detection boxes based on the Hungarian algorithm, including: obtaining the first texture corresponding to the historical detection boxes and the second texture corresponding to the initial detection boxes; matching the globally optimal detection boxes in the initial detection boxes with the historical detection boxes one-to-one according to the first texture, the second texture and the cost function constructed based on the Hungarian algorithm to obtain the target detection boxes, wherein the cost function is constructed based on the texture features and feature similarity of the textures.

[0109] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0110] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the flag detection method provided in the above embodiments.

[0111] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0112] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring historical frames, target frames, and vehicle motion information, wherein the historical frames and target frames are images captured by the vehicle's camera; determining a target area based on the motion information and historical detection boxes in the historical frames, wherein the historical detection boxes are used to mark traffic signs in the historical frames; and generating a target detection box based on the target area and the target frame, wherein the target detection box is used to mark target traffic signs located within the target area in the target frame.

[0113] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: identifying the target traffic sign and obtaining the classification result of the target traffic sign; identifying the traffic auxiliary sign information corresponding to the target traffic sign based on the target area and the target detection box; and outputting the sign detection result corresponding to the target traffic sign based on the classification result and the traffic auxiliary sign information.

[0114] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining a target region based on motion information and historical detection boxes in historical frames, including: determining the first coordinates of the traffic sign marked by the historical detection box in the world coordinate system based on the historical frame; predicting the second coordinates of the traffic sign marked by the historical detection box in the world coordinate system at the time corresponding to the target frame based on the motion information; generating a prediction box corresponding to the historical detection box in the target frame based on the second coordinates; and generating a target region including the prediction box based on the prediction box.

[0115] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: generating a target detection box based on the target region and the target frame, including: cropping the image corresponding to the target region in the target frame to obtain a target image; and performing image recognition on the target image to obtain the target detection box.

[0116] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: performing image recognition on the target image to obtain target detection boxes, including: performing image recognition on the target image to obtain initial detection boxes; establishing an MHT tree structure based on historical detection boxes and the initial detection boxes, wherein the MHT tree structure is used to characterize the weak matching relationship between historical detection boxes and the initial detection boxes; and selecting target detection boxes from the initial detection boxes based on the MHT tree structure.

[0117] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining target detection boxes from the initial detection boxes based on the MHT tree structure, including: obtaining target detection boxes from the initial detection boxes based on the Hungarian algorithm, wherein the Hungarian algorithm is used to correct the weak matching relationship in the MHT tree structure into a one-to-one correspondence between historical detection boxes and target detection boxes.

[0118] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining target detection boxes from the initial detection boxes based on the Hungarian algorithm, including: obtaining the first texture corresponding to the historical detection boxes and the second texture corresponding to the initial detection boxes; matching the globally optimal detection boxes in the initial detection boxes with the historical detection boxes one-to-one according to the first texture, the second texture and the cost function constructed based on the Hungarian algorithm to obtain the target detection boxes, wherein the cost function is constructed based on the texture features and feature similarity of the textures.

[0119] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0125] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting a mark, characterized in that, include: Acquire historical frames, target frames, and vehicle motion information, wherein the historical frames and target frames are images captured by the vehicle's camera; The target region is determined based on the motion information and the historical detection boxes in the historical frames, wherein the historical detection boxes are used to mark traffic signs in the historical frames; The image corresponding to the target region in the target frame is cut out to obtain the target image; Image recognition is performed on the target image to obtain an initial detection box; Based on the historical detection boxes and the initial detection boxes, an MHT tree structure is established, wherein the MHT tree structure is used to characterize the weak matching relationship between the historical detection boxes and the initial detection boxes; Obtain the first texture corresponding to the historical detection boxes and the second texture corresponding to the initial detection boxes; Based on the first texture, the second texture, and the cost function constructed using the Hungarian algorithm, the globally optimal detection box in the initial detection box is matched one-to-one with the historical detection boxes to obtain the target detection box. The cost function is constructed based on the texture features and feature similarity of the texture. The Hungarian algorithm is used to correct the weak matching relationship in the MHT tree structure into a one-to-one correspondence between the historical detection boxes and the target detection box. The weak matching relationship is a one-to-many matching relationship. The target detection box is used to mark the target traffic signs located within the target area in the target frame. The method further includes: obtaining historical attributes of historical detection boxes in the historical frames through an established one-to-one matching relationship; voting using the current attributes of the target detection box and the historical attributes of the historical detection boxes to select the detection result attribute with the most votes, wherein the voting queue is a sliding window of fixed length; identifying the target traffic sign to obtain the classification result of the target traffic sign; responding to the fact that the total number of votes corresponding to the detection result attribute does not exceed a threshold, calling the auxiliary sign recognition model to determine whether the traffic auxiliary sign information corresponding to the target traffic sign contains preset auxiliary information based on the target area and the target detection box, and obtaining a judgment result, wherein the preset auxiliary information is used to determine the lane related to the target traffic sign; and outputting the sign detection result corresponding to the target traffic sign based on the classification result and the judgment result. The method further includes: in response to the initial detection box failing to match the target detection box and the initial detection box not being at the image edge, adding the initial detection box to the prediction box array for predicting traffic signs in the image after the target frame; deleting objects in the prediction box array that have not established a matching relationship for multiple consecutive frames, and merging the two initial detection boxes according to the overlap between any two initial detection boxes in the prediction box array when the overlap is high.

2. The method according to claim 1, characterized in that, The step of determining the target region based on the motion information and the historical detection boxes in the historical frames includes: Based on the historical frames, determine the first coordinates of the traffic sign marked by the historical detection box in the world coordinate system; Based on the motion information, predict the second coordinates of the traffic sign marked by the historical detection box in the world coordinate system at the time corresponding to the target frame; Based on the second coordinates, generate the prediction box corresponding to the historical detection box in the target frame; Based on the predicted bounding box, the target region including the predicted bounding box is generated.

3. A mark detection device, characterized in that, include: The acquisition module is used to acquire historical frames, target frames, and vehicle motion information, wherein the historical frames and the target frames are images captured by the vehicle's camera; The determination module is used to determine the target region based on the motion information and the historical detection boxes in the historical frames, wherein the historical detection boxes are used to mark traffic signs in the historical frames; A generation module is used to cut out the image corresponding to the target region from the target frame to obtain a target image; perform image recognition on the target image to obtain an initial detection box; establish an MHT tree structure based on the historical detection boxes and the initial detection boxes, wherein the MHT tree structure is used to represent the weak matching relationship between the historical detection boxes and the initial detection boxes; obtain the first texture corresponding to each historical detection box and the second texture corresponding to each initial detection box; according to the first texture, the second texture and the cost function constructed based on the Hungarian algorithm, perform a one-to-one matching between the globally optimal detection box in the initial detection box and the historical detection boxes to obtain the target detection box, wherein the cost function is constructed based on the texture features and feature similarity of the textures, the Hungarian algorithm is used to correct the weak matching relationship in the MHT tree structure to a one-to-one correspondence between the historical detection boxes and the target detection boxes, the weak matching relationship is a one-to-many matching relationship, and the target detection box is used to mark the target traffic signs located in the target region in the target frame; The device is further configured to: acquire historical attributes of historical detection boxes in the historical frames through an established one-to-one matching relationship; vote on the current attributes of the target detection box and the historical attributes of the historical detection boxes, selecting the detection result attribute with the most votes, wherein the voting queue is a fixed-length sliding window; identify the target traffic sign and obtain the classification result of the target traffic sign; in response to the total number of votes corresponding to the detection result attribute not exceeding a threshold, invoke an auxiliary sign recognition model to determine whether the traffic auxiliary sign information corresponding to the target traffic sign contains preset auxiliary information based on the target area and the target detection box, wherein the preset auxiliary information is used to determine the lane related to the target traffic sign; and output the sign detection result corresponding to the target traffic sign based on the classification result and the traffic auxiliary sign information. The apparatus is further configured to, in response to the initial detection box failing to match the target detection box and the initial detection box not being at the image edge, add the initial detection box to the prediction box array for predicting traffic signs in the image after the target frame; delete objects in the prediction box array that have not established a matching relationship for multiple consecutive frames; and merge the two initial detection boxes according to the overlap between any two initial detection boxes in the prediction box array, when the overlap is high.

4. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the flag detection method according to any one of claims 1 to 2.

5. A computer device, characterized in that, The computer device includes a memory and a processor, the memory being used to store a program, and the processor being used to run the program stored in the memory, wherein the program, when running, executes the flag detection method according to any one of claims 1 to 2.

Images