A traffic element target detection method and system based on memo learning

Abstract

The application relates to the technical field of multi-target detection, and particularly discloses a traffic element target detection method and system based on memo learning, which comprises the following steps: training a detection model by using a data set; inputting a new historical road image into the trained detection model to output the category of a target; when the proportion of the target in a preset anchor frame does not meet the requirement, adding the new historical road image into a memo; detecting the image in the memo by using a segmentation model to determine the best anchor frame of the target in the image and form a new data set; training the segmentation model by using the new data set; updating the detection model by using the knowledge learned by the trained segmentation model; and inputting a road image to be detected into the updated detection model to output the category of the target in the road image to be detected. The application can improve the accuracy of traffic target detection and lay a foundation for later traffic safety situation analysis, congestion analysis and illegal motive detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target detection, and more particularly to a traffic element target detection method based on memo learning and a traffic element target detection system based on memo learning. BACKGROUND

[0002] With the acceleration of urbanization and the increasing traffic flow, the development of intelligent transportation system becomes more and more important. In the intelligent transportation system, target detection is a crucial link, which can provide real-time and accurate data support for traffic management, safety monitoring and automatic driving, etc. In intelligent transportation, the detection of traffic targets such as lane lines, traffic lights and distant cars is the basis for realizing vehicle navigation, traffic control and accident warning functions. However, these targets are often affected by complex lighting and weather conditions, resulting in poor detection results. Therefore, applying advanced target detection algorithms to improve the detection accuracy and stability of traffic targets has become an important direction for the development of intelligent transportation systems.

[0003] By applying target detection technology in intelligent transportation system, real-time detection and recognition of traffic targets can be realized, providing decision support for traffic management and improving road capacity and safety. At the same time, it can also provide data support for the development of new technologies such as automatic driving, promoting the further upgrading and perfection of intelligent transportation system. Therefore, the application of target detection in intelligent transportation system has very high necessity and importance.

[0004] In early methods, the selection and extraction of features are crucial steps. Commonly used features include edges, corner points, textures, etc., which are extracted by algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), etc. Then, these features are sent to classifiers such as support vector machine (SVM), AdaBoost, etc. for classification decision, so as to determine the existence and location of the target. However, traditional machine learning methods are limited by the ability of hand-designed features, and are difficult to cope with complex and variable scenes and lighting conditions, especially in intelligent transportation system, when facing the detection of small targets such as lane lines, traffic lights and distant cars, the detection accuracy and robustness are often insufficient.

[0005] In recent years, with the rise of deep learning technology, target detection methods based on deep learning gradually replace traditional machine learning methods and become the mainstream technology in the field of target detection. At present, target detection models are mainly divided into two categories: region candidate-based methods (such as R-CNN series) and single detection-based methods (such as YOLO, SSD, etc.). The former generates a series of region candidates and classifies and regresses on these candidate regions to achieve accurate positioning and identification of targets; the latter directly outputs the class and position information of the target through single forward propagation, with higher detection speed and real-time performance. SUMMARY

[0006] In view of the above problems existing in the prior art, the present application provides a traffic element target detection method and system based on memorandum learning, aiming to solve the problem that small targets such as lane lines, traffic lights and distant cars are difficult to detect under complex lighting and weather conditions, thereby quickly detecting and identifying traffic element targets and laying a foundation for later traffic safety situation, congestion analysis and illegal motive detection.

[0007] As a first aspect of the present application, a traffic element target detection method based on memorandum learning is provided, which comprises the following steps:

[0008] Step S1: obtaining historical road images d in a data set D;

[0009] Step S2: constructing a target detection model and training the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ;

[0010] Step S3: inputting a new historical road image b into the trained target detection model S model for detection to output the class of each traffic element target in the new historical road image b;

[0011] Step S4: calculating the proportion of each traffic element target in the new historical road image b in a preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to a review memorandum B;

[0012] Step S5: constructing a deep semantic detection segmentation model and detecting the historical road image b in the review memorandum B through the deep semantic detection segmentation model to obtain the contour coordinate set of each traffic element target in the historical road image b; then determining the best anchor frame of each traffic element target in the historical road image b according to the contour coordinate set of each traffic element target in the historical road image b;

[0013] Step S6: Add the historical road image b and the best anchor boxes of each traffic element target in the historical road image b to the dataset D to form a new dataset D′; then use the historical road images in the new dataset D′ to train the deep semantic detection and segmentation model to obtain the trained deep semantic detection and segmentation model T. model ;

[0014] Step S7: Utilize the trained deep semantic detection and segmentation model T model The learned knowledge applies to the trained target detection model S model Perform periodic updates to obtain the updated object detection model S′. model ;

[0015] Step S8: Input the road image to be detected into the updated target detection model S′ model The system performs detection to output the category of each traffic element target in the road image to be detected.

[0016] Furthermore, the acquisition of historical road images d in dataset D also includes:

[0017] Multiple historical road images d in the dataset D are preprocessed to uniformly adjust their size to 640*640.

[0018] Furthermore, the object detection model is constructed, and the object detection model is trained using historical road images d in the dataset D to obtain the trained object detection model S. model In addition, it also includes:

[0019] The target detection model includes a coarse-grained feature encoder, a fine-grained feature layer, and a detection head decoder;

[0020] The historical road image d in the dataset D is input into the coarse-grained feature encoder for coarse-grained feature extraction to obtain the coarse-grained feature X of the historical road image d;

[0021] The coarse-grained feature X of the historical road image d is input into the fine-grained feature layer for fine-grained feature extraction to obtain the fine-grained feature Y of the historical road image d.

[0022] The fine-grained features Y of the historical road image d are input into the detector head decoder for category prediction to obtain the category of each traffic element target in the historical road image b.

[0023] Further, the coarse-grained feature X of the historical road image d is input into the fine-grained feature layer for fine-grained feature extraction to obtain the fine-grained feature Y of the historical road image d, and the method further comprises the following steps:

[0024] The coarse-grained feature X of the historical road image d is input into a first feature fusion layer p combine1 to obtain a first distance fusion feature X1 of the historical road image d; as shown in formula (1):

[0025] X1=p combine1 (X) (1)

[0026] The first distance fusion feature X1 of the historical road image d is input into a second feature fusion layer p combine2 to obtain a second distance fusion feature X2 of the historical road image d; as shown in formula (2):

[0027] X2=p combine2 (X1) (2)

[0028] The second distance fusion feature X2 of the historical road image d is input into a first dilated convolution layer DilatedCNN1 to obtain a third distance fusion feature X3 of the historical road image d; as shown in formula (3):

[0029] X3=Dilated CNN1(X2) (3)

[0030] The coarse-grained feature X of the historical road image d is input into a second dilated convolution layer DilatedCNN2 to obtain a distance feature X4 of the historical road image d; as shown in formula (4):

[0031] X4=Dilated CNN2(X) (4)

[0032] The third distance fusion feature X3 of the historical road image d and the distance feature X4 of the historical road image d are spliced together to obtain a final fusion feature X of the historical road image d combine ; as shown in formula (5):

[0033] X combine =concat(X3⊙X4) (5)

[0034] The fine-grained feature Y of the historical road image d is obtained using a self-attention mechanism; as shown in formula (6):

[0035]

[0036] Wherein, Q represents a query matrix, K represents a key matrix, V represents a value matrix, d k represents the latitude of the matrix, and the calculation methods of Q, K and V are shown in formulas (7)-(9):

[0037] Q=X combine ·W Q (7)

[0038] K=X combine ·W K (8)

[0039] V=X combine ·W V (9)

[0040] Wherein, W Q , W K and W V are the weight matrix of the query matrix Q, the weight matrix of the key matrix K and the weight matrix of the value matrix V respectively.

[0041] Further, the first feature fusion layer p combine1 and the second feature fusion layer p combine2 have the same structure, and both include a long-distance feature extraction module part dcnn and a short-distance feature extraction module part cnn , wherein the long-distance feature extraction module part dcnn includes a dilated convolution layer, a normalization layer and an activation function, and the short-distance feature extraction module part cnn includes a normal convolution layer, a normalization layer and an activation function.

[0042] Wherein, the structure of the long-distance feature extraction module part dcnn is shown in formula (10):

[0043] part dcnn =Dilated CNN(X)+BN+Relu (10)

[0044] Wherein, the structure of the short-distance feature extraction module part cnn is shown in formula (11):

[0045] part cnn =CNN(X)+BN+Relu (11)

[0046] Wherein, Dilated CNN represents a dilated convolution layer, BN represents a normalization layer, Relu represents an activation function, and CNN represents a normal convolution layer.

[0047] The long-distance feature extraction module partdcnn The near-range feature extraction module part is used to extract long-range features from the historical road image d. cnn Used to extract near-range features of the historical road image d; weighted fusion of the far-range features and near-range features of the historical road image d to obtain the first near-far fusion feature X1 or the second near-far fusion feature X2 of the historical road image d; as shown in formula (12):

[0048] p combine =(1-b)part dcnn +bpart cnn (12)

[0049] Where b is the weight.

[0050] Furthermore, the step of inputting the fine-grained features Y of the historical road image d into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b also includes:

[0051] The fine-grained features Y of the historical road image d are transformed into a 1*n vector through a fully connected layer. in,

[0052] Calculate the vector The predicted class probability of each element is given by formula (13):

[0053]

[0054] Among them, y i Representing vectors The i-th element z i The predicted probability of the category, where N represents the number of categories of the traffic element target, z j It is a vector The j-th element in;

[0055] y i The set of elements with a value greater than 0.8 is used as the category set of traffic element targets in the historical road image b.

[0056] Furthermore, the target detection model is trained using historical road images d from the dataset D to obtain the trained target detection model S. model In addition, it also includes:

[0057] During the training of the object detection model, the total loss function Loss is used. total Backpropagation is performed to update the parameters of the object detection model; wherein, the total loss function Losstotal The calculation formula (14) of the classification loss function Loss

[0058] Loss total = λ1Loss cls + λ2Loss obj (14)

[0059] wherein, Loss cls is a classification loss function, Loss obj is a confidence loss function, and λ1 and λ2 both represent weights;

[0060] The calculation formula (15) of the classification loss function Loss cls is as follows:

[0061]

[0062] The calculation formula (16) of the confidence loss function Loss obj is as follows:

[0063]

[0064] wherein, N represents the number of categories of the traffic element target, N' represents the number of positive samples, and l n represents a real label, and l represents the category prediction probability of the real label.

[0065] Further, the proportion of each traffic element target in the preset anchor frame in the new historical road image b is calculated, and when the proportion does not meet the requirement, the new historical road image b is added to the review memorandum B, and the method further comprises:

[0066] A plurality of variable anchor frames box are preset, and the definition of the variable anchor frame box is as shown in formula (17):

[0067] box = {B[m*n], p} (17)

[0068] wherein, m and n respectively represent the length and width of the variable anchor frame, and p represents the scaling ratio.

[0069] The number of pixel points of each traffic element target category o in the new historical road image b is calculated to obtain the area S o of each traffic element target in the new historical road image b.

[0070] According to the area S oselecting one variable anchor box box capable of covering the traffic element target from a plurality of variable anchor boxes, and then calculating the proportion q of the traffic element target in the variable anchor box in the new historical road image b, and the calculation formula (18) is as follows:

[0071]

[0072] wherein S Box represents the area of the variable anchor box box;

[0073] If the proportion q of the traffic element target in the variable anchor box in the new historical road image b is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the variable anchor box are directly output; if the proportion q of the traffic element target in the variable anchor box in the new historical road image b is not greater than the certain threshold, the length and width of the variable anchor box are reduced according to the scaling ratio p in formula (17), and then the proportion q of the traffic element target in the reduced variable anchor box in the new historical road image b is calculated again.

[0074] If the proportion q of the traffic element target in the reduced variable anchor box is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the reduced variable anchor box are directly output; if the proportion q of the traffic element target in the reduced variable anchor box is still not greater than a certain threshold, the new historical road image b is added to the review memorandum B.

[0075] Further, the step S5 further comprises:

[0076] For the i-th historical road image b i in the review memorandum B, the contour coordinate set W of all traffic element targets in the i-th historical road image b i is represented by formula (19):

[0077] W={w1,w2,...w n} (19)

[0078] wherein w i is the contour coordinate set of the i-th traffic element target in the i-th historical road image b i , and n is the number of traffic element targets in the i-th historical road image b i ; wherein w i ={(x1,y1),(x2,y2),...(x n ,y n )};

[0079] for the i-th historical road image b i a set of contour coordinates w of the i-th traffic element target i traversing all contour coordinates in the set of contour coordinates w of the i-th traffic element target i finding the minimum value x min and the maximum value x max of the x coordinate and the minimum value y min and the maximum value y max of the y coordinate respectively, using the boundary values to determine the minimum circumscribed matrix of the i-th traffic element target in the i-th historical road image b i , and taking the minimum circumscribed matrix as the best anchor frame of the i-th traffic element target in the i-th historical road image b i .

[0080] As a second aspect of the present application, a traffic element target detection system based on memo learning is provided, which comprises:

[0081] an acquisition module configured to acquire historical road images d in a data set D;

[0082] a first training module configured to construct a target detection model and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ;

[0083] a first detection module configured to input a new historical road image b into the trained target detection model S model for detection to output the class of each traffic element target in the new historical road image b;

[0084] a calculation module configured to calculate the proportion of each traffic element target in the new historical road image b in a preset anchor frame, and when the proportion does not meet the requirement, add the new historical road image b into a review memo B;

[0085] a determination module configured to construct a deep semantic detection segmentation model and detect the historical road images b in the review memo B through the deep semantic detection segmentation model to obtain a set of contour coordinates of each traffic element target in the historical road image b, and then determine the best anchor frame of each traffic element target in the historical road image b according to the set of contour coordinates of each traffic element target in the historical road image b;

[0086] a second training module configured to add the historical road image b and the optimal anchor box of each traffic element target in the historical road image b into the data set D to form a new data set D'; and then train the deep semantic detection segmentation model by using the historical road image in the new data set D' to obtain a trained deep semantic detection segmentation model T model ;

[0087] an updating module configured to periodically update the learned knowledge of the trained target detection model S model by using the trained deep semantic detection segmentation model T model to obtain an updated target detection model S' model ;

[0088] a second detection module configured to input a to-be-detected road image into the updated target detection model S' model to detect the to-be-detected road image to output the category of each traffic element target in the to-be-detected road image.

[0089] The traffic element target detection method based on the memo learning has the following beneficial effects: the small traffic element targets such as lane lines, traffic lights and distant cars in bad weather and light conditions can be detected, so that the accuracy of the traffic element target detection is improved, and the foundation for the later traffic safety situation, congestion analysis and illegal motive detection is laid. BRIEF DESCRIPTION OF DRAWINGS

[0090] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, and together with the specific embodiments below, serve to explain the application, but do not constitute a limitation on the application.

[0091] Figure 1 The flowchart of the traffic element target detection method based on the memo learning provided by the application.

[0092] Figure 2 The specific embodiment flowchart of the traffic element target detection method based on the memo learning provided by the application.

[0093] Figure 3 The flowchart of the memo learning provided by the application. DETAILED DESCRIPTION

[0094] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a traffic element target detection method based on memorization learning proposed according to the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0095] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0096] This embodiment provides a traffic element target detection method based on memoization learning, such as Figure 1 As shown, the traffic element target detection method based on memo learning includes the following steps:

[0097] Step S1: Obtain historical road images d from dataset D;

[0098] Preferably, the process of obtaining the historical road image d in the dataset D further includes:

[0099] Multiple historical road images d in the dataset D are preprocessed to uniformly adjust their size to 640*640.

[0100] Step S2: Construct an object detection model and train it using historical road images d from the dataset D to obtain the trained object detection model S. model ;

[0101] Preferably, such as Figure 2 As shown, the object detection model is constructed, and the object detection model is trained using historical road images d in the dataset D to obtain the trained object detection model S. model In addition, it also includes:

[0102] The target detection model comprises a coarse-grained feature encoder, a fine-grained feature layer, and a detection head decoder.

[0103] The historical road image d in the data set D is input into the coarse-grained feature encoder for coarse-grained feature extraction to obtain coarse-grained features X of the historical road image d.

[0104] The coarse-grained features X of the historical road image d are input into the fine-grained feature layer for fine-grained feature extraction to obtain fine-grained features Y of the historical road image d.

[0105] Specifically, the coarse-grained features X of the historical road image d are input into the fine-grained feature layer for fine-grained feature extraction to obtain fine-grained features Y of the historical road image d, and further comprising:

[0106] The coarse-grained features X of the historical road image d are input into a first feature fusion layer p combine1 to obtain first near-far distance fusion features X1 of the historical road image d; as shown in formula (1):

[0107] X1=p combine1 (X) (1)

[0108] The first near-far distance fusion features X1 of the historical road image d are input into a second feature fusion layer p combine2 to obtain second near-far distance fusion features X2 of the historical road image d; as shown in formula (2):

[0109] X2=p combine2 (X1) (2)

[0110] The second near-far distance fusion features X2 of the historical road image d are input into a first dilated convolution layer DilatedCNN1 to obtain third near-far distance fusion features X3 of the historical road image d; as shown in formula (3):

[0111] X3=Dilated CNN1(X2) (3)

[0112] The coarse-grained features X of the historical road image d are input into a second dilated convolution layer DilatedCNN2 to obtain far distance features X4 of the historical road image d; as shown in formula (4):

[0113] X4=Dilated CNN2(X) (4)

[0114] The third near-far distance fusion feature X3 of the historical road image d and the far distance feature X4 of the historical road image d are spliced together to obtain the final fusion feature X of the historical road image d combine ; as shown in formula (5):

[0115] X combine = concat(X3⊙X4) (5)

[0116] The fine-grained feature Y of the historical road image d is obtained using a self-attention mechanism; as shown in formula (6):

[0117]

[0118] Wherein, Q represents a query matrix, K represents a key matrix, V represents a value matrix, d k represents the latitude of the matrix, and the calculation methods of Q, K and V are shown in formulas (7)-(9):

[0119] Q=X combine ·W Q (7)

[0120] K=X combine ·W K (8)

[0121] V=X combine ·W V (9)

[0122] Wherein, W Q , W K and W V are the weight matrix of the query matrix Q, the weight matrix of the key matrix K, and the weight matrix of the value matrix V, respectively.

[0123] Specifically, the structures of the first feature fusion layer p combine1 and the second feature fusion layer p combine2 are the same, and both include a far distance feature extraction module part dcnn and a near distance feature extraction module part cnn , wherein the far distance feature extraction module part dcnn includes a cavity convolution layer, a standardization normalization layer and an activation function, and the near distance feature extraction module part cnn includes a normal convolution layer, a standardization normalization layer and an activation function.

[0124] Wherein, the structure of the far distance feature extraction module part dcnn is shown in formula (10):

[0125] part dcnnDilated CNN(X) + BN + Relu (10)

[0126] The near-distance feature extraction module part cnn The structure is shown in formula (11):

[0127] part cnn = CNN(X) + BN + Relu (11)

[0128] Dilated CNN represents a dilated convolution layer, BN represents a standardization normalization layer, Relu represents an activation function, and CNN represents a normal convolution layer.

[0129] The long-distance feature extraction module part dcnn is used to extract the long-distance feature of the historical road image d, and the near-distance feature extraction module part cnn is used to extract the near-distance feature of the historical road image d; the long-distance feature and the near-distance feature of the historical road image d are weighted and fused to obtain the first long-near distance fusion feature X1 of the historical road image d or the second long-near distance fusion feature X2 of the historical road image d; as shown in formula (12):

[0130] p combine = (1-b) part dcnn +bpart cnn (12)

[0131] Wherein b is a weight.

[0132] The fine-grained feature Y of the historical road image d is input into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b.

[0133] Specifically, the fine-grained feature Y of the historical road image d is input into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b, and further comprising:

[0134] The fine-grained feature Y of the historical road image d is converted into a 1*n vector Wherein,

[0135] The category prediction probability of each element in the vector is calculated, as shown in formula (13):

[0136]

[0137] Wherein y i represents a vector the i-th element in the vector z i , N represents the number of categories of the traffic element target, z j is a vector the j-th element in the vector z

[0138] The element set y i with a value greater than 0.8 is taken as the category set of the traffic element target in the historical road image b.

[0139] Preferably, the historical road image d in the data set D is used to train the target detection model to obtain a trained target detection model S model Further comprising:

[0140] In the training process of the target detection model, a total loss function Loss total is used for back propagation to update the parameters of the target detection model; wherein the calculation formula (14) of the total loss function Loss total is as follows:

[0141] Loss total = λ1Loss cls + λ2Loss obj (14)

[0142] Wherein, Loss cls is a classification loss function, Loss obj is a confidence loss function, and λ1 and λ2 both represent weights.

[0143] The calculation formula (15) of the classification loss function Loss cls is as follows:

[0144]

[0145] The calculation formula (16) of the confidence loss function Loss obj is as follows:

[0146]

[0147] Wherein, N represents the number of categories of the traffic element target, N' represents the number of positive samples, and l n represents the real label, represents the category prediction probability of the real label.

[0148] Step S3: input the new historical road image b into the trained target detection model S model for detection to output the category of each traffic element target in the new historical road image b.

[0149] Step S4: Calculate the proportion of each traffic element target in the new historical road image b within the preset anchor frame. If the proportion does not meet the requirements, add the new historical road image b to the review memo B.

[0150] It should be noted that the present invention pre-sets three variable anchor frames, box1, box2, and box3. The variable anchor frames can be freely selected according to their size and scaled according to the scaling ratio.

[0151] Preferably, such as Figures 2-3 As shown, the step of calculating the proportion of each traffic element target in the new historical road image b within a preset anchor frame, and adding the new historical road image b to the review memo B when the proportion does not meet the requirements, also includes:

[0152] Multiple variable anchor boxes are preset, and the definition of the variable anchor box is shown in formula (17):

[0153] box={B[m*n],p} (17)

[0154] Where m and n represent the length and width of the variable anchor box, respectively, and p represents the scaling ratio; for example, the initial values ​​of the length m of the variable anchor boxes box1, box2, and box3 are 20, 50, and 80, respectively, the initial values ​​of the width n are 20, 50, and 80, respectively, and the initial value of p is set to 0.1.

[0155] Calculate the number of pixels for each traffic element target category o in the new historical road image b to obtain the area S of each traffic element target in the new historical road image b. o ;

[0156] Based on the area S of each traffic element target in the new historical road image b o Select a variable anchor box from multiple variable anchor boxes that can cover the traffic element target, and then calculate the proportion q of the traffic element target in the new historical road image b within the variable anchor box. The calculation formula (18) is as follows:

[0157]

[0158] Among them, S Box This represents the area of ​​the variable anchor box (Box).

[0159] If the proportion q of the traffic element target in the new historical road image b within the variable anchor box is greater than a certain threshold (e.g., 85%), the category o of the traffic element target in the new historical road image b and the position of the variable anchor box are directly output; if the proportion q of the traffic element target in the new historical road image b within the variable anchor box is not greater than the certain threshold, the length and width of the variable anchor box are reduced according to the scaling ratio p in formula (17), and the proportion q of the traffic element target in the new historical road image b within the reduced variable anchor box is calculated again.

[0160] If the proportion q of the traffic element target in the reduced variable anchor box Box is greater than a certain threshold, then the category o of the traffic element target and the position of the reduced variable anchor box Box in the new historical road image b are directly output; if the proportion q of the traffic element target in the reduced variable anchor box Box is still not greater than a certain threshold, then the new historical road image b is added to the review memo B.

[0161] Step S5: Construct a deep semantic detection and segmentation model, and use the deep semantic detection and segmentation model to detect the historical road image b in the review memo B to obtain the contour coordinate set of each traffic element target in the historical road image b; then determine the best anchor box of each traffic element target in the historical road image b based on the contour coordinate set of each traffic element target in the historical road image b; wherein, the deep semantic detection and segmentation model consists of 4 Transformers;

[0162] Preferably, step S5 further includes:

[0163] For the i-th historical road image b in the review memorandum B i The i-th historical road image b i The set of contour coordinates W of all traffic element targets is expressed by formula (19):

[0164] W = {w1, w2, ... w} n} (19)

[0165] Among them, w i For the i-th historical road image b i The set of contour coordinates of the i-th traffic element target in the image, where n is the i-th historical road image b. i The number of traffic element targets in the data; among which, w i ={(x1,y1),(x2,y2),...(x n ,y n )};

[0166] for the i-th historical road image b i a set of contour coordinates w of the i-th traffic element target i , find the minimum value x i and the maximum value x min of the x coordinate and the minimum value y max and the maximum value y min of the y coordinate respectively in the set of contour coordinates w of the i-th traffic element target max , determine the minimum bounding box of the i-th traffic element target in the i-th historical road image b i using the boundary values, and take the minimum bounding box as the best anchor box of the i-th traffic element target in the i-th historical road image b i .

[0167] Step S6: add the historical road image b and the best anchor box of each traffic element target in the historical road image b to the data set D to form a new data set D'; then train the deep semantic detection segmentation model using the historical road images in the new data set D' to obtain a trained deep semantic detection segmentation model T model .

[0168] Step S7: periodically update the trained target detection model S model using the knowledge learned by the trained deep semantic detection segmentation model T model , so that S model has a performance comparable to T model , realizes model compression and acceleration, and thus obtains an updated target detection model S' model .

[0169] Step S8: input the road image to be detected into the updated target detection model S' model for detection to output the category of each traffic element target in the road image to be detected.

[0170] As another embodiment of the present application, a traffic element target detection system based on memo learning is provided, which comprises:

[0171] an acquisition module configured to acquire historical road images d in a data set D;

[0172] a first training module configured to construct a target detection model and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model .

[0173] a first detection module configured to input a new historical road image b into the trained target detection model S model to detect each traffic element target in the new historical road image b and output a category of each traffic element target in the new historical road image b;

[0174] a calculation module configured to calculate a proportion of each traffic element target in the new historical road image b in a preset anchor frame, and add the new historical road image b into a review memorandum B when the proportion does not meet a requirement;

[0175] a determination module configured to construct a deep semantic detection segmentation model, detect each historical road image b in the review memorandum B by using the deep semantic detection segmentation model, and obtain a contour coordinate set of each traffic element target in the historical road image b; and then determine a best anchor frame of each traffic element target in the historical road image b according to the contour coordinate set of each traffic element target in the historical road image b;

[0176] a second training module configured to add the historical road image b and the best anchor frame of each traffic element target in the historical road image b into the data set D to form a new data set D'; and then train the deep semantic detection segmentation model by using the historical road image in the new data set D' to obtain a trained deep semantic detection segmentation model T model ;

[0177] an update module configured to periodically update the trained target detection model S model by using the knowledge learned by the trained deep semantic detection segmentation model T model to obtain an updated target detection model S' model ;

[0178] a second detection module configured to input a to-be-detected road image into the updated target detection model S' model to detect each traffic element target in the to-be-detected road image and output a category of each traffic element target in the to-be-detected road image.

[0179] The traffic element target detection method based on memo learning provided by the embodiment of the present application, (1) realizes accurate detection of different size targets in different scenes through adaptive anchor frame, and adopts a knowledge distillation method to prune the detection model, thereby reducing the complexity of the detection model to adapt to the hardware deployment requirements; (2) introduces a hollow convolution and a convolution network to obtain the relationship between long-distance and close-range pixels, thereby improving the performance of the detection model; (3) a memo learning library is established, and the parameters of the detection model can be optimized regularly; the present application can detect smaller traffic element targets such as lane lines, traffic lights and distant cars under adverse weather and light conditions, thereby improving the accuracy of traffic element target detection and laying a foundation for later traffic safety situation, congestion analysis and illegal motive detection.

[0180] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application still belong to the scope of the technical solution of the present application.

Claims

1. A traffic element target detection method based on memo learning, characterized in that, The traffic element target detection method based on the memorandum learning comprises the following steps: Step S1: obtaining historical road images d in a data set D; Step S2: constructing a target detection model, and training the target detection model by using the historical road images d in the data set D to obtain a trained target detection model S model ; Step S3: input a new historical road image b to the trained target detection model S model to detect in order to output the class of each traffic element target in the new historical road image b; Step S4: calculating the proportion of each traffic element target in the new historical road image b in a preset anchor frame, and adding the new historical road image b to a review memorandum B when the proportion does not meet the requirement; Step S5: constructing a deep semantic detection segmentation model, detecting the historical road images b in the review memorandum B by using the deep semantic detection segmentation model to obtain a contour coordinate set of each traffic element target in the historical road images b, and then determining a best anchor frame of each traffic element target in the historical road images b according to the contour coordinate set of each traffic element target in the historical road images b; Step S6: add the historical road image b and the optimal anchor frame of each traffic element target in the historical road image b into the data set D to form a new data set D'; then train the deep semantic detection segmentation model using the historical road images in the new data set D' to obtain a trained deep semantic detection segmentation model T model ; Step S7: using the trained deep semantic detection segmentation model T model The learned knowledge is used for the trained target detection model S model Periodically updated to obtain an updated target detection model S' model ; Step S8: inputting a to-be-detected road image into the updated target detection model S' for detection to output a category of each traffic element target in the to-be-detected road image. model Step S8: inputting a to-be-detected road image into the updated target detection model S' for detection to output a category of each traffic element target in the to-be-detected road image. 2.The traffic element target detection method based on memo learning according to claim 1, wherein, The historical road images d in the data set D further comprise: The multiple historical road images d in the data set D are preprocessed to uniformly adjust the size of the multiple historical road images d to 640*640. 3.The traffic element target detection method based on memo learning according to claim 1, wherein, The construction target detection model, and the historical road image d in the data set D is used to train the target detection model to obtain the trained target detection model S model In some embodiments, the method further comprises: The target detection model comprises a coarse-grained feature encoder, a fine-grained feature layer and a detection head decoder; The historical road images d in the data set D are input into the coarse-grained feature encoder to extract coarse-grained features, so as to obtain coarse-grained features X of the historical road images d; The coarse-grained features X of the historical road images d are input into the fine-grained feature layer to extract fine-grained features, so as to obtain fine-grained features Y of the historical road images d; The fine-grained features Y of the historical road images d are input into the detection head decoder to perform category prediction, so as to obtain the category of each traffic element target in the historical road image b.

4. The traffic element target detection method based on memo learning according to claim 3, characterized in that, The coarse-grained features X of the historical road images d are input into the fine-grained feature layer to extract fine-grained features, so as to obtain fine-grained features Y of the historical road images d, which further comprises: inputting the coarse-grained feature X of the historical road image d to a first feature fusion layer p combine1 to obtain the first distance fusion feature X1 of the historical road image d; as shown in formula (1): X1= p combine1 (X) (1) input the first distance fusion feature X1 of the historical road image d to a second feature fusion layer p combine2 to obtain the second distance fusion feature X2 of the historical road image d; as shown in formula (2): X2 = p combine2 (X1) (2) The second distance fusion features X2 of the historical road images d are input into a first dilated convolutional layer DilatedCNN1 to obtain third distance fusion features X3 of the historical road images d; as shown in formula (3): X3=Dilated CNN1(X2) (3) The coarse-grained features X of the historical road images d are input into a second dilated convolutional layer DilatedCNN2 to obtain distance features X4 of the historical road images d; as shown in formula (4): X4=Dilated CNN2(X) (4) The third distance fusion feature X3 of the historical road image d and the long-distance feature X4 of the historical road image d are spliced together to obtain the final fusion feature X of the historical road image d combine ; as shown in formula (5): X combine = concat(X3 Θ X4) (5) The fine-grained features Y of the historical road images d are obtained by using a self-attention mechanism; as shown in formula (6): wherein Q represents a query matrix, K represents a key matrix, V represents a value matrix, d k denotes the latitude of the matrix, and the calculation methods of Q, K, and V are shown in equations (7)-(9): Q = X combine • W Q (7) K = X combine • W K (8) V = X combine • W V (9) wherein W Q , W K , and W V are weight matrices for the query matrix Q, the key matrix K, and the value matrix V, respectively.

5. The traffic element target detection method based on memo learning according to claim 4, characterized in that, The first feature fusion layer p combine1 The second feature fusion layer p combine2 The structure is same, and all includes a long-distance feature extraction module part dcnn And a short-distance feature extraction module part cnn Wherein, the long-distance feature extraction module part dcnn Including a hollow convolution layer, a standardization normalization layer and an activation function, the short-distance feature extraction module part cnn Including a normal convolution layer, a standardization normalization layer and an activation function; The remote feature extraction module part dcnn The structure is shown as formula (10): part dcnn = Dilated CNN(X) + BN + Relu (10) The near distance feature extraction module part cnn The structure is shown as formula (11): part cnn = CNN(X) + BN + Relu (11) Wherein, DilatedCNN represents a dilated convolutional layer, BN represents a standardization normalization layer, Relu represents an activation function, and CNN represents a normal convolutional layer; The long-distance feature extraction module part dcnn The long-distance feature extraction module part is used for extracting the long-distance feature of the historical road image d. cnn The long-distance feature extraction module part is used for extracting the long-distance feature of the historical road image d. The long-distance feature extraction module part is used for extracting the long-distance feature of the historical road image d. p combine = (1 - b)part dcnn + bpart cnn (12) Wherein, b is a weight.

6. The traffic element target detection method based on memo learning according to claim 3, characterized in that, The fine-grained features Y of the historical road images d are input into the detection head decoder to perform category prediction, so as to obtain the category of each traffic element target in the historical road image b, which further comprises: convert the fine-grained features Y of the historical road image d into a 1*n vector through a fully connected layer wherein, Compute the vector The class prediction probability for each element in the vector is computed as shown in equation (13): where y i represents the category prediction probability of the i-th element z in the vector i , N represents the number of categories of the traffic element target, and z j is the j-th element in the vector . y i a set of elements greater than 0.8 as a set of classes of traffic element targets in the historical road image b.

7. The traffic element target detection method based on memo learning according to claim 1, wherein, The target detection model is trained by using the historical road images d in the data set D to obtain a trained target detection model S model Further comprises: In the training process of the target detection model, a total loss function Loss total is used, and back propagation is performed to update parameters of the target detection model; wherein, the total loss function Loss total is calculated according to the following formula (14): Loss total = λ1Loss cls + λ2Loss obj (14) wherein Loss cls is a classification loss function, Loss obj is a confidence loss function, and λ1 and λ2 both represent weights; The classification loss function Loss cls The calculation formula (15) is as follows: The confidence loss function Loss obj The calculation formula (16) is as follows: where N represents the number of categories of the traffic element target, N' represents the number of positive samples, l n represents a true label, represents a class prediction probability of a true label. 8.The traffic element target detection method based on memo learning according to claim 1, wherein, The proportion of each traffic element target in the new historical road image b in a preset anchor frame is calculated, and when the proportion does not meet the requirement, the new historical road image b is added to the review memorandum B, and the method further comprises: A plurality of variable anchor frames box are preset, and the definition of the variable anchor frame box is shown in formula (17): box={B[m*n],p} (17) Wherein, m and n represent the length and width of the variable anchor frame respectively, and p represents the scaling ratio; The number of pixel points of each traffic element target category o in the new historical road image b is calculated to obtain the area S of each traffic element target in the new historical road image b o ; According to the area S of each traffic element target in the new historical road image b o From the multiple variable anchor boxes box, a variable anchor box Box is selected that can cover the traffic element target, and then the proportion q of the traffic element target in the variable anchor box Box in the new historical road image b is calculated, and the calculation formula (18) is as follows: where S Box represents the area of the variable anchor box Box. If the proportion q of the traffic element target in the new historical road image b in the variable anchor frame Box is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the variable anchor frame Box are directly output; if the proportion q of the traffic element target in the new historical road image b in the variable anchor frame Box is not greater than the certain threshold, the length and width of the variable anchor frame Box are reduced according to the scaling ratio p in formula (17), and then the proportion q of the traffic element target in the new historical road image b in the reduced variable anchor frame Box is calculated again; If the proportion q of the traffic element target in the reduced variable anchor frame Box is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the reduced variable anchor frame Box are directly output; if the proportion q of the traffic element target in the reduced variable anchor frame Box is still not greater than a certain threshold, the new historical road image b is added to the review memorandum B. 9.The traffic element object detection method based on memo learning according to claim 1, wherein, The step S5 further comprises: For the i-th historical road image b in the review memo B i , the contour coordinate set W of all traffic element targets in the i-th historical road image b i is expressed as formula (19): W = {wl, w2,... w n} (19) wherein w i is a set of contour coordinates of the i-th traffic element target in the i-th historical road image b i , and n is the number of traffic element targets in the i-th historical road image b i ; wherein w i = {(x1,y1),(x2,y2),...(xn,yn)}. n n ​​ For the i-th historical road image b i The set of contour coordinates w of the i-th traffic element target i Iterate through the set of contour coordinates w of the i-th traffic element target. i For all contour coordinates, find the minimum x-coordinate x. min and maximum value x max And the minimum value of the y-coordinate y min and maximum value y max Using these boundary values, the i-th historical road image b is determined. i The minimum bounding matrix of the i-th traffic element target is used as the minimum bounding matrix of the i-th historical road image b. i The optimal anchor frame for the i-th traffic element objective.

10. A memo learning based traffic element object detection system for implementing the memo learning based traffic element object detection method according to any one of claims 1 to 9, characterized by, The traffic element target detection system based on the memorandum learning comprises: An acquisition module is configured to acquire historical road images d in a data set D; The first training module is configured to construct a target detection model, and train the target detection model by using historical road images d in the data set D, so as to obtain a trained target detection model S. model ; The first detection module is configured to input a new historical road image b into the trained target detection model S model to perform detection to output a category of each traffic element target in the new historical road image b. A calculation module is configured to calculate the proportion of each traffic element target in the new historical road image b in a preset anchor frame, and when the proportion does not meet the requirement, the new historical road image b is added to the review memorandum B; A determination module is configured to construct a deep semantic detection segmentation model, and detect the historical road images b in the review memorandum B through the deep semantic detection segmentation model to obtain the contour coordinate set of each traffic element target in the historical road image b; then the best anchor frame of each traffic element target in the historical road image b is determined according to the contour coordinate set of each traffic element target in the historical road image b. a second training module configured to add the historical road image b and the optimal anchor box of each traffic element target in the historical road image b into the data set D to form a new data set D'; and then train the deep semantic detection segmentation model using the historical road images in the new data set D' to obtain a trained deep semantic detection segmentation model T model ; an updating module configured to utilize the trained deep semantic detection segmentation model T model learned knowledge to the trained object detection model S model periodically updated to obtain an updated object detection model S' model ; The second detection module is configured to input a to-be-detected road image into the updated target detection model S' to detect each traffic element target in the to-be-detected road image and output a category of each traffic element target in the to-be-detected road image. model The second detection module is configured to input a to-be-detected road image into the updated target detection model S' to detect each traffic element target in the to-be-detected road image and output a category of each traffic element target

Images