Memo learning-based traffic element target detection method and system

Abstract

The invention relates to the technical field of multi-target detection, and particularly discloses a memo learning-based traffic element target detection method and system, and the method comprises the steps: carrying out the training of a detection model through a data set; inputting a new historical road image into the trained detection model to output the category of the target; when the proportion of the target in the preset anchor frame does not meet the requirement, adding the new historical road image into the memorandum; detecting an image in the memorandum through the segmentation model to determine an optimal anchor frame of a target in the image, and forming a new data set; training the segmentation model by using the new data set; using knowledge learned by the trained segmentation model to update the detection model; and inputting the to-be-detected road image into the updated detection model to output the category of the target in the to-be-detected road image. According to the method, the accuracy of traffic target detection can be improved, and a foundation is laid for later traffic safety situation, congestion analysis and illegal motivation detection.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target detection, and more specifically, to a traffic element target detection method based on memorandum learning and a traffic element target detection system based on memorandum learning. Background Art

[0002] With the acceleration of urbanization and the continuous increase in traffic volume, the development of intelligent transportation systems has become increasingly important. In intelligent transportation systems, target detection is a crucial link, which can provide real-time and accurate data support for fields such as traffic management, safety monitoring and autonomous driving. In intelligent transportation, the detection of traffic targets such as lane lines, traffic lights and distant cars is the basis for realizing functions such as vehicle navigation, traffic control and accident warning. However, these targets are often affected by complex lighting and weather conditions, resulting in poor detection results. Therefore, the application of advanced target detection algorithms to improve the detection accuracy and stability of traffic targets has become an important direction for the development of intelligent transportation systems.

[0003] By applying target detection technology in intelligent transportation systems, real-time detection and identification of traffic targets can be achieved, providing decision support for traffic management, and improving road capacity and safety. At the same time, it can also provide data support for the development of new technologies such as autonomous driving, and promote the further upgrading and improvement of intelligent transportation systems. Therefore, the application of target detection in intelligent transportation systems is extremely necessary and important.

[0004] In early methods, feature selection and extraction is a crucial step. Common features include edges, corners, textures, etc., which are extracted through algorithms such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features). Subsequently, these features are fed into classifiers, such as support vector machines (SVM) and AdaBoost, to make classification decisions to determine the presence and location of the target. However, traditional machine learning methods are limited by the ability to manually design features and are difficult to cope with complex and changing scenes and lighting conditions. Especially in intelligent transportation systems, when faced with the detection of smaller targets such as lane lines, traffic lights, and distant cars, their detection accuracy and robustness are often insufficient.

[0005] In recent years, with the rise of deep learning technology, deep learning-based target detection methods have gradually replaced traditional machine learning methods and become the mainstream technology in the field of target detection. At present, target detection models are mainly divided into two categories: region candidate-based methods (such as the R-CNN series) and single-shot detection-based methods (such as YOLO, SSD, etc.). The former generates a series of region candidates and performs classification and regression on these candidate regions to achieve accurate positioning and recognition of the target; the latter directly outputs the target category and location information through a single forward propagation, which has higher detection speed and real-time performance. Summary of the invention

[0006] In view of the above-mentioned problems existing in the prior art, the present invention provides a traffic element target detection method and system based on memorandum learning, which aims to solve the problem that small targets such as lane lines, traffic lights and distant cars are difficult to detect under complex lighting and weather conditions, thereby quickly detecting and identifying traffic element targets, laying the foundation for later traffic safety situation, congestion analysis and illegal motive detection.

[0007] As a first aspect of the present invention, a traffic element target detection method based on memorandum learning is provided, and the traffic element target detection method based on memorandum learning comprises the following steps:

[0008] Step S1: Obtain a historical road image d in the dataset D;

[0009] Step S2: construct a target detection model, and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ;

[0010] Step S3: Input the new historical road image b into the trained object detection model S model Detection is performed in order to output the category of each traffic element target in the new historical road image b;

[0011] Step S4: calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B;

[0012] Step S5: constructing a deep semantic detection segmentation model, and detecting the historical road image b in the review memo B by using the deep semantic detection segmentation model to obtain a set of contour coordinates of each traffic element target in the historical road image b; then determining the best anchor frame for each traffic element target in the historical road image b according to the set of contour coordinates of each traffic element target in the historical road image b;

[0013] Step S6: adding the historical road image b and the best anchor frame of each traffic element target in the historical road image b to the data set D to form a new data set D′; then using the historical road images in the new data set D′ to train the deep semantic detection segmentation model to obtain a trained deep semantic detection segmentation model T model ;

[0014] Step S7: Using the trained deep semantic detection segmentation model T model The learned knowledge is used to train the target detection model S model Perform regular updates to obtain the updated target detection model S′ model ;

[0015] Step S8: Input the road image to be detected into the updated target detection model S′ model Detection is performed to output the category of each traffic element target in the road image to be detected.

[0016] Furthermore, the acquiring of the historical road image d in the data set D further includes:

[0017] The multiple historical road images d in the data set D are preprocessed to uniformly adjust the sizes of the multiple historical road images d to a size of 640*640.

[0018] Furthermore, the target detection model is constructed, and the target detection model is trained using the historical road images d in the data set D to obtain a trained target detection model S model Also included:

[0019] The target detection model includes a coarse-grained feature encoder, a fine-grained feature layer and a detection head decoder;

[0020] Inputting the historical road image d in the data set D into the coarse-grained feature encoder to extract coarse-grained features, so as to obtain the coarse-grained features X of the historical road image d;

[0021] Inputting the coarse-grained feature X of the historical road image d into the fine-grained feature layer to extract fine-grained features, so as to obtain fine-grained features Y of the historical road image d;

[0022] The fine-grained feature Y of the historical road image d is input into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b.

[0023] Furthermore, the step of inputting the coarse-grained feature X of the historical road image d into the fine-grained feature layer for extracting fine-grained features to obtain the fine-grained feature Y of the historical road image d further includes:

[0024] The coarse-grained features X of the historical road image d are input into the first feature fusion layer p combine1 In order to obtain the first long-short distance fusion feature X1 of the historical road image d; as shown in formula (1):

[0025] X1=p combine1 (X) (1)

[0026] The first long-short distance fusion feature X1 of the historical road image d is input into the second feature fusion layer p combine2 In order to obtain the second long-short distance fusion feature X2 of the historical road image d; as shown in formula (2):

[0027] X2=p combine2 (X1) (2)

[0028] The second near-far distance fusion feature X2 of the historical road image d is input into the first dilated convolutional layer DilatedCNN1 to obtain the third near-far distance fusion feature X3 of the historical road image d; as shown in formula (3):

[0029] X3=Dilated CNN1(X2) (3)

[0030] The coarse-grained feature X of the historical road image d is input into the second dilated convolutional layer DilatedCNN2 to obtain the long-range feature X4 of the historical road image d; as shown in formula (4):

[0031] X4=Dilated CNN2(X) (4)

[0032] The third long-distance and short-distance fusion feature X3 of the historical road image d and the long-distance feature X4 of the historical road image d are spliced ​​together to obtain the final fusion feature X of the historical road image d. combine ; As shown in formula (5):

[0033] X combine =concat(X3⊙X4) (5)

[0034] The self-attention mechanism is used to obtain the fine-grained feature Y of the historical road image d, as shown in formula (6):

[0035]

[0036] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k represents the latitude of the matrix. The calculation methods of Q, K, and V are shown in formulas (7)-(9):

[0037] Q=X combine ·W Q (7)

[0038] K=X combine ·W K (8)

[0039] V=X combine ·W V (9)

[0040] Among them, W Q , W K , W V They are the weight matrices of the query matrix Q, the key matrix K, and the value matrix V respectively.

[0041] Furthermore, the first feature fusion layer p combine1 and the second feature fusion layer p combine2 The structure of is the same, both include the long-distance feature extraction module part dcnn And the close range feature extraction module part cnn , wherein the long-distance feature extraction module part dcnn Including a hole convolution layer, a normalization layer and an activation function, the close-range feature extraction module part cnn Includes ordinary convolutional layer, normalization layer and activation function;

[0042] Among them, the long-distance feature extraction module part dcnn The structure of is shown in formula (10):

[0043] part dcnn =Dilated CNN(X)+BN+Relu (10)

[0044] Among them, the short-range feature extraction module part cnn The structure of is shown in formula (11):

[0045] part cnn =CNN(X)+BN+Relu (11)

[0046] Among them, DilatedCNN represents a dilated convolutional layer, BN represents a normalized layer, Relu represents an activation function, and CNN represents a normal convolutional layer;

[0047] The long-distance feature extraction module partdcnn The module for extracting long-distance features of the historical road image d is used to extract the short-distance features of the historical road image d. cnn Used to extract the short-range features of the historical road image d; weightedly fuse the long-range features and short-range features of the historical road image d to obtain a first long-short range fusion feature X1 of the historical road image d or a second long-short range fusion feature X2 of the historical road image d; as shown in formula (12):

[0048] p combine =(1-b)part dcnn +bpart cnn (12)

[0049] Among them, b is the weight.

[0050] Furthermore, the step of inputting the fine-grained feature Y of the historical road image d into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b further includes:

[0051] The fine-grained feature Y of the historical road image d is converted into a 1*n vector through a fully connected layer in,

[0052] Calculate the vector The category prediction probability of each element in is shown in formula (13):

[0053]

[0054] Among them, y i Representation vector The i-th element z in i The category prediction probability, N represents the number of categories of traffic element targets, z j is a vector The jth element in ;

[0055] y i The element set with a value greater than 0.8 is used as the category set of traffic element targets in the historical road image b.

[0056] Furthermore, the target detection model is trained using the historical road images d in the data set D to obtain a trained target detection model S model Also included:

[0057] In the training process of the target detection model, the total loss function Loss is used. total Back propagation is performed to update the parameters of the target detection model; wherein the total loss function Losstotal The calculation formula (14) is as follows:

[0058] Loss total =λ1Loss cls +λ2Loss obj (14)

[0059] Among them, Loss cls is the classification loss function, Loss obj is the confidence loss function, λ1 and λ2 both represent weights;

[0060] The classification loss function Loss cls The calculation formula (15) is as follows:

[0061]

[0062] The confidence loss function Loss obj The calculation formula (16) is as follows:

[0063]

[0064] Among them, N represents the number of categories of traffic element targets, N′ represents the number of positive samples, and l n represents the true label, Represents the class prediction probability of the true label.

[0065] Furthermore, the calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B, further includes:

[0066] Preset multiple variable anchor boxes, and the definition of the variable anchor box is shown in formula (17):

[0067] box={B[m*n],p} (17)

[0068] Among them, m and n represent the length and width of the variable anchor box respectively, and p represents the scaling ratio;

[0069] Calculate the number of pixels of each traffic element target category o in the new historical road image b to obtain the area S of each traffic element target in the new historical road image b. o ;

[0070] According to the area S of each traffic element target in the new historical road image b o, select a variable anchor box Box that can cover the traffic element target from multiple variable anchor boxes, and then calculate the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box. The calculation formula (18) is as follows:

[0071]

[0072] Among them, S Box Represents the area of ​​the variable anchor box Box;

[0073] If the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box is greater than a certain threshold, directly output the category o of the traffic element target in the new historical road image b and the position of the variable anchor box Box; if the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box is not greater than the certain threshold, reduce the length and width of the variable anchor box Box according to the scaling ratio p in formula (17), and calculate the proportion q of the traffic element target in the new historical road image b in the reduced variable anchor box Box again;

[0074] If the proportion q of the traffic element target in the reduced variable anchor box Box is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the reduced variable anchor box Box are directly output; if the proportion q of the traffic element target in the reduced variable anchor box Box is still not greater than a certain threshold, the new historical road image b is added to the review memo B.

[0075] Furthermore, the step S5 further includes:

[0076] For the i-th historical road image b in the review memo B i , the i-th historical road image b i The contour coordinate set W of all traffic element targets in is expressed as formula (19):

[0077] W={w1,w2,...w n} (19)

[0078] Among them, w i is the i-th historical road image b i The contour coordinate set of the i-th traffic element target in, n is the i-th historical road image b i The number of traffic element targets in i ={(x1,y1),(x2,y2),...(x n ,y n )};

[0079] For the i-th historical road image b i The contour coordinate set w of the i-th traffic element target i , traverse the contour coordinate set w of the i-th traffic element target i For all contour coordinates in , find the minimum x coordinate min and the maximum value x max and the minimum y coordinate y min and the maximum value y max , using these boundary values ​​to determine the i-th historical road image b i The minimum external matrix of the i-th traffic element target in is used as the i-th historical road image b i The best anchor box for the i-th traffic element target in .

[0080] As a second aspect of the present invention, a traffic element target detection system based on memorandum learning is provided, and the traffic element target detection system based on memorandum learning includes:

[0081] An acquisition module, used to acquire a historical road image d in a data set D;

[0082] The first training module is used to construct a target detection model and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ;

[0083] The first detection module is used to input the new historical road image b into the trained target detection model S model Detection is performed in order to output the category of each traffic element target in the new historical road image b;

[0084] A calculation module, used for calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B;

[0085] A determination module is used to construct a deep semantic detection segmentation model, and detect the historical road image b in the review memo B through the deep semantic detection segmentation model to obtain a contour coordinate set of each traffic element target in the historical road image b; then determine the best anchor frame for each traffic element target in the historical road image b according to the contour coordinate set of each traffic element target in the historical road image b;

[0086] The second training module is used to add the historical road image b and the best anchor frame of each traffic element target in the historical road image b to the data set D to form a new data set D′; then the deep semantic detection segmentation model is trained using the historical road images in the new data set D′ to obtain a trained deep semantic detection segmentation model T model ;

[0087] An updating module is used to utilize the trained deep semantic detection segmentation model T model The learned knowledge is used to train the target detection model S model Perform regular updates to obtain the updated target detection model S′ model ;

[0088] The second detection module is used to input the road image to be detected into the updated target detection model S′ model Detection is performed to output the category of each traffic element target in the road image to be detected.

[0089] The traffic element target detection method based on memorandum learning provided by the present invention has the following beneficial effects: it can detect smaller traffic element targets such as lane lines, traffic lights and distant cars under bad weather and lighting conditions, thereby improving the accuracy of traffic element target detection and laying the foundation for later traffic safety situation, congestion analysis and illegal motive detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0091] Figure 1 A flow chart of the traffic element target detection method based on memorandum learning provided by the present invention.

[0092] Figure 2 A flowchart of a specific implementation of the traffic element target detection method based on memorandum learning provided by the present invention.

[0093] Figure 3 A flowchart of memorandum learning provided by the present invention. DETAILED DESCRIPTION

[0094] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, characteristics and effects of a traffic element target detection method based on memorandum learning proposed by the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0095] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0096] In this embodiment, a traffic element target detection method based on memorandum learning is provided. Figure 1 As shown, the traffic element target detection method based on memorandum learning includes the following steps:

[0097] Step S1: Obtain a historical road image d in the dataset D;

[0098] Preferably, the step of acquiring the historical road image d in the data set D further includes:

[0099] The multiple historical road images d in the data set D are preprocessed to uniformly adjust the sizes of the multiple historical road images d to a size of 640*640.

[0100] Step S2: construct a target detection model, and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ;

[0101] Preferably, if Figure 2 As shown, the target detection model is constructed, and the target detection model is trained using the historical road images d in the data set D to obtain the trained target detection model S model Also included:

[0102] The target detection model includes a coarse-grained feature encoder, a fine-grained feature layer and a detection head decoder;

[0103] Inputting the historical road image d in the data set D into the coarse-grained feature encoder to extract coarse-grained features, so as to obtain the coarse-grained features X of the historical road image d;

[0104] Inputting the coarse-grained feature X of the historical road image d into the fine-grained feature layer to extract fine-grained features, so as to obtain fine-grained features Y of the historical road image d;

[0105] Specifically, the step of inputting the coarse-grained feature X of the historical road image d into the fine-grained feature layer to extract fine-grained features to obtain the fine-grained feature Y of the historical road image d further includes:

[0106] The coarse-grained features X of the historical road image d are input into the first feature fusion layer p combine1 In order to obtain the first long-short distance fusion feature X1 of the historical road image d; as shown in formula (1):

[0107] X1=p combine1 (X) (1)

[0108] The first long-short distance fusion feature X1 of the historical road image d is input into the second feature fusion layer p combine2 In order to obtain the second long-short distance fusion feature X2 of the historical road image d; as shown in formula (2):

[0109] X2=p combine2 (X1) (2)

[0110] The second near-far distance fusion feature X2 of the historical road image d is input into the first dilated convolutional layer DilatedCNN1 to obtain the third near-far distance fusion feature X3 of the historical road image d; as shown in formula (3):

[0111] X3=Dilated CNN1(X2) (3)

[0112] The coarse-grained feature X of the historical road image d is input into the second dilated convolutional layer DilatedCNN2 to obtain the long-range feature X4 of the historical road image d; as shown in formula (4):

[0113] X4=Dilated CNN2(X) (4)

[0114] The third long-distance and short-distance fusion feature X3 of the historical road image d and the long-distance feature X4 of the historical road image d are spliced ​​together to obtain the final fusion feature X of the historical road image d. combine ; As shown in formula (5):

[0115] X combine =concat(X3⊙X4) (5)

[0116] The self-attention mechanism is used to obtain the fine-grained feature Y of the historical road image d, as shown in formula (6):

[0117]

[0118] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k represents the latitude of the matrix. The calculation methods of Q, K, and V are shown in formulas (7)-(9):

[0119] Q=X combine ·W Q (7)

[0120] K=X combine ·W K (8)

[0121] V=X combine ·W V (9)

[0122] Among them, W Q , W K , W V They are the weight matrices of the query matrix Q, the key matrix K, and the value matrix V respectively.

[0123] Specifically, the first feature fusion layer p combine1 and the second feature fusion layer p combine2 The structure of is the same, both include the long-distance feature extraction module part dcnn And the close range feature extraction module part cnn , wherein the long-distance feature extraction module part dcnn Including a hole convolution layer, a normalization layer and an activation function, the close-range feature extraction module part cnn Includes ordinary convolutional layer, normalization layer and activation function;

[0124] Among them, the long-distance feature extraction module part dcnn The structure of is shown in formula (10):

[0125] part dcnn=Dilated CNN(X)+BN+Relu (10)

[0126] Among them, the short-range feature extraction module part cnn The structure of is shown in formula (11):

[0127] part cnn =CNN(X)+BN+Relu (11)

[0128] Among them, DilatedCNN represents a dilated convolutional layer, BN represents a normalized layer, Relu represents an activation function, and CNN represents a normal convolutional layer;

[0129] The long-distance feature extraction module part dcnn The module for extracting long-distance features of the historical road image d is used to extract the short-distance features of the historical road image d. cnn Used to extract the short-range features of the historical road image d; weightedly fuse the long-range features and short-range features of the historical road image d to obtain a first long-short range fusion feature X1 of the historical road image d or a second long-short range fusion feature X2 of the historical road image d; as shown in formula (12):

[0130] p combine =(1-b)part dcnn +bpart cnn (12)

[0131] Among them, b is the weight.

[0132] The fine-grained feature Y of the historical road image d is input into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b.

[0133] Specifically, the step of inputting the fine-grained feature Y of the historical road image d into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b further includes:

[0134] The fine-grained feature Y of the historical road image d is converted into a 1*n vector through a fully connected layer in,

[0135] Calculate the vector The category prediction probability of each element in is shown in formula (13):

[0136]

[0137] Among them, y i Representation vector The i-th element z in i The category prediction probability, N represents the number of categories of traffic element targets, z j is a vector The jth element in ;

[0138] y i The element set with a value greater than 0.8 is used as the category set of traffic element targets in the historical road image b.

[0139] Preferably, the target detection model is trained using the historical road images d in the data set D to obtain a trained target detection model S model Also included:

[0140] In the training process of the target detection model, the total loss function Loss is used. total Back propagation is performed to update the parameters of the target detection model; wherein the total loss function Loss total The calculation formula (14) is as follows:

[0141] Loss total =λ1Loss cls +λ2Loss obj (14)

[0142] Among them, Loss cls is the classification loss function, Loss obj is the confidence loss function, λ1 and λ2 both represent weights;

[0143] The classification loss function Loss cls The calculation formula (15) is as follows:

[0144]

[0145] The confidence loss function Loss obj The calculation formula (16) is as follows:

[0146]

[0147] Among them, N represents the number of categories of traffic element targets, N′ represents the number of positive samples, and l n represents the true label, Represents the class prediction probability of the true label.

[0148] Step S3: Input the new historical road image b into the trained object detection model S model Detection is performed in order to output the category of each traffic element target in the new historical road image b;

[0149] Step S4: calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B;

[0150] It should be noted that the present invention presets three variable anchor boxes box1, box2, and box3. The variable anchor boxes can be freely selected according to the size and can be scaled according to the scaling ratio.

[0151] Preferably, if Figure 2-Figure 3 As shown, the proportion of each traffic element target in the new historical road image b in the preset anchor frame is calculated, and when the proportion does not meet the requirement, the new historical road image b is added to the review memo B, and further includes:

[0152] Preset multiple variable anchor boxes, and the definition of the variable anchor box is shown in formula (17):

[0153] box={B[m*n],p} (17)

[0154] Among them, m and n represent the length and width of the variable anchor box respectively, and p represents the scaling ratio; for example, the initial values ​​of the length m of the variable anchor boxes box1, box2, and box3 are 20, 50, and 80 respectively, and the initial values ​​of the width n are 20, 50, and 80 respectively, and the initial value of p is set to 0.1.

[0155] Calculate the number of pixels of each traffic element target category o in the new historical road image b to obtain the area S of each traffic element target in the new historical road image b. o ;

[0156] According to the area S of each traffic element target in the new historical road image b o , select a variable anchor box Box that can cover the traffic element target from multiple variable anchor boxes, and then calculate the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box. The calculation formula (18) is as follows:

[0157]

[0158] Among them, S Box Represents the area of ​​the variable anchor box Box;

[0159] If the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box is greater than a certain threshold (e.g., 85%), the category o of the traffic element target in the new historical road image b and the position of the variable anchor box Box are directly output; if the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box is not greater than the certain threshold, the length and width of the variable anchor box Box are reduced according to the scaling ratio p in formula (17), and the proportion q of the traffic element target in the new historical road image b in the reduced variable anchor box Box is calculated again;

[0160] If the proportion q of the traffic element target in the reduced variable anchor box Box is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the reduced variable anchor box Box are directly output; if the proportion q of the traffic element target in the reduced variable anchor box Box is still not greater than a certain threshold, the new historical road image b is added to the review memo B.

[0161] Step S5: construct a deep semantic detection segmentation model, and detect the historical road image b in the review memo B through the deep semantic detection segmentation model to obtain a set of contour coordinates of each traffic element target in the historical road image b; then determine the best anchor frame for each traffic element target in the historical road image b according to the set of contour coordinates of each traffic element target in the historical road image b; wherein the deep semantic detection segmentation model is composed of 4 Transformers;

[0162] Preferably, the step S5 further includes:

[0163] For the i-th historical road image b in the review memo B i , the i-th historical road image b i The contour coordinate set W of all traffic element targets in is expressed as formula (19):

[0164] W={w1,w2,...w n} (19)

[0165] Among them, w i is the i-th historical road image b i The contour coordinate set of the i-th traffic element target in, n is the i-th historical road image b i The number of traffic element targets in i ={(x1,y1),(x2,y2),...(x n ,y n )};

[0166] For the i-th historical road image b i The contour coordinate set w of the i-th traffic element target i , traverse the contour coordinate set w of the i-th traffic element target i For all contour coordinates in , find the minimum x coordinate min and the maximum value x max and the minimum y coordinate y min and the maximum value y max , using these boundary values ​​to determine the i-th historical road image b i The minimum external matrix of the i-th traffic element target in is used as the i-th historical road image b i The best anchor box for the i-th traffic element target in .

[0167] Step S6: adding the historical road image b and the best anchor frame of each traffic element target in the historical road image b to the data set D to form a new data set D′; then using the historical road images in the new data set D′ to train the deep semantic detection segmentation model to obtain a trained deep semantic detection segmentation model T model ;

[0168] Step S7: Using the trained deep semantic detection segmentation model T model The learned knowledge is used to train the target detection model S model Regular updates are performed to ensure that S model With T model Comparable performance, achieving model compression and acceleration, and then obtaining the updated target detection model S′ model ;

[0169] Step S8: Input the road image to be detected into the updated target detection model S′ model Detection is performed to output the category of each traffic element target in the road image to be detected.

[0170] As another embodiment of the present invention, a traffic element target detection system based on memorandum learning is provided, and the traffic element target detection system based on memorandum learning includes:

[0171] An acquisition module, used to acquire a historical road image d in a data set D;

[0172] The first training module is used to construct a target detection model and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ;

[0173] The first detection module is used to input the new historical road image b into the trained target detection model S model Detection is performed in order to output the category of each traffic element target in the new historical road image b;

[0174] A calculation module, used for calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B;

[0175] A determination module is used to construct a deep semantic detection segmentation model, and detect the historical road image b in the review memo B through the deep semantic detection segmentation model to obtain a contour coordinate set of each traffic element target in the historical road image b; then determine the best anchor frame for each traffic element target in the historical road image b according to the contour coordinate set of each traffic element target in the historical road image b;

[0176] The second training module is used to add the historical road image b and the best anchor frame of each traffic element target in the historical road image b to the data set D to form a new data set D′; then the deep semantic detection segmentation model is trained using the historical road images in the new data set D′ to obtain a trained deep semantic detection segmentation model T model ;

[0177] An updating module is used to utilize the trained deep semantic detection segmentation model T model The learned knowledge is used to train the target detection model S model Perform regular updates to obtain the updated target detection model S′ model ;

[0178] The second detection module is used to input the road image to be detected into the updated target detection model S′ model Detection is performed to output the category of each traffic element target in the road image to be detected.

[0179] The method for detecting traffic element targets based on memorandum learning provided by the embodiment of the present invention (1) realizes accurate detection of targets of different sizes in different scenes through adaptive anchor frames, and uses knowledge distillation method to prune the detection model, thereby reducing the complexity of the detection model to meet the hardware deployment requirements; (2) introduces hole convolution and convolution network to obtain the relationship between long-distance and short-distance pixels, thereby improving the performance of the detection model; (3) establishes a memorandum learning library, which can regularly optimize the parameters of the detection model; the present invention can detect smaller traffic element targets such as lane lines, traffic lights and distant cars under bad weather and lighting conditions, thereby improving the accuracy of traffic element target detection, and laying a foundation for later traffic safety situation, congestion analysis and illegal motive detection.

[0180] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A traffic element target detection method based on memo learning, characterized in that: The traffic element target detection method based on memorandum learning comprises the following steps: Step S1: Obtain a historical road image d in the dataset D; Step S2: construct a target detection model, and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ; Step S3: Input the new historical road image b into the trained object detection model S model Detection is performed in order to output the category of each traffic element target in the new historical road image b; Step S4: calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B; Step S5: constructing a deep semantic detection segmentation model, and detecting the historical road image b in the review memo B by using the deep semantic detection segmentation model to obtain a set of contour coordinates of each traffic element target in the historical road image b; then determining the best anchor frame for each traffic element target in the historical road image b according to the set of contour coordinates of each traffic element target in the historical road image b; Step S6: adding the historical road image b and the best anchor frame of each traffic element target in the historical road image b to the data set D to form a new data set D′; then using the historical road images in the new data set D′ to train the deep semantic detection segmentation model to obtain a trained deep semantic detection segmentation model T model ; Step S7: Using the trained deep semantic detection segmentation model T model The learned knowledge is used to train the target detection model S model Perform regular updates to obtain the updated target detection model S′ model ; Step S8: Input the road image to be detected into the updated target detection model S′ model Detection is performed to output the category of each traffic element target in the road image to be detected.

2. According to claim 1, a traffic element target detection method based on memorandum learning is characterized in that: The step of acquiring the historical road image d in the data set D further includes: The multiple historical road images d in the data set D are preprocessed to uniformly adjust the sizes of the multiple historical road images d to a size of 640*640.

3. The method for traffic element target detection based on memo learning according to claim 1 is characterized in that: The target detection model is constructed, and the target detection model is trained using the historical road images d in the data set D to obtain a trained target detection model S model Also included: The target detection model includes a coarse-grained feature encoder, a fine-grained feature layer and a detection head decoder; Inputting the historical road image d in the data set D into the coarse-grained feature encoder to extract coarse-grained features, so as to obtain the coarse-grained features X of the historical road image d; Inputting the coarse-grained feature X of the historical road image d into the fine-grained feature layer to extract fine-grained features, so as to obtain the fine-grained feature Y of the historical road image d; The fine-grained feature Y of the historical road image d is input into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b.

4. The method for traffic element target detection based on memorandum learning according to claim 3 is characterized in that: The step of inputting the coarse-grained feature X of the historical road image d into the fine-grained feature layer to extract fine-grained features to obtain the fine-grained feature Y of the historical road image d further includes: The coarse-grained features X of the historical road image d are input into the first feature fusion layer p combine1 In order to obtain the first long-short distance fusion feature X1 of the historical road image d; as shown in formula (1): X1=p combine1 (X) (1) The first long-short distance fusion feature X1 of the historical road image d is input into the second feature fusion layer p combine2 In order to obtain the second long-short distance fusion feature X2 of the historical road image d; as shown in formula (2): X2=p combine2 (X1) (2) The second near-far distance fusion feature X2 of the historical road image d is input into the first dilated convolutional layer DilatedCNN1 to obtain the third near-far distance fusion feature X3 of the historical road image d; as shown in formula (3): X3=Dilated CNN1(X2) (3) The coarse-grained feature X of the historical road image d is input into the second dilated convolutional layer DilatedCNN2 to obtain the long-range feature X4 of the historical road image d; as shown in formula (4): X4=Dilated CNN2(X) (4) The third long-distance and short-distance fusion feature X3 of the historical road image d and the long-distance feature X4 of the historical road image d are spliced ​​together to obtain the final fusion feature X of the historical road image d. combine ; As shown in formula (5): X combine =concat(X3⊙X4) (5) The fine-grained feature Y of the historical road image d is obtained using the self-attention mechanism, as shown in formula (6): Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k represents the latitude of the matrix. The calculation methods of Q, K, and V are shown in formulas (7)-(9): Q=X combine ·W Q (7) K=X combine ·W K (8) V=X combine ·W V (9) Among them, W Q , W K , W V They are the weight matrices of the query matrix Q, the key matrix K, and the value matrix V respectively.

5. The method for detecting traffic elements based on memo learning according to claim 4 is characterized in that: The first feature fusion layer p combine1 and the second feature fusion layer p combine2 The structure of is the same, both include the long-distance feature extraction module part dcnn And the close range feature extraction module part cnn , wherein the long-distance feature extraction module part dcnn Including a hole convolution layer, a normalization layer and an activation function, the close-range feature extraction module part cnn Includes ordinary convolutional layer, normalization layer and activation function; Among them, the long-distance feature extraction module part dcnn The structure of is shown in formula (10): part dcnn =Dilated CNN(X)+BN+Relu (10) Among them, the short-range feature extraction module part cnn The structure of is shown in formula (11): share cnn =CNN(X)+BN+Relu (11) Among them, DilatedCNN represents a dilated convolutional layer, BN represents a normalized layer, Relu represents an activation function, and CNN represents a normal convolutional layer; The long-distance feature extraction module part dcnn The module for extracting long-distance features of the historical road image d is used to extract the short-distance features of the historical road image d. cnn Used to extract the short-range features of the historical road image d; weightedly fuse the long-range features and short-range features of the historical road image d to obtain a first long-short range fusion feature X1 of the historical road image d or a second long-short range fusion feature X2 of the historical road image d; as shown in formula (12): p combine =(1-b)part dcnn +bpart cnn (12) Among them, b is the weight.

6. The method for traffic element target detection based on memorandum learning according to claim 3 is characterized in that: The step of inputting the fine-grained feature Y of the historical road image d into the detection head decoder for category prediction to obtain the category of each traffic element target in the historical road image b further includes: The fine-grained feature Y of the historical road image d is converted into a 1*n vector through a fully connected layer in, Calculate the vector The category prediction probability of each element in is shown in formula (13): Among them, y i Representation vector The i-th element z in i The category prediction probability, N represents the number of categories of traffic element targets, z j is a vector The jth element in ; y i The element set with a value greater than 0.8 is used as the category set of traffic element targets in the historical road image b.

7. The method for traffic element target detection based on memorandum learning according to claim 1 is characterized in that: The target detection model is trained using the historical road images d in the data set D to obtain a trained target detection model S model Also included: In the training process of the target detection model, the total loss function Loss is used. total Back propagation is performed to update the parameters of the target detection model; wherein the total loss function Loss total The calculation formula (14) is as follows: Loss total =λ1Loss cls +λ2Loss obj (14) Among them, Loss cls is the classification loss function, Loss obj is the confidence loss function, λ1 and λ2 both represent weights; The classification loss function Loss cls The calculation formula (15) is as follows: The confidence loss function Loss obj The calculation formula (16) is as follows: Among them, N represents the number of categories of traffic element targets, N′ represents the number of positive samples, and l n represents the true label, Represents the class prediction probability of the true label.

8. The method for detecting traffic elements based on memo learning according to claim 1, characterized in that: The calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B, further includes: Preset multiple variable anchor boxes, and the definition of the variable anchor box is shown in formula (17): box={B[m*n],p} (17) Among them, m and n represent the length and width of the variable anchor box respectively, and p represents the scaling ratio; Calculate the number of pixels of each traffic element target category o in the new historical road image b to obtain the area S of each traffic element target in the new historical road image b. o ; According to the area S of each traffic element target in the new historical road image b o , select a variable anchor box Box that can cover the traffic element target from multiple variable anchor boxes, and then calculate the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box. The calculation formula (18) is as follows: Among them, S Box Represents the area of ​​the variable anchor box Box; If the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box is greater than a certain threshold, directly output the category o of the traffic element target in the new historical road image b and the position of the variable anchor box Box; if the proportion q of the traffic element target in the new historical road image b in the variable anchor box Box is not greater than the certain threshold, reduce the length and width of the variable anchor box Box according to the scaling ratio p in formula (17), and calculate the proportion q of the traffic element target in the new historical road image b in the reduced variable anchor box Box again; If the proportion q of the traffic element target in the reduced variable anchor box Box is greater than a certain threshold, the category o of the traffic element target in the new historical road image b and the position of the reduced variable anchor box Box are directly output; if the proportion q of the traffic element target in the reduced variable anchor box Box is still not greater than a certain threshold, the new historical road image b is added to the review memo B.

9. The method for traffic element target detection based on memorandum learning according to claim 1 is characterized in that: The step S5 further includes: For the i-th historical road image b in the review memo B i , the i-th historical road image b i The contour coordinate set W of all traffic element targets in is expressed as formula (19): In={in1, in2,... in n } (19) Among them, w i is the i-th historical road image b i The contour coordinate set of the i-th traffic element target in, n is the i-th historical road image b i The number of traffic element targets in i ={(x1,y1),(x2,y2),...(x n ,y n )}; For the i-th historical road image b i The contour coordinate set w of the i-th traffic element target i , traverse the contour coordinate set w of the i-th traffic element target i For all contour coordinates in , find the minimum x coordinate min and the maximum value x max and the minimum y coordinate y min and the maximum value y max , using these boundary values ​​to determine the i-th historical road image b i The minimum external matrix of the i-th traffic element target in is used as the i-th historical road image b i The best anchor box for the i-th traffic element target in .

10. A traffic element target detection system based on memorandum learning, used to implement the traffic element target detection method based on memorandum learning as described in any one of claims 1 to 9, characterized in that: The traffic element target detection system based on memorandum learning includes: An acquisition module, used to acquire a historical road image d in a data set D; The first training module is used to construct a target detection model and train the target detection model using the historical road images d in the data set D to obtain a trained target detection model S model ; The first detection module is used to input the new historical road image b into the trained target detection model S model Detection is performed in order to output the category of each traffic element target in the new historical road image b; A calculation module, used for calculating the proportion of each traffic element target in the new historical road image b in the preset anchor frame, and when the proportion does not meet the requirement, adding the new historical road image b to the review memo B; A determination module is used to construct a deep semantic detection segmentation model, and detect the historical road image b in the review memo B through the deep semantic detection segmentation model to obtain a contour coordinate set of each traffic element target in the historical road image b; then determine the best anchor frame for each traffic element target in the historical road image b according to the contour coordinate set of each traffic element target in the historical road image b; The second training module is used to add the historical road image b and the best anchor frame of each traffic element target in the historical road image b to the data set D to form a new data set D′; then the deep semantic detection segmentation model is trained using the historical road images in the new data set D′ to obtain a trained deep semantic detection segmentation model T model ; An updating module is used to utilize the trained deep semantic detection segmentation model T model The learned knowledge is used to train the target detection model S model Perform regular updates to obtain the updated target detection model S′ model ; The second detection module is used to input the road image to be detected into the updated target detection model S′ model Detection is performed to output the category of each traffic element target in the road image to be detected.

Images