Real-time target detection method, system and medium

Abstract

The present invention provides a real-time target detection method, system and medium, wherein the target includes one of a vehicle or a pedestrian, and the method includes: obtaining monitoring data information of the current target, wherein the type of the monitoring data of the current target includes video data or a picture frame; performing reasoning based on the type of the monitoring data of the current target and a target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights, detecting and analyzing the current target; judging whether the current target is in a dangerous driving area based on the analysis results; and performing corresponding operations based on the judgment results. The present invention innovatively combines previous background knowledge distillation and adaptive dynamic loss weights, so that the distillation results are more accurate and the detection speed is faster. In addition, the real-time target detection method of the present invention is superior to many traditional knowledge distillation methods, and the detection results are much more accurate than traditional methods, and the missed detection rate is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle and pedestrian detection, and in particular to a real-time target detection method, system and medium based on adaptive dynamic foreground and background knowledge distillation. Background Art

[0002] Autonomous driving has been a hot research area in recent years, incorporating many cutting-edge intelligent technologies. The most critical aspect of autonomous driving is ensuring adequate safety during operation. Therefore, vehicles must accurately and efficiently perceive their surroundings and respond quickly to environmental changes, particularly those affecting pedestrians, vehicles, and surrounding obstacles and landmarks. These key issues can be addressed using object detection technology. By collecting environmental information through onboard cameras and various image sensors, the vehicle's "brain" can determine the presence of nearby objects, their proximity, and other factors, enabling it to make informed decisions. Summary of the Invention

[0003] The main purpose of the present invention is to provide a real-time target detection method, system and medium based on adaptive dynamic background knowledge distillation, aiming to improve the real-time detection accuracy of vehicles and pedestrians and reduce the missed detection rate.

[0004] To achieve the above object, the present invention provides a real-time target detection method, wherein the target includes a vehicle or a pedestrian, and the method comprises the following steps:

[0005] Step S10, obtaining monitoring data information of the current target, where the type of the monitoring data of the current target includes video data or picture frames;

[0006] Step S20: Detect and analyze the current target based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights;

[0007] Step S30, judging whether the current target is in a driving danger zone according to the analysis result;

[0008] Step S40: Execute corresponding operations according to the judgment result.

[0009] A further technical solution of the present invention is that the step S40 of performing corresponding operations according to the judgment result includes:

[0010] Step S401: If the current target is in a dangerous driving area, a vehicle alarm is issued;

[0011] Step S402: outputting the detection result of the current target;

[0012] Step S403: If the current target is in a dangerous driving area, directly output the detection result of the current target.

[0013] A further technical solution of the present invention is that, in step S20, the step of performing reasoning on the current target based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights, detecting and analyzing the current target includes:

[0014] S201, if the type of the monitoring data of the current target is video data, use Gstreamer to encode and decode the video stream, and use the cap=cv2.VideoCapture() and cap.read() methods in OpenCV to read the video frame by frame and extract image frames;

[0015] In step S202 , the image frame is fed into a target detection model that is pre-established by combining previous and background knowledge distillation and adaptive dynamic loss weights to perform reasoning, and the current target is detected and analyzed.

[0016] A further technical solution of the present invention is that, in step S20, the step of performing inference on the current target based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights, further comprises:

[0017] If the type of monitoring data of the current target is an image frame, step S202 is directly executed, and the image frame is sent to the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights to perform reasoning, and the current target is detected and analyzed.

[0018] A further technical solution of the present invention is that the step S30 of determining whether the current target is in a driving danger zone according to the analysis result includes:

[0019] Step S301: If the current target appears in front of the vehicle and the pixel value of the current target is greater than a first preset value, or the current target appears on both sides of the vehicle and the pixel value of the current target is greater than a second preset value, it is determined that the current target is in a driving danger area.

[0020] A further technical solution of the present invention is that, before the step S10 of obtaining the monitoring data information of the current target, the following steps are included:

[0021] Step S00: Preliminarily establish a target detection model by combining previous background knowledge distillation and adaptive dynamic loss weights.

[0022] A further technical solution of the present invention is that the step S00 of pre-establishing the target detection model by combining previous background knowledge distillation and adaptive dynamic loss weights includes:

[0023] Select the Neck part from the object detection network, use the ground-truth mask to split the extracted features into foreground-related features and background-related features, and use L2 Loss between the teacher and the student to effectively transfer the teacher's knowledge to the student. For a given intermediate feature of size H×W, first generate a binary mask M based on the ground-truth:

[0024] M i，j =1[(i, j)∈B] (1)

[0025] Where B is the ground-truth, M∈{0, 1} H×W , the value of position (i, j) is 1 if it belongs to the foreground, and 0 if it belongs to the background; by generating a binary mask M, we can know whether each position belongs to the foreground or background, which can be used to decouple the foreground and background features of the neck;

[0026] Here we assume that T∈R H×W×C and S∈R H×W×C are the intermediate features of students and teachers respectively, then the distillation loss of the intermediate features is:

[0027]

[0028]

[0029]

[0030] in:

[0031] H, W, and C represent the height, width, and number of channels of the feature map, respectively;

[0032] M represents the generated binary mask:

[0033] N obj Indicates the number of elements in the foreground area;

[0034] N bg Represents the number of elements in the background area;

[0035] Formula (3) represents the number of positive sample elements in the foreground area;

[0036] Formula (4) represents the number of negative sample elements in the background area;

[0037] φ represents the adaptive layer, which is used to control the Teacher and Student to produce feature maps of the same scale;

[0038] α obj Represents the weight of the foreground region distillation loss function;

[0039] α bg Represents the weight of the background area distillation loss function;

[0040] Formula (2) represents the distillation loss of intermediate features;

[0041] In this way, we divide the feature maps in the teacher and student networks into foreground and background regions in a mask-based way, and use α obj and α bg Different weights are used to control the proportion of foreground and background distilled knowledge; the final total loss is as follows:

[0042] L=α obj L obj +α bg L bg +βL cls +λL reg (5)

[0043] in:

[0044] L obj , L bg , L cls and L reg They represent foreground distillation loss, background distillation loss, classification loss, and regression loss respectively;

[0045] α obj , α bg , α cls and α reg Represents the foreground distillation loss weight, background distillation loss weight, classification loss weight, and regression loss weight respectively;

[0046] L indicates that the four losses are jointly optimized with different loss weights.

[0047] A further technical solution of the present invention is that the step S00 of pre-establishing the target detection model by combining previous background knowledge distillation and adaptive dynamic loss weights further includes:

[0048] The exponential moving average method is used to calculate the standard deviation of the loss change after each batch, avoiding the time-consuming increase caused by explicit gradient calculation, and the gradient magnitude near a certain point can be obtained. The formula used is:

[0049]

[0050]

[0051] b i ←βb i +(1-β)L i (8)

[0052]

[0053]

[0054] in:

[0055] n means there are n tasks;

[0056] ||g i || represents the gradient of task i;

[0057] w i represents the weight of task i;

[0058] L i represents the loss of task i;

[0059] a i Indicates L i Squared moving average;

[0060] b i Indicates L i Moving average of

[0061] β represents the hyperparameter of the moving average;

[0062] si represents the standard deviation of task i;

[0063] At the same time, in order to prevent the learned weight w from being too large or too small, the sum of the weights needs to be normalized n times each time, as shown in formula (10);

[0064] In order to prevent differences in the calculation of standard deviations due to different loss magnitudes, the coefficient of variation is introduced in formula (9) to eliminate the influence of measurement scale and dimension. Thus, when calculating the independent standard deviation of each task, the loss magnitude is also taken into account. The formula used is:

[0065]

[0066] To achieve the above objectives, the present invention also proposes a real-time target detection system, which includes a memory, a processor, and a real-time target detection program stored on the processor. When the real-time target detection program is run by the processor, the steps of the method described above are executed.

[0067] To achieve the above objectives, the present invention further proposes a computer-readable storage medium, characterized in that the computer-readable storage medium stores a target real-time detection program, and the target real-time detection program executes the steps of the above method when run by a processor.

[0068] The beneficial effects of the real-time target detection method, system and medium of the present invention are: the target detection method of the present invention adopts the above-mentioned technical scheme, and the target includes one of a vehicle or a pedestrian. The method includes the following steps: Step S10, obtaining the monitoring data information of the current target, and the type of the monitoring data of the current target includes video data or picture frames; Step S20, performing reasoning according to the type of the monitoring data of the current target and the target detection model pre-established by combining the previous background knowledge distillation and the adaptive dynamic loss weight, detecting and analyzing the current target; Step S30, judging whether the current target is in a dangerous driving area according to the analysis result; Step S40, performing corresponding operations according to the judgment result, innovatively combining the previous background knowledge distillation and the adaptive dynamic loss weight, so that the distillation result is more accurate and the detection speed is faster, and the real-time target detection method of the present invention is superior to many traditional knowledge distillation methods, and the detection results are also much more accurate than traditional methods, and the missed detection rate is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 1 is a flow chart of a preferred embodiment of the target real-time detection method of the present invention;

[0070] Figure 2 This is a schematic diagram of the background knowledge distillation network;

[0071] Figure 3 This is a schematic diagram of the overall process of a preferred embodiment of the target real-time detection method of the present invention;

[0072] Figure 4 This is a schematic diagram of the vehicle assisted driving system test results.

[0073] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0074] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0075] The present invention proposes a real-time target detection method, focusing on real-time vehicle and pedestrian detection, designs a suitable model, and adopts efficient and real-time adaptive dynamic background knowledge distillation technology to make model detection as fast and accurate as possible.

[0076] At the same time, a real-time target detection system is designed, which can be used in the vehicle assisted driving system. The system monitors in real time whether there are any detected objects in the driving danger zone ahead. If there are any, it will promptly feedback to the driver to remind him to slow down or brake in time; if not, it will drive normally.

[0077] This method innovatively combines the distillation of foreground and background knowledge with adaptive dynamic loss weights, making the distillation results more accurate and the detection speed faster. In addition, this method is superior to many traditional knowledge distillation methods, and the detection results are much more accurate than traditional methods, with a greatly reduced missed detection rate.

[0078] Specifically, if Figure 1 As shown, in a preferred embodiment of the target real-time detection method of the present invention, the target includes one of a vehicle or a pedestrian, and the preferred embodiment of the target real-time detection method of the present invention includes the following steps:

[0079] Step S10: Acquire monitoring data information of the current target, where the type of the monitoring data of the current target includes video data or picture frames.

[0080] Step S20 , performing reasoning based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights, to detect and analyze the current target.

[0081] Specifically, in this embodiment, step S20 specifically includes:

[0082] S201, if the type of the monitoring data of the current target is video data, use Gstreamer to encode and decode the video stream, and use the cap=cv2.VideoCapture() and cap.read() methods in OpenCV to read the video frame by frame and extract image frames.

[0083] In step S202 , the image frame is fed into a target detection model that is pre-established by combining previous and background knowledge distillation and adaptive dynamic loss weights to perform reasoning, and the current target is detected and analyzed.

[0084] In step S203, if the type of the monitoring data of the current target is an image frame, step S202 is directly executed, and the image frame is sent to the target detection model established in advance by combining the previous background knowledge distillation and the adaptive dynamic loss weight to perform reasoning, and the current target is detected and analyzed.

[0085] Step S30: determining whether the current target is in a driving danger zone based on the analysis result.

[0086] Specifically, in this embodiment, step S30 specifically includes:

[0087] Step S301: If the current target appears in front of the vehicle and the pixel value of the current target is greater than a first preset value, or the current target appears on both sides of the vehicle and the pixel value of the current target is greater than a second preset value, it is determined that the current target is in a driving danger area.

[0088] The first preset value can be modified according to different video resolutions, for example, set to 100, and the second preset value can also be modified according to different video resolutions, for example, set to 170.

[0089] Step S40: Execute corresponding operations according to the judgment result.

[0090] Specifically, the step S40 includes:

[0091] Step S401: If the current target is in a driving danger zone, a vehicle alarm is issued.

[0092] It should be noted that, in this embodiment, if the current target is in a dangerous driving area, timely feedback can be given to the driver to remind him to slow down or brake in time.

[0093] Step S402: output the detection result of the current target.

[0094] Step S403: If the current target is in a dangerous driving area, directly output the detection result of the current target.

[0095] It should be noted that, in this embodiment, if the current target is not in the driving danger zone, the vehicle can drive normally.

[0096] Furthermore, in this embodiment, before the step S10 of obtaining the monitoring data information of the current target, the following steps are included:

[0097] Step S00: Preliminarily establish a target detection model by combining previous background knowledge distillation and adaptive dynamic loss weights.

[0098] Specifically, the step S00 of pre-establishing the target detection model by combining the previous background knowledge distillation and the adaptive dynamic loss weight includes:

[0099] Select the Neck part from the object detection network, use the ground-truth mask to split the extracted features into foreground-related features and background-related features, and use L2 Loss between the teacher and the student to effectively transfer the teacher's knowledge to the student. For a given intermediate feature of size H×W, first generate a binary mask M based on the ground-truth:

[0100] M i,j =1[(i,j)∈B] (1)

[0101] Where B is the ground-truth, M∈{0,1} H×W , the value of position (i, j) is 1 if it belongs to the foreground, and 0 if it belongs to the background; by generating a binary mask M, we can know whether each position belongs to the foreground or background, which can then be used to decouple the foreground and background features of the neck.

[0102] Here we assume that T∈R H×W×C and S∈R H×W×C are the intermediate features of students and teachers respectively, then the distillation loss of the intermediate features is:

[0103]

[0104]

[0105]

[0106] in:

[0107] H, W, and C represent the height, width, and number of channels of the feature map, respectively;

[0108] M represents the generated binary mask;

[0109] N obj Indicates the number of elements in the foreground area;

[0110] N bg Represents the number of elements in the background area;

[0111] Formula (3) represents the number of positive sample elements in the foreground area;

[0112] Formula (4) represents the number of negative sample elements in the background area;

[0113] φ represents the adaptive layer, which is used to control the Teacher and Student to produce feature maps of the same scale;

[0114] a obj Represents the weight of the foreground region distillation loss function;

[0115] α bg Represents the weight of the background area distillation loss function;

[0116] Formula (2) represents the distillation loss of intermediate features;

[0117] In this way, we divide the feature maps in the teacher and student networks into foreground and background regions in a mask-based way, and use α obj and α bg Different weights are used to control the proportion of foreground and background distilled knowledge; the final total loss is as follows:

[0118] L=α obj L obj +α bg L bg +βL cls +λL reg (5)

[0119] in:

[0120] L obj , L bg , L cls and L reg Represents foreground distillation loss, background distillation loss, classification loss and regression loss respectively:

[0121] α obj , α bg , α cls and α reg Represents the foreground distillation loss weight, background distillation loss weight, classification loss weight, and regression loss weight respectively;

[0122] L indicates that the four losses are jointly optimized with different loss weights.

[0123] Furthermore, the step S00 of pre-establishing the target detection model by combining the previous background knowledge distillation and the adaptive dynamic loss weight also includes:

[0124] The exponential moving average method is used to calculate the standard deviation of the loss change after each batch, avoiding the time-consuming increase caused by explicit gradient calculation, and the gradient magnitude near a certain point can be obtained. The formula used is:

[0125]

[0126] b i ←βb i +(1-β)L i (8)

[0127]

[0128]

[0129] in:

[0130] n means there are n tasks;

[0131] ‖g i ‖ represents the gradient of task i;

[0132] w i represents the weight of task i;

[0133] L i represents the loss of task i;

[0134] a i Indicates L i Squared moving average;

[0135] b i Indicates L i Moving average of

[0136] β represents the hyperparameter of the moving average;

[0137] si represents the standard deviation of task i;

[0138] At the same time, in order to prevent the learned weight w from being too large or too small, the sum of the weights needs to be normalized n times each time, as shown in formula (10);

[0139] In order to prevent differences in the calculation of standard deviations due to different loss magnitudes, the coefficient of variation is introduced in formula (9) to eliminate the influence of measurement scale and dimension. Thus, when calculating the independent standard deviation of each task, the loss magnitude is also taken into account. The formula used is:

[0140]

[0141] The following combination Figures 2 to 4 The target real-time detection method of the present invention is further elaborated.

[0142] To achieve efficient real-time target detection, this method applies knowledge distillation to target detection. A rational model is designed, and dynamic loss weighting is employed to maximize the effectiveness of knowledge distillation. Finally, a vehicle assisted driving system is designed based on the proposed model.

[0143] The main research contents of the present invention include:

[0144] 1. Distillation of previous background knowledge

[0145] The ground-truth mask is used to segment the extracted features into foreground-related features and background-related features, while the foreground and background knowledge are distilled and fused with different loss weights.

[0146] 2. Adaptive dynamic loss weight

[0147] Because the background knowledge distillation network integrates multiple types of loss, each task requires a different loss weight to achieve optimal results. Based on multiple factors, such as the learning stage, difficulty, and learning effect of the task, we designed and optimized an adaptive dynamic loss weighting method to dynamically adjust the loss weights to balance the learning rate of each task and maximize knowledge.

[0148] 3. Vehicle assisted driving system

[0149] Based on the above optimization model, a vehicle assisted driving system is designed, and reasonable functional design and implementation are carried out according to the functional requirements of the system.

[0150] The above research contents are introduced in detail below.

[0151] 1. Adaptive dynamic background knowledge distillation

[0152] This method can be broken down into two main modules: foreground knowledge distillation and adaptive dynamic loss weighting. Therefore, the following sections will introduce these two modules separately. The first section introduces foreground knowledge distillation, the second section introduces adaptive dynamic loss weighting, and finally the final solution and experimental results.

[0153] 1.1. Distillation of Background Knowledge

[0154] This section mainly introduces the overview of the background knowledge distillation method and the designed model.

[0155] 1.1.1 Overview of Background Knowledge Distillation Methods

[0156] Knowledge distillation is a widely used paradigm in model compression. It primarily uses the output of a complex teacher network to enable a simpler student network model to achieve the same performance as a complex network. The teacher network is a network with a complex structure and large number of parameters but good inference performance; the student network is a network with a simpler structure, lower complexity, and poorer performance.

[0157] Currently, knowledge distillation is widely used in image classification and object detection. However, unlike image classification, object detection is more complex, involving multiple loss functions, and the features upon which semantic information relies are relatively complex. Consequently, many knowledge distillation methods applied to image classification have largely failed in the field of object detection. This is due to the greater complexity of the object detection domain and the significant imbalance in pixels between foreground and background. Most current knowledge distillation methods for object detection focus primarily on the foreground region containing the object. These methods extract knowledge solely from this region, assuming that background regions may introduce significant noise during the distillation process.

[0158] However, there are many research works that have proved that background information also contains a lot of useful knowledge, which is helpful for target detection. Therefore, it is inappropriate to directly abandon the distillation of background areas.

[0159] In knowledge distillation, while using only positive samples can improve model metrics, omitting them and instead using only background regions can also achieve the same metrics. This is because background region distillation effectively reduces the number of false positives. Previous knowledge distillation methods assumed that background information contained too much noise, hindering the performance of the student model. However, background regions also contain useful information, which can help improve student model metrics. Positive sample information helps improve the accuracy of the results, while background information helps reduce false positives.

[0160] However, some related studies have shown that once the background and foreground are integrated together, the distillation performance will deteriorate. This may be because these methods directly integrate the two regions. Therefore, based on the different information contained in the target region and the background region, different weights should be given.

[0161] 1.1.2. Background Knowledge Distillation Network Design

[0162] Based on the above background, we need to design a network that can separate and simultaneously distill foreground and background knowledge, and fuse them with different loss weights. Here is an introduction to the design ideas:

[0163] Generally speaking, an object detection network consists of three or four components: Backbone for feature extraction, Neck for fusing multi-layer features, RPN for generating proposals (used only in two-stage detection methods), and the detection head for classification and bounding box regression. The goal of distillation is to allow the student network to learn the knowledge of the teacher network, which can be intermediate-layer features or soft knowledge from proposals in the detection head.

[0164] Select the Neck part of the network, use the ground-truth mask to segment the extracted features into foreground-related features and background-related features, and use L2 Loss between the teacher and the student to effectively transfer the teacher's knowledge to the student. The specific network structure diagram is as follows Figure 2 shown.

[0165] For a given intermediate feature of size H×w, a binary mask M is first generated based on the ground-truth:

[0166] M i，j =1[(i, j)∈B] (1)

[0167] Where B is the ground-truth, M∈{0, 1} H×W , the value of position (i, j) is 1 if it belongs to the foreground and 0 if it belongs to the background. By generating a binary mask M, we can know whether each position belongs to the foreground or background, which can be used to decouple the foreground and background features of the neck.

[0168] Here we assume that T∈R H×W×C and S∈R H×W×C are the intermediate features of students and teachers respectively, then the distillation loss of the intermediate features is:

[0169]

[0170]

[0171]

[0172] in:

[0173] H, W, and C represent the height, width, and number of channels of the feature map, respectively.

[0174] M represents the generated binary mask.

[0175] N obj The number of elements representing the foreground area.

[0176] N bg The number of elements representing the background area.

[0177] Formula (3) represents the number of positive sample elements in the foreground area.

[0178] Formula (4) represents the number of negative sample elements in the background area.

[0179] φ represents the adaptive layer, which is used to control the Teacher and Student to produce feature maps of the same scale.

[0180] α obj Represents the weight of the foreground region distillation loss function.

[0181] α bg Represents the weight of the background region distillation loss function.

[0182] Formula (2) represents the distillation loss of intermediate features.

[0183] In this way, we divide the feature maps in the teacher and student networks into foreground and background regions in a mask-based way, and use α obj and α bg Different weights are used to control the weight of the background distillation knowledge. The final total loss is as follows:

[0184] L=α obj L obj +α bg L bg +βL cls +λL reg (5)

[0185] in:

[0186] L obj , L bg , L cls and L reg They represent foreground distillation loss, background distillation loss, classification loss, and regression loss respectively.

[0187] α obj , α bg , α cls and α reg They represent the foreground distillation loss weight, background distillation loss weight, classification loss weight, and regression loss weight, respectively.

[0188] L indicates that the four losses are jointly optimized with different loss weights.

[0189] 1.2. Adaptive dynamic loss weight

[0190] This section mainly introduces the overview of the adaptive dynamic loss weight method and the design solution.

[0191] 1.2.1 Overview of Adaptive Dynamic Loss Weight Method

[0192] Because the background knowledge distillation network integrates multiple types of loss, each task needs to be assigned a different loss weight to achieve optimal results. During model training, the total loss of multiple tasks is the weighted sum of the losses of each task. However, the loss weights of different tasks are currently set manually, which can lead to some problems:

[0193] (1) Setting the loss weight requires a lot of manual experience, which is costly and inefficient.

[0194] (2) The balance of gradients of different tasks is not considered. When the gradient of a task is large, it may dominate the direction of gradient update.

[0195] (3) The learning difficulty of the tasks themselves varies. Paying too much attention to simple tasks will limit the model's learning effect on difficult tasks.

[0196] Due to the existence of these problems, it is necessary to dynamically adjust the loss weights from multiple aspects such as the learning stage of the task, the difficulty of learning, and the learning effect, so as to balance the learning rate of each task and maximize knowledge.

[0197] Currently, most adaptive dynamic loss weights are divided based on whether gradients need to be calculated:

[0198] (1) Gradient-based models

[0199] It's often necessary to calculate the relative gradient of the loss for each objective to more precisely determine the training speed of each objective. Representative methods include GradNorm, PCGrad, and the Pareto series. The need to calculate gradients can increase resource usage and slow down the execution speed.

[0200] (2) Non-gradient-based models

[0201] In practice, this method is more feasible and faster to calculate, and is gradually becoming the mainstream method in the industry. Representative methods include uncertainty weighting, DTP, and DWA.

[0202] 1.2.2. Adaptive dynamic loss weight design scheme

[0203] After thoroughly researching related methods and experimenting with several well-known dynamic loss weighting methods, I concluded that an ideal dynamic loss algorithm should not be based on gradients and should not only focus on relative changes in loss but also consider the magnitude of loss across different tasks. Based on this, I designed an efficient and advanced dynamic loss algorithm.

[0204] We know that the standard deviation of a point's neighborhood can approximate the magnitude of the gradient at that point. Therefore, we can use the exponential moving average method to calculate the standard deviation of the loss change after each batch, as shown in Formulas (7), (8), and (9). This approximates the magnitude of the gradient. This clever operation avoids the time-consuming increase in explicit gradient calculations while also obtaining the magnitude of the gradient near a point.

[0205]

[0206] b i ←βb i +(1-β)L i (8)

[0207]

[0208]

[0209] in:

[0210] n means there are n tasks.

[0211] ‖g i ‖ represents the gradient of task i.

[0212] w i represents the weight of task i.

[0213] L i represents the loss of task i.

[0214] a i Indicates L i Squared moving average.

[0215] b i Indicates L i The moving average of .

[0216] β represents the hyperparameter of the moving average.

[0217] si represents the standard deviation of task i.

[0218] At the same time, in order to prevent the learned weight w from being too large or too small, the sum of the weights needs to be normalized n times each time, as shown in formula (2-10).

[0219]

[0220] In order to prevent differences in the calculation of standard deviations due to different loss magnitudes, the present invention further optimizes formula (9). As shown in formula (11), the coefficient of variation is introduced to eliminate the influence of measurement scale and dimension, so that the loss magnitude is cleverly taken into account when calculating the independent standard deviation of each task.

[0221] 1.3 Adaptive Dynamic Background Knowledge Distillation Method

[0222] The final solution is a combination of the background knowledge distillation designed in Section 1.1 and the adaptive dynamic loss weighting method designed in Section 1.2.

[0223] Since the background knowledge distillation network integrates multiple types of loss, in order to achieve the best results, each task needs to be equipped with a different dynamic loss weight. During model training, the total loss of multiple tasks is the weighted sum of the losses of each task. Therefore, I replaced the training loss weight obtained by the adaptive dynamic loss weight method in Section 1.2 with α in formula (5) obj , α bg , β and λ.

[0224] 1.4 Experimental Results

[0225] The DETRAC dataset is used in this experiment, and the specific results are shown in Table 1.

[0226] Table 1 Performance comparison of different optimization methods on the DETRAC dataset

[0227]

[0228]

[0229] As can be seen from Table 1, for the DETRAC dataset, both distilling only foreground information and distilling only background information have a significant improvement in mAP.

[0230] If the foreground and background information are distilled with different weights, since the weights are manually specified and remain unchanged, the change in mAP also brings certain uncertainty.

[0231] If the foreground and background information are all distilled at a 1:1 ratio, the mAP will drop significantly, indicating that the foreground and background information need to be distilled separately with different weights.

[0232] The adaptive dynamic foreground and background distillation I proposed achieved the highest accuracy in mAP and realized the gain combination of foreground and background information.

[0233] 2 Implementation of Vehicle Assisted Driving System

[0234] Nowadays, with the rapid influx of urban population, vehicles and pedestrians can be seen everywhere on the streets, and some people who do not obey traffic rules pose a huge hidden danger to the safety of drivers.

[0235] The vehicle assisted driving system designed in this article is designed to help drivers solve this problem. The system monitors in real time whether there are any objects in the driving danger zone ahead. If there are any, it will promptly provide feedback to the driver to remind them to slow down or brake in time; if not, the system will continue driving normally.

[0236] The system uses the video captured by the vehicle's front camera or local video as input data. The overall flow chart is as follows: Figure 3 As shown in the figure, the system subthread uses Gstreamer to encode and decode the video stream, and uses the OpenCV methods cap = cv2.VideoCapture() and cap.read() to read the video frame by frame and extract image frames. These image frames are then fed into the optimized model for inference to detect and analyze targets such as pedestrians and vehicles. When a target appears in front of the vehicle and its pixel value is greater than 100 (this can be modified based on the video resolution), or when a target appears on either side of the vehicle and its pixel value is greater than 170 (this can be modified based on the video resolution), the system will promptly issue an alarm to remind the driver of the vehicle or pedestrian ahead and to slow down or apply emergency braking. After the relevant operations are completed, the system automatically reads the next frame of the image and performs the same operation.

[0237] At this point, the main functions of the vehicle assisted driving system have been realized. After the detection is completed, the detection results will be transmitted in real time for the driver to view. Figure 4 As shown in the figure, it can be seen that because the distance between the vehicle in front and the driver is too close, a reminder "Driving Advice: Please Slow Down" will be displayed in the upper left corner of the diagram to facilitate the driver to make the next decision.

[0238] The beneficial effects of the real-time target detection method of the present invention are: the target detection method of the present invention adopts the above-mentioned technical scheme, and the target includes one of a vehicle or a pedestrian. The method includes the following steps: Step S10, obtaining the monitoring data information of the current target, and the type of the monitoring data of the current target includes video data or picture frames; Step S20, performing reasoning according to the type of the monitoring data of the current target, and the target detection model pre-established by combining the previous background knowledge distillation and the adaptive dynamic loss weight, detecting and analyzing the current target; Step S30, judging whether the current target is in a dangerous driving area according to the analysis result; Step S40, performing corresponding operations according to the judgment result, innovatively combining the previous background knowledge distillation and the adaptive dynamic loss weight, so that the distillation result is more accurate and the detection speed is faster, and the real-time target detection method of the present invention is superior to many traditional knowledge distillation methods, and the detection results are also much more accurate than traditional methods, and the missed detection rate is greatly reduced.

[0239] To achieve the above-mentioned purpose, the present invention also proposes a real-time target detection system, which includes a memory, a processor, and a real-time target detection program stored on the processor. When the real-time target detection program is run by the processor, the steps of the method described in the above embodiment are executed, which will not be repeated here.

[0240] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, which stores a target real-time detection program. When the target real-time detection program is run by a processor, the steps of the method described in the above embodiment are executed, which will not be repeated here.

[0241] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time target detection method, characterized in that: The target includes one of a vehicle and a pedestrian, and the method includes the following steps: Step S10, obtaining monitoring data information of the current target, where the type of the monitoring data of the current target includes video data or picture frames; Step S20: Detect and analyze the current target based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights; Step S30, judging whether the current target is in a driving danger zone according to the analysis result; Step S40, performing corresponding operations according to the judgment result; The step S10, before the step of obtaining the monitoring data information of the current target, includes: Step S00, pre-establishing a target detection model by combining previous background knowledge distillation and adaptive dynamic loss weights; The step S00 of pre-establishing a target detection model by combining previous background knowledge distillation and adaptive dynamic loss weights includes: Select the Neck part from the object detection network, use the ground-truth mask to split the extracted features into foreground-related features and background-related features, and use L2 Loss between the teacher and the student to effectively transfer the teacher's knowledge to the student. For a given intermediate feature of size H×W, first generate a binary mask M based on the ground-truth: M i，j =1[(i,j)∈B] (1) Where B is the ground-truth, M∈{0, 1} H×W , the value of position (i, j) is 1, indicating that it belongs to the foreground, and 0 indicates the background; by generating a binary mask M, we can know whether each position belongs to the foreground or background, which can be used to decouple the foreground and background features of the neck; The step S00 of pre-establishing the target detection model by combining the previous background knowledge distillation and the adaptive dynamic loss weight also includes: The exponential moving average method is used to calculate the standard deviation of the loss change after each batch, avoiding the time-consuming increase caused by explicit gradient calculation, and the gradient magnitude near a certain point can be obtained. The formula used is: b i ←bb i +(1-β)L i (8) in: n means there are n tasks; w i represents the weight of task i; L i represents the loss of task i; a i Indicates L i Squared moving average; b i Indicates L i Moving average of β represents the hyperparameter of the moving average; s i represents the standard deviation of task i; At the same time, in order to prevent the learned weight w from being too large or too small, the sum of the weights needs to be normalized n times each time, as shown in formula (10); In order to prevent differences in the calculation of standard deviations due to different loss magnitudes, the coefficient of variation is introduced in formula (9) to eliminate the influence of measurement scale and dimension. Thus, when calculating the independent standard deviation of each task, the loss magnitude is also taken into account. The formula used is:

2. The real-time target detection method according to claim 1, characterized in that: The step S40 of performing corresponding operations according to the judgment result includes: Step S401: If the current target is in a dangerous driving area, a vehicle alarm is issued; Step S402: outputting the detection result of the current target; Step S403: If the current target is in a dangerous driving area, directly output the detection result of the current target.

3. The real-time target detection method according to claim 1, characterized in that: In step S20, the steps of performing reasoning on the current target based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights, detecting and analyzing the current target include: S201, if the type of the monitoring data of the current target is video data, use Gstreamer to encode and decode the video stream, and use the cap=cv2.VideoCapture() and cap.read() methods in OpenCV to read the video frame by frame and extract image frames; In step S202 , the image frame is fed into a target detection model that is pre-established by combining previous and background knowledge distillation and adaptive dynamic loss weights to perform reasoning, and the current target is detected and analyzed.

4. The real-time target detection method according to claim 3, characterized in that: In step S20, the step of performing reasoning on the current target based on the type of monitoring data of the current target and the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights, and detecting and analyzing the current target also includes: If the type of monitoring data of the current target is an image frame, step S202 is directly executed, and the image frame is sent to the target detection model pre-established by combining previous background knowledge distillation and adaptive dynamic loss weights to perform reasoning, and the current target is detected and analyzed.

5. The real-time target detection method according to claim 4, characterized in that: The step S30 of determining whether the current target is in a dangerous driving area according to the analysis result includes: Step S301: If the current target appears in front of the vehicle and the pixel value of the current target is greater than a first preset value, or the current target appears on both sides of the vehicle and the pixel value of the current target is greater than a second preset value, it is determined that the current target is in a driving danger area.

6. The real-time target detection method according to claim 1, characterized in that: In step S00, the distillation loss of the intermediate feature is: in: T∈R H×W×C and S∈R H×W×C are the intermediate characteristics of students and teachers respectively; H, W, and C represent the height, width, and number of channels of the feature map, respectively; M represents the generated binary mask; N obj Indicates the number of elements in the foreground area; N bg Represents the number of elements in the background area; Formula (3) represents the number of positive sample elements in the foreground area; Formula (4) represents the number of negative sample elements in the background area; φ represents the adaptive layer, which is used to control the Teacher and Student to produce feature maps of the same scale; α obj Represents the weight of the foreground region distillation loss function; α bg Represents the weight of the background area distillation loss function; Formula (2) represents the distillation loss of intermediate features; The feature maps in the teacher and student networks are divided into foreground and background regions by a mask-based approach, and the foreground and background regions are respectively obj and α bg Different weights are used to control the proportion of foreground and background distilled knowledge; the final total loss is as follows: L=a obj L obj +a bg L bg +βL cls +λL reg (5) in: L obj , L bg , L cls and L reg They represent foreground distillation loss, background distillation loss, classification loss, and regression loss respectively; α obj , α bg , α cls and α reg Represents the foreground distillation loss weight, background distillation loss weight, classification loss weight, and regression loss weight respectively; L indicates that the four losses are jointly optimized with different loss weights.

7. A real-time target detection system, characterized in that: The system includes a memory, a processor, and a real-time target detection program stored on the processor. When the real-time target detection program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are executed.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a target real-time detection program, and when the target real-time detection program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.

Images