Data balance and multitask signal lamp detection method

Abstract

The invention provides a data balance and multi-task signal lamp detection method. The data balance and multi-task signal lamp detection method comprises the steps of data balance, model construction and training test. Data balancing enhances source data and simulation data by collecting a source data set and a background data set. Model construction involves feature encoders, color, direction, and position detection heads. The training test comprises transfer learning, joint loss optimization and single-module and joint test. According to the invention, the accuracy and robustness of signal lamp detection can be improved, and the intelligent level of a traffic supervision system is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation systems, and more particularly to a data balancing and multi-task signal light detection method. Background Art

[0002] In the field of intelligent transportation systems, accurate detection of traffic light status is crucial for traffic flow management, vehicle navigation, and autonomous driving technology. Existing traffic light detection technologies mainly rely on image processing and machine learning algorithms, which often face problems of insufficient accuracy and robustness when dealing with complex traffic scenes and traffic light detection under different lighting conditions. Traditional detection methods may have limitations in recognizing the color, shape, and position of traffic lights, especially when the traffic lights are obscured or there is a lot of background interference. In addition, single-task detection models often have difficulty in simultaneously processing multiple attributes of traffic lights, such as color, direction, and position, which limits the comprehensive performance of the detection system.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the imbalance of the data set causes the model to have insufficient recognition ability for certain categories of traffic lights, and the single-task model cannot effectively integrate various attribute information of traffic lights, resulting in limited detection efficiency and accuracy. Summary of the invention

[0004] The present invention provides a data-balanced and multi-task signal light detection method, comprising: Step S1: Data balancing: performing data collection to obtain a source data set and a background data set, performing a source data enhancement operation on the source data set, generating a simulation data set through data simulation, and performing simulation data enhancement on the simulation data set; Step S2: Traffic light detection model construction: construct a multi-module joint detection structure including a feature encoder, a color detection head, a direction detection head and a position detection head. The feature encoder extracts deep features of the image and passes them to other detection heads. The color detection head outputs the color status of the traffic light. The direction detection head identifies the direction status of the traffic light. The position detection head detects the position information of the traffic light panel. Step S3: Model training and testing: First transfer the learning feature encoder, then use the loss function to optimize the model, and finally perform single module testing, joint testing, error analysis and fine-tuning on the model.

[0005] Furthermore, the data collection in step S1 is specifically as follows: The source dataset is constructed by collecting target sub-images of scarce categories from public datasets and the Internet, and the background dataset is constructed by collecting complete images without traffic lights.

[0006] Furthermore, the source data enhancement in step S1 includes the following operations: Image Scaling: Using Formulas

[0007] To scale, , , , is the image size information, and are the horizontal and vertical scaling factors, respectively; Brightness and contrast adjustment: adjust the overall brightness and contrast of the image; Image affine transformation: according to the formula

[0008] To transform, is the rotation angle, and is a parameter, s is a scaling parameter; Gaussian blur: by formula

[0009] accomplish, Determines the degree of blur.

[0010] Furthermore, the data simulation in step S1 includes: Randomly select source data sub-graphs and background data graphs, select sub-graphs containing traffic light states from the source data, and select background graphs from the background data; Matching detection: Check the matching of the source data sub-image and the background image, mainly based on size matching; Target replacement: After matching, embed the traffic light status in the source data sub-image into the background image.

[0011] Furthermore, the simulation data enhancement in step S1 includes: Random selection of source data: randomly select several groups of data from the simulation data set; Data generation permutation enhancement: by formula

[0012] Processing images and coordinate labels, where

[0013] is the data annotation box coordinate matrix, S is the scaling matrix, and T is the translation matrix; Blank Area Inpainting: Fill the uncovered areas of the image and save it.

[0014] Furthermore, in step S2, the feature encoder adopts a deep convolutional neural network structure, whose network is MobileNet, the input is the original traffic image, and the output is a multi-channel feature map. Specifically, three levels of convolutional layers (3x3 convolution kernels) are used, combined with Batch Normalization and Re LU activation functions. The number of output channels of each layer gradually increases. The number of output channels of the first convolutional layer is 32, the second layer is 64, and the third layer is 128. The size of the feature map is reduced step by step, which is achieved through maximum pooling or stride convolution.

[0015] Furthermore, in step S2, the color detection head outputs a feature map based on the feature encoder, and adopts a two-layer convolution + pooling structure. The first layer of convolution extracts features (64 channels), and the second layer of convolution (128 channels) performs deep color feature extraction. The YOLO detection strategy is used to predict the bounding boxes and category probabilities of red, yellow, and green traffic lights on the feature map. Each grid unit predicts multiple bounding boxes and corresponding confidences, as well as the probability that each bounding box belongs to the three colors of red, yellow, and green. Finally, non-maximum suppression (NMS) is performed to obtain the final color detection result.

[0016] Furthermore, the transfer learning feature encoder in step S3 includes: Load pre-trained weights: Load MobileNet weights from the Image Net pre-trained model; Freeze the weights of the first few layers: Freeze the weights of the lower convolutional layers; Fine-tune high-level convolutional layers: Set high-level convolutional layers to be trainable and fine-tune weights with a preset learning rate.

[0017] Furthermore, the joint loss optimization in step S3 includes: Defining Weighted IOU (WIOU) Loss

[0018] Among them, IOU measures the accuracy of the detection area. is the weight factor; Total loss function

[0019] is the weight parameter of each module; Optimize in the following steps: Initialize the loss term weight and calculate the loss change rate

[0020] The weights are updated accordingly, and the model is optimized through backpropagation. The cosine annealing learning rate decay strategy is also introduced.

[0021] in, is the learning rate of the current training cycle t, is the minimum learning rate, is the maximum learning rate and T is the cycle length.

[0022] Furthermore, the model test in step S3 includes: Single module test: test the color, direction, and position detection head performance respectively; Joint testing: testing the accuracy of multi-task collaboration simultaneously; Error analysis and fine-tuning: Analyze test error cases and fine-tune the optimization model accordingly. Use average intersection-over-union (IoU), recall rate, and precision rate for evaluation. Average IoU evaluates the position detection accuracy, recall rate reflects the traffic light detection coverage, and precision rate evaluates the detection accuracy.

[0023] The above-mentioned embodiments of the present invention have at least the following beneficial effects: The signal light detection method of the present invention can effectively improve the accuracy and robustness of signal light detection through data balancing and multi-task learning strategies. By collecting source data sets and background data sets, and performing data enhancement and simulation, the method can enhance the model's ability to recognize signal lights under different categories and conditions, especially when data is scarce. In addition, the design of the multi-module joint detection structure enables the model to simultaneously process the color, direction and location information of the signal light, thereby improving the comprehensiveness and accuracy of the detection.

[0024] Through transfer learning feature encoders and loss function optimization, the method of the present invention can quickly adapt to new data sets and further improve performance by fine-tuning model parameters. This method can not only reduce the time and resources required for model training, but also further optimize the detection performance of the model through cosine annealing learning rate decay strategy and error analysis and fine-tuning, making traffic light detection more accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which: Figure 1 A flowchart of a data balancing and multi-task signal light detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0027] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0028] It should be noted that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0029] Reference below Figure 1 , Figure 1 The following is a flow chart of a data balancing and multi-task signal light detection method provided by an embodiment of the present invention. Figure 1 As shown, a data balancing and multi-tasking signal light detection method S0 includes: Step S1: Data balancing: performing data collection to obtain a source data set and a background data set, performing a source data enhancement operation on the source data set, generating a simulation data set through data simulation, and performing simulation data enhancement on the simulation data set; Step S2: Traffic light detection model construction: construct a multi-module joint detection structure including a feature encoder, a color detection head, a direction detection head and a position detection head. The feature encoder extracts deep features of the image and passes them to other detection heads. The color detection head outputs the color status of the traffic light. The direction detection head identifies the direction status of the traffic light. The position detection head detects the position information of the traffic light panel. Step S3: Model training and testing: First transfer the learning feature encoder, then use the loss function to optimize the model, and finally perform single module testing, joint testing, error analysis and fine-tuning on the model.

[0030] It should be noted that this embodiment relates to a data-balanced and multi-task traffic light detection method, which first acquires a source data set and a background data set through data acquisition, performs a source data enhancement operation on the source data set, and generates a simulation data set through data simulation and implements simulation data enhancement.

[0031] In this process, the source dataset refers to the image data containing traffic lights, while the background dataset refers to the image data without traffic lights. Data enhancement refers to increasing the diversity of samples in the dataset through a series of technical means to improve the generalization ability of the model.

[0032] Specifically, data collection can be done by collecting target sub-images of scarce categories from public datasets and the Internet to construct a source dataset, while collecting complete images without traffic lights to form a background dataset. Source data enhancement operations include techniques such as image scaling, brightness and contrast adjustment, image affine transformation, and Gaussian blur. For example, image scaling can be achieved through a specific matrix transformation, which involves the size information and scaling factor of the image; brightness and contrast adjustment is to change the brightness and contrast of the image through an algorithm to simulate images under different lighting conditions; image affine transformation involves rotation, translation, and scaling of the image; and Gaussian blur is to achieve the blurring effect of the image through a specific mathematical formula, where the degree of blur is determined by parameters.

[0033] Preferably, in the data simulation step, the source data sub-image and the background data image can be randomly selected for matching detection, mainly based on size matching. In the target replacement step, after matching, the signal light state in the source data sub-image is embedded in the background image to generate a more realistic simulation data set.

[0034] Furthermore, in the simulation data enhancement step, data generation and permutation enhancement technology can be used to process images and coordinate labels through specific formulas, which involve data annotation box coordinate matrix, scaling matrix and translation matrix. In addition, the image uncovered area can be filled with blank area patching technology and then saved to further improve the quality of the simulation data set.

[0035] In some embodiments, data collection in step S1 specifically includes: The source dataset is constructed by collecting target sub-images of scarce categories from public datasets and the Internet, and the background dataset is constructed by collecting complete images without traffic lights. The complete images without traffic lights refer to complete images composed of common traffic lights but not scarce traffic lights (such as the most common green light).

[0036] It should be noted that this embodiment describes in detail the specific steps of data collection, namely, collecting target subgraphs of scarce categories from public datasets and the Internet to construct a source dataset, and collecting complete images without traffic lights to form a background dataset. Here, public datasets refer to existing image collections that can be used for research and development, while network collection refers to the process of searching and downloading relevant images through the Internet. Scarce target subgraphs refer to specific types of traffic light images that are relatively small in number in the dataset but are critical to model training.

[0037] Specifically, during the data collection process, automated scripts can be used to download traffic light images from multiple online databases and image sharing platforms. These images need to be screened to ensure that they contain traffic lights of different types and in different states. At the same time, a large number of images without traffic lights need to be collected. These images will be used as background data sets to simulate scenes without traffic light interference. When constructing the source data set, specific parameters can be set, such as image resolution, size and location of traffic lights, etc., to ensure the diversity and coverage of the data set.

[0038] Preferably, in order to improve the efficiency and quality of data collection, an image recognition algorithm can be used to assist in screening images to ensure that the collected traffic light images meet specific standards. In addition, the automatically downloaded images can be screened again through manual review to eliminate images of low quality or not meeting the requirements.

[0039] Furthermore, when constructing the background dataset, you can consider using high-resolution traffic scene images, which can be obtained from traffic monitoring cameras or downloaded from public traffic image databases. In this way, the authenticity and diversity of the background dataset can be ensured, thereby improving the generalization ability of the model in practical applications.

[0040] In some embodiments, the source data enhancement in step S1 includes the following operations: Image Scaling: Using Formulas

[0041] To scale, , , , is the image size information, and are the horizontal and vertical scaling factors, respectively; Brightness and contrast adjustment: adjust the overall brightness and contrast of the image; Image affine transformation: according to the formula

[0042] To transform, is the rotation angle, and is a parameter, s is a scaling parameter; Gaussian blur: by formula

[0043] accomplish, Determines the degree of blur.

[0044] It should be noted that this embodiment involves source data enhancement operations, including image scaling, brightness contrast adjustment, image affine transformation, and Gaussian blur. These operations are intended to increase the diversity of the data set and improve the model's ability to recognize traffic lights under different conditions. Among them, image scaling refers to changing the size of the image, while brightness contrast adjustment refers to changing the brightness and contrast of the image to simulate images under different lighting conditions. Image affine transformation includes rotation, translation, and scaling, while Gaussian blur is an image blurring technique used to simulate camera shake or motion blur.

[0045] Specifically, image scaling can be achieved by the following formula: ;

[0046] in, and are the horizontal and vertical scaling factors, and is the image size information. Brightness contrast adjustment can be achieved by adjusting the histogram of the image. The specific parameters include brightness offset and contrast coefficient. The formula for image affine transformation is: ;

[0047] in, is the rotation angle, and is the translation parameter, is the scaling parameter. The formula for Gaussian blur is:

[0048] in, Determines the degree of blur.

[0049] Preferably, the scaling factor of the image scaling can be dynamically adjusted according to the size of the traffic light in the image to keep the proportion of the traffic light unchanged. The specific parameters of the brightness contrast adjustment can be set according to the histogram distribution of the image to ensure that the image remains natural after enhancement. The parameters of the image affine transformation can be adjusted according to the needs of the actual scene. For example, the rotation angle can be set according to the angle of the traffic monitoring camera. The degree of Gaussian blur can be adjusted according to the degree of simulated camera shake to more realistically simulate the actual shooting conditions. In addition, other types of image enhancement techniques, such as histogram equalization, sharpening filtering, etc., can also be considered to further improve the diversity of the data set.

[0050] In some embodiments, the data simulation in step S1 includes: Randomly select source data sub-graphs and background data graphs, select sub-graphs containing traffic light states from the source data, and select background graphs from the background data; Matching detection: Check the matching of the source data sub-image and the background image, mainly based on size matching; Target replacement: After matching, embed the traffic light status in the source data sub-image into the background image.

[0051] It should be noted that this embodiment describes in detail the process of data simulation, including randomly selecting source data sub-graphs and background data graphs, as well as matching detection and target replacement operations. Here, data simulation refers to the use of technical means to simulate and generate data close to the actual situation to enhance the generalization ability of the model. The source data sub-graph refers to an image fragment containing a traffic light, while the background data graph refers to a complete traffic scene image without a traffic light.

[0052] Specifically, data simulation first involves randomly selecting a sub-image containing the traffic light state from the source data set, and selecting a background image from the background data set. Matching detection refers to evaluating the compatibility of the two image segments in terms of size, lighting, and texture to ensure that the synthesized image looks natural and seamless. Target replacement is to replace the traffic light state in the source data sub-image with the corresponding position in the background image. In matching detection, size matching is a key factor, which can be achieved by calculating the resolution of the two image segments and the proportion of the traffic light in the image.

[0053] Preferably, the matching detection in the data simulation process can be further refined into multiple sub-steps. For example, the lighting conditions of the source data sub-image and the background data image can be automatically detected by an image processing algorithm, and then the brightness and contrast of the two can be adjusted to achieve better matching.

[0054] Furthermore, an edge detection algorithm can be used to identify the position of the traffic lights in the background image to ensure that the traffic lights in the source data sub-image can be accurately replaced in the correct position in the background image. In the target replacement step, image fusion techniques, such as Poisson fusion or Laplace pyramid fusion, can be used to smooth the edges of the synthesized image and reduce visible seams. These techniques can improve the realism of the synthesized image, thereby improving the effect of model training.

[0055] In some embodiments, the simulation data enhancement in step S1 includes: Random selection of source data: randomly select several groups of data from the simulation data set; Data generation permutation enhancement: by formula

[0056] Processing images and coordinate labels, where

[0057] is the data annotation box coordinate matrix, S is the scaling matrix, and T is the translation matrix; Blank Area Inpainting: Fill the uncovered areas of the image and save it.

[0058] It should be noted that the present embodiment involves specific operations of simulation data enhancement, including random selection of source data, data generation arrangement enhancement, and blank area patching. Simulation data enhancement refers to processing simulation data through a series of technical means to improve the diversity and quality of the data set. Random selection of source data refers to randomly selecting data samples from a simulation data set, while data generation arrangement enhancement refers to transforming these samples to increase the diversity of the data. Blank area patching refers to processing the uncovered areas in the transformed image to ensure the integrity of the image.

[0059] Specifically, the random selection of source data can be done by writing an algorithm to randomly extract several groups of data from the simulation data set, which include images and corresponding coordinate labels. Data generation permutation enhancement refers to scaling, rotating, and translating these images and coordinate labels to generate new data samples. This process can be achieved by the following formula:

[0060] in, is the data annotation box coordinate matrix, is the translation matrix, is the scaling matrix. The specific parameters of these matrices can be set as needed. For example, the scaling matrix can be adjusted according to the size of the image and the position of the traffic light, and the translation matrix can be determined according to the relative position of the traffic light in the image.

[0061] Preferably, the data generation permutation enhancement operation can be further refined, for example, random scaling and rotation angles, as well as translation distances, can be set to simulate different traffic scenarios and signal light states. Blank area patching can be achieved through interpolation methods or deep learning techniques. For example, edge filling techniques can be used to predict and fill blank areas in images, or generative adversarial networks (GANs) can be used to generate realistic background textures. These methods can effectively improve the quality of simulation data and the generalization ability of the model. In addition, other types of image enhancement techniques, such as color jittering, noise addition, etc., can also be considered to further improve the diversity of the data set.

[0062] In some embodiments, the feature encoder in step S2 adopts a deep convolutional neural network structure, whose network is Mobile Net, the input is the original traffic image, and the output is a multi-channel feature map. Specifically, three levels of convolutional layers (3x3 convolution kernels) are used, combined with Batch Normalization and Re LU activation functions, the number of output channels of each layer gradually increases, the number of output channels of the first convolutional layer is 32, the second layer is 64, and the third layer is 128. The size of the feature map is reduced step by step, which is achieved through maximum pooling or stride convolution.

[0063] It should be noted that this embodiment describes the specific structure and function of the feature encoder, which adopts a deep convolutional neural network structure, specifically Mobile Net. Mobile Net is a lightweight deep learning model optimized for mobile and embedded devices. It reduces the model size and computational complexity by using deep separable convolutions. The function of the feature encoder is to extract deep features of the input image and pass them to other detection heads. Here, the feature encoder refers to a network structure that can extract useful feature information from the input data.

[0064] Specifically, the input of the feature encoder is the original traffic image, and the output is a multi-channel feature map. The Mobile Net model uses three levels of convolutional layers, each of which is followed by Batch Normalization and Re LU activation functions. BatchNormalization is a technique used to normalize the output of the intermediate layers of the neural network to improve training speed and performance. The Re LU activation function is a nonlinear function used to increase the nonlinear expression ability of the network. The number of channels output by each convolutional layer gradually increases, with the first convolutional layer outputting 32 channels, the second layer 64 channels, and the third layer 128 channels. The size of the feature map is reduced step by step, which is achieved through maximum pooling or strided convolution, which can reduce the spatial size of the feature map while retaining important feature information.

[0065] Preferably, the specific implementation of the feature encoder can be further refined. For example, the size and number of convolution kernels in the convolution layer can be adjusted to adapt to different image resolutions and feature extraction requirements. In addition, residual connections can be introduced to improve the training effect of deep networks. Residual connections allow signals in the network to directly skip some layers, thereby alleviating the problem of gradient disappearance. Other types of lightweight network structures, such as Shuffle Net or Efficient Net, can also be considered. These network structures are also suitable for feature extraction and have different characteristics and advantages. Through these alternatives, the most suitable feature encoder structure can be selected according to the specific application scenario and performance requirements.

[0066] In some embodiments, in step S2, the color detection head outputs a feature map based on the feature encoder, and adopts a two-layer convolution + pooling structure. The first layer of convolution extracts features (64 channels), and the second layer of convolution (128 channels) performs deep color feature extraction. The YOLO detection strategy is used to predict the bounding boxes and category probabilities of red, yellow, and green traffic lights on the feature map. Each grid unit predicts multiple bounding boxes and corresponding confidences, as well as the probability that each bounding box belongs to the three colors of red, yellow, and green. Finally, non-maximum suppression (NMS) is performed to obtain the final color detection result.

[0067] It should be noted that this embodiment describes in detail the structure and working principle of the color detection head. The color detection head uses a two-layer convolution + pooling structure to extract deep color features based on the feature map output by the feature encoder. Here, the color detection head refers to a network module specifically used to identify the color state of the traffic light, and the feature map refers to the image output by the feature encoder that contains the deep feature information of the image. YOLO's detection strategy refers to the detection method used in the You Only Look Once (YOLO) algorithm, which is a popular real-time target detection system.

[0068] Specifically, the first convolution layer of the color detection head is used to extract features (64 channels), and the second convolution layer (128 channels) performs deeper color feature extraction. In the YOLO detection strategy, each grid cell predicts multiple bounding boxes and corresponding confidences, as well as the probability that each bounding box belongs to the three colors red, yellow, and green. This means that in each grid cell of the feature map, the model predicts multiple possible bounding boxes, each of which has a confidence score indicating the probability that the bounding box contains the object, and a classification probability indicating the color of the object within the bounding box. Non-maximum suppression (NMS) is a post-processing step used to select the best bounding box and remove overlapping predictions.

[0069] Preferably, the operation steps of the color detection head can be further refined. For example, the size of the convolution kernel in the convolution layer can be adjusted to better capture the color characteristics of the signal light. The type and size of the pooling layer can also be adjusted according to actual needs to balance the resolution of the feature map and the efficiency of feature extraction. In the YOLO detection strategy, the number of bounding boxes predicted by each grid cell and the confidence threshold can be adjusted to optimize the detection performance.

[0070] Furthermore, the implementation details of non-maximum suppression (NMS), such as the IOU (intersection over union) threshold, can also be adjusted according to the specific situation to reduce false detection and improve detection accuracy. Other advanced detection strategies, such as Faster R-CNN or SSD, can also be considered, which may have different advantages in handling objects of different scales and improving detection accuracy.

[0071] In some embodiments, the transfer learning feature encoder in step S3 includes: Load pre-trained weights: Load Mobile Net weights from the Image Net pre-trained model; Freeze the weights of the first few layers: Freeze the weights of the lower convolutional layers; Fine-tune high-level convolutional layers: Set high-level convolutional layers to be trainable and fine-tune weights with a preset learning rate.

[0072] It should be noted that this embodiment involves the process of transfer learning feature encoders, including loading pre-trained weights, freezing the weights of the first few layers, and fine-tuning high-level convolutional layers. Transfer learning is a machine learning technique that allows a model to use knowledge pre-trained on a large dataset to improve performance on new tasks. Here, pre-trained weights refer to model parameters trained on large-scale datasets (such as ImageNet), and frozen weights refer to keeping the parameters of certain layers unchanged during training.

[0073] Specifically, the first step of the transfer learning feature encoder is to load the pre-trained weights, that is, load the MobileNet weights from the ImageNet pre-trained model. These weights contain features learned from a large number of images, which can help the model converge faster and improve the generalization ability of new data. Next, freeze the weights of the first few layers, which means that the parameters of these layers will remain unchanged during subsequent training because they usually capture common features of images, such as edges and textures. Finally, fine-tune the high-level convolutional layers, that is, set the high-level convolutional layers to a trainable state and use a specific learning rate to adjust the weights of these layers to suit specific tasks.

[0074] Preferably, the operation steps of the transfer learning feature encoder can be further refined. For example, you can choose to freeze the first few layers of MobileNet, which usually contain basic feature extractors such as depthwise separable convolutional layers. The specific number of layers can be determined based on the structure of the model and the task requirements. During fine-tuning, different learning rate strategies can be used, such as gradually reducing the learning rate, to ensure that the model can fine-tune parameters during training.

[0075] Furthermore, you can also consider using learning rate warm-up, that is, using a lower learning rate at the beginning of training and gradually increasing the learning rate as training progresses to help the model converge better. Alternatives include using different pre-trained models, or adjusting the number of frozen and fine-tuned layers to adapt to different data sets and task requirements.

[0076] In some embodiments, the joint loss optimization in step S3 includes: Defining Weighted IOU (WIOU) Loss

[0077] Among them, IOU measures the accuracy of the detection area. is the weight factor; Total loss function

[0078] is the weight parameter of each module; Optimize in the following steps: Initialize the loss term weight and calculate the loss change rate

[0079] The weights are updated accordingly, and the model is optimized through backpropagation. The cosine annealing learning rate decay strategy is also introduced.

[0080] in, is the learning rate of the current training cycle t, is the minimum learning rate, is the maximum learning rate and T is the cycle length.

[0081] It should be noted that this implementation describes the process of joint loss optimization, which includes defining weighted IOU (WIOU) loss and total loss function, and adopting specific optimization steps. Here, weighted IOU loss is a loss function that measures the accuracy of the detection area, which adjusts the importance of different detection areas by weighted intersection over union (IOU). The total loss function is the weighted sum of all loss terms in model training, which is used to guide the learning process of the model.

[0082] Specifically, the weighted IOU loss is defined as:

[0083] in, Measures the overlap between the detected area and the true area. is the IOU of the background area, is a weight factor used to balance the importance of foreground and background. The total loss function combines multiple loss terms, such as:

[0084] in, is the classification loss, is the bounding box regression loss, is the weight parameter of each module.

[0085] Preferably, the operation steps of joint loss optimization can be further refined. For example, when initializing the weight of the loss term, different weight values ​​can be set according to the importance of the task. For example, for the traffic light detection task, a higher IOU loss weight may be required. When calculating the loss change rate, an exponential moving average can be used to smooth the loss change and reduce fluctuations during training. In addition, a cosine annealing learning rate decay strategy can be introduced, and its formula is:

[0086] in, is the learning rate of the current cycle, is the minimum learning rate, is the maximum learning rate, is the cycle length, is the current training cycle. This learning rate decay strategy can help the model adjust parameters more finely in the later stages of training and improve the convergence performance of the model. Alternatives include using other types of learning rate decay strategies, such as step decay or linear decay, to adapt to different training dynamics.

[0087] In some embodiments, the model testing in step S3 includes: Single module test: test the color, direction, and position detection head performance respectively; Joint testing: testing the accuracy of multi-task collaboration simultaneously; Error analysis and fine-tuning: Analyze test error cases and fine-tune the optimization model accordingly. Use average intersection-over-union (IoU), recall rate, and precision rate for evaluation. Average IoU evaluates the position detection accuracy, recall rate reflects the traffic light detection coverage, and precision rate evaluates the detection accuracy.

[0088] It should be noted that this embodiment involves specific steps of model testing, including single module testing, joint testing, and error analysis and fine-tuning. Single module testing refers to evaluating the performance of color, direction, and position detection heads separately, while joint testing refers to testing the accuracy of multi-task collaboration at the same time. Error analysis and fine-tuning refers to analyzing test error cases and optimizing the model accordingly to improve detection performance.

[0089] Specifically, single-module testing can evaluate the performance of each detection head, for example, by calculating indicators such as accuracy, recall, and F1 score of each detection head. Joint testing evaluates the comprehensive performance of the entire system when all detection heads work simultaneously. Error analysis and fine-tuning involve detailed analysis of error cases in the test results, finding out where the model is deficient, and adjusting the model parameters or structure accordingly. For example, the mean intersection over union (mAP) can be used to evaluate position detection accuracy, the recall rate can reflect the coverage of traffic light detection, and the precision rate can be used to evaluate detection accuracy.

[0090] Preferably, the operational steps of the model test can be further refined. For example, in single-module testing, different thresholds can be set to evaluate the performance of the detection head, such as confidence thresholds and IOU thresholds, to determine the best operating point. In joint testing, a multi-task learning framework, such as an end-to-end training method, can be used to ensure that all detection heads can work together effectively. In the error analysis and fine-tuning stage, the confusion matrix can be used to identify which categories the model performs poorly on, and to increase the training samples of these categories or adjust the model structure in a targeted manner.

[0091] Furthermore, you can also consider using data enhancement techniques, such as image flipping and rotation, to improve the model's adaptability to different scenarios. Alternatives include using different evaluation metrics, such as the AUC value of the detection head, or using different optimization algorithms, such as Adam or RMSprop, to fine-tune the model parameters.

[0092] The above-mentioned embodiments of the present invention have the following beneficial effects: The signal light detection method described in the present invention can improve the accuracy and efficiency of signal light detection. Through the data balancing step, the method can effectively deal with the category imbalance problem in the data set and enhance the model's ability to recognize different signal light states. The multi-module joint detection structure of the feature encoder, color detection head, direction detection head and position detection head can accurately detect the color, direction and position of the signal light at the same time. This multi-task learning strategy can improve the generalization ability of the model, so that it can maintain high accuracy even in complex traffic environments.

[0093] In addition, this method can accelerate the model training process and improve the convergence speed of the model by transferring the learning feature encoder and optimizing the loss function. By defining the weighted IOU loss and the total loss function, the model can pay more attention to the accuracy of the detection area during the training process, thereby improving the accuracy of the detection. The application of the cosine annealing learning rate decay strategy can make the model more stable in the later stage of training, avoid overfitting, and further improve the generalization performance of the model. Finally, through single module testing, joint testing, error analysis and fine-tuning, the high accuracy and robustness of the model in practical applications can be ensured.

[0094] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0095] The above descriptions are only some preferred embodiments of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A data-balanced and multi-tasking signal light detection method, characterized in that: The following steps are involved: Step S1, performing data collection to obtain a source data set and a background data set, performing a source data enhancement operation on the source data set, generating a simulation data set through data simulation, and performing simulation data enhancement on the simulation data set; Step S2, constructing a multi-module joint detection structure including a feature encoder, a color detection head, a direction detection head and a position detection head, wherein the feature encoder extracts deep features of the image and transmits them to other detection heads, the color detection head outputs the color status of the signal light, the direction detection head identifies the direction status of the signal light, and the position detection head detects the position information of the signal light panel; Step S3, transfer learning feature encoder, optimize the model using loss function, perform single module testing, joint testing, error analysis and fine-tuning on the model.

2. The signal light detection method according to claim 1, characterized in that: The data collection in step S1 includes: The source dataset is constructed by collecting target sub-images from public datasets and the Internet, and the complete images without traffic lights are collected to form the background dataset.

3. The signal light detection method according to claim 1, characterized in that: The source data enhancement in step S1 includes the following operations: Using the formula ； Perform image scaling, where , , , is the image size information, and are the horizontal and vertical scaling factors, respectively; Adjust the overall brightness and contrast of the image; According to the formula ； Perform an affine transformation, is the rotation angle, and is a parameter, s is a scaling parameter; By formula Gaussian blur is performed, where Determines the degree of blur.

4. The signal light detection method according to claim 1, characterized in that: The data simulation in step S1 includes: Randomly select source data sub-graphs and background data graphs from the source data set and the background data set, select a sub-graph containing the traffic light state from the source data set, and select a background graph from the background data; Detect the matching of the source data sub-image and the background image based on the size and match them; After matching, the traffic light status in the source data sub-image is embedded in the corresponding background image.

5. The signal light detection method according to claim 1, characterized in that: The simulation data enhancement in step S1 includes: Randomly select several groups of data from the simulation data set; By formula ； Processing images and coordinate labels, where ； is the data annotation box coordinate matrix, S is the scaling matrix, and T is the translation matrix; Fill the uncovered areas of the image and save it.

6. The signal light detection method according to claim 1, characterized in that: In step S2, the feature encoder adopts a deep convolutional neural network structure, whose network is Mobile Net. The input is the original traffic image and the output is a multi-channel feature map. It uses three-level convolutional layers, combined with Batch Normalization and Re LU activation functions. The number of output channels of each layer gradually increases. The number of output channels of the first convolutional layer is 32, the second layer is 64, and the third layer is 128. The size of the feature map decreases step by step, which is achieved through maximum pooling or stride convolution.

7. The signal light detection method according to claim 1, characterized in that: In step S2, the color detection head outputs a feature map based on the feature encoder, and adopts a two-layer convolution and pooling structure. The first layer of convolution extracts features, and the second layer of convolution performs deep color feature extraction. The YOLO detection strategy is used to predict the bounding boxes and category probabilities of red, yellow, and green traffic lights on the feature map. Each grid unit predicts multiple bounding boxes and corresponding confidences, as well as the probability that each bounding box belongs to the three colors of red, yellow, and green. Finally, non-maximum suppression is performed to obtain the final color detection result.

8. The signal light detection method according to claim 1, characterized in that: The transfer learning feature encoder in step S3 includes: Load Mobile Net weights from the Image Net pre-trained model; Freeze the weights of low-level convolutional layers; Set high-level convolutional layers to be trainable and fine-tune the weights with a preset learning rate.

9. The signal light detection method according to claim 1, characterized in that: The joint loss optimization in step S3 includes: Define weighted IOU loss: ； Among them, IOU measures the accuracy of the detection area. is the weight factor; The total loss function is: ； , , are the weight parameters of each module respectively; Initialize the loss term weight and calculate the loss change rate: ； Based on this, the weights are updated, and finally the model is optimized through backpropagation, while the cosine annealing learning rate decay strategy is introduced: ； in, is the learning rate of the current training cycle t, is the minimum learning rate, is the maximum learning rate and T is the cycle length.

10. The signal light detection method according to claim 1, characterized in that: The model test in step S3 includes: Single module test, testing the color, direction, and position detection head performance respectively; Joint testing, testing the accuracy of multi-task collaborative work; Error analysis and fine-tuning: Analyze test error cases and fine-tune the optimization model accordingly. Use average intersection-over-union (IoU), recall rate, and precision rate for evaluation. Average IoU evaluates the position detection accuracy, recall rate reflects the traffic light detection coverage, and precision rate evaluates the detection accuracy.

Images