Coarse-to-fine attention network for optical signal detection and recognition

Abstract

A vehicle light signal detection and recognition method, system, and computer program product includes defining one or more regions of an image of a car using a coarse attention module to generate one or more defined regions, the image including at least one of a brake light and a signal light generated by the car, the one or more regions including an illuminated portion, removing noise from the one or more defined regions using a fine attention module to generate one or more noise-free defined regions, and identifying the at least one of the brake light and the signal light from the one or more noise-free defined regions.

Description

Background Technology

[0001] This invention generally relates to a method for detecting and recognizing vehicle light signals, and more specifically, but not limitingly, to a system, method, and computer program product for detecting and recognizing the semantics of taillight signals for autonomous driving vehicles.

[0002] Taillight detection for autonomous vehicles has garnered increasing attention due to its crucial role in assisting cars to accurately recognize the driving intentions of other vehicles and make rapid driving decisions based on detection outputs. Attention mechanisms have proven helpful in improving the discriminative power of acquisition models. Self-illuminating object detection presents challenges due to its abstract semantics and ambiguous boundaries.

[0003] Traditional techniques that utilize complex image processing and employ prior knowledge of specific datasets neglect the common properties of self-illuminating targets. Previous techniques also produce low-quality proposals because much noise affects the prediction of blurred boundaries and the extraction of semantic information.

[0004] Therefore, the accuracy of bounding box position is limited, and the predicted category, which depends on the quality of the region proposal, will also be significantly affected by obfuscated information. Thus, conventional techniques are not optimal for autonomous driving applications. Summary of the Invention

[0005] Therefore, the inventors have recognized the need in this field and have discovered a novel vehicle headlight signal detection network with two components: a coarse attention module and a fine attention module. These two sub-modules complement each other and can be embedded into various existing neural network-based object detection networks. They can also be trained end-to-end without additional supervision. Therefore, the inventors propose a coarse-to-fine attention (CFA) mechanism to address traditional challenges in dynamically locating information patterns corresponding to abstract regions. Specifically, the newly invented coarse attention module clusters features responsible for bounding box localization and category classification into an expert channel, while the fine attention module extracts patterns from noisy proposed regions.

[0006] In other words, as a result of technological improvements, this invention includes practical applications because it can systematically learn the features of self-illuminating targets (i.e., vehicle taillights) and dynamically extract information patterns from region proposals, allowing for more noise in the proposed regions. This enables fast and accurate acquisition of bounding boxes and predicted categories, which is largely impossible in conventional techniques. Furthermore, the coarse-to-fine attention module of this invention can be integrated into any two-stage detector while maintaining the end-to-end training paradigm, thus addressing the problem of prior art being limited by the specific structure or appearance of the data.

[0007] In one exemplary embodiment, the present invention provides a computer-implemented method for detecting and recognizing vehicle light signals. The method includes: using a coarse attention module to define one or more regions of an image of a vehicle to generate one or more defined regions, the image of the vehicle including at least one of brake lights and signal lights generated by vehicle signals, the one or more regions including illuminated portions; using a fine attention module to remove noise from the one or more defined regions to generate one or more noise-free defined regions; and identifying at least one of the brake lights and signal lights from the one or more noise-free defined regions.

[0008] In another exemplary embodiment, the present invention provides a computer program product including a computer-readable storage medium having program instructions contained therein, the program instructions being executable by a computer to cause the computer to: define one or more regions of an image of a vehicle using a coarse attention module to generate one or more defined regions, the image of the vehicle including at least one of brake lights and turn signals generated by vehicle signals, the one or more regions including illuminated portions; remove noise from the one or more defined regions using a fine attention module to generate one or more noise-free defined regions; and identify at least one of the brake lights and turn signals from the one or more noise-free defined regions.

[0009] In another exemplary embodiment, the present invention provides a vehicle light signal detection and recognition system, the system including a processor and a memory storing instructions for the processor to perform the following operations: using a coarse attention module to define one or more regions of an image of a vehicle to generate one or more defined regions, the image of the vehicle including at least one of brake lights and turn signals generated by vehicle signals, the one or more regions including illuminated portions; using a fine attention module to remove noise from the one or more defined regions to generate one or more noise-free defined regions; and identifying at least one of the brake lights and turn signals from the one or more noise-free defined regions.

[0010] In an alternative embodiment, the coarse attention module receives a smaller feature map from the region of interest (ROI) pooling module, and the fine attention module receives a larger feature map from the ROI pooling module.

[0011] In another alternative embodiment, the coarse attention module includes two branches for computation: an attention score branch and an expected learning score branch. In the attention score learning branch, the coarse attention module converts the smaller feature map into the original feature vector, and in the expected learning score branch, the coarse attention module computes the average of all previous features of the same category as the channel expected attention score (CE score) for each category, which is multiplied by the smaller feature map to obtain a refined feature map.

[0012] In another alternative embodiment, the fine attention module uses the larger feature map and the coarse attention score obtained from the coarse attention module to generate a fine attention map by calculating the average of the scores of all channels, and multiplies the fine attention map with the larger feature map to obtain a refined feature map-2, which is used to generate distinguishing regions for predicting bounding boxes and categories.

[0013] Further details and embodiments of the invention are described below to provide a better understanding of its contribution to the prior art. Nevertheless, the invention is not limited in its application to the details, wording, terminology, illustrations, and / or arrangements set forth in the specification or shown in the drawings. Rather, the invention is capable of being embodied in other ways than those described, and can be practiced and performed in various manners, and should not be considered limiting.

[0014] Therefore, those skilled in the art will understand that the concepts upon which this disclosure is based can be readily used as the basis for designing other structures, methods, and systems for achieving several of the objectives of the invention. It is therefore important that the claims be considered to include these equivalent constructions, provided they do not depart from the spirit and scope of the invention. Attached Figure Description

[0015] Referring to the accompanying drawings, various aspects of the invention will be better understood from the following detailed description of exemplary embodiments thereof, wherein:

[0016] Figure 1 A high-level flowchart of a traffic light detection and recognition method 100 according to an embodiment of the present invention is shown as an example;

[0017] Figures 2 to 7 The system architecture of the coarse-to-fine attention (CFA) mechanism for implementing the method 100 according to an embodiment of the present invention is described exemplarily;

[0018] Figure 8 Experimental results of the CFA mechanism according to an embodiment of the present invention are described exemplarily;

[0019] Figures 9(a) to 9(d) illustrate the tasks of detecting self-luminous objects in an exemplary manner;

[0020] Figure 10 A cloud computing node 10 according to an embodiment of the present invention is depicted;

[0021] Figure 11 A cloud computing environment 50 according to an embodiment of the present invention is described; and

[0022] Figure 12 An abstract model layer according to an embodiment of the present invention is described. Detailed Implementation

[0023] Now refer to Figures 1 to 12 In describing this invention, the same reference numerals always denote the same parts. It should be emphasized that, by convention, the various features in the figures are not necessarily drawn to scale. Instead, for clarity, the sizes of the various features may be arbitrarily enlarged or reduced.

[0024] By introducing Figure 1 The example shown, an embodiment of the vehicle light signal detection and identification method 100 according to the present invention, may include various steps of using a computing device to identify the brake lights and signal lights (i.e., taillights) of a vehicle having a coarse attention module and a fine attention module implemented in a two-level detector.

[0025] Through introduction Figure 10 The example depicted, in which one or more computers of computer system 12 according to an embodiment of the present invention may include memory 28 having instructions stored in a storage system for execution Figure 1 The steps.

[0026] Although one or more embodiments can be implemented in a cloud environment 50 (e.g., Figure 12 This invention can be implemented in a cloud environment, but it should be understood that the invention can be implemented outside of a cloud environment.

[0027] Overall reference Figure 1Object detection is a fundamental task in the computer vision community and has achieved success in many real-world scenarios, including general object detection, road object detection, and face detection. Despite progress, self-illuminating object detection remains an open research problem. Traditional techniques either use complex image processing or rely on prior knowledge of specific datasets, ignoring the common properties of these objects. In this invention, a novel attention mechanism called a coarse-to-fine attention mechanism is described to address the challenges in the self-illuminating object detection task.

[0028] Self-illuminating object detection differs from general object detection for at least the following reasons. First, there is more noise in such scenarios (e.g., the bounding box regressor will be fooled into including light noise from other cars and surrounding streetlights, which in turn causes the signal category classifier to give incorrect predictions). Second, semantic information within the bounding box is important because, for all ground truths, the general outline and appearance look the same, and semantics are key to making correct predictions. Third, the brightness and contrast of the image vary greatly, causing object boundaries to sometimes be blurred and making it difficult to distinguish separate light signals.

[0029] Figures 9(a) through 9(d) exemplarily depict illustrations of a self-illuminating object detection task. Figures 9(a) and 9(b) illustrate the contrast of the light signal and how the background varies greatly (i.e., the signal is easier to identify in 9(a) than in 9(b)). Figure 9(c) depicts the blurred boundaries of the bounding boxes characterizing semantic categories, while Figure 9(d) depicts the ambient light interference noise.

[0030] In a two-stage general object detector, the first stage generates region proposals, while the second stage predicts the exact location of the bounding box and the object category. While this pipeline has been shown to be suitable for general object detection tasks, it leads to poor performance when used to detect self-illuminating targets with significant noise, which affects the prediction of blurred boundaries and the extraction of semantic information. For example, the boundaries of objects are difficult to determine in low-light environments, and the characteristics of traffic light signals vary greatly across different scenes (e.g., in rain or at night). Therefore, the performance of a typical two-stage detector is limited. Due to environmental interference, the region proposal generator (e.g., the Region Proposal Network (RPN) in Faster R-CNN) will generate low-quality proposals. Consequently, the accuracy of the bounding box location is limited, and the predicted category, which is highly dependent on the quality of the region proposals, will also be significantly affected by the obfuscating information.

[0031] To address these limitations, one could (a) learn a better regional proposal network that generates more accurate proposal locations and prospect predictions, or (b) dynamically extract information patterns from regional proposals, allowing for more noise in the proposal regions.

[0032] In the present invention described below, method 100 and architecture follow a second efficient approach to insert attention modules for noise filtering and information delivery. An exemplary motivation is to guide detection by encouraging a set of expert extractors in the network to extract features from low-quality coarse region proposals, followed by fine attention to discover precise regions in the spatial region. With this novel idea, both of the aforementioned problems can be alleviated. On the one hand, precise bounding boxes can be predicted by considering the spatial location of certain patterns generated by the expert extractors during coarse attention, particularly for structured objects where the structure and the correlations between them need to be considered for accurate prediction. Furthermore, information patterns crucial for classifying such self-illuminating targets (e.g., the signal semantics of traffic lights) can be effectively integrated to achieve performance improvements using the attention mechanism of the present invention, named the "Coarse-to-Fine Attention Mechanism" (CFA) in this invention, and can be integrated with a two-stage detector for performance enhancement.

[0033] By utilizing the CFA module, precise sub-parts that are highly correlated with core information are detected, making bounding box location and category predictions more accurate in detection tasks that require extracting semantic information of non-rigid objects or objects from noisy proposed regions.

[0034] Specifically, coarse attention can be interpreted as a channel-dimensional attention mechanism that encourages expert channels to extract information from rough proposals. Modal feature vectors are responsible for distinguishing sub-parts of an object that lack specific semantic appearance, generating modal feature vectors to represent certain patterns in the underlying data in an unsupervised manner. Since coarse attention bridges the channel and spatial dimensions, expert feature extractors are expected to focus on specific spatial patterns. Fine attention, benefiting from the transmission of information to modal feature vectors via expert channels, is followed to predict accurate attention maps in spatial regions. In this way, the region proposal network is implicitly optimized to include desired information with backpropagation (i.e., regions representing semantic meaning and boundaries).

[0035] The coarse-to-fine attention (CFA) module is described in detail below. This module can be directly inserted into faster R-CNNs or other traditional two-stage detectors for self-illuminating object detection tasks. Mathematical proofs and intuitive insights are provided to validate the effectiveness of the CFA mechanism. Furthermore, as... Figure 8As shown, experiments conducted on the CIFAR-10 dataset were used to demonstrate the correctness of the coarse attention module and the performance of the attention mechanism on the dataset was reported, which significantly outperformed the baseline.

[0036] Figure 2 The backbone model of the present invention is provided. The input image 201 of the vehicle taillights is processed by a convolutional neural network (CNN) module 202 for feature extraction and a region proposal network (RPN) module 203 for raw region proposal.

[0037] The CFA module 301 of this invention replaces Figure 2 The pooling portion of the Region of Interest (ROI) in the 204, such as Figure 3 As shown, it is fed into the classification and localization module 205 and provides an output 206. The CFA includes two coupled attention modules: a coarse attention module 301a and a fine attention module 301b. ROI pooling is provided to both the coarse and fine attention modules. The coarse attention module clusters information features into expert channels as central feature vectors and iteratively assigns attention weights to each channel. The fine attention module utilizes information from the coarse attention module and extracts spatial attention regions from the coarse proposal. A central feature vector is generated from coarse attention, followed by fine attention. In this invention, the central feature vector and modal feature vector are used interchangeably; both refer to feature patterns responsible for bounding box localization and category classification.

[0038] Coarse attention bridges the connection between the channel dimension and the spatial dimension by considering the informational patterns underlying the data possessed by each channel. Fine attention fuses the central feature vectors to provide a spatial attention map for bounding box regression and class prediction. CFA can be inserted with any two-stage detector, and the entire network can be optimized in an end-to-end manner. Figure 3 An example of a CFA module integrated with Faster R-CNN is shown. Since fine attention is responsible for generating the spatial attention map, a larger spatial dimension is desired. This is achieved by assigning attention weights to each channel and encouraging central features to be assigned to channels with high attention weights (e.g., expert channels, which extract patterns from noise corresponding to self-illuminating objects in the coarse proposal).

[0039] For example, Figure 4The ROI pooling input is described in more detail. Two distinct feature maps, termed "bigger-feature-map" and "smaller-feature-map," are generated through ROI pooling with different kernels from the original feature map from the RPN. Since the fine attention module focuses on fine-grained information extraction, the size of the feature map used in the fine attention module is, for example, twice that in the coarse attention module (e.g., H2 = 2H3, W2 = 2W3). This invention can generate multiple CAs and FAs of ROI feature maps with different proportions. Therefore, the size of the feature map is not limited.

[0040] Then, as Figure 4 As shown, the CFA module extracts information patterns from the original region proposal to predict more accurate boundaries and categories. With the help of the CFA module, the accuracy of the vehicle headlight signal detection output will be greatly improved. As mentioned above, the CFA module, including a coarse attention module and a fine attention module, can be embedded in any two-stage detector for the detection of self-illuminating objects.

[0041] Figure 5 A coarse attention module (CA) is described, which takes a small feature map from ROI pooling as input. The CA module utilizes a set of expert feature extractors to extract representational modality vectors responsible for locating the unraveled spatial patterns. To filter out interfering noise and assign attention weights to feature learning channels, the CA module clusters informative features into expert channels as central feature vectors and iteratively assigns weights to each channel.

[0042] Since inaccurate region proposals contain confusing information, this invention aims to train a set of feature extractors responsible for the desired information. However, direct training of high-dimensional neural networks will non-selectively consider noisy patterns outside the informational sub-regions. Motivated by this, this invention discovers the desired portion within a coarse region by passing important features to certain channels and learning the modal feature vectors of each part.

[0043] Expert feature extractors are encouraged to focus on specific patterns by forcing the generation of similar modality vectors using the same category containing similar semantic information. The expected spatial location favors bounding box localization and is symmetric (i.e., iteratively). Through backpropagation, relevant distinguishable features can be passed to the clustering of specific patterns in a manner similar to a mixture model to update the modality vectors and dynamically send useful information responsible for that pattern. To extract informative features from coarsely proposed regions, the motivation is to optimize a set of expert feature extractors by assigning channels with high attention weights and to regularize informative features to cluster to the corresponding expert channels by encouraging experts to extract the central features of each class.

[0044] Each channel of the feature is position-aware, and all instances of the same class share a central feature through several perfect pattern extractors. The aforementioned property does not apply to features located in each channel, but it encourages important information to cluster into several channels and implicitly separates features from sparse representations through backpropagation. That is, the CA module adds regularization by dynamically weighting the importance of channels to model the expected modality vector for each class. Channels with high attention scores embed certain patterns while avoiding the integration of noisy features.

[0045] The CA module implements the above idea by averaging previously trained samples across specific channels to generate candidate modality features for each category in each channel. Then, importance weights for each channel are dynamically assigned by calculating similarity to hypothetical modality features of the same category and dissimilarity to other categories. This clustering regularizes discriminative features into a set of experts that share similarity with the mixture model, iteratively updating the modality vectors and weight assignments via an expectation-maximization (EM) algorithm. As training time increases, central features of important patterns can be obtained through channels with high attention scores optimized in an end-to-end manner.

[0046] like Figure 5 As shown, to learn an expert feature extractor, the central feature of each class is approximated by averaging all generated features on the same channel of the same class. The score of each channel of the current sample is needed to ensure that important features are extracted from channels with high scores.

[0047] By encouraging the sharing of similarity between the features of expert channels and the central features of the same class, while distinguishing the central features of different classes, the channel score S of sample i with class c can be calculated as follows:

[0048]

[0049] Among them, F i A is a feature of the current sample i. c By using the buffer and F i The average center feature of class C is calculated by averaging the center features, where C is the total number of classes, and 1 / C-1 controls the contribution weight of samples with different labels. The above process is similar to the EM algorithm. First, weighted channel attention scores are generated as the "expectation step" before optimization. Then, informative features are passed to channels with high scores, and the parameters of the RPN are optimized as the "maximization step." The network is optimized to fit the channel attention scores, making the training process stable and avoiding average feature buffering during testing.

[0050] The CA module consists of two branches: an attention score branch (which, as discussed later, is fed into the FA module) and an expected score learning branch.

[0051] In the attention score learning branch, the coarse attention module transforms the smaller feature map into values ​​called "raw feature vectors" through global average pooling (GAP), which are used to compute coarse attention scores (CA scores) of the feature map through two fully connected layers (FC). The CA scores then serve as input to the fine attention module.

[0052] In the expected score learning branch, with 8 categories and 512 channels, the coarse attention module calculates the average of all previous features for the same category as the channel expected attention score (CE score) for each category, which is multiplied by a smaller feature map to obtain a refined feature map-1. The refined feature map-1 is then output to the classification and localization modules, as discussed later.

[0053] Next, the CA module will demonstrate how to perform mathematical operations. Starting with global average pooling, it depicts discriminative image regions by generating class activation maps (CAMs) for classifying specific classes. More specifically, the characteristics of sample I with class c after the GAP are:

[0054]

[0055] Where d represents the index of the channel dimension, f d (x, y) represents the feature map of channel d at position (x, y). Then, the original classification output score O is calculated following the FC layer. org :

[0056]

[0057] Among them W d This represents the weight of the FC layer of class c at channel dimension d. Using the proposed coarse attention mechanism, each channel is assigned an attention weight S, as shown in equation (1), and the corresponding output score is:

[0058]

[0059] By eliminating in S d In The output fraction O can be calculated as:

[0060]

[0061] Among them, f d These are the characteristics of the current sample in channel d, and It is the central feature at channel d of category c. In the implementation of this invention, it is approximated by averaging all previously trained samples with category c at channel c. And it is stored in a buffer during each iteration. Therefore, and It is treated as a constant vector.

[0062] Therefore, optimization can be viewed as two main steps. First, the central features and attention weights are calculated, and then the parameters of the classifier are optimized. For each channel d, this invention first calculates... and These are respectively used as modal feature vectors and channel attention weights. Because the weight W d These are parameters corresponding to the same class c, so we want to maximize O. When the attention score of channel d is high, W... d F d The forecast is also high, and W d It is optimized accordingly. Therefore, during training, F is encouraged. d Modal features containing the same category are used to make the output score change with the expert channel weight W. d And it becomes larger. Thus, the central features and those responsible for generating F... d Parameters and W d It is iteratively optimized in a manner similar to the EM algorithm.

[0063] refer to Figure 6 Furthermore, to refine localization, the Fine Attention (FA) module generates a refined feature map to locate precisely discriminative regions for accurate bounding box and class prediction. A refined attention map is generated by averaging the scores across all channels using a larger feature map and a coarse attention score obtained from the previous CA module. This refined attention map is then multiplied by the larger feature map to obtain a refined feature map-2, which can be used to generate discriminative regions for bounding box and class prediction. This refined feature map-2 is then fed into the classification and localization modules, as described below.

[0064] In other words, coarse attention establishes a connection between the channel dimension and the spatial dimension by clustering distinguishable spatial features into a set of channels. However, to better understand semantics and thus more accurately predict bounding boxes and categories, a fine attention module is needed to locate the precise spatial regions responsible for semantic categories and basic fact bounding boxes. Since this invention has already obtained several experts (i.e., via the CA module) focusing on specific central features of interest, it can generate a spatial attention mask by weighted summing of the spatial regions focused on by each expert, thus giving attention to sub-regions containing explanations of semantic categories and bounding box locations. In fact, the fine attention module generates a spatial attention map to locate the distinguishable portions responsible for predicting bounding boxes and semantic categories.

[0065] As mentioned above, the architecture of the fine attention module is in Figure 6As shown in the diagram, the input feature map is generated from the RPN through ROI pooling. Then, the predicted channel attention scores obtained from the previous CA module are used to generate a spatial attention mask, and the features are refined accordingly. That is:

[0066]

[0067] Where f(x, y) is the input feature, d is the channel index, and S d σ is the channel attention score at channel d generated by the coarse attention module, and σ is the sigmoid activation function.

[0068] like Figure 7 As shown, in the classification and localization module 205, two refined feature maps from the coarse attention module and the fine attention module are converted into values ​​by the fully connected layer, which can be used to calculate the coordinates of the bounding box and the category of the object in the bounding box.

[0069] The total loss for the self-illuminating object detection task is a weighted sum of classification and regression losses. For the classification loss, a multi-task loss function and a non-maximum suppression (NMS) method are used to calculate the classification loss. For the regression loss, a residual fitting method is used to calculate the regression loss. Specifically, regression learning is used to calculate the deviation between the actual values ​​of the bounding boxes (actual values) and the ground truth values ​​of the bounding boxes (ground truths).

[0070] Turning Figure 1 , Figure 1 An exemplary method flow of the coarse-to-fine attention mechanism of the present invention for a self-illuminating object detection task is described. CFA is designed to address the challenges of representing objects, including bounding boxes and semantic meaning, mitigating problems caused by noise in the coarse proposal region (e.g., if surrounding streetlights are also included in the proposal region, they will cause confusion).

[0071] In step 101, the present invention includes receiving, by a computing device, an image of a vehicle including brake lights and / or signal lights generated from vehicle signals (i.e., taillights including brake lights and signal lights).

[0072] In step 102, the present invention includes using a computing device to define one or more regions of an image, including the illuminated portion, using a coarse attention module to generate one or more defined regions.

[0073] In step 103, the invention includes removing noise from one or more bounded regions using a fine attention module via a computing device to generate one or more noise-free bounded regions. The output from the coarse attention module is used in the fine attention module.

[0074] In step 104, the present invention includes identifying brake lights and / or signal lights from one or more noise-free defined areas using a computing device.

[0075] Therefore, the present invention includes: a coarse attention module for filtering interfering noise in the environment and assigning attention weights to each feature learning channel, which is beneficial for subsequent bounding box and category prediction; a fine attention module for locating precisely discriminative regions for accurate prediction of bounding boxes and categories; and an end-to-end learning network that combines a classic object detection network with novel coarse and fine attention modules for self-illuminating object detection.

[0076] In fact, such as Figure 8 As shown, due to the challenges of locating light signals and extracting semantic information, the inventors have adopted the Vehicle Light Signals (VLS) dataset for results analysis. The VLS dataset contains four common vehicle behaviors: moving forward, braking, turning left, and turning right. Each behavior signal is divided into two cases: daytime and nighttime, because the lighting signals differ between day and night. Data was collected from a dashcam by uniformly sampling 15 frames from a 15-minute video. The VLS dataset includes 7720 images, 8 categories, and 10571 instances. During the day and night, bounding boxes are classified by moving forward, braking, turning left, and turning right, respectively. In the experiments, 60% of the samples were randomly selected as training data, 20% as validation, and 20% as test data; this ratio can be fixed for different models in other experiments.

[0077] For example, the models were implemented using the deep learning framework Caffe. These models were trained on an NVIDIA GTX 1080Ti. For all models, the initial learning rate was set to 10. -3 Momentum was set to 0.9, and weight decay was 5 × 10⁻⁶. -4 In this task, the system takes a raw image as input and outputs a classification result containing the vehicle's location and the headlight signals in that image. Therefore, the mean accuracy (AP) for each class and the mean average accuracy (MAP) for object detection are used as performance evaluation metrics.

[0078] Most existing methods used in optical signal detection tasks employ general object detectors. Previous work first evaluated the performance of a faster R-CNN for a popular two-stage general object detector with different backbones, and then evaluated the performance when the coarse-to-fine attention mechanism of this invention is integrated. Results are as follows... Figure 8 As shown, for the same backbone network, the detector outperforms the original model when trained with CFA, especially when the backbone network performs poorly. In other words, the CFA module of this invention described herein has the ability to extract informative features from low-quality proposals with more noise or interference.

[0079] In an exemplary aspect, using a cloud computing environment

[0080] While this detailed description includes exemplary embodiments of the invention in a cloud computing environment, it should be understood that implementation of the teachings described herein is not limited to such a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0081] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing power, storage devices, applications, virtual machines, and services) with minimal management effort or interaction with service providers, allowing for the rapid provisioning and release of these resources. This cloud model can include at least five features, at least three service models, and at least four deployment models.

[0082] The features are as follows:

[0083] On-demand self-service: Cloud consumers can automatically and unilaterally allocate computing power, such as server time and network storage, as needed, without having to interact manually with service providers.

[0084] Extensive network access: Functionality is available via the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0085] Resource pooling: Leveraging a multi-tenant model, provider computing resources are pooled to serve multiple consumers, dynamically allocating and reallocating different physical and virtual resources based on demand. There is a sense of location independence because consumers typically cannot control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0086] Rapid and flexible: Features can be configured quickly and flexibly, and in some cases can be automatically configured to scale out quickly and released quickly to scale in. For consumers, the available configuration options appear unlimited, and any quantity can be purchased at any time.

[0087] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a certain level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0088] The service model is as follows:

[0089] Software as a Service (SaaS): The functionality provided to consumers is the use of applications from a provider running on cloud infrastructure. These applications can be accessed from various client circuits via thin client interfaces, such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionality, with the exception of potentially limited user-specific application configuration settings.

[0090] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or purchased by the consumer onto cloud infrastructure. These applications are created using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage, but they can control the deployed applications and, if any, the configuration of the application hosting environment.

[0091] Infrastructure as a Service (IaaS): This provides consumers with processing, storage, networking, and other basic computing resources on which consumers can deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they have control over the operating system, storage, deployed applications, and may have limited control over selected network components (such as host firewalls).

[0092] The deployment model is as follows:

[0093] Private cloud: The cloud infrastructure is operated by only one organization. It may be managed by the organization or a third party, and it may exist internally or externally.

[0094] Community cloud: A cloud infrastructure shared by multiple organizations that supports a specific community with common concerns (e.g., mission, security needs, policy and regulatory compliance considerations). It may be managed by an organization or a third party and may exist internally or externally.

[0095] Public cloud: Cloud infrastructure that is open to the public or large industry organizations and is owned by organizations that sell cloud services.

[0096] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies to enable data and application portability (e.g., cloud bursts to achieve load balancing between clouds).

[0097] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprised of interconnected networks of nodes.

[0098] Now for reference Figure 10The diagram illustrates an example of a cloud computing node. Cloud computing node 10 is merely an example of a suitable node and is not intended to impose any limitation on the scope or functionality of the embodiments of the invention described herein. In any case, cloud computing node 10 can be implemented and / or perform any of the functions set forth herein.

[0099] Although cloud computing node 10 is described as computer system / server 12, it is understood to be capable of operating with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop circuits, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or circuits.

[0100] The computer system / server 12 can be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system / server 12 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing circuitry linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage circuitry.

[0101] Now for reference Figure 10 The computer system / server 12 is shown in the form of a general-purpose computing circuit. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples various system components, including system memory 28, to the processor 16.

[0102] Bus 18 represents any one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0103] Computer system / server 12 typically includes various computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and non-volatile media, removable and non-removable media.

[0104] System memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, and generally referred to as "hard disk drives"). Although not shown, disk drives for reading from and writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In this case, each may be connected to bus 18 via one or more data media interfaces. As will be further described below, memory 28 may include a computer program product storing one or more program modules 42, which include computer-readable instructions configured to perform one or more features of the present invention.

[0105] By way of example and not limitation, a program / utility 40 having a set (at least one) of program modules 42, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 28. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may be adapted to be implemented in a networked environment. In some embodiments, program modules 42 are generally adapted to perform one or more functions and / or methods of the present invention.

[0106] The computer system / server 12 can also communicate with one or more external devices 14, such as a keyboard, indicator circuitry, other peripherals, such as a display 24, etc., and one or more components that facilitate interaction with the computer system / server 12. This communication can be achieved through input / output (I / O) interface 22 and / or any circuitry (e.g., network interface card, modem, etc.). This enables the computer system / server 12 to communicate with one or more other computing circuits. For example, the computer system / server 12 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via network adapter 20. As shown, network adapter 20 communicates with other components of the computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with the computer system / server 12. Examples include, but are not limited to: microcode, circuit drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0107] Now for reference Figure 11 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, and local computing circuits used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N, can communicate with the cloud computing nodes 10. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds as described above, or combinations thereof. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, for which cloud consumers do not need to maintain resources on their local computing circuits. It should be understood that... Figure 11 The types of computing circuits 54A-N shown are for illustrative purposes only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized circuit via any type of network and / or network-addressable connection (e.g., using a web browser).

[0108] Now for reference Figure 12 This demonstrates a cloud computing environment of 50 ( Figure 11 This provides a set of exemplary functional abstraction layers. It should be understood beforehand that... Figure 12 The components, layers, and functions shown are illustrative only, and embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0109] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; storage circuitry 65; and networking and interconnection components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0110] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0111] In one example, management layer 80 can provide the following functions: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks in the cloud computing environment. Metering and Pricing 82 provides cost tracking when resources are used in the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks and protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to meet required service levels. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-planning and procurement of cloud resources based on the anticipated future demand for cloud resources according to the SLA.

[0112] Workload layer 90 provides examples of functionalities that can be utilized in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis and processing 94; transaction processing 95; and the vehicle light signal detection and recognition method 100 according to the present invention.

[0113] This invention can be any possible system, method, and / or computer program product at any level of integration technical detail. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0114] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punched cards or raised structures in recesses on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being a transient signal, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0115] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded via a network to an external computer or external storage device, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0116] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute the computer-readable program instructions by utilizing the status information of the computer-readable program instructions to personalize the electronic circuitry and thereby perform aspects of the invention.

[0117] This document describes various aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of manufacture comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0119] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device, thereby producing a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions marked in the blocks may occur in a non-linear order. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0121] The description of various embodiments of the present invention is given for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0122] Furthermore, the applicant's intent is to include equivalents of all claim elements, and any modification to any claim of this application shall not be construed as a waiver of any interest or right to equivalents of any element or feature of the modified claim.

Claims

1. A computer-implemented method for detecting and recognizing vehicle headlight signals, the method comprising: A coarse attention module is used to define one or more regions of an image of a vehicle to generate one or more defined regions, the image including at least one of brake lights and turn signals generated by vehicle signals, the one or more regions including illuminated portions, wherein the coarse attention module processes smaller feature maps to output a refined feature map-1 and a coarse attention score, i.e., a CA score; The fine attention module removes noise from the one or more bounded regions to generate one or more noise-free bounded regions, wherein the fine attention module processes the larger feature map and the coarse attention score to output a refined feature map-2; and Identify at least one of the brake lights and the signal lights from the one or more noise-free defined regions, wherein the refined feature map-1 and the refined feature map-2 are processed by a classification and localization module to calculate the coordinates of the bounding box and the category of the object in the bounding box.

2. The method according to claim 1, wherein, The coarse attention module and the fine attention module use neural networks.

3. The method according to claim 1, wherein, The coarse attention module encourages a set of expert extractors in the network to extract features from low-quality coarse region proposals, and then the fine attention module follows fine attention to guide detection to discover accurate regions in the spatial region.

4. The method according to claim 3, wherein, The bounding box is predicted based on the spatial location of a specific pattern generated by the expert extractor through the coarse attention module.

5. The method according to claim 1, wherein, The coarse attention module performs coarse attention in the channel dimension, which encourages expert channels to extract information from the coarse proposals. The modal feature vectors are generated by the coarse attention module to represent certain patterns in the underlying data in an unsupervised manner.

6. The method according to claim 1, wherein, The coarse-to-fine attention CFA module includes the coarse attention module and the fine attention module.

7. The method according to claim 1, wherein, The coarse attention module clusters information features into expert channels as central feature vectors and iteratively assigns attention weights to each channel. The fine attention module utilizes information from the coarse attention module and extracts the spatial attention region from the coarse proposal.

8. The method according to claim 1, wherein, The coarse attention module receives smaller feature maps from the Region of Interest (ROI) pooling module, and The fine attention module receives a larger feature map from the ROI pooling module.

9. The method according to claim 8, wherein, The larger feature map is twice the size of the smaller feature map.

10. The method according to claim 8, wherein, The coarse attention module extracts the expression mode vector from the smaller feature map, and The coarse attention module filters out interference noise and assigns attention weights to the feature learning channels by clustering information features into the expert channel as the central feature vector, and iteratively assigns weights to each channel.

11. The method according to claim 8, wherein, The coarse attention module includes two branches for computation: an attention score learning branch and an expected score learning branch. In the attention score learning branch, the coarse attention module converts the smaller feature map into the original feature vector, and In the expected score learning branch, the coarse attention module calculates the average of all previous features of the same category as the channel expected attention score, i.e., CE score, for each category, and multiplies it by the smaller feature map to obtain the refined feature map-1.

12. The method according to claim 8, wherein, The fine attention module generates refined feature maps to locate precise distinguishing regions for predicting bounding boxes and categories.

13. The method according to claim 11, wherein, The fine attention module uses the larger feature map and the coarse attention score obtained from the coarse attention module to generate a fine attention map by calculating the average of the scores across all channels. The fine attention map is multiplied by the larger feature map to obtain a refined feature map-2, which is used to generate distinguishing regions for predicting bounding boxes and categories.

14. The method of claim 13, further comprising a classification and localization module, the classification and localization module receiving the refined feature map-1 from the coarse attention module and the refined feature map-2 from the fine attention module, and converting the refined feature map-1 and the refined feature map-2 into values ​​for calculating the coordinates of the bounding box and the category of the object in the bounding box through a fully connected layer.

15. The method according to claim 1, embodied in a cloud computing environment.

16. A computer program product comprising a computer-readable storage medium having program instructions executable by a computer to cause the computer to perform the method of any one of claims 1-15.

17. A vehicle headlight signal detection and recognition system, the system comprising: processor; as well as A memory that stores instructions that cause the processor to perform the method described in any one of claims 1-15.

Images