Apparatus and method for soft attention based saliency map generation for object detection

Abstract

The present invention describes an apparatus and method for detecting objects present in an image. The method states receiving an input image (602) including at least one object to be detected, and generating a soft attention image by processing the received image using a soft attention technique (604). The method further states processing the received image (606) with a first CNN model to extract a first feature map, and processing the soft attention image (608) with a second CNN model to extract a second feature map. The method further states integrating the first and second feature maps to generate a stacked feature map (610), processing the stacked feature map (612) to generate a saliency map, and detecting at least one object based on the saliency map (614).

Description

[Technical field]

[0001] The subject matter described herein generally discloses an apparatus and method for detecting objects present within an image. In other words, the present application discloses techniques for soft-attention based saliency map generation for object detection. [Background technology]

[0002] There are many algorithms for object detection in images, such as Region-based Convolutional Neural Networks (R-CNN), Fast R-CNN, Faster R-CNN, YOLOv2, and Single Shot Detector (SSD). These state-of-the-art algorithms search the entire image for the object of interest. These algorithms are computationally inefficient and biologically impossible. This leads to lower evaluation metric performance and longer training periods.

[0003] In real life, when humans search for a particular object in an image, rapid eye movements occur between fixation points, called saccades, before arriving at the target object. Eye movement trajectories vary depending on the human observer's object of interest. Each fixation point indicates an area of ​​the image to which the human "attended." Human visual attention is assumed to move along with eye movements. However, existing techniques tend to process the entire input frame instead of focusing / searching for discriminatory features of the input frame.

[0004] Therefore, there is a need for an attention-based model for localizing objects in images. Summary of the Invention [Means for solving the problem]

[0005] The present disclosure overcomes one or more shortcomings of the prior art and provides additional advantages as described throughout the present disclosure. Further features and advantages are realized through the techniques of the present disclosure. Other embodiments and aspects of the present disclosure are described in detail herein and are considered a part of the claimed disclosure.

[0006] In one non-limiting embodiment of the present disclosure, a method for detecting an object present in an image is disclosed. The method includes receiving an input image including at least one object to be detected, and generating a soft attention image by modifying the brightness of the received image using a soft attention technique. The method further states that the received input image is processed with a first CNN model to extract a first feature map, and the soft attention image is processed with a second CNN model to extract a second feature map. The method further states that the first and second feature maps are integrated to generate a stacked feature map, the stacked feature map is processed to generate a saliency map, and at least one object is detected based on the saliency map.

[0007] In another non-limiting embodiment, the present disclosure states that generating a soft attention image includes multiplying a brightness channel of a received input image with a multivariate Gaussian function, such that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0008] In another non-limiting embodiment, the present disclosure states that the soft attention image is generated by converting the received input image into an HSV image including a luminance channel, multiplying the luminance channel of the HSV image by a multivariate Gaussian function such that the luminance of the HSV image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function, and after modifying the luminance of the HSV image, converting the modified HSV image into an RGB image to generate the soft attention image.

[0009] In another non-limiting embodiment, the present disclosure states that detecting at least one object based on a saliency map includes detecting a highest order activity of the saliency map, the highest order activity corresponding to a position of the object.

[0010] In yet another non-limiting embodiment, the present disclosure states that detecting the highest order activity in the saliency map includes creating multiple contours in the saliency map, calculating the density of each contour by calculating the average pixel value inside the contour, and the contour with the highest density is selected as the location of the object.

[0011] In yet another non-limiting embodiment, the present disclosure states that the first and second feature maps are generated using one or more feature detectors using convolution operations on the received image and the soft attention image, respectively, and the first and second feature maps are sets of matrices, the sets of matrices including pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image.

[0012] In yet another non-limiting embodiment, the present disclosure states that the stacked feature map includes a combined matrix obtained from a collection of matrices of the first and second feature maps, the combined matrix includes combined pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image. The saliency map includes a saliency matrix obtained from the combined matrix, the saliency matrix includes salient pixel values ​​that are regions within the received input image that indicate at least one object to be detected in the received input image.

[0013] In yet another non-limiting embodiment, the present disclosure discloses an apparatus for detecting an object present in an image. The apparatus comprises an input / output (I / O) interface configured to receive an input image including at least one object to be detected. The apparatus further comprises a soft attention image generation unit operatively coupled to the I / O interface. The soft attention generation unit is configured to generate a soft attention image by modifying a brightness of the received image using a soft attention technique. The apparatus further comprises an attention search unit operatively coupled to the I / O interface and the soft attention image generation unit. The attention search unit is configured to process the received image using a first CNN model to extract a first feature map, process the soft attention image using a second CNN model to extract a second feature map, merge the first and second feature maps using a merge model to generate a stacked feature map, and process the stacked feature map using a map generation model to generate a saliency map. The apparatus further comprises a processing unit operatively coupled to the I / O interface, the soft attention image generation unit, and the attention search unit. The processing unit is configured to detect at least one object based on the saliency map.

[0014] In yet another non-limiting embodiment, the present disclosure states that to generate a soft attention image, the soft attention image generation unit is configured to multiply the brightness channel of the received input image by a multivariate Gaussian function, such that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0015] In yet another non-limiting embodiment, the present disclosure states that to generate a soft attention image, the soft attention image generation unit is configured to convert the received input image into an HSV image including a luminance channel, multiply the luminance channel of the HSV image by a multivariate Gaussian function, such that the luminance of the HSV image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function, and after modifying the luminance of the HSV image, convert the modified HSV image into an RGB image to generate the soft attention image.

[0016] In yet another non-limiting embodiment of the present disclosure, to detect at least one object based on the saliency map, the processing unit is configured to detect a highest order activity in the saliency map, where the highest order activity corresponds to a position of the object.

[0017] In yet another non-limiting embodiment of the present disclosure, to detect the highest order activity in the saliency map, the processing unit is configured to create multiple contours in the saliency map and calculate the density of each contour by calculating the average pixel value inside the contour, and the contour with the highest density is selected as the location of the object.

[0018] In yet another non-limiting embodiment of the present disclosure, the present application discloses that the first and second feature maps are generated using one or more feature detectors using a convolution operation on the received input image and the soft attention image, respectively. The first and second feature maps are a set of matrices, and the set of matrices includes pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image.

[0019] In yet another non-limiting embodiment, the present disclosure states that the stacked feature map includes an integrated matrix obtained from a collection of matrices of the first and second feature maps, the integrated matrix including integrated pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image, and the saliency map includes a saliency matrix obtained from the integrated matrix, the saliency matrix including salient pixel values ​​that are regions within the received input image that are indicative of at least one object to be detected in the received input image.

[0020] The above summary is illustrative only and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments and features described above, further aspects, embodiments and features will become apparent by reference to the drawings and the following detailed description.

[0021] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate several exemplary embodiments and, together with the following description, serve to explain the disclosed embodiments. In the drawings, the leftmost digit(s) of a reference number identifies the figure in which that reference number first appears. The same numbers are used throughout the figures to refer to like features and elements. Some embodiments of apparatus and / or methods according to embodiments of the present subject matter will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0022] [Figure 1] 1 illustrates an apparatus for detecting an object present in an image according to an embodiment of the present disclosure. [Figure 2a] 13A-13D illustrate exemplary RGB and soft attention images, respectively, according to an embodiment of the present disclosure. [Figure 2b] 13A-13D illustrate exemplary RGB and soft attention images, respectively, according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates an architecture for generating a saliency map, according to an embodiment of the present disclosure. [Figure 4a]4 shows performance measurements of the proposed device according to an embodiment of the present subject matter. [Figure 4b] 4 shows performance measurements of the proposed device according to an embodiment of the present subject matter. [Figure 5a] 1 shows detection results of the proposed device and the prior art according to an embodiment of the present subject matter. [Figure 5b] 1 shows detection results of the proposed device and the prior art according to an embodiment of the present subject matter. [Figure 6] 1 shows a flowchart of a method for detecting an object present in an image, according to an embodiment of the present subject matter.

[0023] It should be understood by those skilled in the art that the block diagrams herein represent conceptual views of exemplary apparatus embodying the principles of the present subject matter. Similarly, it should be understood that flow charts, flow diagrams, state transition diagrams, pseudocode, and the like are substantially represented in a computer-readable medium and represent various processes that may be executed by such a computer or processor, whether or not a computer or processor is explicitly depicted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0025] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are described in detail below, however, it is to be understood that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0026] The terms "comprises," "comprising," "include(s)," or any other variation thereof, are intended to mean a non-exclusive inclusion, and an apparatus, device, or method that includes a list of elements or steps does not include only those elements or steps, but may include other elements or steps that are not expressly listed or that are inherent to such apparatus or device or method. In other words, one or more elements of a device or apparatus following "comprising" does not exclude the presence of other or additional elements in the device or apparatus, unless there are further constraints.

[0027] In the following detailed description of the embodiments of the present disclosure, reference is made to the accompanying drawings which form a part hereof and which show, by way of illustration, specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, it being understood that other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. Thus, the following description should not be construed in a limiting sense.

[0028] The present invention will be described in detail in the remainder of this specification with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail, so as to obscure the description with unnecessary details.

[0029] There are various approaches to detect objects present in an image, such as Region-based Convolutional Neural Network (R-CNN), Fast R-CNN, Faster R-CNN, YOLOv2, and Single Shot Detector (SSD). In R-CNN, the network receives an input of about 2000 candidate object regions from an image. The region proposals are generated by a selective search algorithm. Each region proposal is scaled to the size of the CNN's input. After scaling, each region proposal is fed into the CNN to extract features. The extracted features are then used to classify and predict bounding box coordinates using linear models and regression of support vector machines.

[0030] Furthermore, in Faster R-CNN technique, region proposals are generated from the selective search algorithm as done in R-CNN technique. These region proposals are fed into a Visual Geometry Group (VGG) convolutional network. Since each pixel in the output of the CNN network corresponds to a patch (region of interest) in the input image, the regions of interest in the input image (RoI) are projected with different sizes in the fifth convolutional layer output. Each RoI is scaled to a 7x7 square. Max pooling is performed on each of these RoIs to obtain a 49-dimensional vector of RoIs. Each of these 49-dimensional vectors is connected to a fully connected layer that gives a representation of this region proposal. The fully connected layer goes through a softmax activation to compute a probability distribution over the possible object classes. Along with this, a recurrent layer is used to predict new bounding boxes.

[0031] Furthermore, in Faster R-CNN, region proposals are generated using a CNN-based approach instead of a selective search algorithm. A full image is given as input to VGG-Net. The last layer of VGG-Net has an output of dimensions (h×w×512). A 3×3 kernel is applied to each of all (h×w) cells of the 512 feature maps, which results in a 512-dimensional vector of cells. We assume that we are given a number of bounding boxes (called anchor boxes), denoted by "k", of different sizes and aspect ratios, and a 512-dimensional vector for each location. The network then predicts the probability that a given anchor box centered at this location contains a possible object class. Each cell at the output of the last convolutional layer corresponds to a patch of the original image (due to the region of influence at the input of each pixel of the output of the last convolutional layer). Each of these anchor boxes is at the center of the corresponding patch. The classifier outputs 1 for anchor boxes that have a reasonable overlap (intersection on union greater than 0.7) with the true bounding box. Similarly, the regression model predicts the true bounding box from the anchor boxes. Once the region proposals are generated in this way, the rest is the same as in Fast R-CNN.

[0032] YOLO (You Only Look Once) uses an end-to-end training deep neural network for object detection. The network takes as input a full image and its corresponding labels to predict the location of objects in new images. Before training, a label for each image is created by creating (S x S) grids across the image. Each grid is assigned two random bounding boxes for their coordinates and the class of the object. If the center of the object falls within a grid, then that grid serves to detect the object. YOLO is a real-time object detector due to its fast speed.

[0033] However, all these object detection algorithms lack the human attention mechanism for detecting objects in an image and do not impart the logic of the attention mechanism used by the human brain, and in these techniques, the complete network is not amenable to end-to-end training.

[0034] The present disclosure discloses a hybrid model for object detection while addressing the above-mentioned problem statement. The hybrid model relies on giving attention to important discriminative features of the input frame and therefore does not focus on other irrelevant features in the input frame, just like the human brain functions. The hybrid model uses "attention search" techniques and focuses primarily on mimicking the function of the brain, thereby focusing only on specific features whose relevance to the output is most important and therefore not on features that do not affect the output. In this way, the hybrid model saves a lot of computational cost by only looking at the features required for the final output. The hybrid model utilizes Temporal Difference Error (TDE) as a primary learning rule to achieve optimal object detection. This mechanism also reduces the overall training time.

[0035] Referring to FIG. 1, an apparatus 100 for detecting an object in an image according to an embodiment of the present disclosure is shown. In an embodiment, the apparatus 100 may include various entities, such as, but not limited to, an attention search unit 102, an input / output (I / O) interface 112, a processing unit 114, and a soft attention image generation unit 120. These entities are communicatively and operatively coupled to each other via a wired or wireless network. In an embodiment, the apparatus 100 may use an attention search-based deep Q neural network for object localization. In another embodiment, the attention search unit 102 comprises a first convolutional neural network (CNN) model 104, a second CNN model 106, a map generation model 108, and an integrated model 110. In an embodiment, the CNN models 104, 106 are a kind of neural network model that extracts a higher-level representation of image content. Unlike traditional image recognition, where a user must define image features by himself, CNN takes raw pixel data of an image and automatically extracts features. In yet another embodiment, the processing unit 114 may comprise one or more processors 116 and a memory 118 .

[0036] The I / O interface 112 may receive an image including at least one object to be detected by the device 100. The received image may be stored in a database (not shown in FIG. 1). In an embodiment, the image may be acquired by any imaging device, which may be part of the device 100 or any external imaging device. The received image is an RGB image. An RGB image, also called a true color image, may be stored as an M*N*I data array that defines red, green, and blue color components for each individual pixel. An exemplary RGB image is shown in FIG. 2a.

[0037] The soft attention image generation unit 120 may modify the brightness of the received input image to generate the soft attention image. In an embodiment, to generate the soft attention image, the image generation unit 120 may convert the received RGB image into an HSV image including a brightness channel. HSV stands for "hue, saturation, value". HSV is used to measure the hue angle, saturation (or amount of grayness) and brightness (or value) of a color. The image generation unit 120 may modify the brightness of the HSV image by multiplying the brightness channel of the received input image by a multivariate Gaussian f(y,x) (shown in the following equation). The brightness of the HSV image is determined by the center (y 1 ,x 1 ) has high brightness, and the center (y 1 ,x 1 ) to gradually decrease in brightness as it moves away from the center. The location of the center is considered random. In an embodiment, the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. After modifying the brightness of the HSV image, the image is converted back to an RGB image, as shown in FIG. 2b. The generated RGB image is a soft attention image that can be stored in a database.

number

[0038] Also, the first CNN model 104 of the attention search unit 102 takes the received image (original image) as input, and the second CNN model 106 takes the generated soft attention image as input. The received image and the soft attention image are processed in parallel by the first and second CNN models by passing through six convolution layers to extract features. The first CNN model 104 may process the received original image to generate a first feature map, and the second CNN model 106 may process the generated soft attention image to generate a second feature map. In an embodiment, the feature map is generated by applying a filter or feature detector to the input image or the feature map of a previous input layer in the CNN using a convolution operation. The feature map is a collection of matrices, and the value of a pixel is calculated based on a high-level understanding of the presented input image corresponding to its color, texture, object shape, and brightness, etc. The features may include, but are not limited to, at least hue, saturation, brightness, and color.

[0039] The output feature map is passed to the integration model 110, which stacks the first and second feature maps to output a stacked feature map. In an embodiment, the integration model 110 may include a convolution layer that takes both feature maps (first and second features) as input and outputs a stacked map with dimensions 39×63×2. The stacked feature map is processed by the map generation model 108 to generate a saliency map. In an embodiment, the map generation model 108 may be a deep Q-learning model that may include one or more convolution layers. The deep Q-learning model may process the stacked map to generate a saliency map with dimensions 39×63×1, as shown in FIG. 3. In an embodiment, the stacked feature map includes an integration matrix obtained from a set of matrices of the first and second feature maps, the integration matrix including integrated pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image. In another embodiment, the saliency map includes a saliency matrix obtained from the integration matrix, and the saliency matrix includes salient pixel values ​​that are regions within the received input image that indicate at least one object to be detected in the received input image. Figure 3 illustrates an architecture for generating a saliency map according to an embodiment of the present disclosure.

[0040] The deep Q-learning model of the apparatus 100 can be trained to generate a saliency map using a deep Q-learning process. For each input image of the dataset for training, a reward matrix with dimensions 640×1024×1 is created. In the reward matrix, pixel coordinates where a traffic sign (object) overlaps with a soft attention are given a value of “1”, pixel coordinates where only a traffic sign is present are given a value of “0.5”, and elsewhere a value of “0”. The reward matrix R t+1 is resized to 39×63×1 to match the dimensions of the saliency map output from the network.

number

[0041] where (x,y) are the pixel coordinates of the input image. The model is trained on frames of RGB images. The dimensions of both inputs are 640x1024x3. To train the network, the Temporal Difference Error (TDE) between the saliency maps of two consecutive frames at time 't' and 't+1' is calculated using Equation 2 shown below.

number

[0042] In the above equation, 'y' is the discount factor. The time-difference error is summed with a regularization cost using the L2-norm of the weights. The total cost of a batch of 32 image frames is then minimized by using a backpropagation algorithm with an optimizer to update the network weights. The hyperparameters such as learning rate, sigma, gamma, and regularization factor beta are fine-tuned to values ​​of 0.00015, 25, 0.4, and 0.00001, respectively, to reach the best result with the minimum total cost. The model is trained for 54 epochs.

[0043] During testing, a frame containing a traffic sign is given to the device. The traffic sign is the object to be detected. The device outputs a saliency map of 39x63x1 dimensions, which is resized to 640x1024x1 dimensions to match the height and width of the input image. The output saliency map has values ​​ranging from 0 to 1. The highest order activity in the saliency map corresponds to the detected location of the traffic sign in the given image. Additionally, one or more contours are created in the saliency map. In an embodiment, the OpenCV library is used to create the contours in the saliency map. The density of each contour is calculated as the average pixel value inside the contour. The densest contour is selected as the traffic sign detection. In an embodiment, OpenCV basically finds locations with pixel values ​​close to 1 and creates a peak at all such high values ​​by collecting pixel values ​​around them. In this way, there are multiple peaks or contours in the map. Then, an average pixel value is calculated for each such contour. After calculating the average pixel value for each contour, one of the contours has the highest average pixel value that is considered to be the location of the target object.

[0044] Referring again to Figure 1, after generating the saliency map, the saliency map is processed by processing unit 114 to detect at least one object. To detect the object, processing unit 114 may create multiple contours on the saliency map and calculate the density of each contour by calculating the average pixel value inside the contour. The contour with the highest density is selected as the location of the object.

[0045] In this way, the device 100 can detect objects in an image with high speed and accuracy. Furthermore, the training time of the device is also reduced compared to the prior art. The performance of the device 100 is compared to other object detection algorithms on the same training and testing datasets.

[0046] In an embodiment, the center of the multivariate Gaussian function is first determined to be random, and then determined using the output of the generated saliency map. To that end, the disclosed model is trained to generate the most salient region at the location of the target object in the image using a given reward system. The location of the most salient region is taken as the center location. Values ​​are high at the center of the multivariate Gaussian, and low values ​​far from the center of the multivariate Gaussian. After multiplying such a matrix with a "value" or "intensity" channel, this creates the same bias scenario in the intensity channel. After converting such an HSV image to an RGB image, the center of the Gaussian allows high values ​​at the target location and low values ​​at non-target locations, thus creating a soft attention version of the image frame. This biased nature of the soft attention input helps the model learn the same bias nature towards the target location and predict high-order salient regions at the target location.

[0047] The device disclosed in this disclosure successfully achieves a 23.52% increase in Mean Average Precision (mAP) compared to YOLOv2, the state-of-the-art object detection model. Not only does the device significantly outperform YOLOv2, but the device also requires significantly less time to train.

[0048] The disclosed device has a 23.52% higher mAP than YOLOv2 when tested on a dataset of 1,893 frames of road video containing both day and night images, as shown in Table 1. The device requires approximately 18 hours of training to reach an mAP of 41.53%, whereas YOLOv2 requires approximately 24 hours of training to reach an mAP of 18.01% on the same test dataset.

[0049] [Table 1]

[0050] Figures 4a and 4b show that for the disclosed device, the mAP percentage of the validation dataset (3,785 image frames) increases per epoch as the absolute value of the training loss decreases. Figure 4a shows the training loss (absolute value) and validation mAP (%) of the disclosed device per epoch, and Figure 4b shows the training loss (absolute value) and validation mAP (%) of YOLOv2 per iteration in blue and red, respectively.

[0051] Moreover, the proposed device not only detects objects such as traffic signs at very far distances, but also detects these signs even when they are hard to see due to bad weather during the day or light from oncoming vehicles at night, unlike the conventional techniques as shown in Figures 5a-5c and 5e-5g. The device also gives more accurate detection than the conventional techniques, as shown in Figures 5d and 5h. Figures 5a-5d show test detection images of the disclosed device 100, where the red bounding box shows the detection box and the blue bounding box shows the ground truth box of the traffic sign. The detection images during testing of the conventional technique YOLOv2 are shown in Figures 5e-5h, where the pink bounding box is the detection box of the traffic sign and the blue is the ground truth bounding box of the traffic sign.

[0052] In this way, the proposed device is able to detect objects in an image in a more accurate and efficient way. Moreover, the device requires less time to train, which also improves the performance of the device. Moreover, the device is also able to detect objects at a very large distance and can detect objects even when they are not very visible.

[0053] 6 shows a flowchart of a method 600 for detecting objects in an image according to an embodiment of the present disclosure. In step 602, the method 600 states that an input image including at least one object to be detected is received. In an embodiment, the image may be acquired by any imaging device, which may be part of the apparatus 100 or may be any external imaging device.

[0054] The method of step 604 states that the soft attention image is generated by modifying the brightness of the received image. In an embodiment, the soft attention image can be generated by multiplying the brightness channel of the received input image by a multivariate Gaussian function, where the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. In another embodiment, the received input image can be an RGB image that can be processed to generate the soft attention image. To generate the soft attention image, the received RGB image can be converted to an HSV image that includes a brightness channel. HSV stands for "hue, saturation, value." HSV is used to measure the hue angle, saturation (or amount of grayness) and brightness (or value) of a color. The HSV image is further processed to modify the brightness of the HSV image by multiplying the brightness channel of the HSV image by a multivariate Gaussian function, where the brightness of the HSV image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. Furthermore, the image is converted back to an RGB image to generate a soft attention image.

[0055] In step 606, the method 600 states that the received image is processed using the first CNN model 104 to extract a first feature map. In step 608, the method states that the soft-attention image is processed using the second CNN model 106 to extract a second feature map. In an embodiment, the first and second feature maps are generated using one or more feature detectors or filters using convolution operations on the received input image and the soft-attention image, respectively. The extracted feature map is a collection of matrices, and pixel values ​​represent one or more of color, texture, shape, and brightness. The received image and the soft-attention image are processed in parallel by the first and second CNN models by passing through six convolution layers to extract features.

[0056] The method in step 610 states that the first and second feature maps are merged to generate a stacked feature map. In an embodiment, the first and second feature maps may be generated by a merged model 110 including a convolutional layer that takes both feature maps as input to output a stacked map of 39×63×2 dimensions. In an embodiment, the stacked feature map includes a merged matrix obtained from a set of matrices of the first and second feature maps, the merged matrix including merged pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image.

[0057] In step 612, the method 600 states that it processes the stacked feature map using a Deep Q learning model to generate a saliency map. The stacked feature map may be processed by the map generation model 108, which is a Deep Q learning model. The model may include one or more convolutional layers that process the stacked map to generate a saliency map of 39x63x1 dimensions. In an embodiment, the saliency map includes a saliency matrix obtained from the integration matrix, the saliency matrix includes salient pixel values ​​that are regions within the received input image that indicate at least one object to be detected in the received input image. In step 614, the method further states that it detects an object based on the saliency map. The saliency map may be processed by the processing unit 114 to detect at least one object. According to the method, detecting at least one object based on the saliency map includes detecting a highest order activity in the saliency map, the highest order activity corresponding to a location of the object. In an embodiment, detecting the highest order activity in the saliency map includes creating multiple contours in the saliency map and calculating the density of each contour by calculating the average pixel value inside the contour. The contour with the highest density is selected as the location of the object. In an embodiment, the OpenCV library is used to create the contours in the saliency map. The density of each contour is calculated as the average pixel value inside the contour. The densest contour is selected as the traffic sign detection. In an embodiment, OpenCV basically finds locations with pixel values ​​close to 1 and creates a peak at all such high values ​​by collecting pixel values ​​around them. In this way, there are multiple peaks or contours in the map. An average pixel value is then calculated for each such contour. After calculating the average pixel value for each contour, whichever contour has the highest average pixel value is considered to be the location of the object of interest.

[0058] In this way, the method is able to detect objects in an image with high speed and high accuracy, and furthermore, the method detects objects at very large distances and also when they are not very visible.

[0059] The illustrated steps are presented to explain the illustrated exemplary embodiments, and it is to be expected that ongoing technological developments will change the manner in which certain functions are performed. These examples are presented herein for purposes of illustration and not limitation. Moreover, the boundaries of functional building blocks have been arbitrarily defined herein for convenience of description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

[0060] Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to those of ordinary skill in the relevant art based on the teachings contained herein, and such alternatives are within the scope of the disclosed embodiments.

[0061] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory capable of storing processor-readable information or data. Thus, a computer-readable storage medium may store instructions executed by one or more processors, including instructions that cause the processor(s) to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROM, DVDs, flash drives, disks, and any other known physical storage medium.

[0062] Suitable processors / controllers include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine.

[0063] The following examples describe further technical aspects of the above-mentioned devices, systems, and methods, and should not be construed as claims. The following examples can be further combined with any of the above-mentioned devices, systems, and methods, or any of the originally filed claims.

[0064] Example 1 is a method for detecting objects present in an image, the method including receiving an input image including at least one object to be detected (602), generating a soft-attention image by modifying a brightness of the received input image (604), processing the received input image with a first CNN model (606) to extract a first feature map, processing the soft-attention image with a second CNN model (608) to extract a second feature map, integrating the first and second feature maps to generate a stacked feature map (610), processing the stacked feature map (612) to generate a saliency map, and detecting at least one object based on the saliency map (614).

[0065] In Example 2, the subject matter of Example 1 may optionally include generating the soft attention image including multiplying a brightness channel of the received input image with a multivariate Gaussian function, such that the brightness of the received input image is modified to be high at a center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0066] In Example 3, the subject matter of Examples 1 and 2 may optionally include converting the received input image into an HSV image including a luminance channel; multiplying the luminance channel of the HSV image with a multivariate Gaussian function such that the luminance of the HSV image is modified to be high at a center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function; and after modifying the luminance of the HSV image, converting the modified HSV image into an RGB image to generate the soft attention image.

[0067] In Example 4, the subject matter of Examples 1-3 may optionally include detecting a highest order activity in the saliency map, where the highest order activity corresponds to a location of the object.

[0068] In Example 5, the subject matter of Examples 1-4 may optionally include creating multiple contours in the saliency map and calculating the density of each contour by calculating the average pixel value within the contour, and the contour with the highest density is selected as the location of the object in the received input image.

[0069] In Example 6, the subject matter of Examples 1-5 may optionally include first and second feature maps having a set of matrices, the set of matrices including pixel values ​​representing at least one of color, texture, shape, and lightness of the received input image and the soft attention image.

[0070] In Example 7, the subject matter of Examples 1-6 may optionally include an integrated matrix obtained from the collection of matrices of the first and second feature maps, the integrated matrix including integrated pixel values ​​representing at least one of color, texture, shape, and lightness of both the received input image and the soft attention image, and the saliency map including a saliency matrix obtained from the integrated matrix, the saliency matrix including salient pixel values ​​that are regions within the received input image that are indicative of at least one object to be detected in the received input image.

[0071] Example 8 is an apparatus for detecting an object present in an image, the apparatus comprising: an input / output (I / O) interface (112) configured to receive an input image including at least one object to be detected; a soft attention image generation unit (120) operatively coupled to the I / O interface (112), the soft attention generation unit (120) configured to generate a soft attention image by modifying a brightness of the received input image; and an attention search unit (102) operatively coupled to the I / O interface (112) and the soft attention image generation unit (120), the attention search unit (102) being configured to generate a first CNN model (104). to extract a first feature map, process the soft-attention image using a second CNN model (106) to extract a second feature map, integrate the first and second feature maps using an integration model (110) to generate a stacked feature map, and process the stacked feature map using a map generation model (108) to generate a saliency map; and a processing unit (114) operatively coupled to the I / O interface (112), the soft-attention image generation unit (120), and the attention search unit (102), wherein the processing unit (114) is configured to detect at least one object based on the saliency map.

[0072] In Example 9, the subject matter of Example 8 may optionally include multiplying a brightness channel of the received input image with a multivariate Gaussian function, such that the brightness of the received input image is modified to be high at a center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0073] In Example 10, the subject matter of Examples 8 and 9 may optionally include, to generate the soft attention image, the soft attention image generation unit (120) is configured to convert the received input image into an HSV image including an intensity channel, multiply the intensity channel of the HSV image by a multivariate Gaussian function such that the intensity of the HSV image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function, and after modifying the intensity of the HSV image, convert the modified HSV image into an RGB image to generate the soft attention image.

[0074] In Example 11, the subject matter of Examples 8-10 can optionally include the processing unit (114) being configured to detect a highest order activity in the saliency map, the highest order activity corresponding to a position of the object.

[0075] In Example 12, the subject matter of Examples 8-11 can optionally include creating multiple contours in the saliency map and calculating the density of each contour by calculating the average pixel value inside the contour, and the contour with the highest density is selected as the location of the object.

[0076] In Example 13, the subject matter of Examples 8-12 may optionally include, where the first and second feature maps are generated using one or more feature detectors using convolution operations on the received input image and the soft attention image, respectively, and the first and second feature maps are sets of matrices, the sets of matrices including pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image.

[0077] In Example 14, the subject matter of Examples 8-13 may optionally include a matrix obtained from a collection of first and second feature map matrices, where the integrated matrix includes integrated pixel values ​​representing at least one of color, texture, shape, and lightness of both the received input image and the soft attention image, and the saliency map includes a saliency matrix obtained from the integrated matrix, where the saliency matrix includes salient pixel values ​​that are regions within the received input image that are indicative of at least one object to be detected in the received input image.

Claims

1. 1. A method for detecting an object present in an image, comprising: Receiving an input image including at least one object to be detected (602); generating a soft attention image by modifying the brightness of the received input image (604); processing (606) the received input image with a first convolutional neural network (CNN) model to extract a first feature map; processing (608) the soft-attention image with a second CNN model to extract a second feature map; Merging (610) the first and second feature maps to generate a stacked feature map; processing 612 the stacked feature map to generate a saliency map; and Detecting the at least one object based on the saliency map (614); A method comprising:

2. The generating of the soft attention image includes: multiplying the brightness channel of a received input image by a multivariate Gaussian function such that the brightness of the received input image is altered to be high at a center of the multivariate Gaussian function and to decrease with distance from the center of the multivariate Gaussian function; The method of claim 1 , comprising:

3. The generating of the soft attention image includes: converting the received input image into an HSV image including a luminance channel; multiplying the luminance channel of the HSV image by a multivariate Gaussian function such that luminance of the HSV image is altered to be high at a center of the multivariate Gaussian function and to decrease with distance from the center of the multivariate Gaussian function; and modifying the luminosity of the HSV image and then converting the modified HSV image to an RGB image to generate the soft attention image; The method of claim 1 or 2, comprising:

4. Detecting the at least one object based on the saliency map includes: detecting a highest order activity in the saliency map, the highest order activity corresponding to a position of the object; The method according to any one of claims 1 to 3, comprising:

5. Detecting the highest order activity in the saliency map includes: generating a plurality of contours in the saliency map; and calculating the density of each contour by calculating the average pixel value inside the contour; The method according to any one of claims 1 to 4, wherein the contour having the highest density is selected as the location of the object in the received input image.

6. The first and second feature maps are generated using one or more feature detectors that use a convolution operation on the received input image and the soft attention image, respectively; The method of any one of claims 1 to 5, wherein the first and second feature maps are sets of matrices, the sets of matrices comprising pixel values ​​representing at least one of color, texture, shape, and lightness of the received input image and the soft attention image.

7. the stacked feature map includes a combined matrix obtained from the set of matrices of the first and second feature maps, the combined matrix including combined pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image; 7. The method of claim 1, wherein the saliency map comprises a saliency matrix obtained from the integration matrix, the saliency matrix comprising salient pixel values ​​that are indicative of regions within the received input image of the at least one object to be detected in the received input image.

8. An apparatus for detecting an object present in an image, comprising: an input / output (I / O) interface (112) configured to receive an input image including at least one object to be detected; a soft attention image generation unit (120) operatively coupled to the I / O interface (112), the soft attention generation unit (120) configured to generate a soft attention image by modifying a brightness of the received input image; an attention search unit (102) operatively coupled to the I / O interface (112) and the soft attention image generation unit (120), the attention search unit (102) comprising: processing the received input image with a first convolutional neural network (CNN) model (104) to extract a first feature map; processing the soft-attention image using a second CNN model (106) to extract a second feature map; integrating the first and second feature maps using an integration model (110) to generate a stacked feature map; and processing the stacked feature map using a map generation model (108) to generate a saliency map. An attention search unit (102); a processing unit (114) operatively coupled to the I / O interface (112), the soft attention image generation unit (120), and the attention search unit (102), the processing unit (114) configured to detect the at least one object based on the saliency map; 13. An apparatus comprising:

9. To generate the soft attention image, the soft attention image generation unit (120) comprises:

9. The apparatus of claim 8, configured to multiply an intensity channel of the received input image by a multivariate Gaussian function such that intensity of the received input image is modified to be high at a center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

10. To generate the soft attention image, the soft attention image generation unit (120) comprises: converting the received input image into an HSV image including a luminance channel; multiplying the luminance channel of the HSV image by a multivariate Gaussian function, such that luminance of the HSV image is altered to be high at a center of the multivariate Gaussian function and to decrease with distance from the center of the multivariate Gaussian function; 10. An apparatus according to claim 8 or 9, configured to modify the lightness of the HSV image and then convert the modified HSV image into an RGB image to generate the soft attention image.

11. The apparatus of any one of claims 8 to 10, wherein for detecting the at least one object based on the saliency map, the processing unit (114) is configured to detect a highest order activity in the saliency map, the highest order activity corresponding to a position of the object.

12. To detect the highest order activity in the saliency map, the processing unit (114) creating a plurality of contours in the saliency map; configured to calculate the density of each contour by calculating the average pixel value within said contour; An apparatus according to any one of claims 8 to 11, wherein the contour having the highest density is selected as the location of the object.

13. The first and second feature maps are generated using one or more feature detectors using a convolution operation on the received input image and a soft attention image, respectively; The apparatus of any one of claims 8 to 12, wherein the first and second feature maps are sets of matrices, the sets of matrices comprising pixel values ​​representing at least one of color, texture, shape, and lightness of the received input image and the soft attention image.

14. the stacked feature map includes a combined matrix obtained from the set of matrices of the first and second feature maps, the combined matrix including combined pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image; The apparatus of any one of claims 8 to 13, wherein the saliency map comprises a saliency matrix obtained from the integration matrix, the saliency matrix comprising salient pixel values ​​that are indicative of regions within the received input image of the at least one object to be detected in the received input image.

Images