Soft attention-based saliency map generation apparatus for object detection and method thereof

Abstract

Apparatus and methods for detecting objects present in an image are described. The method narrates receiving (602) an input image comprising at least one object to be detected and generating (604) a soft attention image by processing the received image using a soft attention technique. The method further narrates processing (606) the received image using a first CNN model to extract a first feature map, and processing (608) the soft attention image using a second CNN model to extract a second feature map. The method further describes integrating (610) the first feature map and the second feature map to generate a stacked feature map, processing (612) the stacked feature map to generate a saliency map, and detecting (614) at least one object based on the saliency map.

Description

Technical Field

[0001] The subject matter described herein generally discloses an apparatus and method for detecting objects present in an image. In other words, the present application discloses a soft-attention-based saliency map generation technique for object detection. Background Art

[0002] There are many algorithms for object detection in images such as Region-based Convolutional Neural Network (R-CNN), Fast R-CNN, Faster R-CNN, YOLOv2, Single Shot Detector (SSD), etc. These state-of-the-art algorithms search for the target object throughout the image. These algorithms are computationally inefficient and biologically implausible. This results in low performance on evaluation metrics and long training duration.

[0003] In real life, when humans search for a specific object in an image, the eyes move quickly between fixations (called saccades) before settling on the target object. The trajectory of the eye movement varies depending on the object of interest to the human observer. Each fixation point represents the area in the image that the human is "focusing on". Human visual attention is believed to move as the eyes move. However, existing techniques tend to process the entire input frame instead of focusing / searching for discriminative features in the input frame.

[0004] Therefore, an attention-based model is needed to localize objects in images. Summary of the invention

[0005] The present disclosure overcomes one or more shortcomings of the prior art and provides the additional advantages discussed in the present disclosure. Additional features and advantages are achieved through the technology of the present disclosure. Other embodiments and aspects of the present disclosure are described in detail herein, and these embodiments and aspects are considered to be a part of the present disclosure claimed.

[0006] In one non-limiting embodiment of the present disclosure, a method for detecting an object present in an image is disclosed. The method includes receiving an input image including at least one object to be detected, and generating a soft attention image by modifying the brightness of the received image using a soft attention technique. The method further describes processing the received input image using a first CNN model to extract a first feature map, and processing the soft attention image using a second CNN model to extract a second feature map. The method further describes integrating the first feature map and the second feature map to generate a stacked feature map, processing the stacked feature map to generate a saliency map, and detecting at least one object based on the saliency map.

[0007] In another non-limiting embodiment, the present disclosure describes generating a soft attention image comprising multiplying a brightness channel of a received input image by a multivariate Gaussian function such that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases.

[0008] In another non-limiting embodiment, the present disclosure describes that a soft attention image is generated by: converting the received input image into an HSV image including a brightness channel; multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases; after modifying the brightness of the HSV image, converting the modified HSV image into an RGB image to generate the soft attention image.

[0009] In another non-limiting embodiment, the present disclosure recites that detecting at least one object based on a saliency map includes detecting a highest activity in the saliency map, wherein the highest activity corresponds to a location of the object.

[0010] In yet another non-limiting embodiment, the present disclosure recites that detecting the highest activity in a saliency map includes creating a plurality of contours on the saliency map; calculating the density of each contour by calculating the average pixel value within each contour, wherein the contour with the highest density is selected as the location of the object.

[0011] In yet another non-limiting embodiment, the present disclosure describes that the first feature map and the second feature map are generated by performing convolution operations on the received image and the soft attention image, respectively, using one or more feature detectors, and the first feature map and the second feature map are sets of matrices, wherein the set of matrices includes pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and the soft attention image.

[0012] In yet another non-limiting embodiment, the present disclosure recites that the stacked feature map includes a merge matrix obtained from the matrix set of the first feature map and the second feature map, wherein the merge matrix includes merged pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image. The saliency map includes saliency matrices obtained from the merge matrices, wherein the saliency pixel values ​​of the saliency matrices are regions within the received input image indicating at least one object to be detected in the received input image.

[0013] In yet another non-limiting embodiment, the present disclosure discloses a device for detecting an object present in an image. The device includes an input / output (I / O) interface configured to receive an input image including at least one object to be detected. The device further includes a soft attention image generation unit operatively coupled to the I / O interface. The soft attention generation unit is configured to generate a soft attention image by modifying the brightness of the received image using a soft attention technique. The device further includes an attention search unit operatively coupled to the I / O interface and the soft attention image generation unit. The attention search unit is configured to process the received image using a first CNN model to extract a first feature map, process the soft attention image using a second CNN model to extract a second feature map, integrate the first feature map and the second feature map using an integration model to generate a stacked feature map, and process the stacked feature map using a graph generation model to generate the saliency map. The device further includes a processing unit operatively coupled to the I / O interface, the soft attention image generation unit, and the attention search unit. The processing unit is configured to detect the at least one object based on the saliency map.

[0014] In yet another non-limiting embodiment, the present disclosure describes generating a soft attention image, wherein the soft attention image generation unit is configured to multiply the brightness channel of the received input image by a multivariate Gaussian function so that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases.

[0015] In another non-limiting embodiment, the present disclosure describes that in order to generate a soft attention image, the soft attention image generation unit is configured to convert the received input image into an HSV image including a brightness channel; multiply the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases; and after modifying the brightness of the HSV image, convert the modified HSV image into an RGB image to generate the soft attention image.

[0016] In yet another non-limiting embodiment of the present disclosure, to detect the at least one object based on the saliency map, the processing unit is configured to detect a highest activity in the saliency map, wherein the highest activity corresponds to a location of the object.

[0017] In yet another non-limiting embodiment of the present disclosure, to detect the highest activity in the saliency map, the processing unit is configured to create a plurality of contours on the saliency map, and calculate the density of each contour by calculating the average pixel value within each contour, and select the contour with the highest density as the location of the object.

[0018] In another non-limiting embodiment of the present disclosure, the present application discloses that the first feature map and the second feature map are generated by performing convolution operations on the received input image and the soft attention image using one or more feature detectors, respectively. The first feature map and the second feature map are sets of matrices, wherein the matrix set includes pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and the soft attention image.

[0020] In yet another non-limiting embodiment, the present disclosure recites that the stacked feature map comprises a merged matrix obtained from a set of matrices of the first feature map and the second feature map, wherein the merged matrix comprises merged pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image, and the saliency map comprises saliency matrices obtained from these merged matrices, wherein the saliency pixel values ​​of these saliency matrices are areas within the received input image indicating at least one object to be detected in the received input image.

[0021] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated into and constitute a part of the present disclosure, illustrate exemplary embodiments and, together with the description, are used to explain the disclosed embodiments. In the drawings, the leftmost digit(s) of a reference number indicates the drawing in which the reference number first appears. The same numerals are used throughout the drawings to refer to similar features and components. Some embodiments of the apparatus and / or method according to embodiments of the present subject matter will now be described by way of example only and with reference to the accompanying drawings, in which:

[0023] Figure 1 An apparatus for detecting an object present in an image according to an embodiment of the present disclosure is illustrated.

[0024] Figure 2a to Figure 2b Exemplary RGB and soft-attention images according to embodiments of the present disclosure are illustrated, respectively.

[0025] Figure 3 An architecture for generating a saliency map according to an embodiment of the present disclosure is illustrated.

[0026] Figure 4a to Figure 4b Performance measurements of the proposed device according to embodiments of the present subject matter are shown.

[0027] Figure 5a to Figure 5b Detection results of the proposed apparatus according to an embodiment of the present subject matter and conventional techniques are shown.

[0028] Figure 6 A flow chart of a method for detecting an object present in an image according to an embodiment of the present subject matter is illustrated.

[0029] Those skilled in the art will appreciate that any block diagram herein represents a conceptual view of an illustrative device that embodies the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocodes, etc. represent various processes that can be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown. DETAILED DESCRIPTION

[0030] In this document, the word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any “exemplary” embodiment or implementation of the subject matter described herein is not necessarily to be construed as preferred or advantageous over other embodiments.

[0031] Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the accompanying drawings and will be described in detail below. However, it should be understood that it is not intended to limit the present disclosure to the particular form disclosed, but rather, the present disclosure includes all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0032] The terms "comprises, comprising, include(s)" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a configuration, apparatus, or method that includes a series of components or steps includes not only these components or steps, but may also include other components or steps not expressly listed or inherent to these configurations, apparatuses, or methods. In other words, without more constraints, one or more elements in a device or apparatus preceded by "comprises...a" does not exclude the presence of other or additional elements in the device or apparatus.

[0033] In the following detailed description of the embodiments of the present disclosure, reference is made to the accompanying drawings that form a part of the present disclosure, and in the accompanying drawings, specific embodiments in which the present disclosure can be practiced are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, and it should be understood that other embodiments may be used and may be modified without departing from the scope of the present disclosure. Therefore, the following description is not to be considered as limiting.

[0034] The present invention will be described below with reference to the accompanying drawings. In the following description, well-known functions or constructions will not be described in detail because they would obscure the description with unnecessary detail.

[0035] There are various techniques to detect objects present in an image such as Region-based Convolutional Neural Network (R-CNN), Fast R-CNN, Faster R-CNN, YOLOv2, Single Shot Detector (SSD), etc. In R-CNN, the network takes an input of around 2000 object region proposals from a single image. These region proposals are generated using a selective search algorithm. Each region proposal is scaled to the CNN input size. After scaling, each region proposal is fed into a CNN to extract features. The extracted features are then classified and predicted in terms of bounding box coordinates using a linear model using support vector machines and regression.

[0036] Further, in the Fast R-CNN technique, region proposals are generated by a selective search algorithm, just as it was done in the R-CNN technique. These region proposals are fed into the Visual Geometry Group (VGG) convolutional network. Since each pixel in the output of the CNN network corresponds to a patch (region of interest) in the input image, the region of interest (RoI) in the input image is projected in the output of the 5th convolutional layer at different sizes. Each RoI is scaled into a 7×7 square. Max pooling is performed on each of these RoIs, resulting in a 49-dimensional RoI vector. Each 49-dimensional vector is connected to a fully connected layer, giving a representation of the region proposal. The fully connected layer is passed to a soft-max activation to calculate the probability distribution of possible object classes. At the same time, a regression layer is used to predict new bounding boxes.

[0037] Further, in Faster R-CNN, region proposals are generated using a CNN-based approach instead of a selective search algorithm. The full image is used as input to VGG-Net. The output size of the last layer of VGG-Net is (h×w×512). A 3×3 kernel is applied to each (h×w) cell in all 512 feature maps, resulting in a 512-dimensional vector of cells. Assume that the number of bounding boxes (called anchor boxes) of different sizes and aspect ratios (denoted by "k") and a 512-dimensional vector for each location are given. The network then predicts the probability that a given anchor box centered at that location contains a possible object class. Each cell in the output of the last convolutional layer corresponds to a patch in the original image (due to the influence region in the input of each pixel in the output of the last convolutional layer). Each of the anchor boxes is centered on the corresponding patch. The classifier output of 1 indicates that these anchor boxes have reasonable overlap with the true bounding box (intersection-over-union ratio > 0.7). Similarly, the regression model predicts the true bounding box based on the anchor box. After generating region proposals in this way, the rest is the same as Fast R-CNN.

[0038] In YOLO (Look Only Once), an end-to-end trained deep neural network is used for object detection. The network takes a full image as input and uses the corresponding label to predict the location of objects in new images. Before training, the label of each image is created by making (S×S) grids over the entire image; each grid is assigned two random bounding boxes with their coordinates and the object class. If the center of an object falls within a grid, then that grid is responsible for detecting that object. YOLO is fast and is a real-time object detector.

[0039] However, all these object detection algorithms lack the attention mechanism that humans use to detect objects in images, nor do they impart the logic of attention mechanism that is done by the human brain. Moreover, in these techniques, the complete network does not follow end-to-end training.

[0040] The present disclosure discloses a hybrid model for object detection while solving the above problem statement. The hybrid model relies on focusing attention on key discriminative features in the input frame, thereby ignoring other irrelevant features in the input frame, just like how the human brain works. The hybrid model uses the "attention search" technique, which mainly focuses on mimicking the function of the brain, whereby it only focuses on certain features that are most relevant to the output, and ignores those that do not affect the output. In this way, the hybrid model only needs to focus on the features required for the final output, thereby saving a lot of computational costs. The hybrid model uses temporal difference error (TDE) as the main learning rule to provide optimal object detection. This mechanism also reduces the overall training time.

[0041] refer to Figure 1, illustrates an apparatus 100 for detecting (multiple) objects in an image according to an embodiment of the present disclosure. In one embodiment, the apparatus 100 may include various entities, such as an attention search unit 102, an input / output (I / O) interface 112, a processing unit 114, and a soft attention image generation unit 120, but is not limited thereto. These entities communicate and are operatively coupled to each other via a wired or wireless network. In an embodiment, the apparatus 100 may use a deep Q neural network based on attention search for object localization. In another embodiment, the attention search unit 102 includes a first convolutional neural network (CNN) model 104, a second CNN model 106, a graph generation model 108, and an integration model 110. In an embodiment, the CNN models 104, 106 are neural network models that extract a higher-level representation of image content. Unlike traditional image recognition in which a user must define image features by himself, CNN obtains raw pixel data of an image and automatically extracts features. In yet another embodiment, the processing unit 114 may include one or more processors 116 and a memory 118.

[0042] The I / O interface 112 may receive an image including at least one object to be detected by the apparatus 100. The received image may be stored in a database ( Figure 1 100). In an embodiment, the image may be captured by any image capture device, which may be part of the apparatus 100 or any external imaging device. The received image is an RGB image. An RGB image, also known as a true color image, may be stored as an M*N*I data array, defining the red, green, and blue color components of each individual pixel. Figure 2a An exemplary RGB image is shown.

[0043] The soft attention image generation unit 120 can modify the brightness of the received input image to generate a soft attention image. In an embodiment, in order to generate the soft attention image, the image generation unit 120 can convert the received RGB image into an HSV image including a brightness channel. HSV stands for "hue, saturation, value". HSV is used to measure the hue angle, saturation (or amount of grayscale), and brightness (or brightness) of a color. The image generation unit 120 can modify the brightness of the HSV image by multiplying the brightness channel of the received input image by a multivariate Gaussian f(y, x) (as shown in the equation below). The brightness of the HSV image is modified so that the brightness is higher at the center (y1, x1) of the multivariate Gaussian and gradually decreases as it moves away from the center (y1, x1). The center position is considered to be random. In an embodiment, the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases. After modifying the brightness of the HSV image, the image is converted back to an RGB image, as shown in the following equation. Figure 2bThe generated RGB image is a soft attention image, which can be stored in the database. where x and y represent the pixel locations of the Gaussian matrix, and σ represents the standard deviation or the radius of the circle.

[0044] Further, the first CNN model 104 of the attention search unit 102 takes the received image (original image) as input, and the second CNN model 106 takes the generated soft attention image as input. The received image and the soft attention image are processed in parallel by the first and second CNN models through six convolutional layers to extract features. The first CNN model 104 can process the original received image to generate a first feature map, and the second CNN model 106 can process the generated soft attention image to generate a second feature map. In an embodiment, the feature map is generated by applying a filter or feature detector to the input image or the feature map of the previous input layer of the CNN using a convolution operation. The feature map is a collection of matrices in which the values ​​of the pixels are calculated based on a high-level understanding of the presented input image corresponding to its color, texture, shape and brightness of the object, etc. The feature may include at least hue, saturation, brightness and color, but is not limited thereto.

[0045] The output feature map is passed to the integration model 110, which stacks the first feature map and the second feature map to output a stacked feature map. In an embodiment, the integration model 110 may include a convolutional layer that takes the two feature maps (the first feature and the second feature) as input and outputs a stacked map of size 39×63×2. The stacked feature map is processed by the graph generation model 108 to generate a saliency map. In an embodiment, the graph generation model 108 may be a deep Q learning model, which may include one or more convolutional layers. The deep Q learning model may process the stacked map to generate a saliency map of size 39×63×1, such as Figure 3 As shown. In an embodiment, the stacked feature map includes a merged matrix obtained from a matrix set of the first feature map and the second feature map, wherein the merged matrix includes merged pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and the soft attention image. In another embodiment, the saliency map includes saliency matrices obtained from these merged matrices, wherein the saliency pixel values ​​of these saliency matrices are areas within the received input image indicating at least one object to be detected in the received input image. Figure 3 An architecture for generating a saliency map according to an embodiment of the present disclosure is illustrated.

[0046] The deep Q-learning model of the apparatus 100 may be trained to generate a saliency map using a deep Q-learning process. A reward matrix of size 640×1024×1 is created for each input image in the dataset for training. In the reward matrix, a value of “1” is assigned to pixel coordinates where a traffic sign (object) and soft attention overlap, a value of “0.5” is assigned to pixel coordinates where only a traffic sign exists, and a value of “0” is assigned to other locations. Afterwards, the reward matrix R t+1 The size of is resized to 39×63×1 to match the size of the saliency map as the output of the network.

[0047] Here, (x, y) are the pixel coordinates in the input image. The model is trained on frames of RGB images. The dimensions of both inputs are 640×1024×3. To train the network, the temporal differential error (TDE) between the saliency maps of two consecutive frames at time ‘t’ and ‘t+1’ is calculated using Equation 2 shown below. TDE=(R t+1 +γQ t+1 -Q t ) (2)

[0048] In the above equation, “γ” is the discount factor. The temporal difference error is added with the regularization cost using the L2 norm of the weights. After that, the network weights are updated by using the back-propagation algorithm with an optimizer to minimize the total cost for a batch of 32 image frames. The hyper parameters such as learning rate, σ, γ, regularization factor β are fine-tuned to the values ​​of 0.00015, 25, 0.4, 0.00001 respectively to achieve the best result with the minimum total cost. The model is trained for 54 epochs.

[0049] During testing, the device is provided with a frame containing a traffic sign. The traffic sign is the object to be detected. The device outputs a saliency map of size 39×63×1, which is resized to 640×1024×1 to match the height and width of the input image. The values ​​of the output saliency map are in the range of 0 to 1. The highest activity in the saliency map corresponds to the location where the traffic sign is detected in the given image. Further, one or more contours are created on the saliency map. In an embodiment, the contours are created on the saliency map using the OpenCV library. The density of each contour is calculated as the average pixel value within the contour. The contour with the highest density is selected as the detection of the traffic sign. In an embodiment, OpenCV basically finds the location where the pixel value is close to 1 and forms a peak on all these high values ​​by collecting the pixel values ​​around these high values. In this way, there will be multiple peaks or contours on the map. The average pixel value of each such contour is then calculated. After calculating the average pixel value of each contour, the contour with the highest average pixel value is considered as the location of the target object.

[0050] Reference again Figure 1 After generating the saliency map, the processing unit 114 processes the saliency map to detect at least one object. To detect the object, the processing unit 114 may create multiple contours on the saliency map and calculate the density of each contour by calculating the average pixel value within each contour. The contour with the highest density is selected as the location of the object.

[0051] In this way, the apparatus 100 can detect objects in images quickly and with high accuracy. Furthermore, the training time of the apparatus is also shortened compared to conventional techniques. The performance of the apparatus 100 is compared with other object detection algorithms on the same training and test data sets.

[0052] In an embodiment, the center of the multivariate Gaussian function is initially determined to be random and then determined using the generated saliency map output. To this end, the disclosed model is trained to generate the most salient region at the location of the target object in the image using a given reward system. The location of the most salient region is considered to be the center location. The values ​​are higher at the center of the multivariate Gaussian and lower at the location away from the center of the multivariate Gaussian. After multiplying this matrix with the "brightness" or "luminance" channel, the same biased scenario is generated in the luminance channel. After converting such an HSV image back to an RGB image, the Gaussian center will produce high values ​​at the target location and low values ​​at non-target locations, thus creating a soft attention version of the image frame. This bias in the soft attention input helps the model learn the same bias towards the target location and predict a highly salient region at the target location.

[0053] Compared with the most advanced object detection model YOLOv2, the device disclosed in the present disclosure successfully improved the mean average precision (mAP) by 23.52%. Not only does the performance of the device far exceed that of YOLOv2, but the training time required by the device is also greatly reduced.

[0054] As shown in Table 1, the disclosed device achieves 23.52% higher mAP than YOLOv2 on a test dataset of 1,893 frames of road videos containing both daytime and nighttime images. The device requires approximately 18 hours of training to achieve a mAP of 41.53%, while YOLOv2 requires approximately 24 hours of training to achieve a mAP of 18.01% on the same test dataset.

[0067] Figure 4a and Figure 4b It is shown that, for the disclosed device, the mAP percentage on the validation dataset (3,785 image frames) increases every epoch as the absolute value of the training loss decreases. Figure 4aThe training loss (absolute value) and validation mAP (%) of the disclosed device per epoch are shown, and Figure 4b The training loss (absolute value) and validation mAP (%) of YOLOv2 at each iteration are shown in blue and red, respectively.

[0068] Furthermore, unlike conventional techniques, the proposed device can not only detect objects at great distances (e.g., traffic signs), but can also detect these signs when they are not very visible during the day due to bad weather conditions or at night due to the lights of passing vehicles, e.g. Figure 5a to Figure 5c and Figure 5e to Figure 5g As shown. Figure 5d and Figure 5h As shown, the device can also provide more accurate detection than traditional techniques. Figure 5a to Figure 5d FIG. 1 shows a test detection image of the disclosed apparatus 100, wherein the red bounding box represents the detection box and the blue bounding box represents the ground truth box of the traffic sign. Figure 5e to Figure 5h As shown, the pink bounding box is the detection box of the traffic sign, and the blue one is the ground truth bounding box of the traffic sign.

[0069] In this way, the proposed device can detect objects in images more accurately and efficiently. In addition, the device requires less training time, which also improves the performance of the device. Further, the device can also detect objects at a long distance and even when the objects are not obvious.

[0070] Figure 6 A flow chart of a method 600 for detecting an object within an image according to an embodiment of the present disclosure is illustrated. In step 602, the method 600 recites receiving an input image including at least one object to be detected. In an embodiment, the image may be captured by any image capture device, which may be part of the apparatus 100 or any external imaging device.

[0071] In step 604, the method describes generating a soft attention image by modifying the brightness of the received image. In an embodiment, the soft attention image can be generated by multiplying the brightness channel of the received input image by a multivariate Gaussian function so that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases. In another embodiment, the received input image can be an RGB image, which can be processed to generate a soft attention image. In order to generate the soft attention image, the received RGB image can be converted to an HSV image including a brightness channel. HSV stands for "hue, saturation, brightness". HSV is used to measure the hue angle, saturation (or amount of grayscale) and brightness (or lightness) of a color. The HSV image is further processed to modify the brightness of the HSV image by multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases. Further, the image is converted back to an RGB image to generate a soft attention image.

[0072] In step 606, method 600 describes processing the received image using the first CNN model 104 to extract a first feature map. In step 608, the method describes processing the soft attention image using the second CNN model 106 to extract a second feature map. In an embodiment, the first feature map and the second feature map are generated by convolution operations on the received input image and the soft attention image, respectively, using one or more feature detectors or filters. The extracted feature map is a set of matrices in which pixel values ​​represent one or more of color, texture, shape, and brightness. The received image and the soft attention image are processed in parallel by the first and second CNN models through six convolutional layers to extract features.

[0073] In step 610, the method recites integrating the first feature map and the second feature map to generate a stacked feature map. In an embodiment, the first feature map and the second feature map may be generated by an integration model 110, which includes a convolutional layer that takes the two feature maps as input to output a stacked map of size 39×63×2. In an embodiment, the stacked feature map includes a merge matrix obtained from the set of matrices of the first feature map and the second feature map, wherein the merge matrix includes merged pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image.

[0074] In step 612, method 600 describes processing the stacked feature map using a deep Q-learning model to generate a saliency map. The stacked feature map can be processed by the map generation model 108, which is a deep Q-learning model. The model may include one or more convolutional layers that process the stacked map to generate a saliency map of size 39×63×1. In an embodiment, the saliency map includes a saliency matrix obtained from the merged matrices, wherein the saliency pixel values ​​of the saliency matrices are regions within the received input image indicating at least one object to be detected in the received input image. In step 614, the method further describes detecting an object based on the saliency map. The processing unit 114 may process the saliency map to detect at least one object. According to the method, detecting at least one object based on the saliency map includes detecting the highest activity in the saliency map, and the highest activity corresponds to the location of the object. In an embodiment, detecting the highest activity in the saliency map includes creating multiple contours on the saliency map, and calculating the density of each contour by calculating the average pixel value within each contour. The contour with the highest density is selected as the location of the object. In an embodiment, the OpenCV library is used to create contours on the saliency map. The density of each contour is calculated as the average pixel value within the contour. The contour with the highest density is selected as the detection of the traffic sign. In an embodiment, OpenCV basically finds the position where the pixel value is close to 1, and forms a peak on all these high values ​​by collecting the pixel values ​​around these high values. In this way, there will be multiple peaks or contours on the map. The average pixel value of each such contour is then calculated. After calculating the average pixel value of each contour, the contour with the highest average pixel value is regarded as the position of the target object.

[0075] In this way, the method can detect objects in an image quickly and with high accuracy. Furthermore, the method can detect objects at great distances, even when the objects are not obvious.

[0076] The steps shown are set forth to illustrate the exemplary embodiments shown, and it should be expected that ongoing technological developments will change the manner in which specific functions are performed. The examples provided herein are for illustrative purposes, not limitation. Further, for ease of description, the boundaries of functional building blocks are arbitrarily defined herein. Alternative boundaries may be defined as long as the specified functions and their relationships are appropriately performed.

[0077] Based on the teachings contained herein, alternatives (including equivalents, extensions, variations, deviations, etc. of those described herein) will be apparent to those skilled in the relevant art. These alternatives fall within the scope of the disclosed embodiments.

[0078] In addition, one or more computer-readable storage media can be utilized to implement embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor can be stored. Therefore, a computer-readable storage medium can store instructions for execution by one or more processors, including instructions for causing (multiple) processors to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carriers and transient signals, i.e., non-transient. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disk drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.

[0079] Suitable processors / controllers include, for example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or state machine. The following examples describe further technical aspects of the above-described devices, systems, and methods and should not be construed as claims. The following examples may be combined with any of the devices, systems, and methods described above or with any of the claims originally filed. Example 1 is a method for detecting an object present in an image, the method comprising: receiving (602) an input image including at least one object to be detected; generating (604) a soft attention image by modifying the brightness of the received input image; processing (606) the received input image using a first CNN model to extract a first feature map; processing (608) the soft attention image using a second CNN model to extract a second feature map; integrating (610) the first feature map and the second feature map to generate a stacked feature map; processing (612) the stacked feature map to generate a saliency map; and detecting (614) the at least one object based on the saliency map. In Example 2, the subject matter of Example 1 may optionally include: generating the soft attention image includes multiplying the brightness channel of the received input image by a multivariate Gaussian function, so that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases. In Example 3, the subject matter of Examples 1 and 2 may optionally include: converting the received input image into an HSV image including a brightness channel, multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases; and after modifying the brightness of the HSV image, converting the modified HSV image into an RGB image to generate the soft attention image. In Example 4, the subject matter of Examples 1 to 3 may optionally include detecting a highest activity in the saliency map, wherein the highest activity corresponds to a location of the object. In Example 5, the subject matter of Examples 1 to 4 may optionally include creating multiple contours on the saliency map; and calculating the density of each contour by calculating the average pixel value within each contour, wherein the contour with the highest density is selected as the position of the object in the received input image. In Example 6, the subject matter of Examples 1 to 5 may optionally include: the first feature map and the second feature map have a set of matrices, wherein the set of matrices includes pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image. In Example 7, the subject matter of Examples 1 to 6 may optionally include a merged matrix obtained from the set of matrices of the first feature map and the second feature map, wherein the merged matrix includes merged pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image, and wherein the saliency map includes saliency matrices obtained from these merged matrices, wherein the saliency pixel values ​​of these saliency matrices are areas within the received input image indicating at least one object to be detected in the received input image. Example 8 is a device for detecting an object present in an image, the device comprising: an input / output (I / O) interface (112) configured to receive an input image including at least one object to be detected; a soft attention image generation unit (120) operatively coupled to the I / O interface (112), wherein the soft attention image generation unit (120) is configured to generate a soft attention image by modifying the brightness of the received input image; an attention search unit (102) operatively coupled to the I / O interface (112) and the soft attention image generation unit (120), wherein the attention search unit (102) is configured to: use the first A CNN model (104) processes the received input image to extract a first feature map; uses a second CNN model (106) to process the soft attention image to extract a second feature map; uses an integration model (110) to integrate the first feature map and the second feature map to generate a stacked feature map; and uses a graph generation model (108) to process the stacked feature map to generate a saliency map; and a processing unit (114) operatively coupled to the I / O interface (112), the soft attention image generation unit (120) and the attention search unit (102), wherein the processing unit (114) is configured to detect the at least one object based on the saliency map. In Example 9, the subject matter of Example 8 may optionally include multiplying the brightness channel of the received input image by a multivariate Gaussian function so that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases. In Example 10, the subject matter of Examples 8 and 9 may optionally include generating the soft attention image, the soft attention image generating unit (120) being configured to: convert the received input image into an HSV image including a brightness channel, multiply the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases; and after modifying the brightness of the HSV image, convert the modified HSV image into an RGB image to generate the soft attention image. In Example 11, the subject matter of Examples 8 to 10 may optionally include: the processing unit (114) is configured to detect a highest activity in the saliency map, wherein the highest activity corresponds to the location of the object. In Example 12, the subject matter of Examples 8 to 11 may optionally include creating multiple contours on the saliency map, and calculating the density of each contour by calculating the average pixel value within each contour, wherein the contour with the highest density is selected as the location of the object. In Example 13, the subject matter of Examples 8 to 12 may optionally include: the first feature map and the second feature map are respectively generated by performing convolution operations on the received input image and the soft attention image using one or more feature detectors, wherein the first feature map and the second feature map are sets of matrices, wherein the matrix set includes pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image. In Example 14, the subject matter of Examples 8 to 13 may optionally include matrices obtained from a set of matrices of the first feature map and the second feature map, wherein the merged matrix includes merged pixel values ​​representing at least one of color, texture, shape, and brightness of both the received input image and the soft attention image, and wherein the saliency map includes saliency matrices obtained from these merged matrices, wherein the saliency pixel values ​​of these saliency matrices are areas within the received input image indicating at least one object to be detected in the received input image.

Claims

1. A method for detecting an object present in an image, the method comprising: Receiving (602) an input image including at least one object to be detected; generating (604) a soft attention image by modifying the brightness of the received input image; Processing (606) the received input image using a first convolutional neural network (CNN) model to extract a first feature map; Processing (608) the soft attention image using a second CNN model to extract a second feature map; integrating (610) the first feature map and the second feature map to generate a stacked feature map; processing (612) the stacked feature map to generate a saliency map; as well as The at least one object is detected (614) based on the saliency map.

2. The method of claim 1, wherein: Generating the soft attention image includes: The brightness channel of the received input image is multiplied by a multivariate Gaussian function so that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases.

3. The method according to claims 1 and 2, wherein: Generating the soft attention image includes: Convert the received input image into an HSV image including a brightness channel; Multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases; as well as After modifying the brightness of the HSV image, the modified HSV image is converted into an RGB image to generate the soft attention image.

4. The method according to claims 1 to 3, wherein: Detecting the at least one object based on the saliency map includes: A highest activity in the saliency map is detected, wherein the highest activity corresponds to the location of the object.

5. The method according to claims 1 to 4, wherein: Detecting the highest activity in the saliency map involves: creating a plurality of contours on the saliency map; and The density of each contour is calculated by taking the average pixel value within each contour, Therein, the contour with the highest density is selected as the position of the object in the received input image.

6. The method according to claims 1 to 5, wherein: The first feature map and the second feature map are generated by performing convolution operations on the received input image and the soft attention image using one or more feature detectors, respectively. The first feature map and the second feature map are a set of matrices, wherein the matrix set includes pixel values ​​representing at least one of the color, texture, shape and brightness of the received input image and the soft attention image.

7. The method according to claims 1 to 6, in, The stacked feature map includes a merge matrix obtained from the set of matrices of the first feature map and the second feature map, wherein the merge matrix includes merged pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image, and The saliency map includes saliency matrices obtained from the merged matrices, wherein saliency pixel values ​​of the saliency matrices are regions within the received input image indicating at least one object to be detected in the received input image.

8. A device for detecting an object present in an image, the device comprising: An input / output (I / O) interface (112) configured to receive an input image including at least one object to be detected; a soft attention image generation unit (120) operatively coupled to the I / O interface (112), wherein the soft attention image generation unit (120) is configured to generate the soft attention image by modifying the brightness of the received input image; An attention search unit (102) operatively coupled to the I / O interface (112) and the soft attention image generation unit (120), wherein the attention search unit (102) is configured to: Processing the received input image using a first convolutional neural network (CNN) model (104) to extract a first feature map; Processing the soft attention image using a second CNN model (106) to extract a second feature map; integrating the first feature map and the second feature map using an integration model (110) to generate a stacked feature map; and Processing the stacked feature map using a graph generation model (108) to generate a saliency map; and A processing unit (114) operatively coupled to the I / O interface (112), the soft attention image generation unit (120), and the attention search unit (102), wherein the processing unit (114) is configured to detect the at least one object based on the saliency map.

9. The device of claim 8, wherein: In order to generate the soft attention image, the soft attention image generating unit (120) is configured to: The brightness channel of the received input image is multiplied by a multivariate Gaussian function so that the brightness of the received input image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases.

10. The device as claimed in claims 8 and 9, wherein In order to generate the soft attention image, the soft attention image generating unit (120) is configured to: Convert the received input image into an HSV image including a brightness channel; Multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is modified to be higher at the center of the multivariate Gaussian function and decreases as the distance from the center of the multivariate Gaussian function increases; as well as After modifying the brightness of the HSV image, the modified HSV image is converted into an RGB image to generate the soft attention image.

11. The device according to claims 8 to 10, wherein: In order to detect the at least one object based on the saliency map, the processing unit (114) is configured to detect a highest activity in the saliency map, wherein the highest activity corresponds to the position of the object.

12. The device as claimed in claims 8 to 11, wherein: In order to detect the highest activity in the saliency map, the processing unit (114) is configured to: creating a plurality of contours on the saliency map; and The density of each contour is calculated by taking the average pixel value within each contour, Among them, the contour with the highest density is selected as the location of the object.

13. The device as claimed in claims 8 to 12, wherein: The first feature map and the second feature map are generated by performing convolution operations on the received input image and the soft attention image using one or more feature detectors, respectively. The first feature map and the second feature map are a set of matrices, wherein the matrix set includes pixel values ​​representing at least one of the color, texture, shape and brightness of the received input image and the soft attention image.

14. The device as claimed in claims 8 to 13, in, The stacked feature map includes a merge matrix obtained from the set of matrices of the first feature map and the second feature map, wherein the merge matrix includes merged pixel values ​​representing at least one of color, texture, shape, and brightness of the received input image and the soft attention image, and The saliency map includes saliency matrices obtained from the merged matrices, wherein saliency pixel values ​​of the saliency matrices are regions within the received input image indicating at least one object to be detected in the received input image.

Images