Apparatus and method for generating soft attention-based splendor maps for object detection.

Abstract

The present invention describes an apparatus and method for detecting objects present in an image. The method states receiving an input image (602) including at least one object to be detected, and generating a soft attention image by processing the received image using a soft attention technique (604). The method further states processing the received image (606) with a first CNN model to extract a first feature map, and processing the soft attention image (608) with a second CNN model to extract a second feature map. The method further states integrating the first and second feature maps to generate a stacked feature map (610), processing the stacked feature map (612) to generate a saliency map, and detecting at least one object based on the saliency map (614).

Description

Technical Field

[0001] The subject matter described in this specification generally discloses an apparatus and a method for detecting an object existing inside an image. In other words, this application discloses a technique for generating a saliency map based on soft attention for object detection.

Background Art

[0002] There are many algorithms for object detection in images, such as region-based convolutional neural network (R-CNN), Fast R-CNN, Faster R-CNN, YOLOv2, and single-shot detector (SSD). These state-of-the-art algorithms search for target objects across the entire image. These algorithms are computationally inefficient and biologically infeasible. This leads to lower performance of evaluation metrics and a longer training period.

[0003] In real life, when a human searches for a specific object in an image, rapid eye movements occur between fixed points called saccades before reaching the target object. The trajectory of the eye movement varies according to the object of interest of the human observer. Each fixation point indicates the area of the image that the human has "paid attention" to. It is assumed that human visual attention moves along with the eye movement. However, existing technologies tend to process the entire input frame rather than focusing on / searching for the discriminative features of the input frame.

[0004] Therefore, an attention-based model for locating objects in images is needed.

Summary of the Invention

Means for Solving the Problems

[0005] This disclosure overcomes one or more drawbacks of the prior art and provides additional advantages described throughout this disclosure. Further features and advantages are realized through the technology of this disclosure. Other embodiments and aspects of this disclosure are described in detail herein and are deemed to be part of the claimed disclosure.

[0006] In one non-limiting embodiment of the present disclosure, a method for detecting objects present in an image is disclosed. The method includes receiving an input image containing at least one object to be detected, and generating a soft attention image by changing the brightness of the received image using a soft attention technique. The method further states that it processes the received input image using a first CNN model to extract a first feature map, and processes the soft attention image using a second CNN model to extract a second feature map. The method further states that it combines the first and second feature maps to generate a stacked feature map, processes the stacked feature map to generate a splendor map, and detects at least one object based on the splendor map.

[0007] In another, non-limiting embodiment, the Disclosure states that generating a soft attention image includes multiplying the brightness channel of a received input image by a multivariate Gaussian function such that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0008] In another non-limiting embodiment, the disclosure states that a soft attention image is generated by: converting a received input image into an HSV image including a brightness channel; multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is increased at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function; and, after the brightness of the HSV image has been modified, converting the modified HSV image into an RGB image to generate a soft attention image.

[0009] In another, non-limiting embodiment, the disclosure states that detecting at least one object based on a splendor map includes detecting the highest-order activity in the splendor map, where the highest-order activity corresponds to the location of the object.

[0010] In yet another, non-limiting embodiment, the disclosure states that detecting the highest-order activity in a splendor map includes creating multiple contours in the splendor map, calculating the density of each contour by calculating the average pixel value inside the contours, and selecting the contour with the highest density as the location of the object.

[0011] In yet another non-limiting embodiment, the disclosure states that the first and second feature maps are generated using one or more feature detectors that perform convolution operations on the received image and soft attention image, respectively, and that the first and second feature maps are sets of matrices, the sets of matrices containing pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and soft attention image.

[0012] In yet another non-limiting embodiment, the disclosure states that the stacked feature map includes a combined matrix obtained from a set of matrices of the first and second feature maps, the combined matrix includes combined pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image. The splendor map includes a splendor matrix obtained from the combined matrix, the splendor matrix includes splendor pixel values ​​which are regions within the received input image that indicate at least one object to be detected within the received input image.

[0013] In yet another non-limiting embodiment, the disclosure discloses a device for detecting objects present in an image. The device comprises an input / output (I / O) interface configured to receive an input image containing at least one object to be detected. The device further comprises a soft attention image generation unit operably coupled to the I / O interface. The soft attention generation unit is configured to generate a soft attention image by modifying the brightness of the received image using soft attention techniques. The device further comprises an attention search unit operably coupled to the I / O interface and the soft attention image generation unit. The attention search unit is configured to process the received image using a first CNN model to extract a first feature map, process the soft attention image using a second CNN model to extract a second feature map, integrate the first and second feature maps using an integration model to generate a stacked feature map, and process the stacked feature map using a map generation model to generate a splendor map. The device further comprises a processing unit operably coupled to the I / O interface, the soft attention image generation unit, and the attention search unit. The processing unit is configured to detect at least one object based on the spleness map.

[0014] In yet another, non-limiting embodiment, the Disclosure states that, in order to generate a soft attention image, the soft attention image generation unit is configured to multiply the brightness channel of a received input image by a multivariate Gaussian function so that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0015] In yet another non-limiting embodiment, the Disclosure states that, in order to generate a soft attention image, the soft attention image generation unit is configured to convert a received input image into an HSV image including a brightness channel, multiply the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is increased at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function, and after the brightness of the HSV image has been modified, convert the modified HSV image into an RGB image to generate a soft attention image.

[0016] In yet another, less restrictive embodiment of the present disclosure, in order to detect at least one object based on a splendor map, the processing unit is configured to detect the highest-order activity in the splendor map, the highest-order activity corresponding to the location of the object.

[0017] In yet another, non-limiting embodiment of the present disclosure, in order to detect the highest-order activity in the splendor map, the processing unit is configured to create multiple contours in the splendor map and calculate the density of each contour by calculating the average pixel value inside the contour. The contour with the highest density is selected as the location of the object.

[0018] In yet another non-limiting embodiment of the present disclosure, the application discloses that the first and second feature maps are generated using one or more feature detectors using convolution operations on the received input image and soft attention image, respectively. The first and second feature maps are sets of matrices, each set of matrices containing pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and soft attention image.

[0019] In yet another non-limiting embodiment, the Disclosure states that a stacked feature map includes a combined matrix obtained from a set of matrices of first and second feature maps, the combined matrix includes combined pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image, and a spleness map includes a spleness matrix obtained from the combined matrix, the spleness matrix includes spleness pixel values ​​which are regions within the received input image that indicate at least one object to be detected in the received input image.

[0020] The above overview is merely illustrative and not intended to be limiting. Further embodiments, features, and characteristics beyond those described above should become apparent with reference to the drawings and the detailed description below.

[0021] The accompanying drawings, incorporated into and constituting part of this disclosure, illustrate several exemplary embodiments and serve to illustrate the embodiments disclosed in conjunction with the following description. In these drawings, the leftmost digit of the reference number identifies the drawing in which the reference number first appears. The same numbering is used throughout all drawings to refer to similar features and elements. Hereinafter, several embodiments of apparatus and / or methods according to embodiments of the subject matter are described with reference to the accompanying drawings merely as examples. [Brief explanation of the drawing]

[0022] [Figure 1] This disclosure describes an apparatus for detecting objects present in an image, according to an embodiment of this disclosure. [Figure 2a] Exemplary RGB and soft attention images according to embodiments of this disclosure are shown, respectively. [Figure 2b] Exemplary RGB and soft attention images according to embodiments of this disclosure are shown, respectively. [Figure 3] This disclosure describes an architecture for generating a spleness map according to an embodiment of this disclosure. [Figure 4a]Shows the measured performance values of the proposed device according to an embodiment of the present subject matter. [Figure 4b] Shows the measured performance values of the proposed device according to an embodiment of the present subject matter. [Figure 5a] Shows the detection results of the proposed device and the prior art according to an embodiment of the present subject matter. [Figure 5b] Shows the detection results of the proposed device and the prior art according to an embodiment of the present subject matter. [Figure 6] Shows a flowchart of a method for detecting an object existing in an image according to an embodiment of the present subject matter.

[0023] It should be understood by those skilled in the art that the block diagrams in this specification represent conceptual diagrams of exemplary devices embodying the principles of the present subject matter. Similarly, flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. are substantially represented in a computer-readable medium and represent various processes that can be executed by such a computer or processor, regardless of whether such a computer or processor is explicitly shown.

Mode for Carrying Out the Invention

[0024] As used herein, the term "exemplary" is used in this specification to mean "serving as an example, instance or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" should not be construed as necessarily being more preferred or advantageous than other embodiments.

[0025] While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail below. However, it is not intended to limit the present disclosure to the specific forms disclosed, and on the contrary, it should be understood that the present disclosure encompasses all modifications, equivalents, and alternative forms falling within the scope of the present disclosure.

[0026] The terms “comprises,” “comprising,” “include(s),” or any other variation thereof are intended to mean non-exclusive inclusion, and an apparatus, device, or method containing a list of elements or steps does not mean that it contains only those elements or steps, but may contain other elements or steps that are not expressly enumerated or that are specific to such apparatus, device, or method. In other words, one or more elements of a device or device following “contains” do not, unless further constraints, exclude the presence of other or additional elements in that device or device.

[0027] In the following detailed description of embodiments of the Disclosure, references are made to the accompanying drawings which form part thereof and illustrate specific embodiments in which the Disclosure may be carried out. These embodiments are described in sufficient detail to enable those skilled in the art to carry out the Disclosure, and it should be understood that other embodiments may be used and modified without departing from the scope of the Disclosure. Therefore, the following description should not be construed as limiting.

[0028] The present invention is described in detail below this specification with reference to the accompanying drawings. In the following description, well-known functions or structures are not described in detail, as this would obscure the explanation with unnecessary details.

[0029] There are various methods for detecting objects present in an image, such as Region-based Convolutional Neural Network (R-CNN), Fast R-CNN, Faster R-CNN, YOLOv2, and Single Shot Detector (SSD). In R-CNN, the network receives approximately 2000 object region candidate inputs from a single image. Region proposals are generated by a selective search algorithm. Each region proposal is scaled to the size of the CNN's input. After scaling, each region proposal is fed into the CNN for feature extraction. The extracted features are then used to classify and predict bounding box coordinates using linear models and regressions of support vector machines.

[0030] Furthermore, in the fast R-CNN technique, region proposals are generated from a selective search algorithm, as is done in the R-CNN technique. These region proposals are fed into a Visual Geometry Group (VGG) convolutional network. Since each pixel in the output of the CNN network corresponds to a patch (region of interest) in the input image, the regions of interest in the input image (RoI) are projected at different sizes in the output of the fifth convolutional layer. Each RoI is scaled to a 7x7 square. Maximum pooling is performed on each of these RoIs to obtain a 49-dimensional vector of RoIs. Each of these 49-dimensional vectors is connected to a fully connected layer that gives a representation of this region proposal. The fully connected layer computes a probability distribution across possible object classes through softmax activation. Along with this, a regression layer is used to predict a new bounding box.

[0031] Furthermore, Faster R-CNN uses a CNN-based method instead of a selective search algorithm to generate region proposals. A full image is given as input to VGG-Net. The last layer of VGG-Net has an output of dimension (h × w × 512). A 3 × 3 kernel is applied to each of the (h × w) cells of the 512 feature maps, which gives rise to a 512-dimensional vector of cells. Assume that there is a number of bounding boxes (called anchor boxes) denoted by "k" with different sizes and aspect ratios, and a 512-dimensional vector for each location. The network then predicts the probability that a given anchor box centered at this location contains a possible object class. Each cell in the output of the last convolutional layer corresponds to a patch of the original image (due to the region in which each pixel in the input of the output of the last convolutional layer is affected). Each of these anchor boxes is at the center of the corresponding patch. The classifier outputs 1 for anchor boxes that have a reasonable overlap (intersection on unions greater than 0.7) with the true bounding box. Similarly, the regression model predicts the true bounding box from the anchor boxes. Once the region proposal is generated in this way, the rest of the process is the same as with Fast R-CNN.

[0032] YOLO (You Only Look Once) uses an end-to-end trained deep neural network for object detection. The network takes a full image and its corresponding label as input and predicts the location of objects in a new image. Before training, the labels for each image are created by creating (S×S) grids across the entire image. Each grid is assigned two random bounding boxes: their coordinates and the object's class. If the center of an object falls within a grid, that grid is considered to have detected an object. Due to its high speed, YOLO is a real-time object detector.

[0033] However, all of these object detection algorithms lack the human attentional mechanisms for detecting objects in images and do not provide the logic of the attentional mechanisms used by the human brain. Furthermore, in these techniques, the complete network does not undergo end-to-end training.

[0034] This disclosure addresses the problem described above and discloses a hybrid model for object detection. The hybrid model relies on paying attention to important discriminative features of the input frame and therefore does not focus on other irrelevant features in the input frame, just as the human brain works. The hybrid model uses an "attentional search" technique and focuses primarily on mimicking brain function, thereby focusing only on specific features that are most relevant to the output and therefore not on features that do not affect the output. In this way, the hybrid model avoids significant computational costs by only looking at the features necessary for the final output. The hybrid model utilizes time-difference error (TDE) as a primary learning rule to achieve optimal object detection. This mechanism also reduces the overall training time.

[0035] Referring to Figure 1, an apparatus 100 for detecting objects in an image according to an embodiment of the present disclosure is shown. In one embodiment, the apparatus 100 may include, but is not limited to, various entities such as an attention search unit 102, an input / output (I / O) interface 112, a processing unit 114, and a soft attention image generation unit 120. These entities are coupled together to communicate and operate with each other via a wired or wireless network. In an embodiment, the apparatus 100 may use an attention search-based deep Q neural network for object location. In another embodiment, the attention search unit 102 comprises a first convolutional neural network (CNN) model 104, a second CNN model 106, a map generation model 108, and an integration model 110. In an embodiment, CNN models 104, 106 are a type of neural network model that extracts higher-order representations of the image content. Unlike traditional image recognition where the user must define the image features themselves, CNNs take raw pixel data of an image and automatically extract features. In yet another embodiment, the processing unit 114 may comprise one or more processors 116 and memory 118.

[0036] The I / O interface 112 can receive an image containing at least one object to be detected by the device 100. The received image may be stored in a database (not shown in Figure 1). In embodiments, the image may be part of the device 100 or may be acquired by any imaging device, which may be any external imaging device. The received image is an RGB image. An RGB image, also called a true color image, can be stored as an M*N*I data array that defines the red, green, and blue color components for each individual pixel. An exemplary RGB image is shown in Figure 2a.

[0037] The soft attention image generation unit 120 can generate a soft attention image by changing the brightness of the received input image. In an embodiment, to generate a soft attention image, the image generation unit 120 can convert the received RGB image into an HSV image that includes a brightness channel. HSV stands for "Hue, Saturation, Value". HSV is used to measure the hue angle, saturation (or amount of grayness), and brightness (or value) of a color. The image generation unit 120 can change the brightness of the HSV image by multiplying the brightness channel of the received input image by a multivariate Gaussian f(y,x) (shown in the following formula). The brightness of the HSV image is changed so that the brightness is high at the center (y1,x1) of the multivariate Gaussian distribution and gradually decreases as you move away from the center (y1,x1). The position of the center is considered to be random. In an embodiment, the brightness of the received input image is changed so that it is high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. After changing the brightness of the HSV image, the image is converted back to an RGB image, as shown in Figure 2b. The generated RGB image is a soft attention image that can be stored in a database.

number

[0038] Furthermore, the first CNN model 104 of the attention search unit 102 takes the received image (original image) as input, and the second CNN model 106 takes the generated soft attention image as input. The received image and the soft attention image are processed in parallel by the first and second CNN models by passing through six convolutional layers to extract features. The first CNN model 104 may process the received original image to generate a first feature map, and the second CNN model 106 may process the generated soft attention image to generate a second feature map. In the embodiment, the feature map is generated by applying a filter or feature detector to the input image or the feature map of the previous input layer in the CNN using a convolution operation. The feature map is a set of matrices, and the pixel values ​​are calculated based on a high level of understanding of the presented input image, corresponding to its color, texture, object shape, and brightness, etc. Features may include, but are not limited to, hue, saturation, brightness, and color.

[0039] The output feature maps are passed to the integration model 110, which stacks the first and second feature maps to output a stacked feature map. In an embodiment, the integration model 110 may include a convolutional layer that takes both feature maps (first and second features) as input and outputs a 39 × 63 × 2-dimensional stacked map. The stacked feature map is processed by the map generation model 108 to generate a splendor map. In an embodiment, the map generation model 108 may be a deep Q-learning model that includes one or more convolutional layers. The deep Q-learning model may process the stacked map to generate a 39 × 63 × 1-dimensional splendor map, as shown in Figure 3. In an embodiment, the stacked feature map includes an integration matrix obtained from the set of matrices of the first and second feature maps, the integration matrix includes integrated pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image. In another embodiment, the splendor map includes a splendor matrix obtained from a merged matrix, the splendor matrix including splendor pixel values ​​which are regions within the received input image that indicate at least one object to be detected in the received input image. Figure 3 shows an architecture for generating a splendor map according to an embodiment of the present disclosure.

[0040] The deep Q-learning model of device 100 can be trained to generate spleness maps using the deep Q-learning process. For each input image in the training dataset, a reward matrix with dimensions of 640 × 1024 × 1 is created. In the reward matrix, the value "1" is assigned to pixel coordinates where a traffic sign (object) and soft attention overlap, the value "0.5" is assigned to pixel coordinates where only a traffic sign exists, and the value "0" is assigned to other locations. Subsequently, the reward matrix R t+1 It is resized to 39 × 63 × 1 to match the dimensions of the spleness map, which is the output from the network.

number

[0041] Here, (x,y) are the pixel coordinates of the input image. The model is trained on frames of RGB images. The dimensions of both inputs are 640 × 1024 × 3. To train the network, the time difference error (TDE) between the spleness maps of two consecutive frames at times 't' and 't+1' is calculated using Equation 2 shown below.

number

[0042] In the above equation, 'y' is the discount factor. The time difference error is added to the regularization cost using the L2 norm of the weights. The total cost of a batch of 32 image frames is then minimized by using a backpropagation algorithm with an optimizer to update the network weights. Hyperparameters such as the learning rate, sigma, gamma, and regularization coefficient beta are fine-tuned to values ​​of 0.00015, 25, 0.4, and 0.00001, respectively, to achieve the best result with the minimum total cost. The model is trained for 54 epochs.

[0043] During the test, a frame containing a traffic sign is provided to the device. The traffic sign is the object to be detected. The device outputs a 39×63×1-dimensional splendor map, which is resized to 640×1024×1-dimensional to match the height and width of the input image. The output splendor map has values ​​in the range of 0 to 1. The highest-order activity in the splendor map corresponds to the detected location of the traffic sign in the given image. In addition, one or more contours are created in the splendor map. In the embodiment, the OpenCV library is used to create contours in the splendor map. The density of each contour is calculated as the average pixel value inside the contour. The contour with the highest density is selected as the traffic sign detection. In the embodiment, OpenCV essentially finds locations with pixel values ​​close to 1 and creates peaks at all such high values ​​by collecting pixel values ​​around them. In this way, there are multiple peaks or contours in the map. Next, the average pixel value is calculated for each such contour. After calculating the average pixel value for each contour, any contour has the highest average pixel value that is considered to be the location of the target object.

[0044] Referring again to Figure 1, after generating the spleness map, the spleness map is processed by the processing unit 114 to detect at least one object. To detect an object, the processing unit 114 can create multiple contours on the spleness map and calculate the density of each contour by calculating the average pixel value inside the contour. The contour with the highest density is selected as the location of the object.

[0045] In this way, the device 100 can detect objects in an image quickly and with high accuracy. Furthermore, the training time of the device is also reduced compared to conventional techniques. The performance of device 100 is compared with other object detection algorithms on the same training and testing dataset.

[0046] In the embodiment, the center of the multivariate Gaussian function is first determined to be random, and then determined using the output of the generated splendor map. To this end, the disclosed model is trained to generate the most splendid region at the location of the target object in the image using a given reward system. The location of that most splendid region is taken as the center position. The multivariate Gaussian has high values ​​at the center and low values ​​farther away from the center. After multiplying such a matrix by a "value" or "splendor" channel, this creates a scenario with the same bias in the splendor channel. After converting such an HSV image to an RGB image, the Gaussian center allows for high values ​​at the target location and low values ​​at non-target locations, thus creating a soft attention version of the image frame. This biased nature of the soft attention input helps the model learn the same bias towards the target location and predict higher-order splendid regions at the target location.

[0047] The device disclosed in this disclosure successfully achieves a 23.52% increase in Mean Average Precision (mAP) compared to YOLOv2, a state-of-the-art object detection model. Not only is the device significantly more powerful than YOLOv2, but it also requires considerably less training time.

[0048] The disclosed device, as shown in Table 1, exhibits a 23.52% higher mAP (maturity performance) than YOLOv2 when tested with a dataset of 1,893 frames of road video including both day and night images. The device requires approximately 18 hours of training to reach an mAP of 41.53%, compared to approximately 24 hours of training to reach an mAP of 18.01% on the same test dataset.

[0049] [Table 1]

[0050] Figures 4a and 4b show that for the disclosed apparatus, the mAP percentage of the validation dataset (3,785 image frames) increases per epoch as the absolute value of the training loss decreases. Figure 4a shows the training loss (absolute value) and validation mAP (%) of the disclosed apparatus per epoch, and Figure 4b shows the training loss (absolute value) and validation mAP (%) of YOLOv2 per iteration in blue and red, respectively.

[0051] Furthermore, the proposed device not only detects objects such as traffic signs at very long distances, but also detects these signs even when they are difficult to see due to bad weather during the day or light from oncoming vehicles at night, unlike conventional techniques as shown in Figures 5a-5c and 5e-5g. The device also provides more accurate detection than conventional techniques, as shown in Figures 5d and 5h. Figures 5a-5d show test detection images of the disclosed device 100, where the red boundary box represents the detection box and the blue boundary box represents the ground truth box of the traffic sign. Figures 5e-5h show test detection images of the conventional technique YOLOv2, where the pink boundary box is the detection box of the traffic sign and the blue boundary box is the ground truth boundary box of the traffic sign.

[0052] In this way, the proposed device can detect objects in images in a more accurate and efficient manner. Furthermore, the device requires less training time, which also improves the device's performance. In addition, the device can also detect objects at very long distances, and can detect objects even when they are not clearly visible.

[0053] Figure 6 shows a flowchart of a method 600 for detecting an object in an image according to an embodiment of the present disclosure. In step 602, the method 600 states that it receives an input image containing at least one object to be detected. In embodiments, the image may be part of apparatus 100 or may be acquired by any imaging device which may be any external imaging device.

[0054] The method of step 604 states that a soft attention image is generated by changing the brightness of a received image. In one embodiment, a soft attention image can be generated by multiplying the brightness channel of a received input image by a multivariate Gaussian function, so that the brightness of the received input image is changed to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. In another embodiment, the received input image may be an RGB image that can be processed to generate a soft attention image. To generate a soft attention image, the received RGB image can be converted to an HSV image that includes a brightness channel. HSV stands for "Hue, Saturation, Value". HSV is used to measure the hue angle, saturation (or amount of grayness), and brightness (or value) of a color. The HSV image is further processed to change the brightness of the HSV image by multiplying the brightness channel of the HSV image by a multivariate Gaussian function, so that the brightness of the HSV image is changed to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. Furthermore, the image is converted back to an RGB image to generate a soft attention image.

[0055] In step 606, method 600 declares that it processes the received image using a first CNN model 104 to extract a first feature map. In step 608, the method declares that it processes the soft attention image using a second CNN model 106 to extract a second feature map. In the embodiment, the first and second feature maps are generated using one or more feature detectors or filters by performing convolutional operations on the received input image and the soft attention image, respectively. The extracted feature map is a set of matrices, where pixel values ​​represent one or more of color, texture, shape, and brightness. The received image and the soft attention image are processed in parallel by the first and second CNN models by passing through six convolutional layers to extract features.

[0056] The method in step 610 states that the first and second feature maps are merged to generate a stacked feature map. In an embodiment, the first and second feature maps may be generated by a merging model 110 which includes a convolutional layer that takes both feature maps as input to output a 39 × 63 × 2-dimensional stacked map. In an embodiment, the stacked feature map includes a merged matrix obtained from a set of matrices of the first and second feature maps, the merged matrix includes merged pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image.

[0057] In step 612, method 600 states that it processes a stacked feature map using a Deep Q-learned model to generate a splendor map. The stacked feature map may be processed by a map generation model 108, which is a Deep Q-learned model. The model may include one or more convolutional layers that process the stacked map to generate a 39 × 63 × 1-dimensional splendor map. In embodiments, the splendor map includes a splendor matrix obtained from a combined matrix, the splendor matrix includes splendor pixel values, which are regions within the received input image that indicate at least one object to be detected in the received input image. In step 614, method further states that it detects objects based on the splendor map. The splendor map is processed by a processing unit 114 so that at least one object can be detected. According to this method, detecting at least one object based on the splendor map includes detecting the highest-order activity in the splendor map, where the highest-order activity corresponds to the location of the object. In the embodiment, detecting the highest-order activity in the splendor map involves creating multiple contours in the splendor map and calculating the density of each contour by calculating the average pixel value inside the contour. The contour with the highest density is selected as the location of the object. In the embodiment, contours are created in the splendor map using the OpenCV library. The density of each contour is calculated as the average pixel value inside the contour. The contour with the highest density is selected as the detection of a traffic sign. In the embodiment, OpenCV essentially finds locations with pixel values ​​close to 1 and creates peaks at all such high values ​​by collecting pixel values ​​around them. In this way, there are multiple peaks or contours in the map. Next, the average pixel value is calculated for each such contour. After calculating the average pixel value for each contour, any contour has the highest average pixel value that is considered to be the location of the target object.

[0058] In this way, this method can detect objects in an image quickly and with high accuracy. Furthermore, this method can detect objects at very great distances, and even when they are not clearly visible.

[0059] The illustrated steps are presented to illustrate the exemplary embodiments shown, and it should be anticipated that ongoing technological developments will alter the manner in which specific functions are performed. These examples are presented herein for illustrative purposes only and are not limiting. Furthermore, the boundaries of functional configuration blocks are arbitrarily defined herein for the convenience of explanation. Alternative boundaries can be defined as long as the specified functions and their relationships are adequately performed.

[0060] Alternative forms (including equivalents, extensions, modifications, and deviations of those described herein) will be apparent to those skilled in the art in the relevant field based on the teachings contained herein. Such alternative forms are included within the scope of the disclosed embodiments.

[0061] Furthermore, one or more computer-readable storage media may be used when implementing embodiments consistent with this disclosure. Computer-readable storage media refers to any type of physical memory capable of storing processor-readable information or data. Thus, computer-readable storage media can store instructions executed by one or more processors, including instructions causing a processor(s) to perform steps or stages consistent with some embodiments described herein. The term “computer-readable media” should be understood to include tangible items, excluding carrier and transient signals, i.e., non-transient. Some examples include random-access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, and any other known physical storage media.

[0062] Suitable processors / controllers include, for example, general-purpose processors, dedicated processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines.

[0063] The following examples illustrate further technical aspects of the devices, systems, and methods described above and should not be construed as claims. These examples may be further combined with any of the devices, systems, and methods described above, or any of the original claims.

[0064] Example 1 is a method for detecting objects present in an image, comprising: receiving an input image containing at least one object to be detected (602); generating a soft attention image by changing the brightness of the received input image (604); processing the received input image using a first CNN model (606) to extract a first feature map; processing the soft attention image using a second CNN model (608) to extract a second feature map; integrating the first and second feature maps to generate a stacked feature map (610); processing the stacked feature map (612) to generate a splendor map; and detecting at least one object based on the splendor map (614).

[0065] In Example 2, the subject of Example 1 may optionally include generating a soft attention image by multiplying the brightness channel of the received input image by a multivariate Gaussian function, such that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0066] In Example 3, the themes of Examples 1 and 2 may optionally include: converting a received input image into an HSV image including a brightness channel; multiplying the brightness channel of the HSV image by a multivariate Gaussian function so that the brightness of the HSV image is increased at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function; and, after changing the brightness of the HSV image, converting the modified HSV image into an RGB image to generate a soft attention image.

[0067] In Example 4, the subject of Examples 1-3 may optionally include detecting the highest-order activity in the splendor map, wherein the highest-order activity corresponds to the location of an object.

[0068] In Example 5, the themes of Examples 1-4 may optionally include creating multiple contours in the spleness map and calculating the density of each contour by calculating the average pixel value inside the contour, with the contour having the highest density being selected as the location of the object in the received input image.

[0069] In Example 6, the subjects of Examples 1-5 may optionally include first and second feature maps having a set of matrices, the set of matrices containing pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and soft attention image.

[0070] In Example 7, the subject of Examples 1-6 may optionally include a unified matrix obtained from a set of first and second feature map matrices, the unified matrix containing unified pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image; the splendor map includes a splendor matrix obtained from the unified matrix, the splendor matrix containing splendor pixel values ​​which are regions within the received input image that indicate at least one object to be detected in the received input image.

[0071] Example 8 is a device for detecting objects present in an image, comprising an input / output (I / O) interface (112) configured to receive an input image containing at least one object to be detected, a soft attention image generation unit (120) operably coupled to the I / O interface (112), the soft attention image generation unit (120) configured to generate a soft attention image by changing the brightness of the received input image, and an attention search unit (102) operably coupled to the I / O interface (112) and the soft attention image generation unit (120), the attention search unit (102) being a first CNN model (104) The apparatus includes an attention search unit (102) configured to process an input image received using a CNN model (106) to extract a first feature map, process a soft attention image using a second CNN model (106) to extract a second feature map, integrate the first and second feature maps using an integration model (110) to generate a stacked feature map, and process the stacked feature map using a map generation model (108) to generate a splendor map; an I / O interface (112); a soft attention image generation unit (120); and a processing unit (114) operably coupled with the attention search unit (102), wherein the processing unit (114) is configured to detect at least one object based on the splendor map.

[0072] In Example 9, the subject of Example 8 may optionally include multiplying the brightness channel of the received input image by a multivariate Gaussian function so that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function.

[0073] In Example 10, the subject matter of Examples 8 and 9 may optionally be configured to generate a soft attention image. This could include the soft attention image generation unit (120) converting a received input image into an HSV image including a brightness channel, multiplying the brightness channel of the HSV image by a multivariate Gaussian function, modifying the brightness of the HSV image so that it is high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function, and then converting the modified HSV image into an RGB image to generate a soft attention image.

[0074] In Example 11, the themes of Examples 8-10 may optionally include the configuration in which the processing unit (114) detects the highest-order activity in the splendor map, the highest-order activity corresponding to the location of an object.

[0075] In Example 12, the themes of Examples 8-11 can optionally include creating multiple contours in the splendor map and calculating the density of each contour by calculating the average pixel value inside the contour, with the contour having the highest density being selected as the object's location.

[0076] In Example 13, the themes of Examples 8-12 may optionally include the fact that the first and second feature maps are generated using one or more feature detectors that perform convolution operations on the received input image and soft attention image, respectively, and the first and second feature maps are sets of matrices, the sets of matrices containing pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and soft attention image.

[0077] In Example 14, the subject matter of Examples 8-13 may optionally include a matrix obtained from a set of first and second feature map matrices, the unified matrix including unified pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image, the spleness map including a spleness matrix obtained from the unified matrix, the spleness matrix including spleness pixel values ​​which are regions within the received input image that indicate at least one object to be detected in the received input image. While this application relates to the invention described in the claims, it also includes the following other aspects. 1. A method for detecting objects present in an image, The system receives an input image containing at least one object to be detected (602), (604) A soft attention image is generated by changing the brightness of the received input image. The received input image is processed (606) using a first convolutional neural network (CNN) model to extract a first feature map. The soft attention image is processed using a second CNN model (608) to extract a second feature map. Integrating the first and second feature maps to generate a stacked feature map (610), The stacked feature map is processed (612) to generate a sampling map, and Detecting the at least one object based on the sampling map (614), Methods that include... 2. The generation of the aforementioned soft attention image is The brightness channel of the received input image is multiplied by a multivariate Gaussian function, so that the brightness of the received input image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. The method described in item 1 above, including the method described in item 1 above. 3. The generation of the aforementioned soft attention image is The received input image is converted into an HSV image including a brightness channel. The brightness channel of the HSV image is multiplied by a multivariate Gaussian function, so that the brightness of the HSV image is modified to be high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. and After changing the brightness of the HSV image, the modified HSV image is converted to an RGB image to generate the soft attention image. The method described in 1 or 2 above, including the method described in 1 or 2 above. 4. Detecting the at least one object based on the aforementioned sampling map is, To detect the highest-order activity in the spleness map, wherein the highest-order activity corresponds to the location of the object. The method described in any one of the above 1 to 3, including the method described in 1 to 3 above. 5. Detecting the highest-order activity in the aforementioned spleness map is, Creating multiple contours in the aforementioned spleness map, and This includes calculating the density of each contour by calculating the average pixel value inside the contour, The method according to any one of 1 to 4 above, wherein the contour having the highest density is selected as the position of the object in the received input image. 6. The first and second feature maps are generated using one or more feature detectors that perform convolution operations on the received input image and the soft attention image, respectively. The method according to any one of 1 to 5 above, wherein the first and second feature maps are a set of matrices, and the set of matrices includes pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and the soft attention image. 7. The stacked feature map includes a unified matrix obtained from the set of matrices of the first and second feature maps, the unified matrix includes unified pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image. The method according to any one of 1 to 6 above, wherein the spleness map includes a spleness matrix obtained from the unified matrix, and the spleness matrix includes spleness pixel values ​​which are regions within the received input image that indicate the at least one object to be detected in the received input image. 8. A device for detecting objects present in an image, An input / output (I / O) interface (112) configured to receive an input image containing at least one object to be detected, A soft attention image generation unit (120) is operably coupled to the I / O interface (112), wherein the soft attention image generation unit (120) is configured to generate a soft attention image by changing the brightness of the received input image, An attention search unit (102) is operably coupled to the I / O interface (112) and the soft attention image generation unit (120), wherein the attention search unit (102) is The received input image is processed using a first convolutional neural network (CNN) model (104) to extract a first feature map. The soft attention image is processed using a second CNN model (106) to extract a second feature map. Using the integrated model (110), the first and second feature maps are integrated to generate a stacked feature map. The system is configured to process the stacked feature map using a map generation model (108) to generate a sampling map. Attention search unit (102), A processing unit (114) operably coupled with the I / O interface (112), the soft attention image generation unit (120), and the attention search unit (102), wherein the processing unit (114) is configured to detect the at least one object based on the spleness map, A device including a device. 9. In order to generate the aforementioned soft attention image, the soft attention image generation unit (120) The apparatus according to 8, wherein the brightness channel of the received input image is multiplied by a multivariate Gaussian function, and the brightness of the received input image is modified so that it is high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. 10. In order to generate the aforementioned soft attention image, the soft attention image generation unit (120) The received input image is converted into an HSV image including a brightness channel. The brightness channel of the HSV image is multiplied by a multivariate Gaussian function, and the brightness of the HSV image is modified so that it is high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. The apparatus according to 8 or 9 above, configured to change the brightness of the HSV image, and then convert the changed HSV image to an RGB image to generate the soft attention image. 11. To detect the at least one object based on the spleness map, the processing unit (114) is configured to detect the highest-order activity in the spleness map, wherein the highest-order activity corresponds to the location of the object, according to any one of the devices described in 8 to 10 above. 12. To detect the highest-order activity in the spleness map, the processing unit (114) Create multiple contours in the aforementioned spleness map, The system is configured to calculate the density of each contour by calculating the average pixel value inside the contour, The apparatus according to any one of 8 to 11 above, wherein the contour having the highest density is selected as the position of the object. 13. The first and second feature maps are generated using one or more feature detectors that perform convolution operations on the received input image and soft attention image, respectively. The apparatus according to any one of 8 to 12 above, wherein the first and second feature maps are a set of matrices, and the set of matrices includes pixel values ​​representing at least one of the color, texture, shape, and brightness of the received input image and the soft attention image. 14. The stacked feature map includes a unified matrix obtained from the set of matrices of the first and second feature maps, the unified matrix includes unified pixel values ​​representing at least one of the color, texture, shape, and brightness of both the received input image and the soft attention image. The apparatus according to any one of 8 to 13, wherein the spleness map includes a spleness matrix obtained from the unified matrix, the spleness matrix includes spleness pixel values ​​which are regions within the received input image that indicate the at least one object to be detected in the received input image.

Claims

1. A method for detecting objects present in an image, (a) A step (602) of receiving an input image which includes at least one object to be detected, wherein the input image is an RGB image, (b) A step (604) to generate a soft attention image for identifying an area of ​​an image to be paid attention to, the following: (b-1) A step of converting the received input image into an HSV image including a brightness channel, (b-2) A step of multiplying the brightness channel of an HSV image by a multivariate Gaussian function, wherein the brightness of the HSV image is modified so that it is high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function, (b-3) After changing the brightness of the HSV image, a step is made to convert the modified HSV image into an RGB image in order to generate a soft attention image. A process (604) including, (c) A step (606) of processing the received input image using a first convolutional neural network (CNN) model in order to extract a first feature map. (d) A step (608) of processing the soft attention image using a second CNN model in order to extract a second feature map. (e) A step (610) of integrating the first and second feature maps in order to generate a stacked feature map, (f) A step (612) of processing the stacked feature map in order to generate a sampling map, and (g) A step (614) of detecting the at least one object based on the sampling map, Methods that include...

2. Detecting the at least one object based on the sampling map is, To detect the highest-order activity in the spleness map, wherein the highest-order activity corresponds to the location of the object. The method according to claim 1, including the method described in claim 1.

3. Detecting the highest-order activity in the aforementioned spleness map is, Creating multiple contours in the aforementioned spleness map, and This includes calculating the density of each contour by calculating the average pixel value inside the contour, The method according to claim 2, wherein the contour having the highest density is selected as the position of the object in the received input image.

4. The first and second feature maps are generated using one or more feature detectors that perform convolution operations on the received input image and the soft attention image, respectively. The method according to claim 1, wherein the first and second feature maps are a set of matrices, the set of matrices includes pixel values ​​generated based on at least one piece of information from the color, texture, shape, and brightness of the received input image and the soft attention image.

5. The stacked feature map includes a unified matrix obtained from the set of matrices of the first and second feature maps, the unified matrix includes unified pixel values ​​generated based on at least one piece of information from the color, texture, shape, and brightness of both the received input image and the soft attention image. The method according to claim 4, wherein the spleness map includes a spleness matrix obtained from the unified matrix, the spleness matrix includes spleness pixel values ​​which are regions within the received input image that indicate the at least one object to be detected in the received input image.

6. A device for detecting objects present in an image, An input / output (I / O) interface (112) configured to receive an input image containing at least one object to be detected, wherein the input image is an RGB image, A soft attention image generation unit (120) is operably coupled to the I / O interface (112), wherein the soft attention image generation unit (120) is configured to generate a soft attention image for identifying an area of ​​an image to be noticed, and in order to generate the soft attention image, - It is configured to change the brightness of the received input image. moreover, - The received input image is converted into an HSV image including a brightness channel. - The brightness channel of the HSV image is multiplied by a multivariate Gaussian function, and the brightness of the HSV image is modified so that it is high at the center of the multivariate Gaussian function and decreases with distance from the center of the multivariate Gaussian function. - After changing the brightness of the HSV image, convert the modified HSV image to an RGB image in order to generate a soft attention image. A soft attention image generation unit (120) is configured as follows, An attention search unit (102) is operably coupled to the I / O interface (112) and the soft attention image generation unit (120), wherein the attention search unit (102) is The received input image is processed using a first convolutional neural network (CNN) model (104) to extract a first feature map. The soft attention image is processed using a second CNN model (106) to extract a second feature map. The first and second feature maps are combined using the integrated model (110) to generate a stacked feature map. The system is configured to process the stacked feature map using a map generation model (108) to generate a sampling map. Attention search unit (102), A processing unit (114) operably coupled with the I / O interface (112), the soft attention image generation unit (120), and the attention search unit (102), wherein the processing unit (114) is configured to detect the at least one object based on the spleness map, A device including a device.

7. The apparatus according to claim 6, wherein, in order to detect the at least one object based on the spleness map, the processing unit (114) is configured to detect the highest-order activity in the spleness map, the highest-order activity corresponding to the location of the object.

8. To detect the highest-order activity in the spleness map, the processing unit (114) Create multiple contours in the aforementioned spleness map, The system is configured to calculate the density of each contour by calculating the average pixel value inside the contour, The apparatus according to claim 7, wherein the contour having the highest density is selected as the position of the object.

9. The first and second feature maps are generated using one or more feature detectors that perform convolution operations on the received input image and soft attention image, respectively. The apparatus according to claim 6, wherein the first and second feature maps are a set of matrices, the set of matrices includes pixel values ​​generated based on at least one piece of information from the color, texture, shape, and brightness of the received input image and the soft attention image.

10. The stacked feature map includes a unified matrix obtained from the set of matrices of the first and second feature maps, the unified matrix includes unified pixel values ​​generated based on at least one piece of information from the color, texture, shape, and brightness of both the received input image and the soft attention image. The apparatus according to claim 9, wherein the spleness map includes a spleness matrix obtained from the unified matrix, the spleness matrix includes spleness pixel values ​​which are regions within the received input image that indicate the at least one object to be detected in the received input image.

Images