Single instruction multiple data processing unit and method for performing object detection in an image

Abstract

The present invention relates to a single instruction multiple data processing unit and a method for performing object detection in an image. The single instruction multiple data processing unit comprises: a set of computing units, the set of computing units being operable to execute a plurality of classifier sequences in parallel, each classifier sequence comprising a plurality of classifier programs, and each classifier program comprising the same instructions as other classifier programs in each of the plurality of classifier sequences; wherein each computing unit is configured to independently maintain data identifying image regions and features being tested on the computing unit, and each classifier program is arranged to access the data, test the identified features against the identified image regions, and update the data, so that the computing unit is operable to test different features against different image regions simultaneously.

Description

[0001] This application is a divisional application of the original invention patent application with application number 201510919951.3 (application date: December 11, 2015, invention name: single instruction multiple data processing unit and method for performing object detection in an image). Technical Field

[0002] The present invention relates to a SIMD processing unit and a method for performing object detection. Background Art

[0003] Many modern methods for performing automatic face detection are based on the Viola-Jones object detection framework, which is described in the paper "Robust realtime face detection" by P. Viola and M. Jones (International Journal of Computer Vision, Volume 57, Chapter 2, Pages 137-154, 2004). The Viola-Jones framework operates on a set of image regions or "sub-windows" defined for an image, each of which has a different position, scale, or rotation angle within the image to allow detection of faces at different positions or with different sizes and rotation angles. A cascade of binary classifiers operates on each sub-window to detect whether the sub-window is likely to surround a face in the image. Each binary classifier performs a test on the sub-window to determine whether the sub-window satisfies one or more simple visual features (commonly referred to as "Haar-like features"). If the sub-window satisfies one or more simple visual features, the binary classifier accepts (passes) the sub-window and moves to the next binary classifier in the cascade. When all binary classifiers in the cascade accept a subwindow, the subwindow becomes a candidate for a face in the image being searched. If any binary classifier in the cascade rejects a subwindow, no further processing is performed on that subwindow, the cascade terminates and cascade processing begins again with the next subwindow.

[0004] Figure 2 Four visual features 201-204 that are commonly used in the Viola-Jones framework are shown in FIG. Figure 2Each of the features 201 to 204 shown in 201 visually represents how the pixel values ​​of the subwindow are processed to test the subwindow. For example, feature 201 can represent the composition of the first binary classifier in the cascade of classifiers and is calculated by subtracting the sum of the pixel values ​​of the subwindows located in the shadow area of ​​the feature (the right hand side of the subwindow) from the sum of the pixel values ​​of the subwindows located in the unshaded area of ​​the feature (the left hand side of the subwindow). If the feature is evaluated to a value exceeding a predefined threshold (usually established by training a face detector on a test image), the subwindow is considered to satisfy the visual feature and the binary classifier accepts the subwindow. Typically, a binary classifier operates on a subwindow of an image (e.g., the pixel values ​​can be brightness values) that has been processed to represent only brightness or grayscale information. In this way, the cascaded binary classifier works to identify specific patterns of contrast in an image that indicate facial features.

[0005] In order to improve the performance of the face detection system, binary classifier operations performed according to the Viola-Jones object detection framework can be performed in parallel at a graphics processing unit (GPU) by assigning multiple groups of threads to the GPU. However, this method may cause the parallel processing elements of the GPU to be idle for a large proportion of the time. This is because the parallel processing of a group of threads is generally not completed until the processing of each thread of the group of threads is completed, and if the sub-window does not accept (fail) the first binary classifier of the cascade, then any given thread operating on the sub-window can be terminated almost immediately, or if it can complete the processing of all binary classifiers of the cascade, then the sub-window represents a face candidate. This underutilization of processing resources presents difficulties in performing face detection in real time using the Viola-Jones framework, especially for mobile and low-power platforms.

[0006] Previous efforts to address this problem have attempted to decompose the performance of the binary classifier into multiple stages, such as the Nvidia CUDA implementation described in "GPU Computing Gems", Wen-mei W. Hwu, Elsevier, 2011, pp. 534-541. However, this only partially addresses the problem, introduces additional overhead for compressing data between multiple stages, and has the disadvantage of low efficiency during the earlier processing stages. Summary of the invention

[0007] According to a first aspect of the present invention, there is provided a single instruction multiple data processing unit for performing object detection in an image by testing a plurality of object features in a plurality of image regions, the single instruction multiple data processing unit comprising:

[0008] a set of computing units operable to execute in parallel a plurality of classifier sequences, each classifier sequence comprising a plurality of classifier programs, and each classifier program comprising the same instructions as other classifier programs in each of the plurality of classifier sequences;

[0009] wherein each computing unit is configured to independently maintain data identifying the image region and features being tested on the computing unit, and each classifier program is arranged to access the data, test the identified features against the identified image region, and update the data so that the computing unit can operate to simultaneously test different features against different image regions.

[0010] Each classifier sequence may be configured to perform a sequence of tests for object features in the image region, each classifier program of the classifier sequence being configured to perform the sequence of tests.

[0011] Each classifier program can be configured to accept or not accept image regions, and includes instructions configured to:

[0012] When an image region is accepted, updating data maintained for the computational unit executing the classifier process so that the next classifier process in the sequence of classifiers tests the next feature for the same image region;

[0013] When an image region is not accepted, data maintained for the computational unit executing the classifier procedure is updated so that the next classifier procedure in the classifier sequence tests the first feature for the next image region.

[0014] The single instruction multiple data processing unit may also include a memory accessible to the group of computing units, the memory being arranged to store an index representing a search range passing through the multiple image regions, and each classifier program including instructions configured to perform the following operations: when an image region is not accepted, updating the data maintained for the computing unit that is executing the classifier program by atomically incrementing the index to update the search range, and causing an identifier of the next image region to be returned to the classifier program.

[0015] Each classifier program is configured to use its data identifying image regions to look up data in the identified regions in the image for which object detection is being performed.

[0016] Each classifier program of the classifier sequence is a single program arranged to be executed iteratively at its computing unit to test the identified features against the identified regions and update the data.

[0017] The single instruction multiple data processing unit may further include: another group of computing units, which is capable of operating to execute the multiple classifier sequences, wherein the group of computing units forms a first computing group and the other group of computing units forms a second computing group, and wherein the multiple image regions are divided into multiple subsets of image regions, and each of the first computing group and the second computing group is assigned a different one of the multiple subsets of image regions for testing.

[0018] The single instruction multiple data processing unit may also include: a global memory, the global memory storing a global index, the global index representing the search range performed by the first calculation group and the second calculation group through the multiple subsets of the image area; wherein each classifier program includes instructions configured to perform the following operations: when the subset of the image area is exhausted for its associated calculation group, a new subset of the image area is assigned to the associated calculation group according to the global index.

[0019] Each classifier program includes instructions configured to perform the following operations: by atomically updating the global index, the global index assigns a new subset of the image area to the relevant calculation group to update the search range through the multiple subsets of the image area, and causes the identifier of the next subset of the image area to be returned to the classifier program.

[0020] The index may be updated when a new subset of the image area is assigned to the first computation group.

[0021] The memory of the first computing group can be accessed by the computing group but not by other computing groups.

[0022] Each image region differs from the other images in one or more of its position, size, and orientation in the frame.

[0023] Each object feature is a Haar-like feature, and each classifier program is configured to determine whether the feature satisfies a predefined threshold.

[0024] A machine readable code may be provided that defines a single instruction multiple data processing unit. A machine readable storage medium having non-transitory machine readable code encoded thereon that defines a single instruction multiple data processing unit may be provided.

[0025] According to a second aspect of the present invention, there is provided a method for performing object detection in an image at a single instruction multiple data processor, the single instruction multiple data processor having a set of computing units, the set of computing units being arranged to execute a plurality of classifier sequences in parallel, each classifier sequence comprising a plurality of classifier programs, the plurality of classifier programs having the same instructions as other classifier programs in each of the plurality of classifier sequences, and the method comprising the following steps:

[0026] executing the classifier procedure of each sequence in parallel at each computing unit in the set of computing units to test for a plurality of object features in a plurality of image regions;

[0027] Each classifier procedure:

[0028] independently maintaining data identifying image regions and features tested by said classifier procedure;

[0029] accessing said data;

[0030] testing the identified features against the identified image region; and

[0031] The data is updated such that the computing unit is operable to test different features simultaneously for different image regions.

[0032] When testing the identified features against the identified image regions, each classifier procedure may:

[0033] accepting the image region and updating data identifying the image region and the feature being tested so that the next classifier process in the sequence of classifiers tests the next feature for the same image region; or

[0034] The image region is not accepted, and data identifying the image region and feature being tested is updated so that the next classifier process in the sequence of classifiers tests the first feature for the next image region.

[0035] The method also includes the following steps: maintaining an index representing a search range of the set of computing units through the plurality of image regions, and when an image region is not accepted, each classifier program updates the search range by atomically incrementing the index to update data identifying the image region and features being tested, and returning an identifier of the next image region to the classifier program.

[0036] Each classifier program may use its data of identified image regions to look up data in the identified image regions in the image for which object detection is being performed.

[0037] The method may comprise the steps of iteratively executing each classifier procedure at its computing unit to test the identified features against the identified regions and to update the data.

[0038] Each classifier program may be executed at a computing unit of a group of computing units in the plurality of groups of computing units, and the method may further comprise the steps of:

[0039] dividing the plurality of image regions into a plurality of subsets of image regions; and

[0040] A different one of the plurality of subsets of the image regions is allocated for testing at each group of computing units.

[0041] The method may further comprise the following steps:

[0042] maintaining a global index representing a search range by the plurality of groups of computing units across the plurality of subsets of the image region; and

[0043] When the classifier procedure exhausts said subset of image regions for its associated calculation group, a new subset of image regions is caused to be assigned to said associated calculation group according to said global index.

[0044] Each classifier program may update a search range through the plurality of subsets of image regions by atomically updating the global index so that a new subset of image regions is assigned to the relevant computation group and so that an identifier of a next subset of image regions is returned to the classifier program.

[0045] Machine readable code for implementing the method may be provided.A machine readable storage medium having non-transitory machine readable code encoded thereon for implementing the method may be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0047] Figure 1 The processing of a cascade of classifiers at a compute unit of a conventional SIMD processor is shown.

[0048] Figure 2 Shown are four Viola-Jones feature types that are commonly used in a cascade of classifiers.

[0049] Figure 3 is a schematic diagram of a processing unit suitable for performing object detection in parallel at a set of computing units.

[0050] Figure 4 It is shown in Figure 3 Flowchart of the operation of a classifier program operating at a computing unit of a processing unit shown in .

[0051] Figure 5 Shown in Figure 3 4. Processing of a cascade of classifiers at the computational units of the processing units shown in FIG.

[0052] Figure 6 Sub-windows of a given size at multiple positions within a frame are shown.

[0053] Figure 7 is a schematic diagram of a processing unit suitable for performing object detection in parallel at multiple groups of computational units. DETAILED DESCRIPTION

[0054] The following description is provided by way of example to enable those skilled in the art to make and use the invention. The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0055] A single instruction multiple data (SIMD) processing unit and method for performing object detection at a SIMD processor are provided. In the examples described herein, the object detection performed is the detection of faces in an image, but the principles described herein are not limited to the detection of faces. It will be apparent that the same principles can be applied to the detection of other object types in an image (e.g., other parts of the human body, car license plates, and animals, to name a few examples). The examples set forth herein are described with respect to the Viola-Jones framework for performing object detection. However, it will be appreciated that the principles described herein apply to any object detection system that utilizes a cascade classifier to identify candidate objects in an image.

[0056] Performing object detection using a sequence of classifiers as described herein includes testing one or more object features in a subject image region (hereinafter also referred to as a sub-window), the sub-window representing the region of the image where object detection is to be performed.

[0057] By way of context, it is instructive to consider how a conventional SIMD processor might perform parallel processing of a sequence of classifiers for detecting faces in an image according to the Viola-Jones framework. Figure 1An example of such conventional processing is shown in FIG. In this example, each sequence includes six stages, each of which tests object features F0 to F5 for each sub-window defined for the image. The SIMD processor includes four computing units, each of which runs the same instruction in parallel for different sub-windows, so that at any given time, all computing units evaluate the same feature (or the computing unit can be idle after its classifier sequence terminates). Typically, there are a large number of sub-windows defined for the subject image. The computing units of the SIMD processor are forced to execute the same instruction simultaneously, so that the sub-windows are assigned to computing units CU0 to CU3 for simultaneous processing in groups of four sub-windows at a time.

[0058] exist Figure 1 , the processing of the first group of sub-windows and the second group of sub-windows at computing units CU0 to CU3 is shown. In the first group, it can be seen that sub-window SW0 accepts all features F0 to F5 of the classifier sequence executed by computing unit CU0. Thus, sub-window SW0 represents a candidate window where a face may be present. Each of sub-windows SW1-SW3 does not accept the classifier feature test, so that the classifier sequence running at its corresponding computing unit is terminated. This makes those computing units idle until computing unit CU0 continues to perform useful work. For example, sub-window SW2 processed at computing unit CU2 does not accept the second classifier feature F1, and is therefore idle, while other computing units execute one or more of features F2-F5.

[0059] Only when the processing of the first set of sub-windows is terminated at all computing units can the loop configured to execute the instructions of the classifier sequence be started again. Figure 1 As shown in , a second group of sub-windows SW4-SW7 are assigned to the computing units, and the classifier sequence is executed for each of those sub-windows simultaneously. In the second group, window SW5 accepts all feature tests F0 to F5, and sub-windows SW4, SW6 and SW7 are terminated before the sequence is completed, so that the corresponding computing units CU0, CU2 and CU3 are idle during one or more instruction sequences about feature tests F2 to F5.

[0060] exist Figure 1 In the example shown in , it can be seen that the computational units of the conventional SIMD processor are idle for more than 30% of the processor time available for processing the first and second groups of sub-windows SW0 to SW7.

[0061] Figure 3, a SIMD processing unit 301 for performing object detection in an image is shown. In one example, the processing unit 301 can be part of a graphics processing unit (GPU). In this example, the object detection being performed is the detection of a face in a subwindow 303 of an image 304. Each subwindow represents a region of an image. The processing unit 301 includes a SIMD processor 302 having a plurality of computing units 309-312. In this example, there are four processing units, but more generally, there may be any number of computing units. The computing units together form a SIMD processor that is arranged to execute the same instructions in parallel (i.e., "in step") on different data. The instructions executed by each computing unit specifically implement a sequence of classifier programs, each of which is configured to test one or more object features for a region of the image where object detection will be performed.

[0062] The processing unit 301 is configured to receive a sub-window 303 representing a region of an image 304 where object detection is to be performed. The image 304 may be, for example, a frame from a digital camera pipeline, such as a still image frame or a frame of a video stream. The image 304 may be a derivative of a source image in which an object is to be detected: for example, the image 304 may be a brightness image pyramid as described below.

[0063] The processing unit 301 is configured to perform object detection in an image by running a sequence of classifiers in parallel at its computational units, each operating on a different sub-window 303. Each classifier sequence tests a variety of image features in the sub-window assigned to it. The classifier sequence may be a Viola-Jones cascade of classifiers, and the features may be Figure 2 In this example, the classifiers are binary classifiers: each binary classifier of the sequence is tested against a subwindow, and the subwindow either accepts or rejects the test. More generally, each classifier can be any suitable kind of classifier that produces any number of possible outcomes, including a ternary classifier (which has three possible outcomes).

[0064] Each classifier sequence executed at the computational units 309-312 of the SIMD processor 302 includes the same instructions. In other words, each computational unit executes the same instructions on different data. Each computational unit is arranged to operate on a different sub-window so that the sub-windows to be searched are processed in parallel. The sub-windows 303 may be generated for the frame 304 in any suitable manner and Figure 3In the example shown in , sub-windows 303 are stored at a sub-window pool 306 that can be accessed by a classifier program running at a computing unit. In this example, sub-windows 306 are saved at a memory 305 that can be accessed by the processing unit, but more generally, at least a portion of the sub-window pool can be saved outside the processing unit, such as, saved at a memory of a graphics processing unit (GPU) and / or saved at a memory of a host system that implements the processing unit.

[0065] Each sub-window represents an area of ​​image 304 where object detection will be performed. Sub-windows 303, for example, can each carry a portion of data obtained from image 304 (on which the classifier program operates). Alternatively, the sub-windows can be references (such as pointers or storage addresses) to representations of all or part of image 304, such that each window identifies an area of ​​image 304. For example, a sub-window can indicate one or more of: a set of coordinates in image 304, a measure of rotation, and a measure of size, from which one or more of the above items can identify the image area represented by the sub-window. All or part of image 304 or its derivatives can be stored at memory 305 for access by the classifier program.

[0066] A preprocessor 308 may alternatively be provided for processing the data representing the image 304 into a form in which object detection can be effectively performed. Such processing may include image and / or color space transformations. Figure 3 In the example shown in , preprocessor 308 is arranged outside the processing unit, such as, at another processing entity of the GPU implementing the processing unit. More generally, the preprocessor can be arranged at any suitable location, including at the host data processing system including the processing unit or at the processing unit itself.

[0067] In a face detection system, non-color information in an image is usually important, color information is usually less important, and for the purpose of face detection, color information can usually be discarded to reduce the image size and thus reduce the required processing resources. For example, image 304 may be a high-resolution sRGB color frame received from a camera pipeline. In order to reduce the size of the search space, the preprocessor 308 can be configured to scale the image to a lower pixel resolution (e.g., 640×480 pixels). Additionally or alternatively, the preprocessor can be configured to transform image 304 to another color space, such as a YUV color space, whose Y-component (brightness) carries brightness information, and face detection is usually performed on brightness information. In other examples, a custom color space can be used.

[0068] It is advantageous for preprocessor 308 to be configured to generate a luminance image having a resolution that provides a balance between maintaining sufficient detail to allow reliable identification of faces in the image and limiting the search space to a size that can be searched within a desired time frame (e.g., in real time between frames provided by the camera pipeline). When objects other than faces are detected, it is advantageous to process the received image to form an image other than a luminance image. For example, if the object is known to be a particular color, the received frame can be processed to emphasize that color and / or discard information about other colors that are not significantly helpful in identifying the object.

[0069] Sub-window 303 may be defined for processing unit 301 in any suitable manner, including external to the processing unit (e.g., in a Figure 3 The subwindow 303 may be defined at a preprocessor 308 (shown in FIG. 1 ) and provided to the processing unit, as well as defined at the processing unit itself. Data defining aspects of the subwindow 303 (e.g., position, size, and angle with respect to a predetermined direction) may be provided to the processing unit together with or separately from data representing one or more regions of the image 304 identified by the subwindow. The position and size of the subwindow 303 may be predefined for the processing unit. For example, a fixed set of image regions defined in terms of pixel coordinates may be maintained at a subwindow pool 306 to enable the processing unit to identify each region of the image 304 where object detection will be performed. Alternatively, subwindows may be generated for an image based on features of the image (e.g., subwindows may be generated for locations in the image that are identified as likely to contain faces based on color or shape).

[0070] Typically, when using the Viola-Jones framework, a predefined set of sub-windows is used for a given size (possibly resized) image on which object detection will be performed. For example, for a 640×480 pixel image, a set of sub-windows may be generated that are reduced from a sub-window of size equal to the image to a sub-window of size 20×20 pixels. Each sub-window of a given size may typically occupy many possible unique positions within the frame. For example, Figure 6 A set of possible positions of sub-windows 303 along the diagonal of image 304 is shown. The total number of sub-windows to be searched for a 640x480 source frame can be in the thousands. If all possible rotations of the sub-windows are also searched, it will be appreciated that the search space will become very large in practice.

[0071] A variety of techniques are used in the art to define a set of subwindows for an image of a given size that provides a good balance between object detection performance and the processing time required to search all subwindows of the set. Some techniques predefine a set of subwindows that are advantageous for a given working image size. Other techniques define subwindows of different densities for those areas of the image that appear likely to contain a face, compared to those areas of the image that appear unlikely to contain a face.

[0072] To avoid confusing these issues, the preprocessor 308 is configured to process the image 304 as an image pyramid or texture map (mipmap) comprising images (typically luminance images) at multiple different scales. Object detection can then be efficiently performed in an image pyramid or texture map at multiple different scales in a single pass by defining a subwindow 303 for the image pyramid / texture map as a whole. Techniques for forming image pyramids and texture maps are well known in the art. For a 640×480 pixel luminance image, an image pyramid with six levels is shown to provide good performance, with the resolution of each level being a factor of 2 smaller than the level above (e.g., the image pyramid includes luminance images having a size of 1 / 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32 of the size of the base luminance image - in this case 640×480 pixels). Scale factors other than 2 can be used, but a scale factor of 2 is convenient because it matches the scale factors commonly used in computer graphics, where image pyramids are also known as texture maps. In the case where the processing unit is implemented in a GPU, the texture mapping hardware of the GPU may be used to accelerate the lookup of pixel data in the image pyramid.

[0073] The Viola-Jones framework requires that the image on which object detection is to be performed be represented in the form of a complete image, where a given pixel value at coordinates (x, y) is the sum of the pixel values ​​in the rectangle defined by the corners (0, 0) and (x-1, y-1). The preprocessor 308 can be configured to provide an image or image pyramid / texture map represented in the form of a complete image. This greatly improves the efficiency of the Viola-Jones summation calculation performed to evaluate each binary classifier.

[0074] Those skilled in the art will appreciate that a classifier sequence suitable for identifying an object in an image can be determined based on visual features of the object (such as patterns that typically provide contrast in an image when viewed from multiple viewpoints). Once a classifier sequence is defined, appropriate thresholds for use with the classifier can be determined empirically to obtain a desired hit rate (correctly identifying an object) and / or miss rate (failure to identify an object). For example, a learning algorithm such as AbaBoost can be used on a set of test images to train a classifier and achieve desired performance characteristics.

[0075] The Viola-Jones framework provides a suitable sequence of classifiers for detecting faces in images in the form of a cascade of binary classifiers, but any sequence of classifiers for implementing object detection may be used. Under the Viola-Jones framework, the complexity of the binary classifiers increases as the cascade progresses, which has the advantage of initially rejecting a large number of sub-windows but maintaining a high match rate for each classifier of the cascade to accept a given sub-window.

[0076] As about Figure 1 As explained, evaluating subwindows in parallel at a conventional SIMD processor results in significant underutilization of the computational units. This is because for a given set of subwindows assigned to a processor, some classifier sequences running at the processor's computational units terminate earlier than other classifier sequences (e.g., when one of the classifier sequences does not accept a feature test and rejects the subwindow), and a new cascade may not have begun evaluating a new subwindow until all computational units have completed processing and the cycle of instructions running at the processor's computational units can begin again. At any given time during object detection processing at a conventional SIMD processor, a large percentage of the computational units are likely to be idle.

[0077] The following references Figure 3 , Figure 4 and Figure 5 Techniques for performing object detection on SIMD processors that address this problem are described. Figure 3In the processing unit 301 of , each of the computing units 309-312 is configured to evaluate the classifier sequence in parallel for the subwindow. Each classifier sequence can include, for example, one or more features of the Viola-Jones framework for testing the subwindow. Executing the classifier sequence at the computing unit includes: running multiple classifier programs, each of which can operate to test the window for the existence of object features. Each computing unit executes the classifier program, and the classifier program includes the same instructions as the program running at all other computing units. Moreover, each classifier program can evaluate any object feature on any subwindow, which means that the same classifier program can be used at each level of the classifier sequence and used in parallel on each computing unit. Because the same classifier program including the same instructions is used to test any feature in the classifier sequence at any computing unit, the computing units of the SIMD processor can test different features in the sequence at the same time. By providing different subwindows to each computing unit, the SIMD processor can also perform object detection in parallel in the subwindows. For example, the program at computation unit 309 may test the first feature of the sequence against a first window defined for the image while the program at computation unit 310 concurrently tests the first feature of the sequence against a second sub-window defined for the image.

[0078] The classifier programs are also configured to independently acquire new subwindows from the subwindow pool 306 upon completion of processing of their current subwindows. This is accomplished by (a) maintaining data defining the subwindows on which the computing unit is operating and the image features being tested, for each computing unit, and (b) including instructions in the instructions defining the classifier programs to acquire new subwindows once processing of the current subwindow is completed. Thus, subwindows are not assigned to computing units in groups as is the case with conventional SIMD processors. Instead, each classifier program running at each computing unit individually determines a new subwindow to be evaluated when a new subwindow is required.

[0079] Each computing unit maintains data 313-316 defining the sub-window in which the computing unit is operating and the object characteristics being tested by the classifier program running at that computing unit. Figure 3 The location of data 315-316 is schematically shown in : the data may be maintained at the computing unit, memory 305, or any other data storage accessible to the classifier program running at the computing unit.

[0080] The program can be configured to take new work from the sub-window pool 306 when an object feature test for a sub-window (for which it is being evaluated) is not acceptable or when the program completes a classifier sequence. Because all processing units of the SIMD processor 302 simultaneously run a common set of instructions at each level of the classifier sequence, a set of computational units can be continuously utilized until the sub-window pool 306 is exhausted and all sub-windows 303 defined for the image 304 are searched. When the sub-window pool is exhausted, no more sub-windows are provided to the computational units, and object detection in the image 304 is complete.

[0081] In order to keep track of which subwindows 303 have been searched by the computing unit of the SIMD processor and which have not been searched, the index 307 can be kept in the form of a shared variable in the memory 305, and the memory 305 can be accessed by the program running at the computing unit 309-312. The index indicates the search range through the subwindow pool 306. Each classifier program is configured to update the index when requesting or receiving a new subwindow from the subwindow pool 306 to reflect that the scope of searching through the subwindow has progressed. For example, each subwindow can be assigned an identifier, and the subwindows are processed in sequence from the subwindow with the lowest value identifier and work upward. By configuring each program to increment the index when it requests a new subwindow, the index represents the highest value identifier assigned for processing, and therefore indicates that the subset of the subwindows of the subwindow pool that will also be assigned to the program for processing. It will be thought that this is only one of many possible examples of the index for keeping the search range of the set of subwindows indicating the subwindows passing through the frame.

[0082] In order to ensure that the index represents the collective progress made by the computing unit, it is preferred to use atomic instructions to increment the index by each program. Increasing the index atomically ensures that no conflict occurs between the writes of the program at each computing unit. Otherwise, a conflict may occur, because SIMD processing may mean that more than one computing unit may attempt to accurately modify the index simultaneously (when the instructions are identical and the pace is consistent on each computing unit). The index can represent the index of the next subwindow in the pool 306 to be processed. When the index reaches the total number of subwindows in the subwindow pool, the pool is exhausted. In other examples, the index indicating the search range through the subwindow pool can be reduced to zero from the total number of subwindows to count down, or the increment / decrement of the index can be an amount other than 1. The index described herein only represents a mechanism that is applicable to keep tracking the search range through the subwindow pool.

[0083] Figure 3The memory 305 shown in is a schematic representation of memory, which may include one or more physical and / or logical portions located at different points of the processing unit 301 or at a data processing system supporting the processing unit 301. For example, the memory 305 may include registers local to each computing unit, memories local to multiple groups of computing units, global memories accessible to all computing units, and one or more memories provided outside the processing unit (e.g., at a host data processing system). The data representing the child window pool 306 and the index 307 may be distributed at one or more portions of the memory 305 in any suitable manner.

[0084] An example of a classifier program running at each of the computational units 309-312 is shown below in pseudo code. F is an identifier of a particular feature in the classifier sequence being evaluated by the program, S is an identifier of the subwindow being operated on by the program, and R is the result (i.e., acceptance or non-acceptance) returned by evaluating feature F for subwindow S. The values ​​F and S for each computational unit are maintained at data 313-316. In this example, there are 50 feature tests (F=0 to 49) in the sequence, and there are 1000 subwindows in the pool 306 processed at the SIMD processor 102.

[0085] At the beginning of the classifier sequence, variables are initialized by setting F=0 and S=atomic_inc(workindex) so that each program receives a different subwindow and starts evaluating the first feature. The instruction "S=atomic_inc(workindex)" performs an atomic increment of index 307, with the parameter workindex starting at zero, so that when a classifier program running on the computing unit calls "atomic_inc(workindex)", one of the multiple programs receives subwindow 1, another receives subwindow 2, etc. In other words, although all classifier programs running on the computing unit request subwindow identifiers at the same time, the atomic increment function handles the contention and ensures that all classifier programs are assigned different subwindows.

[0086] Then, each program executes the following instructions simultaneously and in lockstep:

[0087] iR=eval(F,S)

[0088] ii.if(R==1)F=F+1

[0089] iii.if(F==50){faceFoundAt(S);}

[0090] iv.if(F==50OR R==0){F=0;S=atomic_inc(workindex);if(S>=1000)return;}

[0091] Line (i) evaluates feature F for subwindow S, and the program using index S identifies the subwindow in pool 306 to be evaluated. If the evaluation returns "accept" (R=1), Line (ii) causes the program to move to the next feature in the sequence. Line (iii) checks whether the last feature in the sequence (number 49) has been successfully evaluated (only then can F have the value 50); if so, an object candidate is found in the subwindow being processed, and the program calls the output function "faceFoundAt(S)". Line (iv) determines whether a face is detected (F==50) or the evaluation is not acceptable (R==0). If so, the program is reinitialized to process the next subwindow in the subwindow pool starting with feature zero. If the subwindow pool is exhausted, the thread terminates processing. In this way, the same set of instructions (i)-(iv) above is executed to evaluate any feature of the sequence for any subwindow of the pool, and is iterated to work through the features for each subwindow and through the available subwindows.

[0092] Figure 4 The operation of the above exemplary classifier program is also shown in the flowchart of FIG. At 401, the classifier program 400 evaluates the feature F and the sub-window S by the program according to the values ​​for F and S maintained at 410 and 411. If the feature is accepted at step 402, then at step 403, the value F410 is incremented, and at 404, the program checks to see if the classifier sequence is completed for the sub-window. If all features of the sequence have been evaluated and accepted, the program returns a face detected at 405; if not accepted, the program is iterated to continue evaluating the next feature for the current sub-window. When the feature is not accepted or a face is detected, the feature value F is reset at 406 to identify the first feature in the sequence, at 407, the sub-window value S is incremented, and a check is performed at 408 to determine if the sub-window pool has been exhausted. Once the sub-window pool is exhausted, the program terminates, otherwise the program is iterated to continue evaluating features for sub-windows according to the values ​​of F and S maintained at 410 and 411.

[0093] Figure 5A simple example of the improved efficiency of SIMD processor 302 over conventional SIMD processors is shown in FIG. In this example, the sub-window pool includes eight sub-windows S0 to S7, and the classifier sequence includes six features F0 to F5. Each of the computational units 309 to 312 of SIMD processor 302 runs the classifier program according to the above principles, and keeps track of the processing range through the sub-window pool through index 307. In this example, Figure 5 The features F0-F5 shown in Figure 1 The features F0-F5 in are the same, and the subwindows S0-S7 are the same as Figure 1 The sub-windows SW0-SW7 shown in FIG. are identical. Figure 1 and Figure 5 In , the evaluation result for each given feature and subwindow pair is the same.

[0094] When initialized, the four computing units receive the first four sub-windows S0-S3 (e.g., from an atomic operation on the index) and begin evaluating the first feature F0 of each sequence. Then, the program running at each computing unit continues to operate in the manner described herein. Sub-windows are not assigned to computing units in groups; rather, when a sub-window is required by a program running at a computing unit, a new sub-window is requested. For example, it can be seen that when sub-window S2 does not accept the second feature F1, the program running at computing unit 311 first receives the new sub-window. Then, computing unit 311 immediately begins evaluating the classifier sequence for the next available sub-window S4 from the sub-window pool. Subsequently, sub-window S4 does not accept the evaluation of the fifth feature F4 at computing unit 311. At this point, all sub-windows of the sub-window pool have been exhausted, and the program running at computing unit 311 terminates, so that the computing unit is idle during two iterations of the program running in parallel at SIMD processor 302.

[0095] exist Figure 5 In , all eight sub-windows of the sub-window pool are processed by processing unit 301, and only 3 of the total 36 iterations of the program running at the computing unit are idle. This is equivalent to about 8% idle time at the computing unit, which is much lower than the more than 30% idle time experienced by the computing unit of a traditional SIMD processor when evaluating the same features for the same sub-window. It will be appreciated that processing unit 301 can perform object detection in parallel at the computing unit, which can have substantially greater efficiency than a SIMD processor configured in a traditional manner.

[0096] Since there is an instruction to check whether a new subwindow is required and to keep the data identifying the current subwindow and features for each program, there is little overhead in each iteration of the program executed at the computing unit of the SIMD processor 302. There is no such overhead in the operation of a traditional SIMD processor that is assigned with multiple groups of subwindows for processing. However, the negative effect on performance caused by this overhead is fully less than the positive effect on performance obtained from the idle time far lower than the traditional SIMD processor experienced by the computing unit of the SIMD processor 302.

[0097] Index 307 is described above as a shared variable that can be accessed by all computing units of the processing unit. However, when there may be a large number of programs competing for access to the variable (for example, far more than the four programs described in the above example), and / or the variable is maintained in the global memory of the GPU or the memory of the host data processing system, such a variable may be slow to access. In order to address these problems, it is advantageous that the sub-window pool 306 can be divided into multiple subsets of sub-windows and the computing units of the SIMD processor are logically grouped so that each subset of sub-windows is processed by a different group of computing units. There can be far more subsets of sub-windows than there are multiple groups of computing units. The above problems can be alleviated by assigning subsets of sub-windows to multiple groups of computing units. Figure 7 One exemplary way of doing this is shown in .

[0098] Figure 7 1 shows a SIMD processing unit 700 configured to utilize multiple groups of SIMD computing units, showing three groups of computing units 701-703, each of which can include any number of computing units of a SIMD processor. The computing units can belong to one or more different SIMD processors - for example, each group of computing units can be in the same Figure 3 The SIMD processors are configured in the same manner as the SIMD processors 302 in FIG. Each computational unit executes a classifier program configured to operate in the manner described above.

[0099] Each group of computing units (or "compute group") has corresponding local memory 707-709, which programs running at the computing units of the group can access with lower latency than the latency of all computing groups accessing global memory 704. The local memory of a given computing group can be accessed by that computing group but not by other computing groups. Figure 7In an example of , each local memory includes a local subset 710-712 of subwindows for the computational group (which may be in the form of references to portions of an image, image region identifiers, or the image data itself) and a local index 713-715 indicating a search range through the subset of subwindows. In another example, each local memory stores only a local index, and the subset of subwindows is stored elsewhere (such as in a memory external to the processing unit or in a global memory). The classifier program running at a group of computational units processes the subwindows of its corresponding subset, and the subwindows are sorted as described above with respect to Figures 3 to 5 Describes how to request new child windows from this collection individually.

[0100] Each calculation group can represent Figure 3 For example, computation group 701 may be SIMD processor 302, local memory 707 may be memory 305, local collection 710 may be sub-window pool 306, and local index 713 may be index 307. However, in Figure 7 In the arrangement shown in , the subset of sub-windows operated on by computation group 707 represents only a portion of the entire pool of sub-windows defined for image 304 .

[0101] Each computing group works individually by a subset of subwindows. When a computing group exhausts a subset of its subwindows, a new subset of subwindows is assigned to the computing group from a global subwindow pool 706 maintained in a global memory 704. This can be implemented in any suitable manner, including by monitoring each local index in a local index 713-715 to determine when the corresponding local subwindow of a subwindow is exhausted and making the new subset of subwindows be written to a suitable local collection 710-712. Such monitoring can, for example, be performed by a process running at a GPU implementing a computing group. In another example, a classifier program running at a multi-group computing unit is provided with another instruction requesting a new subset of subwindows from a global memory 706 when the local subset of a subwindow is exhausted.

[0102] For example, step (iii) of the classifier program described in the above pseudocode can be modified to request that a new subset of subwindows be assigned to the calculation group, rather than terminating. The corresponding local index can also be reset or updated so that the index starts to be counted by the subwindows in the new subset. The global memory 706 may include a global index 705, which indicates the search range of the global subwindow pool of the image that will perform object detection. The classifier program can update the global index 705 by using an atomic instruction executed in response to the local subwindow of the subwindow being exhausted, so that the new subset of the subwindow is written to its local memory. The atomic instruction can be used to increment the global index to reflect the search range through the subwindow pool and to write the new subset of the subwindow to the local memory. When the global index indicates that the global subwindow pool has been exhausted, a set of subwindows can no longer be provided for processing, and the program executing the atomic instruction can terminate at this time.

[0103] Although accesses to global memory by programs are slower than local memory, these accesses are infrequent because they occur only when a subset of subwindows is exhausted. Moreover, programs running at the computational units of each group can be continuously occupied because any program can request that a new subset of subwindows be written to local memory when the current set is exhausted.

[0104] In this example, the processing unit is implemented on the GPU, and then an API (such as OpenCL) can be used to provide a classifier program, and the computing unit can be a thread running on the GPU. OpenCL enables threads to be grouped into "workgroups", and each workgroup can be provided with a local memory that only the workgroup can access. OpenCL also enables the provision of global memory that all threads and workgroups can access, but this is slower to access than local memory. The architecture of GPUs (and some CPUs) also tends to be arranged so that they have multiple sets of grouped computing units, which can be in the form of multiple cores or computing clusters. Individual cores or clusters do not share local memory, and therefore cannot quickly access the same data in local memory. Some memories that can be accessed by all cores or clusters can be set, but these memories are farther away from the core / cluster and cannot be quickly accessed. Thus, the above-mentioned computing groups can be configured so that they correspond to the core / cluster of the device, enabling fast access to data that is local to the core / cluster, and slower access to data that is shared between all computing groups but is less frequently needed.

[0105] In some examples, the pool of sub-windows can be divided into multiple subsets so that the sub-windows of each subset refer to some common region of the image (or its derivatives, such as an image pyramid or texture map) on which object detection will be performed. This minimizes the image data copied to the local memory for classifier program operation and improves the efficiency of any cache processing operated at the local memory of each computation group.

[0106] In the above reference Figure 7 In the alternative example described, the allocation of subsets of subwindows to computing groups can be partially handled by the scheduling system of the SIMD processor itself. For example, some SIMD processors (such as GPUs and some CPUs) have multiple computing units that operate in a SIMD manner, and the programs running on these computing units are in the form of threads. Compared with the existence of computing units, these SIMD processors are generally able to have more threads in the system. In other words, not all threads can be executed at the same time. On the contrary, when resources become available, the SIMD processor can switch between threads (for example, if a thread becomes idle, the thread is turned off and another thread is connected).

[0107] In such a case, the subwindows may be divided into subsets, as described above, and each subset may be assigned to a group of threads. There will be more groups of threads than there are multiple computation groups on a processing unit. The processing unit will start executing as many thread groups as possible, and once one of these thread groups has finished evaluating its subset of subwindows, it is turned off and replaced by another thread group assigned a different subset of subwindows. In this way, all subsets of subwindows will be evaluated, and the use of global memory is avoided. However, this relies on the thread scheduling of the processing unit, which is more dedicated to the individual processing units performing object detection.

[0108] Object detection performed according to the principles described herein is useful in many types of devices. In particular, the position of a subwindow accepted as a face candidate can be optimized as an autofocus point by the autofocus system of the camera pipeline, with the goal of ensuring that the face in the image is in focus. The detected face information can be used in a video encoding device to inform the encoder of the location of the face in the video stream and allow the appearance quality of the encoded video to be improved by preferentially increasing the encoding quality of the faces in the video stream. The detected face information can be provided as metadata for a photo captured by a camera to indicate the location of the face in the photo.

[0109] The SIMD processing units described herein may be used to perform object detection in a camera pipeline of, for example, a digital camera, a smart phone, a tablet computer, or any other kind of device that includes a camera (such as a security camera). More generally, the SIMD processing unit may receive frames from any suitable source, including from a storage subsystem that provides a recorded video stream or a series of one or more digital images or from a communication subsystem that provides a received video stream or a series of one or more digital images.

[0110] When some platforms provide SIMD processors, images can be divided into tiles for processing (e.g., for processing on a particular type of GPU). It will be apparent to those skilled in the art that the principles of performing object detection described herein can be easily extended to images represented as a collection of tiles, each tile having a defined collection of sub-windows.

[0111] The terms software, program, and program code may include executable code for a processor (e.g., a CPU and / or GPU), firmware, bytecode, a programming language (such as C or OpenCL), and a module for a reconfigurable logic device (FPGA). Machine-readable code may include software / program code, as well as code for defining a hardware representation of an integrated circuit at any level, including at the register transfer level (RTL), in high-level circuit representations (such as Verilog or VHDL) and low-level representations (such as OASIS and GDSII).

[0112] The term SIMD processor will be understood to refer to any type of processor that can operate to execute the same instruction in parallel to different data, and includes single instruction, multi-thread (SIMT) processors and other variants. The computing unit described herein forms a part of one or more SIMD processors or can be aggregated to represent one or more SIMD processors. The computing unit can be provided by any suitable processing entity (including CPU, GPU, or fixed function or programmable hardware). Software and programs can be stored in a machine-readable medium (such as, integrated circuit memory or optical or magnetic memory) in a non-temporary form. The machine-readable medium may include multiple memories, such as, on-chip memory, computer working memory, and non-volatile storage devices.

[0113] Thus, the applicant discloses each corresponding feature described herein individually and any combination of two or more such features, in the sense that such features or combinations can be implemented as a whole based on this specification in combination with the common knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problems described herein, and without limiting the scope of the claims. The applicant points out that various aspects of the present invention may be composed of any such corresponding features or combinations of features. In view of the foregoing description, it will be apparent to those skilled in the art that various modifications within the scope of the present invention may be made.

Claims

1. A single instruction multiple data processing unit for performing object detection in an image by testing multiple object features in multiple image regions, the single instruction multiple data processing unit include: a set of computing units operable to execute in parallel a plurality of classifier sequences, each classifier sequence comprising a plurality of classifier programs, and each classifier program comprising the same instructions as other classifier programs in each of the plurality of classifier sequences; wherein the set of computing units is configured to simultaneously test different features for different image regions by using data identifying the image regions and features being tested; wherein each classifier program is configured to accept or reject an image region when testing the identified features against the identified image region; and A memory accessible to the set of computing units and arranged to store an index representing a search range passing through the plurality of image regions, and each classifier program includes instructions configured to: when an image region is not accepted, update the data on which the classifier program is being executed by atomically incrementing the index to update the search range and cause an identifier of a next image region to be returned to the classifier program.

2. The single instruction multiple data processing unit according to claim 1, each classifier sequence is configured to perform a sequence of tests on object features in the image area, and each classifier program of the classifier sequence is configured to perform the sequence of tests.

3. The SIMD processing unit of claim 1 , wherein each classifier program comprises instructions configured to: When an image region is accepted, updating data maintained for the computational unit executing the classifier process so that the next classifier process in the sequence of classifiers tests the next feature for the same image region; When an image region is not accepted, data maintained for the computational unit executing the classifier procedure is updated so that the next classifier procedure in the classifier sequence tests the identified features for the next image region.

4. A single instruction multiple data processing unit according to claim 1, each classifier program is configured to use its data of the identified image region to look up data in the identified image region in the image for which object detection is being performed.

5. A single instruction multiple data processing unit according to claim 1, each classifier program of the classifier sequence is a single program, and the single program is arranged to be iteratively executed at its computing unit to test the identified features against the identified area and update the data.

6. The single instruction multiple data processing unit according to claim 1, wherein the single instruction multiple data processing unit further comprises: include: another group of computing units, the another group of computing units being operable to execute the plurality of classifier sequences, wherein the one group of computing units forms a first computing group and the another group of computing units forms a second computing group, and wherein the plurality of image regions are divided into a plurality of subsets of image regions and each of the first computing group and the second computing group is assigned a different one of the plurality of subsets of image regions for testing.

7. The single instruction multiple data processing unit according to claim 6, in, The processing unit further includes: a global memory storing a global index, the global index representing a search range of the plurality of subsets of the image region passed by the first calculation group and the second calculation group; Therein, each classifier program comprises instructions configured to perform the following operations: when the subset of image regions is exhausted for its associated computation group, a new subset of image regions is assigned to the associated computation group according to the global index.

8. According to the single instruction multiple data processing unit according to claim 7, each classifier program includes instructions configured to perform the following operations: by atomically updating the global index, the global index assigns a new subset of the image area to the relevant calculation group to update the search range passing through the multiple subsets of the image area, and the identifier of the next subset of the image area is returned to the classifier program.

9. A single instruction multiple data processing unit according to claim 6, further comprising a memory accessible by the first computing group but not accessible by the second computing group, the memory being arranged to store an index representing a search range through the plurality of image regions by the first computing group; in, Each classifier program of the first computing group is configured to accept or not accept an image region, and includes instructions configured to perform the following operations: when an image region is not accepted, the data maintained for the computing unit that is executing the classifier program is updated by atomically incrementing the index to update the search range, and an identifier of the next image region is returned to the classifier program.

10. The SIMD processing unit of claim 1, each image region differing from the other image regions in one or more of its position, size and orientation in the frame.

11. The SIMD processing unit of claim 2, wherein each object feature is a Haar-like feature, and each classifier program is configured to determine whether the feature satisfies a predefined threshold.

12. A method for performing object detection in an image at a single instruction multiple data processor, the single instruction multiple data processor having a set of computing units arranged to execute a plurality of classifier sequences in parallel, each classifier sequence comprising a plurality of classifier programs having the same instructions as other classifier programs in each of the plurality of classifier sequences, and the method The following steps are involved: executing the classifier procedure of each sequence in parallel at each computing unit in the set of computing units to test for a plurality of object features in a plurality of image regions, such that the set of computing units is configured to simultaneously test different features for different image regions by using data identifying the image regions and features being tested; wherein, when testing the identified features against the identified image region, accepting or rejecting the identified image region; and Maintaining an index representing a search range by the set of computing units through the plurality of image regions, each classifier program updating data identifying the image region and the feature being tested by atomically incrementing the index when an image region is not accepted to update the search range and causing an identifier of the next image region to be returned to the classifier program.

13. The method of claim 12, wherein each classifier program performs the following operations: Upon accepting the image region, updating data identifying the image region and the feature being tested so that a next classifier process in the sequence of classifiers tests the next feature for the same image region; or When the image region is not accepted, data identifying the image region and the feature being tested is updated so that the next classifier process in the sequence of classifiers tests the identified feature against the next image region.

14. The method of claim 12, each classifier program uses its data of identified image regions to look up data in the identified image regions in the image for which object detection is being performed.

15. The method according to claim 12, wherein The following steps are involved: Each classifier program is iteratively executed at its computational unit to test the identified features against the identified regions and update the data.

16. The method of claim 12, wherein each classifier program is executed at a computing unit of a group of computing units in a plurality of groups of computing units, and wherein the method further The following steps are involved: dividing the plurality of image regions into a plurality of subsets of image regions; as well as A different one of the plurality of subsets of the image regions is allocated for testing at each group of computing units.

17. The method according to claim 16, further comprising: The following steps are involved: maintaining a global index representing a search range by the plurality of groups of computing units across the plurality of subsets of the image region; as well as When the classifier procedure exhausts said subset of image regions for its associated calculation group, a new subset of image regions is caused to be assigned to said associated calculation group according to said global index.

18. According to the method of claim 17, each classifier program atomically updates the global index so that a new subset of the image area is assigned to the relevant calculation group to update the search range through the multiple subsets of the image area, and the identifier of the next subset of the image area is returned to the classifier program.

Images