Detection of a complex object using a cascade of classification devices

Abstract

Method for identifying a complex object (220) in a query image (210), wherein the method comprises that: a processor (20) is used to perform the steps that: several pixel sections from the query image (210) are processed by a cascade (520) of classification devices (505, 510, 515), each classification device (505, 510, 515) of the cascade (520) being designed to identify at least one distinguishing feature characteristic of a part of a complex object (220), each subsequent classification device (510, 515) of the cascade (520) identifying a number of distinguishing features greater than the number of distinguishing features identified by previous classification devices (505, 510) of the cascade (520); wherein after processing a pixel segment, after a final classification unit (515) of the cascade (520) has identified the part of the complex object (220), another pixel segment is selected from the query image (210) for processing, the selection of the further pixel segment being based on probability data, wherein each pixel of a pixel segment is assigned a probability value between zero and one in response to output values ​​of each classification unit (505, 510, 515) processing the pixel segment, and after identifying the part of the complex object (220), the further pixel segment is selected by computing the argument of the maximum of a probability function for the next part of the complex object (220); and wherein parts of the complex object (220) identified in different pixel sections of the query image (210) are combined to identify a complete complex object (220).

Description

BACKGROUND OF THE PRESENT INVENTION

[0001] Many different applications utilize computer-based object detection systems and methods that require high accuracy, achieved in near real-time. Examples of such applications include active vehicle safety systems, intelligent monitoring systems, and robotics.

[0002] In the field of vehicle safety, for example, accurate high-speed identification of pedestrians or objects on the roadway enables an automated safety system to take necessary measures to avoid a collision, or enables the automated system to alert the driver, allowing the driver to take necessary precautions to avoid a collision.

[0003] Mikolajczyk, Krystian; Schmid, Cordelia; Zisserman, Andrew; Human detection based on a probabilistic assembly of robust part detectors; in: European Conference on Computer Vision; Springer, Berlin, Heidelberg, 2004, pp. 69-82 discloses a method for detecting humans in images, in which models for an arrangement of human parts are created and a coarse-fine cascade of classifiers is used to successively reduce the search space. The cascade of classifiers begins with the fastest classifier at the lowest scaling level. The detection results of the candidates classified as body parts are combined using a probabilistic model, with the candidate with the highest confidence level (greater than a threshold) being classified as the body part. Adjacent candidates are evaluated using geometric relationships between parts.If a candidate is accepted as a body part, closely overlapping neighbors are removed. SUMMARY

[0004] The object of the invention is to provide an improved method and system for identifying a complex object in a query image.

[0005] To solve the problem, a method with the features of claim 1 and a system with the features of claim 4 are provided. Advantageous embodiments of the invention can be found in the dependent claims, the description, and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The subject matter considered as the invention is specifically set out and clearly claimed in the final part of the application. However, the invention can best be understood with regard to its components, features, operating methods and advantages by reference to the following detailed description and the accompanying drawings, in which: Fig. 1 a schematic block diagram of a system for detecting a complex object using a cascade of classification devices according to an embodiment of the present invention; Fig. 2 is a query image with a complex object to be identified; Fig. 3 is an exemplary complex object whose parts have been marked for learning to use by classification devices of a cascade of classification devices. Fig. 4 a graphical representation of features from which distinguishing features are derived for use by each of three classification devices of a cascade of classification devices when identifying features belonging to a part of a complex object, according to an embodiment of the present invention; Fig. 5 a cascade of classification devices with three classification devices, each classification device identifying its respective set of learned distinguishing features characteristic of a distinguishing feature of a part corresponding to the one in Fig. 2 shown to be the complex object belonging to, according to an embodiment of the present invention; Fig. 6 a processing design of the cascade of classification devices of Fig. 5 for three object parts from multiple locations, wherein each subsequent classification device processes a pixel section under the condition that previous classification devices successfully identified their respective distinguishing features, according to an embodiment of the present invention; Fig. 7 a flowchart that represents the method for identifying additional pixel sections likely to contain additional parts of a complex object based on learned positional relationships with respect to an identified part, according to a non-inventive embodiment of the present invention. Fig. 8 a flowchart representing the method for identifying additional pixel sections likely to contain additional parts of a complex object based on a calculated probability with respect to an identified part, according to an embodiment of the present invention; Fig. 9 the query image of Fig. 2, in which several search windows enclosing pixel sections are distributed at different locations prior to successful identification of a part of a complex object and at a first preferred location after successful identification of the part, according to an embodiment of the present invention; Fig. 10 the query image of Fig. Figure 9, in which several search windows enclosing pixel sections are distributed at different locations prior to successful identification of a part and at a second preferred location after successful identification of a part, according to an embodiment of the present invention; Fig. 11 the query image of Fig. 2, in which a search window encloses a pixel section that has been rejected with regard to future attempts to identify relevant features, and a search window is distributed when searching for parts of a complex object at a preferred location based on a successful identification of two object parts, according to an embodiment of the present invention; Fig. 12 the query image of Fig. 2 with a partially obscured complex object in which the search window encloses pixel sections likely to contain another part of the object, based on a previously identified part, according to an embodiment of the present invention; Fig. 13 the query image of Fig. 2 with a complex object at a reduced scale, in which the search window encloses pixel sections that are likely to contain another part of the object, based on a previously identified part, according to an embodiment of the present invention; and Fig. Figure 14 shows a non-volatile, computer-readable medium with instructions stored on it for identifying a complex object using a cascade of classification devices in a query image according to an embodiment of the present invention.

[0007] It should be noted that, for the sake of simplicity and clarity, elements shown in the figures may not be drawn to scale, and reference symbols may be repeated in different figures to indicate identical, corresponding, or analogous elements. DETAILED DESCRIPTION OF THE PRESENT INVENTION

[0008] Numerous details are provided in the following detailed description to offer a thorough understanding of the invention. However, those skilled in the art will understand that the present invention can be carried out without these specific details. Furthermore, widely known methods, procedures, and components have not been described in detail to avoid obscuring the present invention.

[0009] It should be noted that the following terms are used in this application.

[0010] “Complex object” refers to an object present in an image that requires multiple templates for description or identification due to various complexities associated with the object. These complexities may include object parts with differing anthropometric relationships to one another, large size variations within a given classification, partial occlusion, and multiple views. Typical examples include, but are not limited to, people, animals, or vehicles. For the purposes of this application, and without diminishing its general applicability, a person is highlighted as an example of a complex object.

[0011] "Classification device" refers to a function (e.g., a computer-executable function) designed to identify image object parts based on distinguishing features characteristic of parts belonging to complex objects. These distinguishing features can typically be processed to, for example, generate an output value that is compared to a threshold derived analogously from a model image to determine a "match." Such a match can be based, for example, on imaging parameters such as pixel intensities, basic geometric shapes, and / or other image parameters.

[0012] “Cascade of classification facilities” refers to several successive classification facilities.

[0013] "Pixel section" refers to an area of ​​pixels.

[0014] “Distinguishing features” refer to parameters of such image pixels, such as intensity gradients, average intensities, pixel colors, and represent a feature of the image content.

[0015] “Anthropometric relationship” refers to the relative size, arrangement, and orientation of body parts in humans.

[0016] "Collaborative search" refers to the selection of pixel sections in a query image based on a previous successful identification or classification of at least part of a complex object.

[0017] According to embodiments of the present invention, a method for detecting a complex object using a cascade of classification devices may include identifying a pixel segment in a query image and processing this segment using a cascade of classification devices while searching for learned characteristic features. As mentioned above, the cascade of classification devices may comprise a sequence of classification devices, each of which may be configured to identify its respective set of characteristic features.Each subsequent classification unit in the cascade searches for a larger number of characteristic features for the same object segment and is designed to identify its respective set of distinguishing features only after a previously deployed classification unit has successfully identified its respective characteristic features. If this is not achieved, no subsequent classification unit processes the pixel segment, and that particular segment is rejected and marked as an area lacking the required distinguishing features. Another pixel segment from the query image can then be selected on a random or semi-random basis. In other embodiments, an adjacent segment or any other segment can be selected as the next segment to be processed.When prior classification units identify their respective sets of characteristic features, subsequent classification units process the pixel set until an object part is identified. Upon detection, the object part location, together with learned spatial relationships between object parts in a model object image, serves as the basis for distributing additional pixel segments in the query image that are likely to contain additional object parts. According to the invention, a data mapping is employed, whereby the maximum of an argument of a probability function is used to select an additional pixel set with the highest probability of containing an object part.

[0018] The overall computational savings achieved through the reduced number of classification operations for each part and the reduced number of search locations according to embodiments of the present invention enable highly accurate identification of complex objects in near real-time. Accordingly, the method and system according to the present invention are applicable to a wide variety of real-world applications requiring accurate and rapid identification of complex objects, such as active vehicle safety features, intelligent monitoring systems, and robotics.

[0019] Now, referring to the characters... Fig. Figure 1 shows a schematic diagram of a system for detecting a complex object using a cascade of classification devices according to an embodiment of the present invention. The system 100 for detecting a complex object can comprise one or more computer vision sensors 10 (e.g., cameras, video cameras, digital cameras, or other image acquisition devices). The computer vision sensor 10 can capture an image that may include one or more objects and / or features. Images can also be input into the system 100 in other ways, for example, as downloads from other computers, databases, or systems. The object detection system 100 can comprise one or more processors or controllers 20, a memory 30, a non-volatile long-term memory 40, input devices 50, and output devices 60.Non-restrictive examples of input devices 50 include, for example, a touchscreen, a capacitive input device, a keyboard, a microphone, a pointing device, a button, a switch, or other device. Non-restrictive examples of output devices include a display screen, an audio device such as a speaker or headphones. The input devices 50 and the output devices 60 may be combined into a single device.

[0020] The processor or controller 20 can be, for example, a central processing unit (CPU), a chip, or any suitable computing device. The processor or controller 20 can comprise multiple processors and can include general-purpose processors and / or dedicated processors, such as graphics processing chips. The processor 20 can execute code or instructions stored, for example, in memory 30 or long-term memory 40 to execute embodiments of the present invention.

[0021] Memory 30 can represent random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate memory chip (DDR memory chip), flash memory, volatile memory, non-volatile memory, cache memory, buffer, short-term memory unit, long-term memory unit, or other suitable memory or storage units. Memory 30 can represent or comprise multiple memory units.

[0022] The non-volatile long-term storage device 40 can, for example, be or comprise a hard disk, a floppy disk, a compact disk drive (CD drive), a CD-Recordable drive (CD-R drive), a universal serial bus (USB) device, or any other suitable removable and / or fixed storage unit, and can comprise several or a combination of such units. It should be noted that image data, code, and other relevant data structures are stored in the storage and / or repository devices mentioned above.

[0023] Fig. Figure 2 is a query image 210 containing a complex object 220 of a person, which is to be classified by identifying different parts: a head 240, a back 250, and a foot 260. It should be noted that for the purposes of this application, a person is used as a non-restrictive example of a complex object.

[0024] Fig. Figure 3 shows an image of a model 330 of a complex object, from which sets of distinguishing features for each part and anthropometric relationships between the parts can be extracted. The model 330 of a complex object is divided into pixel sections or image areas that contain object parts. In the non-restrictive example of Fig. 3. The complex object is a person 330, in which three independent parts have been identified: a head 340, a back 350, and a foot 360. It should be noted that a wide variety of complex objects are suitable models that can be used to teach level classification facilities. Such models include animate and inanimate objects, objects with a large number of parts, objects with parts whose geometric relationship to one another differs, objects that are partially obscured, and all objects being viewed from different angles or distances, as mentioned above.

[0025] Fig. Figure 4 shows three graphical representations 405, 410, and 415 of features derived from a front view of an image example (not shown). These features are used in the training of successive classification devices in a cascade according to embodiments of the present invention. A feature selection algorithm can be applied to an image example to obtain the graphical representations 405, 410, and 415, which can be further processed to identify distinguishing features most characteristic of features belonging to an example.For example, the feature selection algorithm can generate ideal distinguishing features based on only two pixel areas, 406 and 407, for use by a first classification facility; ideal distinguishing features based on pixel areas 411–413 for use by a second classification facility; and seven additional pixel areas, collectively labeled as 414, for use by a third classification facility. In this way, each classification facility in a cascade of three can identify distinguishing features of an object part belonging to the complex object with increasing accuracy and clarity.

[0026] It should be noted that there are many pixel or image parameters that can be used to extract the most effective feature-identifying distinguishing features, and some examples include histograms of gradients (HoGs), integral channel features, and hair features. Furthermore, it should be noted that in the example of Fig. Four frontal facial features can be identified from a sample image; however, according to the specific view of the object part to be identified, features can be extracted from side views of the sample images.

[0027] Fig. Figure 5 shows a cascade with three classification devices designed to use the learned distinguishing features on a stepwise basis to identify a part 240 of a complex object, according to embodiments of the present invention.

[0028] As mentioned above, each successive classification facility searches object part 240 to identify its respective set of distinguishing features. In the present non-restrictive example, a first-level classification facility 505 checks the candidate object part 240 for distinguishing features derived from the graphical representation 405. If they are not found, the identified pixel segment is rejected, and the system 100 either distributes additional search areas in the query image 210 or applies the first-level classification facility 505 to additional pixel segments of the parts of the complex object in the queue.If the first classification unit 505 identifies this first set of distinguishing features, a second classification unit 510 searches for a second set of distinguishing features derived from the graphical representation 410. If classification unit 510 does not identify them, this pixel segment object is also rejected, as mentioned above. If a match is found, a third classification unit 515 is applied and attempts to identify the distinguishing features derived from the graphical representation 415. If no match is identified, the searched pixel segment is rejected; however, if a match is identified, the object segment 240 is considered to have been identified by the cascade 520 of classification units.It should be noted that in embodiments of the present invention, any cascade of classification devices comprising any number of classification devices employing any number of distinguishing features may be considered.

[0029] It should be noted that if a pixel segment is rejected, the pixel segment found not to contain the distinguishing features is marked as an unrealizable area with respect to that particular object part, in order to avoid unnecessary searches in that area with respect to the part for which a rejection occurred. It should be noted that the present invention includes embodiments in which pixel segments are rejected with respect to a particular part, and searches for additional object parts are still possible.

[0030] Fig. Figure 6 shows an example of a classification device processing pixel segments at five different locations I-V, wherein five separate cascades with three classification devices 1-3 are used to identify three parts 1-3 of a complex object, according to embodiments of the present invention. As shown, the classification devices 1a determine that the desired features are not present in the content from locations I and III, and therefore there is no further processing by the remaining classification devices 1b and 1c with respect to the content from these locations. The classification devices 1b proceed with processing content from the remaining locations II, IV, and V.Classification unit 1b determines that the content from location V also lacks the desired characteristics, and therefore classification units 1c proceed only with processing the content from locations II and IV. Classification unit 1c determines that the content from location IV also lacks the desired characteristics, and classification unit 1, which processes the content from location II, identifies the desired characteristics, and therefore part 1 is considered to be located at location II.

[0031] The search for part 2 of a complex object can be continued at several (e.g., five) different locations, with respective pixel segments from locations VI–X being processed by a separate cascade of three classification units, 2a–2c. The content from locations VII and VIII is rejected by classification unit 2a, and thus processing continues through classification unit 2b with respect to the content from the remaining locations VI, VIII, and X. Classification unit 2b rejects the content from location VIII, and thus processing continues through classification unit 2c with respect to content derived from locations V1 and X. Classification unit 2c rejects the content derived from location VI, while classification unit 2c identifies the relevant features in the content derived from location X.Since all three classification agencies 2a - 2c identified the relevant features in the content derived from location X, Part 2 is considered identified.

[0032] The search for part 3 proceeds with five cascades of three classification units, each 3a-3c, with respect to content derived from locations XI-XV. Classification unit 3a rejects the content derived from location XIII, so processing continues with pixel segments derived from the remaining locations XI-XIII and XV. Classification unit 3b rejects the content derived from location XIII, and classification units 3c continue processing content derived from the remaining locations XI-XII and XV, then reject the content derived from locations XII and XV. The remaining classification unit 3c identifies the relevant features with respect to the content derived from location XI. Again, part 3 is considered identified at location XI, since all three classification units 3a-3c have identified the relevant features in the content derived from that location.

[0033] Fig. Figure 7 is a flowchart showing the procedure described above with the additional steps of distributing additional search areas or pixel sections for remaining object parts after classifying an object part.

[0034] In particular, in step 710 according to an embodiment of the present invention, a first pixel section can be selected from the query image 210, e.g. on a random basis, according to embodiments of the invention.

[0035] In step 715, provided that all preceding classification facilities in the cascade have identified their respective sets of characteristic features, subsequent classification facilities can be applied to each part. In step 720, if all respective sets of characteristic features of all classification facilities have been identified, an object part is considered classified or identified, as described above. However, if not all respective sets of characteristic features have been identified, this pixel segment is marked as "Rejected" in step 721, and a new pixel segment is selected from query image 210 on a random or semi-random basis in step 710. Again, as shown in step 715, successive classification facilities process the newly selected pixel segment.If all classification devices successfully identified their respective characteristic features, an object part was classified as shown in step 725, and an additional pixel segment was selected from the query image based on learned spatial relationships between the previously identified object part (if any) and the part to be identified, as shown in step 730. After selecting a new pixel segment likely to contain the additional object part, the process was repeated by applying subsequent classification devices associated with the additional part, as shown in step 715.

[0036] The in Fig. The procedure shown in point 8 is analogous to the one in Fig. 7 methods shown, with an alternative way of selecting additional pixel sections that are likely to contain additional object parts, using a probability mapping as shown in step 830.

[0037] Specifically, a probability value between zero and one is assigned to each pixel in response to the output values ​​of each classification device that processes a particular pixel segment. After identifying an object segment, the probability mapping is updated accordingly, and a pixel segment is selected by calculating the argument of the maximum (argmax) of a probability function for the next object segment, or equivalently: ArgmaxPn+1Prob(Pn+1|P'n+1,P1,…,Pn) where: P n the probability mapping of detecting part n = 1...N is; P n+1The previous probability mapping is.

[0038] Areas with probability values ​​that are smaller than a predefined value are rejected by setting the probability values ​​to zero.

[0039] Fig. 9 and Fig. 10 are query images 210 of Fig. 2 with superimposed search windows that specify areas to be searched for an object part. In various embodiments, a system for detecting a complex object using a cascade of classification devices according to one embodiment of the present invention can be configured to distribute search windows that enclose an area substantially corresponding to the area of ​​the learned object part. In a non-restrictive example, search windows 970 and 975 enclose areas corresponding to areas that represent a learned head 340 and a learned back 350, respectively. Fig. 3. Furthermore, the search windows 970 and 975 can be distributed in multiple locations, with a section of the new search area overlapping a section of the previously searched area as shown, or by a procedure that is completely random for either the first selected pixel section or two substitute sections that were rejected as not having the relevant distinguishing features.

[0040] Once an object part has been identified, it is used as the basis for distributing additional search areas that are most likely to contain the requested object part, as mentioned above. Some non-inventive embodiments apply a learned anthropometric relationship to the identified part to direct the subsequent search area to pixel regions that are most likely to contain the additional part, as mentioned above. Inventive embodiments use the location of the identified part as a priori data when determining "maxarg" of a probability function for all parts, as mentioned above. A window 980 indicates that the head 240 ( Fig. 2) was located, and therefore search windows 990 and 1090 ( Fig. 10) distributed in areas most likely to contain the back 250, since these areas show the anthropometric relationship of these parts in model image 330 of Fig. 3. Since both sides of object 220 satisfy the learned anthropometric relationship, the areas of both search windows 990 and 1090 are identified as suitable pixel sections to search.

[0041] In some embodiments of the present invention, when probability images are inserted, both areas enclosed in windows 990 and 1090 can be determined to have a high probability of containing back 250, given the updated probability data. It should be noted that the scope of protection of the present invention includes any plurality of searches.

[0042] Fig. Figure 11 shows an embodiment in which pixel segments are distributed based on the successful identification or classification of multiple object parts. For example, both the head 240 and the foot 260 ( Fig. 3) identified in search windows 1110 and 1120, respectively. Search window 1190 is identified based on learned anthropometric relationships between each of these parts from model image 330, which is shown in Fig. 3 is shown, or updated probability data is distributed. It should be noted that embodiments in which additional search areas are distributed based on any number of previously identified object parts are included within the scope of protection of the present invention.

[0043] In some embodiments of the present invention, computational efficiency is further optimized by reducing search redundancy. Window 1100 is a window that marks a rejected pixel section or area after any one of the classification devices in a cascade has determined that no distinguishing features are present in the section.

[0044] Fig. 12 and Fig. Figure 13 shows applications of the above-described cascade classification device-supported search for a complex object that is partially obscured or has a reduced scale, according to embodiments of the present invention. In particular, the head 240 is identified in window 1210, and window 1220 is identified as a possible location for the foot 260 based on either a learned anthropometric relationship between the head 340 and the foot 360 of Fig. 3 or distributed based on probability data given the identified head 240 as mentioned above.

[0045] Fig.Figure 14 shows a non-restrictive, computer-readable medium containing executable code to configure a computer system to perform the above-described search for complex objects in an image supported by a cascade classification device according to embodiments of the present invention.

[0046] According to the invention, a complete object is identified by combining object parts that have been identified in different pixel sections.

[0047] It should be noted that search areas can be distributed based on any number of successfully identified object parts according to the specific embodiment. Furthermore, it should be noted that a search using circular, triangular, and polygonal search windows is within the scope of protection of the present invention.

[0048] While certain features of the invention have been presented and described herein, many variations, substitutions, modifications, and equivalents will now become apparent to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such variations and modifications inherent in the concept of the invention.

Claims

[1] Method for identifying a complex object (220) in a query image (210), wherein the method comprises that: a processor (20) is used to perform the steps that: several pixel sections from the query image (210) are processed by a cascade (520) of classification devices (505, 510, 515), each classification device (505, 510, 515) of the cascade (520) being designed to identify at least one distinguishing feature characteristic of a part of a complex object (220), each subsequent classification device (510, 515) of the cascade (520) identifying a number of distinguishing features greater than the number of distinguishing features identified by previous classification devices (505, 510) of the cascade (520); wherein after processing a pixel segment, after a final classification unit (515) of the cascade (520) has identified the part of the complex object (220), another pixel segment is selected from the query image (210) for processing, the selection of the further pixel segment being based on probability data, wherein each pixel of a pixel segment is assigned a probability value between zero and one in response to output values ​​of each classification unit (505, 510, 515) processing the pixel segment, and after identifying the part of the complex object (220), the further pixel segment is selected by computing the argument of the maximum of a probability function for the next part of the complex object (220); and wherein parts of the complex object (220) identified in different pixel sections of the query image (210) are combined to identify a complete complex object (220). [2] Method according to claim 1, further comprising identifying at least one distinguishing feature of a part of an exemplary complex image, wherein the distinguishing features are characteristic of the part of the complex object (220). [3] Method according to claim 1, further comprising marking a searched pixel section as to be neglected when selecting future pixel sections if it is determined that the searched pixel section does not contain the distinguishing features that characterize part of the complex object (220). [4] System (100) for identifying a complex object (220) in a query image (210), wherein the system (100) comprises: a processor (20) designed to: to process several pixel sections from the query image (210) with a cascade (520) of classification devices (505, 510, 515), wherein each of the classification devices (505, 510, 515) of the cascade (520) is designed to identify at least one distinguishing feature that is characteristic of a part of the complex object (220), wherein each subsequent classification device (510, 515) of the cascade (520) uses a number of distinguishing features that is greater than the number of distinguishing features used by previous classification devices (505, 510) of the cascade (520); wherein after processing a pixel segment, after a final classification unit (515) of the cascade (520) has identified the part of the complex object (220), another pixel segment is selected from the query image (210) for processing, the selection of the further pixel segment being based on probability data, wherein each pixel of a pixel segment is assigned a probability value between zero and one in response to output values ​​of each classification unit (505, 510, 515) processing the pixel segment, and after identifying the part of the complex object (220), the further pixel segment is selected by computing the argument of the maximum of a probability function for the next part of the complex object (220); and wherein parts of the complex object (220) identified in different pixel sections of the query image (210) are combined to identify a complete complex object (220). [5] System (100) according to claim 4, wherein the processor (20) is further configured to identify distinguishing features of a part of an exemplary complex image, wherein the distinguishing feature is characteristic of the part of the complex object (220). [6] System (100) according to claim 4, wherein the processor (20) is further configured to mark a searched pixel section as to be neglected when selecting future pixel sections if it is found that the distinguishing features that characterize a feature of part of the complex object (220) are not present in the searched pixel section.

Images