Progressive multi-stage object detection method

Abstract

The progressive multi-step object detection method according to the present invention is configured such that an open vocabulary object detection model detects objects in an original image according to a first OVD prompt, and if there are undetected objects in the original image, additionally detects objects in a mask image according to a new OVD prompt. According to the present invention, the probability of detecting all objects within the original image without omission is increased, thereby effectively improving the low recall problem (i.e., the problem of failing to detect some objects) of conventional open vocabulary object detection models.

Description

Technology Field

[0001] The present invention relates to a method for progressively detecting objects contained in an original image over multiple iterations. Background Technology

[0002] Generally, object detection models play the role of identifying the locations of objects contained in an input image and classifying those objects. These object detection models can be broadly classified into closed-set object detection models and open vocabulary object detection models (hereinafter also referred to as 'OVD').

[0003] First, representative examples of closed-set object detection models include YOLO (You Only Look Once), RCNN (Region-based Convolutional Neural Network), and Faster-RCNN.

[0004] Since closed-set object detection models are trained to identify and classify only objects of predefined classes, their detection performance drops significantly for objects outside of those classes. In other words, for a closed-set object detection model to detect objects of a new class, it must be retrained using a new dataset for that class. In this regard, closed-set object detection models suffer from limited scalability and flexibility.

[0005] Next, Open Vocabulary Object Detection (OVD) models are not limited to pre-trained classes and can detect objects corresponding to natural language prompts simply by inputting them into the model. Representative OVD models include Grounding-DINO and ViLD, and such models can achieve a certain level of object detection performance even for objects in classes that have not been pre-trained.

[0006] However, open vocabulary object detection models often fail to detect objects of the class corresponding to the input prompt. In this regard, open vocabulary object detection models have the problem of low object detection recall.

[0007] In addition, open vocabulary object detection models frequently detect irrelevant objects with characteristics unrelated to the input prompt. That is, open vocabulary object detection models can detect not only objects corresponding to the input prompt but also objects unrelated to the input prompt, which results in a problem of low precision in object detection.

[0008] Ultimately, existing open vocabulary object detection models have limitations in that they struggle to simultaneously achieve both recall and precision in object detection. For example, given an input prompt such as 'car,' open vocabulary object detection models may fail to detect the 'car' in the input image (i.e., low object detection recall) or detect objects unrelated to the 'car,' such as people or buildings (i.e., low object detection precision).

[0009] Meanwhile, the following patent document 1 discloses an object detection device and control method capable of detecting small objects in real time. Prior art literature

[0010] Registered Patent Publication No. 10-2051032 (2019.11.26.) The problem to be solved

[0011] The purpose of the present invention is to provide an object detection method capable of improving the recall and precision of object detection by combining an open vocabulary object detection model and a vision language model to progressively detect objects included in an original image over multiple iterations.

[0012] However, the technical problems that the present invention aims to solve are not limited to those described above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description of the invention below. means of solving the problem

[0013] To achieve the above objectives, the progressive multi-step object detection method according to the present invention comprises: (a) a step in which a processor receives an original image; (b) a step in which the processor inputs the original image and a first OVD prompt into an open vocabulary object detection model; (c) a step in which the open vocabulary object detection model detects an object in the original image according to the first OVD prompt; (d) a step in which the processor generates an n-th mask image by masking the area of ​​the detected object; (e) a step in which the processor (200) inputs the n-th mask image into a first vision language model; (f) a step in which the first vision language model determines whether there is an undetected object in the n-th mask image; (g) a step in which, if the first vision language model determines that there is an undetected object in the n-th mask image, the first vision language model generates an n+1 OVD prompt for additional object detection; (h) a step in which the processor generates a new OVD prompt based on the first OVD prompt to the n+1st OVD prompt and inputs the nth mask image and the new OVD prompt into the open vocabulary object detection model; (i) a step in which the open vocabulary object detection model additionally detects an object in the nth mask image according to the new OVD prompt; and (j) a step in which the processor determines whether a preset termination condition is satisfied and steps (d) to (j) are repeated until the preset termination condition is satisfied (where n is a natural number applied in ascending order starting from 1 based on the number of times steps (d) to (j) are performed).

[0014] The progressive multi-step object detection method according to the present invention may further include, after step (j), (k) a step in which the processor crops the detected object to generate a cropped object image; (l) a step in which the processor inputs the cropped object image and a second VLM prompt into a second vision language model (520); (m) a step in which the second vision language model classifies the objects included in the cropped object image by class according to the second VLM prompt; and (n) a step in which the processor outputs only the objects classified by class corresponding to the second VLM prompt as the final object detection result among the objects classified by class.

[0015] The first OVD prompt above includes a phrase representing a super-conceptual attribute of an object, and the second VLM prompt above may include a phrase representing a sub-conceptual attribute of an object in comparison with the first OVD prompt.

[0016] The above termination condition may be that the first vision language model determines that no object exists in the nth mask image.

[0017] The above termination condition may be that the number of times steps (d) through (j) are performed satisfies a predetermined number. Effects of the invention

[0018] According to the present invention, object detection is not limited to a single instance based on a first OVD prompt, but is repeatedly performed based on a new OVD prompt generated by a first vision language model and a processor, thereby enabling additional detection of objects that were omitted during the initial detection process (i.e., undetected objects). As a result, the present invention increases the likelihood that all objects within the original image will be detected without omission, thereby effectively improving the low recall problem (i.e., the problem of failing to detect some objects) of conventional open vocabulary object detection models.

[0019] In addition, according to the present invention, even if some objects are falsely detected, the falsely detected objects can be effectively filtered out through the class-based classification process of the second vision language model and the filtering process of the processor. As a result, the present invention can effectively improve the low precision problem of conventional open vocabulary object detection models (i.e., the problem of detecting objects other than the objects to be detected), and the precision of object detection can be significantly improved due to the reduction in the false detection rate.

[0020] However, the effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0021] FIG. 1 is a schematic diagram showing a progressive multiple object detection device according to one embodiment of the present invention. FIGS. 2 to 4 are flowcharts illustrating a progressive multiple-step object detection method performed by the device of FIG. 1. Figure 5 is a diagram illustrating an exemplary original image received by a processor. Figure 6 is a diagram showing an open vocabulary object detection model detecting an object corresponding to 'car' in the original image of Figure 5. Figure 7 is a diagram showing the first mask image generated by the processor by covering the area of ​​the detected object in the original image. Figure 8 is a diagram showing an open vocabulary object detection model additionally detecting an object in the first mask image. Figure 9 is a diagram showing the process of generating a second mask image by masking the detected objects in the original image. Figure 10 is a diagram showing the process of generating a cropped object image. Figure 11 is a diagram showing the processor outputting the final object detection result. Specific details for implementing the invention

[0022] Hereinafter, a progressive multi-pass object detection method according to an embodiment of the present invention will be described in detail with reference to the attached drawings. The attached drawings are provided merely as examples to ensure that the technical concept of the present invention is sufficiently conveyed to a person skilled in the art, and the present invention is not limited to the drawings presented below and can be embodied in various other forms.

[0023] FIG. 1 is a schematic diagram showing a progressive multi-pass object detection device (1000) according to an embodiment of the present invention, and FIGS. 2 to 4 are flowcharts showing a progressive multi-pass object detection method performed by the device (1000) of FIG. 1. Hereinafter, a progressive multi-pass object detection method according to an embodiment of the present invention will be described with reference to FIGS. 1 to 4.

[0024] As shown in FIG. 1, a progressive multi-pass object detection device (1000) according to one embodiment of the present invention may include an input unit (100), a processor (200), a memory (300), an open vocabulary object detection model (400), a first vision language model (510), and a second vision language model (520). And a progressive multi-pass object detection method according to one embodiment of the present invention may be performed by the device (1000) of FIG. 1.

[0025] The input unit (100) may include input means such as a keyboard or a mouse. The user can input an original image, a first OVD prompt, and a second VLM prompt through the input unit (100). As a result, the input unit (100) can receive the original image, the first OVD prompt, and the second VLM prompt. Here, the first OVD prompt refers to a prompt input to the open vocabulary object detection model (400), and the second VLM prompt refers to a prompt input to the second vision language model (520). In the description of the present invention, for the clear distinction of terms, the image input to the input unit (100) will be referred to as the original image.

[0026] Furthermore, the input unit (100) may include a communication module for communication connection with a camera (CCTV, drone camera, etc.). In this case, the input unit (100) may receive an original image (i.e., an image captured by the camera) from the camera.

[0027] The original image, the first OVD prompt, and the second VLM prompt input into the input unit (100) can be transmitted to the processor (200). As a result, the processor (200) receives the original image, the first OVD prompt, and the second VLM prompt (S100), and the memory (300) can store the original image, the first OVD prompt, and the second VLM prompt.

[0028] FIG. 5 is a diagram illustrating an original image received by a processor (200) as an example. In the following description, the first OVD prompt is to be described as an example of a 'car'. Here, the first OVD prompt corresponds to a prompt for the open vocabulary object detection model (400) to detect a 'car' in the original image. The processor (200) can receive the original image of FIG. 5 and the first OVD prompt, which is 'car', from the input unit (100).

[0029] After step S100 above, the processor (200) can input the original image and the first OVD prompt into the open vocabulary object detection model (400) (S200). That is, the processor (200) can input the original image of FIG. 5 and the first OVD prompt called 'car' into the open vocabulary object detection model (400).

[0030] As described above, the open vocabulary object detection model (400) is not limited to classes that have been learned in advance and can detect objects corresponding to an input prompt given in natural language (in the example above, the first OVD prompt, 'car'). Representative open vocabulary object detection models include models such as Grounding-DINO and ViLD. The open vocabulary object detection model can achieve a certain level of object detection performance even for objects of classes that have not been learned in advance.

[0031] The open vocabulary object detection model (400) may be stored in memory (300) in advance. In this case, when the processor (200) performs step S200, it may load the open vocabulary object detection model (400) stored in memory (300) and then input the original image and the first OVD prompt into the open vocabulary object detection model (400).

[0032] Alternatively, the open vocabulary object detection model (400) may be stored in an external database (not shown). In this case, when the processor (200) performs step S200, it may load the open vocabulary object detection model (400) stored in the external database and then input the original image and the first OVD prompt into the open vocabulary object detection model (400).

[0033] When an original image and a first OVD prompt are input to the open vocabulary object detection model (400), the open vocabulary object detection model (400) detects an object in the original image according to the first OVD prompt (S300). That is, the open vocabulary object detection model (400) can detect an object corresponding to 'car' in the original image of FIG. 5. In the present invention, the fact that the open vocabulary object detection model (400) detects an object means that it identifies the location of an object included in the input image (i.e., the original image or the nth mask image) and classifies the class of the object.

[0034] FIG. 6 is a diagram showing the open vocabulary object detection model (400) detecting an object corresponding to 'car' in the original image of FIG. 5.

[0035] The open vocabulary object detection model (400) can output the object's location 'A' and the object's class 'car' as a result of the object detection when it determines that an object located approximately in the center of the original image of FIG. 5 is a 'car' according to the first OVD prompt.

[0036] Here, the object location 'A' output by the open vocabulary object detection model (400) may be a coordinate value defining a bounding box surrounding the object. However, the object location does not necessarily have to be a coordinate value defining a bounding box, and may be a coordinate value defining various shapes such as a polygon. Also, the object class 'automobile' output by the open vocabulary object detection model (400) may be the same as the first OVD prompt.

[0037] Although there are two cars in Fig. 6, the open vocabulary object detection model (400) may fail to detect both cars in only one step (i.e., step S300). That is, when the open vocabulary object detection model (400) detects objects in the original image according to the first OVD prompt, there may be objects that were not detected (i.e., undetected objects).

[0038] To solve these problems, after step S300, the processor (200) can generate the nth mask image by covering the area of ​​the detected object (S400).

[0039] More specifically, the processor (200) can cover the area of ​​the detected object in the original image in various ways, such as covering it with a specific color (e.g., black), blurring it, or mosaicking it. In the present invention, the image generated by covering the area of ​​the detected object is defined as a mask image.

[0040] In the present invention, n is a natural number applied in ascending order starting from 1 based on the number of times steps S400 to S1000 are performed. That is, if steps S400 to S1000 are performed once, n is 1; if steps S400 to S1000 are performed twice, n is 2; and if steps S400 to S1000 are performed three times, n is 3. Accordingly, the mask image generated by the processor (200) in the above step S400 becomes the first mask image.

[0041] FIG. 7 is a diagram showing the first mask image generated by the processor (200) by covering the area of ​​the detected object in the original image.

[0042] When the open vocabulary object detection model (400) detects an object in the original image and outputs the location of the object 'A', the processor (200) can generate a first mask image as shown in FIG. 7 by covering the area of ​​'A' in the original image (e.g., covering it with black).

[0043] Afterwards, the processor (200) can input the first mask image into the first vision language model (510) (S500).

[0044] At this time, the processor (200) can input a first VLM prompt into the first vision language model (510) along with a first mask image. The first VLM prompt may be a prompt intended to determine whether there is an undetected object (e.g., 'car') in the first mask image.

[0045] The first vision language model (510) and the first VLM prompt may be stored in memory (300) in advance. In this case, when the processor (200) performs step S500, it may retrieve the first vision language model and the first VLM prompt stored in memory (300), and then input the first mask image and the first VLM prompt into the first vision language model (510).

[0046] Alternatively, the first vision language model (510) may be stored in an external database instead of memory (300). In this case, when the processor (200) performs step S500, it may load the first vision language model stored in the external database and then input the first mask image and the first VLM prompt into the first vision language model (510).

[0047] When a first mask image and a first VLM prompt are input into the first vision language model (510), the first vision language model (510) determines whether there is an undetected object in the first mask image according to the first VLM prompt (S600).

[0048] Since the area of ​​the object detected in step S300 is obscured in the first mask image (see FIG. 7), the first vision language model (510) can determine with high accuracy whether there is an undetected object by focusing on the area other than the obscured area.

[0049] According to FIG. 7, a car exists as an undetected object in the lower right corner of the first mask image. Accordingly, the first vision language model (510) can determine that there is an undetected object in the first mask image.

[0050] In this way, if the first vision language model (510) determines that there is an undetected object in the first mask image, it can generate an n+1 OVD prompt for additional object detection (S700).

[0051] As described above, since n is a natural number applied in ascending order from 1 based on the number of times steps S400 to S1000 are performed, when the number of times steps S400 to S1000 are performed is 1, the prompt generated by the first vision language model (510) becomes the second OVD prompt.

[0052] The second OVD prompt corresponds to a prompt that enables the open vocabulary object detection model (400) to better detect a specific object called a 'car' in the first mask image. For example, the first vision language model (510) can generate the second OVD prompt as a 'white car' by taking into account that the color of the undetected object in the first mask image is white.

[0053] Unlike the previous case, if the first vision language model (510) determines that there are no undetected objects in the first mask image, the S1000 step may be performed instead of the S700 step after the S600 step, and a detailed explanation of this will be given later.

[0054] When the first vision language model (510) generates a second OVD prompt for object detection, the processor (200) can generate a new OVD prompt based on the first OVD prompt to the n+1 OVD prompt, and input the first mask image and the new OVD prompt into the open vocabulary object detection model (400) (S800).

[0055] Here, since n is 1 when the number of times steps S400 to S1000 are performed is 1, the processor (200) generates a new OVD prompt based on the first OVD prompt to the second OVD prompt.

[0056] When the first OVD prompt received by the processor (200) in step S100 is 'car' and the second OVD prompt generated by the first vision language model (510) in step S700 is 'white car', the processor (200) can combine the first OVD prompt and the second OVD prompt to generate a new OVD prompt called 'car', 'white car'. Then, the processor (200) can input the first mask image and the new OVD prompt (i.e., 'car', 'white car') into the open vocabulary object detection model (400).

[0057] Even if the first OVD prompt is somewhat abstract or general, the conditions for object detection can be automatically supplemented through a new OVD prompt generated by the processor (200). According to the present invention, the possibility of an undetected object occurring due to the ambiguity of the first OVD prompt (or the n+1 OVD prompt) can be significantly reduced.

[0058] When a first mask image and a new OVD prompt are input to the open vocabulary object detection model (400), the open vocabulary object detection model (400) additionally detects objects in the first mask image according to the new OVD prompt (S900). That is, the open vocabulary object detection model (400) can additionally detect objects corresponding to 'car' or 'white car' in the first mask image of FIG. 7.

[0059] FIG. 8 is a diagram showing the open vocabulary object detection model (400) additionally detecting an object in the first mask image.

[0060] The open vocabulary object detection model (400) can determine that the object located in the upper left corner of the first mask image of FIG. 8 is a 'white car' according to the new OVD prompt. In this case, the open vocabulary object detection model (400) can output the object's location 'B' and the object's class 'white car' as a result of the object detection. Here, the object located approximately in the upper left corner of the first mask image of FIG. 8 is actually a 'person' and not a 'white car'. That is, the open vocabulary object detection model (400) can detect an object that does not correspond to the new OVD prompt (i.e., false object detection), and a solution to the problem of such false object detection will be described later.

[0061] Additionally, the open vocabulary object detection model (400) can determine that the object located in the lower right corner of the first mask image of FIG. 8 is a 'white car' according to the new OVD prompt. In this case, the open vocabulary object detection model (400) can output the object's location 'C' and the object's class 'white car' as a result of the object detection.

[0062] Here, the object locations 'B' and 'C' output by the open vocabulary object detection model (400) may be coordinate values ​​that define a bounding box surrounding the object. However, even in this case, the object location does not necessarily have to be a coordinate value that defines a bounding box, but may be a coordinate value that defines various shapes, such as a polygon. Also, the object class 'white car' may be included in the new OVD prompt (e.g., 'car', 'white car').

[0063] After the above S900 step, the processor (200) can determine whether the preset termination condition has been satisfied (S1000).

[0064] Here, the termination condition may be pre-set in memory (300). When the processor (200) performs step S1000, it may retrieve the termination condition pre-set in memory (300) and determine whether the current state satisfies the termination condition.

[0065] As an example, a termination condition set in memory (300) may be that the first vision language model (510) determines that there are no longer any objects in the nth mask image.

[0066] That is, if the first vision language model (510) determines that there are no longer any objects in the first mask image as a result of performing the S600 step, the S1000 step may be performed instead of the S700 step after the S600 step, and at this time, the processor (200) may determine that the pre-set termination condition is satisfied and perform the S1100 step.

[0067] More specifically, the first vision language model (510) may not generate an n+1 OVD prompt for additional object detection after performing the S600 step, but rather determine whether the processor (200) has satisfied a preset termination condition. Here, since the preset termination condition is that the first vision language model (510) determines that no object exists in the n-th mask image, the processor (200) may determine that the preset termination condition has been satisfied and perform the S1100 step.

[0068] In contrast, if the first vision language model (510) determines that an object exists in the first mask image as a result of performing the above S600 step, the processor (200) and the first vision language model (510) may repeat steps S400 through S1000 until the first vision language model (510) determines that no object exists in the nth mask image.

[0069] As another example, the termination condition set in memory (300) may be that the number of times steps S400 to S1000 are performed satisfies a predetermined number. The predetermined number of termination conditions stored in memory (300) may be one or more.

[0070] When the predetermined number of times regarding the termination condition set in memory (300) is 1, if steps S400 to S1000 are performed once, the processor (200) determines that the predetermined termination condition is satisfied and can perform step S1100.

[0071] When the predetermined number of times regarding the termination condition set in memory (300) is 2, if steps S400 to S1000 are performed once, the processor (200) and the first vision language model (510) can perform steps S400 to S1000 again. Afterward, if steps S400 to S1000 are performed again, the processor (200) determines that the predetermined termination condition is satisfied and can perform step S1100.

[0072] Additionally, when the predetermined number of times regarding the termination condition set in memory (300) is 3, if steps S400 to S1000 are performed once or twice, the processor (200) and the first vision language model (510) may perform steps S400 to S1000 again. Afterward, if steps S400 to S1000 are performed a total of 3 times, the processor (200) determines that the predetermined termination condition is satisfied and may perform step S1100.

[0073] As another example, the termination condition set in memory (300) may be a condition that satisfies both the determination that the first vision language model (510) no longer has an object in the nth mask image and the fact that the number of times steps S400 to S1000 are performed is greater than a predetermined number.

[0074] In the following, we will explain the case where steps S400 to S1000 are performed a second time after being performed once.

[0075] If the processor (200) determines in the above S1000 step that the pre-set termination condition is not satisfied, it can generate a second mask image by covering the area of ​​the object detected so far (S400).

[0076] FIG. 9 is a diagram showing the process of generating a second mask image by the processor (200) covering the previously detected object in the original image. That is, the processor (200) can generate a second mask image as shown in FIG. 9 by covering the area of ​​the object detected in step S300 and the area of ​​the object detected in step S900 in the original image.

[0077] However, it is not necessary to perform masking processing on the original image. That is, the processor (200) may generate a second mask image such as FIG. 9 by masking the area of ​​the object detected in step S900 in the first mask image.

[0078] When the open vocabulary object detection model (400) detects objects and outputs the locations of objects 'A', 'B', and 'C', the processor (200) can generate a second mask image as shown in FIG. 9 by covering the locations of objects, such as the 'A' area, 'B' area, and 'C' area (e.g., covering them with black).

[0079] Afterwards, the processor (200) can input the second mask image into the first vision language model (510) (S500).

[0080] At this time, the processor (200) can input a first VLM prompt along with a second mask image into the first vision language model (510). At this time, the first VLM prompt may be a prompt intended to determine whether there is an undetected object (e.g., 'car', 'white car') in the first mask image.

[0081] When a second mask image and a first VLM prompt are input into the first vision language model (510), the first vision language model (510) determines whether there is an undetected object in the second mask image according to the first VLM prompt (S600).

[0082] Since the area of ​​the object detected in step S300 and the area of ​​the object detected in step S900 are obscured in the second mask image (see FIG. 9), the first vision language model (510) can determine with high accuracy whether there is an undetected object by focusing on the area other than the obscured area.

[0083] According to FIG. 9, there are no longer any cars in the second mask image. Accordingly, the first vision language model (510) can determine that there are no undetected objects in the second mask image.

[0084] In contrast, if the first vision language model (510) determines that there is an undetected object in the second mask image, the first vision language model (510) generates a third OVD prompt for additional object detection (corresponding to step S700). Here, the third OVD prompt corresponds to a prompt that enables the open vocabulary object detection model (400) to better detect a specific object called a 'car' in the second mask image. For example, if the first vision language model (510) determines that the undetected object in the second mask image is next to a tree, it can generate the third OVD prompt as 'car next to a tree'.

[0085] Subsequently, the processor (200) generates a new OVD prompt based on the first to third OVD prompts and inputs the second mask image and the new OVD prompt into the open vocabulary object detection model (400) (corresponding to step S800). At this time, the processor (200) can combine the first to third OVD prompts to generate a new OVD prompt such as 'car', 'white car', and 'car next to a tree', and input the new OVD prompt together with the second mask image into the open vocabulary object detection model (400).

[0086] Subsequently, the open vocabulary object detection model (400) additionally detects objects in the second mask image according to the new OVD prompt (corresponding to step S900).

[0087] In contrast, if the first vision language model (510) determines that there are no undetected objects in the second mask image, instead of the S700 step being performed after the S600 step, the S1000 step in which the processor (200) determines whether the pre-set termination condition is satisfied may be performed.

[0088] In step S1000, if the first vision language model (510) determines that there are no longer any objects in the n-th mask image, which is a preset termination condition, the processor (200) determines that the preset termination condition is satisfied and can perform step S1100.

[0089] Alternatively, if the number of times steps S400 to S1000 are performed is 2, which is a preset termination condition, the processor (200) may determine that the preset termination condition is satisfied and perform step S1100.

[0090] For reference, if the processor (200) determines in step S1000 that the pre-set termination condition is not satisfied, a third mask image can be generated by covering the area of ​​the object detected so far (see step S400).

[0091] In contrast, if the processor (200) determines at step S1000 that a pre-set termination condition is satisfied, the objects detected so far (e.g., objects detected at step S300 of the first round and objects detected at step S900 of the first round) can be cropped to generate cropped object images (S1100). When the processor (200) crops the detected objects to generate cropped object images, the second vision language model (520) can classify the class of the objects in the cropped object images with high accuracy.

[0092] FIG. 10 is a diagram showing the processor (200) generating cropped object images. According to the example described above, the open vocabulary object detection model (400) detected one object in the original image according to the first OVD prompt and detected two objects in the first mask image according to the new OVD prompt. In this case, the processor (200) can crop one object in the original image (see FIG. 10a) and crop two objects in the first mask image (see FIG. 10b and FIG. 10c) to generate a total of three cropped object images.

[0093] Subsequently, the processor (200) can input the cropped object image and the second VLM prompt into the second vision language model (520) (S1200). Here, the second VLM prompt may be a prompt for an object that the user of the device (1000) truly wants to detect in the original image. In the following description, the second VLM prompt will be described using the example of an 'SUV' (Sport Utility Vehicle), which is a type of automobile.

[0094] The first OVD prompt described above includes a phrase representing a super-conceptual attribute of an object (e.g., 'automobile'), and the second VLM prompt may include a phrase representing a sub-conceptual attribute of an object (e.g., 'SUV') in comparison to the first OVD prompt.

[0095] Here, object attributes refer to characteristics that define or identify an object, such as its kind, shape, color, size, action, and location.

[0096] In addition, super-concept attributes of an object refer to attributes that represent the object in a more comprehensive category (e.g., 'car', 'animal', 'home appliance', etc.), and sub-concept attributes refer to attributes that limit the category to a more specific one than the super-concept attributes (e.g., 'SUV', 'black cat', 'wall-mounted air conditioner', etc.).

[0097] In this way, if the first OVD prompt includes a phrase representing a higher-level conceptual attribute of an object, the open vocabulary object detection model (400) can significantly reduce the number of undetected objects because it comprehensively detects objects in the original image through the first OVD prompt. Furthermore, if the second VLM prompt includes a phrase representing a lower-level conceptual attribute of an object, the processor (200) can ultimately significantly reduce the number of false positive objects because it can output only the objects that the user truly intends to detect among the objects detected by the open vocabulary object detection model (400).

[0098] The second vision language model (520) may be stored in memory (300) in advance. In this case, when the processor (200) performs step S1200, it may load the second vision language model stored in memory (300) and then input the crop object image and the second VLM prompt into the second vision language model (520).

[0099] Alternatively, the second vision language model (520) may be stored in an external database. In this case, when the processor (200) performs step S1200, it may load the second vision language model stored in the external database and then input the crop object image and the second VLM prompt into the second vision language model (520).

[0100] The processor (200) can input cropped object images one by one into the second vision language model (520). Alternatively, the processor (200) can input multiple cropped object images into the second vision language model (520) at once.

[0101] When a cropped object image and a second VLM prompt are input to the second vision language model (520), the second vision language model (520) classifies the objects contained in the cropped object image by class according to the second VLM prompt (S1300). Here, the second vision language model (520) may be a model trained to classify the objects contained in an image by class according to the prompt when an image and a prompt are input.

[0102] The object included in the cropped object image of FIG. 10a is an SUV, which is a type of automobile. Accordingly, when the second vision language model (520) receives the cropped object image of FIG. 10a and the second VLM prompt, it can classify the object included in the cropped object image of FIG. 10a as an 'SUV' according to the second VLM prompt.

[0103] In contrast, the object included in the cropped object image of FIG. 10b is a person. Accordingly, when the second vision language model (520) receives the cropped object image of FIG. 10b and the second VLM prompt, it can classify the object included in the cropped object image of FIG. 10b as 'OTHER'. Here, 'OTHER' means a class that does not correspond to the second VLM prompt (e.g., 'SUV').

[0104] Additionally, the object included in the cropped object image of FIG. 10c is a car, but it is a sedan rather than an SUV. Accordingly, when the second vision language model (520) receives the cropped object image of FIG. 10c and the second VLM prompt, the object included in the cropped object image of FIG. 10c can also be classified as 'OTHER'.

[0105] In this way, when the second vision language model (520) classifies the objects included in the cropped object image by class according to the second VLM prompt, the processor (200) outputs only the objects classified by class corresponding to the second VLM prompt as the final object detection result among the objects classified by class (S1400).

[0106] In the above example, the object included in the cropped object image of FIG. 10a was classified as 'SUV' corresponding to the second VLM prompt. In contrast, the object included in the cropped object images of FIG. 10b and FIG. 10c was classified as 'OTHER'. Accordingly, the processor (200) outputs the object included in the cropped object image of FIG. 10a as the final object detection result, and excludes the object included in the cropped object images of FIG. 10b and FIG. 10c from the final object detection result.

[0107] FIG. 11 is a diagram showing the processor (200) outputting the final object detection result.

[0108] The processor (200) can generate an output image such as FIG. 11 by processing only the object according to FIG. 10a in the original image into a bounding box. Here, the bounding box of FIG. 11 indicates the location of the object finally detected. Subsequently, the processor (200) can display the generated output image on the screen of the device (1000), and at this time, the processor (200) can also display the class of the object finally detected (e.g., 'SUV') on the screen of the device (1000).

[0109] As described above, although the present invention has been explained by limited embodiments and drawings, the present invention is not limited to the above embodiments, and various modifications and variations are possible from this description by those skilled in the art to which the present invention belongs. For example, although the first vision language model (510) and the second vision language model (520) have been described above, the first vision language model (510) and the second vision language model (520) may have the same model itself but different functions. Alternatively, the first vision language model (510) and the second vision language model (520) may have different models itself and different functions. Therefore, the technical concept of the present invention should be understood only by the claims, and all equivalent or equivalent variations thereof shall be considered to fall within the scope of the technical concept of the present invention. Explanation of the symbols

[0110] 100: Input section 200: Processor 300: Memory 400: Open Vocabulary Object Detection Model 510: The first vision language model 520: Second Vision Language Model 1000: Incremental multi-cycle object detection device

Claims

Claim 1 (a) A step in which a processor receives an original image; (b) A step in which the processor inputs the original image and a first OVD prompt into an open vocabulary object detection model; (c) A step in which the open vocabulary object detection model detects an object in the original image according to the first OVD prompt; (d) A step in which the processor generates an n-th mask image by masking the area of ​​the detected object; (e) A step in which the processor inputs the n-th mask image into a first vision language model; (f) A step in which the first vision language model determines whether there is an undetected object in the n-th mask image; (g) A step in which, if the first vision language model determines that there is an undetected object in the n-th mask image, a step in which the n+1-th OVD prompt is generated for additional object detection; (h) A step in which the processor generates a new OVD prompt based on the first OVD prompt to the n+1-th OVD prompt, and inputs the n-th mask image and the new OVD prompt into the open vocabulary object detection model; (i) A step in which the open vocabulary object detection model A step of additionally detecting objects in the nth mask image according to the new OVD prompt; (j) a step in which the processor determines whether a preset termination condition is satisfied, and steps (d) through (j) are repeated until the preset termination condition is satisfied; (k) a step in which the processor crops the detected objects to generate a cropped object image; (l) a step in which the processor inputs the cropped object image and a second VLM prompt into a second vision language model; (m) a step in which the second vision language model classifies the objects included in the cropped object image by class according to the second VLM prompt.and (n) a step in which the processor outputs only the objects classified by the class corresponding to the second VLM prompt as the final object detection result among the objects classified by the class. (wherein n is a natural number applied in ascending order starting from 1 based on the number of times steps (d) through (j) are performed); Claim 2 delete Claim 3 A progressive multiple object detection method according to claim 1, characterized in that the first OVD prompt includes a phrase representing a higher-level conceptual attribute of an object, and the second VLM prompt includes a phrase representing a lower-level conceptual attribute of an object when compared with the first OVD prompt. Claim 4 A progressive multiple object detection method according to claim 1, wherein the termination condition is that the first vision language model determines that no object exists in the nth mask image. Claim 5 A progressive multiple object detection method according to claim 1, wherein the termination condition is that the number of times steps (d) through (j) are performed satisfies a predetermined number.

Images