Method and apparatus for determining that a detected object belongs to a particular class

Abstract

Methods and apparatus for determining that a detected object belongs to a particular class are disclosed. Using a first object detector, detecting an object in a first image frame of a video sequence is performed; and a first probability score indicative of a probability that the detected object belongs to a particular class is determined. Using a second object detector, detecting an object in a second image frame of the video sequence is performed; and a second probability score indicative of a probability that the detected object belongs to the particular class is determined. In a case where the first probability score is below a first classification threshold and the second probability score is above a second classification threshold, one or more of decreasing the first classification threshold and increasing the first probability score is performed. The detected object in the first image frame is determined to belong to the particular class on a condition that the increased first probability score is above the first classification threshold, the first probability score is above the decreased first classification threshold, or the increased first probability score is above the decreased first classification threshold.

Description

Technical Field

[0001] This invention relates to target detection, and more specifically, to determining that a detected target belongs to a particular category. Background Technology

[0002] Object detection in image frames of a video sequence, using an object detector (e.g., using a machine learning model that may or may not be based on a neural network), involves identifying the location of an object within an image frame and determining the category to which the detected object belongs. In some cases, it is desirable to quickly detect objects of a specific category, and to perform detection in each image frame in a video sequence where at least a portion of one such object is visible. One such scenario is, for example, where objects belonging to a specific category should be occluded in each image frame. However, when a portion of an object is occluded in an image frame (e.g., an image frame where the object is only partially within the captured scene or an image frame where the object is temporarily partially occluded in the captured scene), it can be difficult to determine whether the detected object in the image frame belongs to a specific category. One solution is, for example, to enhance the object detector used, enabling it to determine that the detected object in the image frame belongs to a specific category even when a portion of the object is occluded. However, this may be impossible, and if possible, it would increase the processing time required for the enhanced object detector to perform this determination, which is detrimental when fast object detection is desired. Furthermore, it should be noted that even without enhancements, the object detector may have a processing time per image frame that, depending on the frame rate of the video sequence, the number of image frames that can be processed per time unit is less than the number of images per time unit. Therefore, further increasing the processing time of the object detector is undesirable.

[0003] Therefore, there is still a need to improve the determination of whether detected targets belong to a specific category. Summary of the Invention

[0004] The purpose of this invention is to overcome or at least mitigate the problems and defects of the prior art.

[0005] According to a first aspect, a method for determining that a detected target belongs to a specific category is provided. The method includes performing a first target detection using a first target detector. The first target detection includes: detecting a target in a first image frame of a video sequence; and determining a first probability score indicating the probability that the detected target in the first image frame belongs to a specific category. The method further includes performing a second target detection using a second target detector. The second target detection includes: detecting a target in a second image frame of a video sequence; and determining a second probability score indicating the probability that the detected target in the second image frame belongs to a specific category, wherein the second image frame follows the first image frame in the video sequence. Compared to the second target detector, the first target detector has higher target detection accuracy, and the first target detector requires a first processing time for performing the first target detection, which is longer than the second processing time required by the second target detector for performing the second target detection. The second target detection is performed in parallel with the first target detection. The method further includes: if the determined first probability score is lower than a first classification threshold and the determined second probability score is higher than a second classification threshold, performing one or more of decreasing the first classification threshold and increasing the first probability score. The method further includes: under the condition that the increased first probability score is higher than the first classification threshold, the first probability score is higher than the decreased first classification threshold, or the increased first probability score is higher than the decreased first classification threshold, the first target detector determines that the detected target in the first image frame belongs to a specific category.

[0006] This invention utilizes the understanding that a second object detector can be introduced to perform second object detection, wherein the second object detector requires a second processing time for performing the second object detection, which is shorter than the first processing time required by the first object detector to perform the first object detection. Therefore, the second object detector can perform the second object detection in a second image frame before the first object detector completes the first object detection in the first image frame, and thus, the first object detector can take into account the result of the second object detection. Since the second image frame follows the first image frame, any object partially occluded in the first image frame may be fully visible or at least less occluded in the second image frame. Therefore, even if the second object detector is inferior to the first object detector in object detection for the same image frame, the result of the second object detection can be more decisive regarding the probability that the object belongs to a particular category.

[0007] By including a second target detector with a second processing time shorter than that of the first target detector, second target detection of a second image frame after the first image frame can be completed before the first target detection in the first image frame is completed, and the result of the second target detection can be used to enhance the result from the first target detection. Therefore, even if the first target detector itself cannot determine that the detected target belongs to a specific category, the first target detector can also determine that the target belongs to a specific category.

[0008] It should be noted that even if the first approach may lead to an increase in the number of times the first target detector incorrectly determines that the detected target belongs to a specific category, this is acceptable because it will also lead to a decrease in the number of times the detected target actually belongs to a specific category but the first target detector is unsure that it belongs to a specific category.

[0009] In one embodiment, in the first target detection, a target is detected in a first region of a first image frame, and in the second target detection, a target is detected in a second region of a second image frame. Then, further, conditioned that the distance from the second region to the first region is less than a distance threshold, one or more of the following actions are performed: decreasing a first classification threshold and increasing a first probability score.

[0010] By introducing this further condition, the number of times the first object detector incorrectly determines that a detected target belongs to a specific category can be reduced. Specifically, if a detected target in the second image frame is located at a position longer than a distance threshold from a detected target in the first image frame, the detection is unlikely to be related to the same target, and therefore, since the result from the second object detector is unlikely to be related to the detection of the first object detector, the first classification threshold should not be lowered and the first probability score increased.

[0011] The distance threshold can increase as the temporal distance between the first and second image frames in the video sequence increases. This is advantageous if the detected target is expected to move at a certain speed such that the distance between the second and first regions will depend on the time between the first and second image frames (i.e., the time between capturing the first and second image frames). The increase in the distance threshold can also depend on a specific target category. This is advantageous because different categories of targets (such as people and cars) will move at different expected speeds. Additionally or alternatively, the method may further include: obtaining the speed of the detected target in the first image frame, wherein the increase in the distance threshold depends on the obtained speed. By obtaining the speed, it is possible to estimate the expected distance from the second region to the first region, and the distance threshold can be set based on this estimated distance, for example, set to the estimated distance or a given percentage above the estimated distance, to cover any errors in the obtained speed and variations related to the obtained speed.

[0012] The method may further include performing a third object detection using a second object detector. The third object detection includes: detecting an object in a third image frame of the video sequence; and determining a third probability score indicating the probability that the detected object in the third image frame belongs to a specific category, wherein the third image frame follows the first image frame in the video sequence. The first processing time is longer than the second processing time plus the third processing time required by the second object detector to perform the third object detection. The third object detection is performed sequentially with the second object detection and in parallel with the first object detection. Then, further, conditioned that the determined third probability score is higher than a second classification threshold, one or more of decreasing the first classification threshold and increasing the first probability score are performed.

[0013] When the time taken for the second object detector to perform object detection in both the second and third image frames is shorter than the time taken for the first object detector to perform object detection in the first image frame, it is beneficial to use the results of the second object detector's object detection in both the second and third image frames to adjust the first object detector's object detection in the first image frame. Specifically, if either the second probability score or the third probability score is lower than the second classification threshold, the result from the second object detector is no result, and therefore, the object detection of the first object detector is only adjusted by decreasing the first classification threshold and increasing the first probability score when both the second probability score and the third probability score are higher than the second classification threshold.

[0014] The method may further include using a first target detector to mask detected targets that are identified as belonging to a specific category in a first image frame.

[0015] The masking envisioned herein may, for example, cover the target with a solid / opaque or semi-transparent color, including applying motion blur to the target so that the target is no longer easily identifiable, and / or, for example, forcibly pixelating and / or blurring the target in the output image stream so that the target is no longer easily identifiable, etc. In other envisioned embodiments, masking may include making the target itself completely or partially transparent, so that the background is visible through the target. This may be possible if, for example, an image frame with no background in front of the target is available (from, for example, an earlier image frame).

[0016] By enabling the first target detector to determine that the detected target belongs to a specific category when the first target detector itself cannot determine that the detected target belongs to a specific category and therefore will not be occluded, the detected target that is determined to belong to the specific category is occluded in the first image frame.

[0017] According to the second aspect, a non-transitory computer-readable storage medium is provided, having instructions stored thereon, which, when executed in a device having a processor, are used to implement the method according to the first aspect or the method according to the second aspect.

[0018] The optional additional features described above in the method according to the first aspect, where applicable, also apply to non-transitory computer-readable storage media according to the second aspect. To avoid unnecessary repetition, refer to the foregoing.

[0019] According to a third aspect, an apparatus is provided for determining that a detected target belongs to a specific category. The apparatus includes a first target detector configured to perform a first target detection, a second target detector configured to perform a second target detection, and circuitry configured to perform a change function. The first target detection includes: detecting a target in a first image frame of a video sequence; and determining a first probability score indicating the probability that the detected target in the first image frame belongs to a specific category. The second target detection includes: detecting a target in a second image frame of a video sequence; and determining a second probability score indicating the probability that the detected target in the second image frame belongs to a specific category, wherein the second image frame follows the first image frame in the video sequence. The first target detector has higher target detection accuracy than the second target detector, and the first target detector requires a first processing time for performing the first target detection, which is longer than the second processing time required by the second target detector for performing the second target detection. The second target detection is performed in parallel with the first target detection. The change function is configured to perform one or more of decreasing the first classification threshold and increasing the first probability score if the determined first probability score is lower than a first classification threshold and the determined second probability score is higher than the second classification threshold. The first target detector is further configured to determine that the detected target in the first image frame belongs to a specific category if the increased first probability score is higher than a first classification threshold, the first probability score is higher than a decreased first classification threshold, or the increased first probability score is higher than a decreased first classification threshold.

[0020] Where applicable, the aforementioned optional additional features of the method according to the first aspect also apply to the apparatus according to the fourth aspect. To avoid unnecessary repetition, refer to the foregoing.

[0021] The further scope of the invention will become apparent from the detailed description given below. However, it should be understood that although preferred embodiments of the invention are shown, various variations and modifications within the scope of the invention will become apparent to those skilled in the art from the detailed description; therefore, the detailed description and specific examples are given by way of illustration only.

[0022] Therefore, it should be understood that the present invention is not limited to such devices and methods because certain components of the described apparatus or the operation of the described method can be modified. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It must be noted that, as used in the specification and appended claims, the words “a,” “the,” and “the” are intended to mean the presence of one or more elements unless the context clearly specifies otherwise. Thus, for example, a reference to “a unit” or “the unit” can include several devices, etc. Furthermore, the words “comprising,” “including,” “containing,” and similar wording do not exclude other elements or steps. Attached Figure Description

[0023] The above and other aspects of the invention will now be described in more detail with reference to the accompanying drawings. The drawings should not be considered limiting, but rather for illustration and understanding.

[0024] Figure 1 A flowchart is shown related to an embodiment of a method for determining that a detected target belongs to a specific category.

[0025] Figures 2a to 2f The illustration shows six image frames, in which the car is temporarily obscured by buildings to varying degrees in the first three image frames, and is fully visible in the last three image frames.

[0026] Figure 3 The diagram illustrates an embodiment of a device for determining whether a detected target belongs to a specific category. Detailed Implementation

[0027] The invention will now be described below with reference to the accompanying drawings, in which presently preferred embodiments of the invention are illustrated. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0028] This invention is applicable to scenarios where it is desirable to detect a target of a specific category in each image frame of a video sequence, where at least a portion of such a target is visible. One such scenario is, for example, where targets belonging to a specific category should be occluded in each image frame. Other examples include scenarios where the number of targets belonging to a specific category should be counted, scenarios where an alarm should be triggered upon detection of a target belonging to a specific category, and scenarios where targets belonging to a specific category should be identified if they remain in the scene for a predetermined period of time. Targets of a specific category should also be detected in image frames when a first target detector has difficulty detecting targets of a specific category, for example, in image frames where a portion of the target is occluded (e.g., in image frames where the target only partially enters the captured scene or in image frames where the target is temporarily partially occluded in the captured scene). Other situations where the first target detector has difficulty detecting targets of a specific category include when lighting conditions change between image frames and when reflections in the window become stronger.

[0029] The present invention is further applicable to scenarios where object detection needs to be performed in real time, such as scenarios where object detection is performed in image frames of a video sequence and where the detected objects in the video sequence are masked in real time before the video sequence is played back.

[0030] The present invention is particularly advantageous in scenarios where the processing time of the first target detector for processing image frames is longer than the time between image frames of the video sequence according to the frame rate of the video sequence.

[0031] Now about Figure 1 Flowcharts and Figures 2a to 2f The illustration shows six image frames to describe an embodiment of method 100 for determining whether a detected target belongs to a specific category.

[0032] Method 100 includes performing a first object detection S110 using a first object detector. The first object detector can be, for example, an object detector using any suitable machine learning model. Such a machine learning model can be based on a neural network. For example, convolutional neural networks are well known, and object detectors using such convolutional neural networks provide the category to which the detected object belongs, along with a bounding box, as a result. In alternative instance segmentation or panoptic segmentation, a mask is involved instead of a bounding box. It should be noted that this mask received from the object detector using the segmentation is different from a mask intended to blur or otherwise anonymize the object. Instead, it is a mask indicating which pixels are associated with the detected object. From the segmentation, the probability of whether the object belongs to a particular category is received for each pixel. Each pixel from the segmentation can be associated with a corresponding pixel in an image frame, or it can be associated with a corresponding pixel block (i.e., two or more adjacent pixels) in an image frame. In the latter case, resolution reduction is performed during segmentation. Alternatively, a non-neural network-based machine learning model, such as the Viola–Jones object detection framework, can be used.

[0033] The first target detection S110 includes: detecting a target in a first image frame of the video sequence S112; and determining a first probability score S114 indicating the probability that the detected target in the first image frame belongs to a specific category. Detection S112 may include identifying the location of the detected target in the first image frame, and this location may be indicated by a bounding box, which is a box (preferably the smallest box) surrounding the detected target in the first image frame. The first probability score may, for example, be a value between zero (“0”) and one (“1”) or a percentage value between zero percent (“0%”) and one hundred percent (“100%”). The first target detection S110 may also include determining one or more additional probability scores indicating the probability that the detected target in the first image frame belongs to a category other than the specific category.

[0034] The method further includes performing a second target detection S120 using a second target detector. The second target detector can be any of the types described with respect to the first target detector, but it is suitable for having shorter processing times and generally lower detection accuracy. The second target detector can be further specialized, for example, for detecting specific categories of targets, analyzing infrared spectra, etc.

[0035] The second object detection includes: detecting an S122 object in a second image frame of a video sequence; and determining a second probability score that S124 indicates the probability that the detected object in the second image frame belongs to a specific category, where, in the video sequence, the second image frame comes after the first image frame. The detection S122 may include identifying the position of the detected object in the second image frame, and the position may be indicated by a bounding box, which is a box (preferably, the smallest box) that encloses the detected object in the second image frame. The second probability score may be, for example, a value between zero ("0") and one ("1") or a percentage value between zero percent ("0%") and one hundred percent ("100%"). The second object detection S120 may also include determining one or more additional probability scores that indicate the corresponding probabilities that the detected object in the second image frame belongs to other categories than the specific category.

[0036] Compared with the second object detector, the first object detector has higher object detection accuracy. In addition, the first object detector requires a first processing time for performing the first object detection, and compared with the second processing time required by the second object detector for performing the second object detection, the first processing time is longer. Since achieving higher accuracy generally requires longer processing time, the higher object detection accuracy and longer processing time of the first object detector are usually correlated. Introducing the second object detector enables the second object detection to be performed in the second image frame in a time shorter than the first processing time of the first object detector. Therefore, it is allowed that the object detection accuracy of the second object detector is lower in order to achieve a shorter processing time of the second object detector.

[0037] The second object detection S120 is performed in parallel with the first object detection S110. However, since the second processing time required by the second object detector for performing the second object detection S120 is shorter than the first processing time required by the first object detector for performing the first object detection S110, if the first object detection S110 and the second object detection S120 start simultaneously, a part of the first object detection S110 will be performed after the second object detection S120 is completed.

[0038] Under the condition C135 that the determined first probability score is lower than the first classification threshold (PS1 < CT1) and the determined second probability score is higher than the second classification threshold (PS2 > CT2), perform one or both of the actions of reducing the first classification threshold S140 to a reduced first classification threshold and increasing the first probability score to an increased first probability score (PSI1). Both of these actions will make the first object detector more inclined to determine that the detected object belongs to a specific category.

[0039] The increase in the first probability score and / or the decrease in the first classification threshold can be a fixed value. Alternatively, the increase in the first probability score and / or the decrease in the first classification threshold can be a value that depends on how much the determined second probability score is higher than the second classification threshold. For example, the increase in the first probability score and / or the decrease in the first classification threshold can be the same as the difference between the determined second probability score and the second classification threshold. Alternatively, it can be a percentage of the difference between the determined second probability score and the second classification threshold. Typically, only one of increasing the first probability score and decreasing the first classification threshold is performed. If both increasing the first probability score and decreasing the first classification threshold are performed, the increase and decrease in value can each be half of what would be achieved if only increasing the first probability score and decreasing the first classification threshold were performed.

[0040] The first classification threshold is typically a threshold for a first probability score. If the score is higher than this threshold, the first object detector will determine that the detected target belongs to a specific category, even without considering the result of the second object detection performed by the second object detector on the second image frame. Preferably, if the determined first probability score is higher than the first classification threshold, none of the actions are performed.

[0041] The second classification threshold is typically a threshold for the second probability score; if the score is below this threshold, the second object detector determines that the detected target does not belong to a specific category. Preferably, if the determined second probability score is (equal to or lower than) the second classification threshold, none of the actions are performed.

[0042] Under the condition C145 that the increased first probability score is higher than the first classification threshold (PSI1>CT1), the first probability score is higher than the decreased first classification threshold (PS1>CTR1), or the increased first probability score is higher than the decreased first classification threshold (PSI1>CTR1), the first target detector determines that the detected target in the first image frame of S150 belongs to a specific category.

[0043] If a detected target is partially occluded in the first image frame but fully visible or at least less occluded in the second image frame, the second probability score determined by the second target detector may be higher than the second classification threshold. This means that the detected target in the second image frame is determined by the second target detector to belong to a specific category. This classification by the second target detector is then used to make the first target detector more inclined to determine that the detected target in the first image frame belongs to the specific category, either by increasing the first probability score or by decreasing the first classification threshold.

[0044] The first and second target detectors can be any target detector capable of detecting targets and determining probability scores indicating the probability that a detected target in a first image frame belongs to a specific category. Preferably, the first and second target detectors should further be capable of identifying the location of the detected target. One or both of the first and second target detectors can be configured as follows... Figure 3 Further description of the software or hardware implementation,

[0045] The positions of the detected targets in the first and second image frames can be determined separately. Then, further, assuming the distance from the second region to the first region is less than a distance threshold, one or both of the actions in S140—lowering the first classification threshold and increasing the first probability score—can be performed. If the detected target in the second image frame is located at a position farther than the detected target in the first image frame than the distance threshold, the detection is unlikely to be related to the same target. Therefore, since the result from the second target detector is unlikely to be related to the detection of the first target detector, neither of the two actions should be performed.

[0046] The distance threshold can increase with the temporal distance between the first and second image frames in the video sequence. For a detected target moving at a specific speed, the distance from the second region to the first region will depend on the time between capturing the first and second image frames. Since different categories of targets (e.g., people and cars) will move at different expected speeds, the increase in the distance threshold can depend on the specific target category. If the speed of the detected target is obtained in the first image frame, the increase in the distance threshold can depend on the obtained speed. By obtaining the speed, the expected distance between the second and first regions can be estimated, and the distance threshold can be set based on the estimated distance, for example, set to the estimated distance or a given percentage above the estimated distance, to cover any errors in the obtained speed and variations related to the obtained speed. The speed can be obtained by means of an external sensor (e.g., radar or lidar), or if the detected target has been successfully tracked in a previous frame between the first and second image frames, the speed can be obtained by means of data from the tracker.

[0047] The method may further include performing a third object detection using a second object detector. The third object detection includes: detecting an object in a third image frame of the video sequence; and determining a third probability score indicating the probability that the detected object in the third image frame belongs to a specific category, wherein the third image frame follows the first image frame in the video sequence. In this embodiment, the first processing time is longer than the second processing time plus the third processing time required by the second object detector to perform the third object detection. The third object detection is performed sequentially with the second object detection and in parallel with the first object detection. Therefore, the second object detector can perform both the second and third object detections before the first object detector completes the first object detection, and the results from both the second and third object detections can be used for the first object detection. Specifically, then, further, conditioned that the determined third probability score is higher than a second classification threshold, one or more of decreasing the first classification threshold and increasing the first probability score are performed.

[0048] Method 100 may further include using a first target detector to mask detected targets in the first image frame that are determined to belong to a specific category in S160.

[0049] In some embodiments, the first processing time of the first object detector is longer than the time between image frames of the video sequence according to the frame rate of the video sequence. In such embodiments, if object detection should be performed by the first object detector in real time, the first object detector can only perform object detection in a subset of image frames of the video sequence. If, for example, the first processing time of the first object detector is six times the time for each image frame of the video sequence according to the frame rate of the video sequence, then if object detection should be performed in real time, the first object detector can only perform object detection in every sixth video frame. In such embodiments, the second object detector may then have a processing time for performing object detection in one image frame, such that it can perform object detection in five image frames in a shorter time than the first processing time of the first object detector. In this case, for each group of six consecutive image frames, the first object detector will perform the first object detection in the first video frame, and this object detection will use the result of the second object detector performing object detection in the following five image frames in parallel with the first object detection performed by the first object detector in the first image frame.

[0050] Now turn to the illustration showing six image frames 201 to 206. Figures 2a to 2fIn this method, a target in the form of a car 210 is temporarily occluded to varying degrees by a building 220 in the first three image frames 201 to 203, and is fully visible in the latter three image frames 204 to 206. An embodiment of method 100 will now be described, wherein, depending on the frame rate of the video sequence, the first processing time of the first target detector is six times the time of each image frame of the video sequence, and the second target detector has a processing time for performing target detection in one image frame, such that it can perform target detection in five image frames in a shorter time than the first processing time of the first target detector. Specifically, the first target detector will perform a first target detection in the first video frame 201, and this target detection will use the result of the second target detector performing target detection in the latter five image frames 202 to 206 in parallel with the first target detection performed by the first target detector in the first image frame 201. In some embodiments, the second target detector may also perform target detection in the first image frame 201. This can be particularly advantageous when the second target detector is dedicated, such as when the second target detector can analyze the infrared spectrum and therefore may be better suited for dark conditions. If the second object detector has a processing time for performing object detection in one image frame, such that it can perform object detection in six image frames in a shorter time than the first processing time of the first object detector, then the second object detector can perform object detection in all six image frames 201 to 206.

[0051] exist Figure 2a The image shows a first image frame 201 in which a car 210 is partially occluded by a building 220. This is the first image frame where a portion of the car 210 is visible, following a previous image frame in which the car 210 is completely occluded by the building 220. A first object detector performs a first object detection in the first image frame 201. The object 210 is detected by the first object detector. Then, the first object detector determines a first probability score indicating the probability that the object 210 belongs to the "car" category.

[0052] exist Figure 2b The image shows a second image frame 202, in which the car 210 has moved relative to its position in the first image frame 201, such that a smaller portion of the car 210 is occluded by the building 220. A second object detector performs a second object detection in the second image frame 202 in parallel with the first object detector, which performs a first object detection in the first image frame 201. The target 210 is detected by the second object detector, and then a second probability score indicating the probability that the target 210 belongs to the "car" category is determined by the second object detector.

[0053] exist Figure 2cThe image shows a third image frame 203, in which the car 210 has moved relative to the second image frame 202, causing a smaller portion of the car 210 to be occluded by the building 220. Following the second object detection performed in the second image frame 202 and in parallel with the first object detection performed in the first image frame 201, the second object detector performs a third object detection in the third image frame 203. The target 210 is detected by the second object detector, and then a third probability score indicating the probability that the target 210 belongs to the "car" category is determined by the second object detector.

[0054] exist Figure 2d The image shows a fourth image frame 204, in which car 210 has moved relative to the third image frame 203, making car 210 fully visible. Following the third object detection performed in the third image frame 203 and in parallel with the first object detection performed by the first object detector in the first image frame 201, the second object detector performs a fourth object detection in the fourth image frame 204. Target 210 is detected by the second object detector, and then the second object detector determines a fourth probability score indicating the probability that target 210 belongs to the "car" category.

[0055] exist Figure 2e The image shows a fifth image frame 205, in which the car 210 has moved relative to the fourth image frame 204. Following the fourth target detection performed in the fourth image frame 204 and in parallel with the first target detection performed by the first target detector in the first image frame 201, the second target detector performs a fifth target detection in the fifth image frame 205. Target 210 is detected by the second target detector, and then a fifth probability score indicating the probability that target 210 belongs to the "car" category is determined by the second target detector.

[0056] exist Figure 2f The image shows a sixth image frame 206, in which the car 210 has moved relative to the fifth image frame 205. Following the fifth target detection performed in the fifth image frame 205 and in parallel with the first target detection performed by the first target detector in the first image frame 201, the second target detector performs a sixth target detection in the sixth image frame 206. Target 210 is detected by the second target detector, and then the second target detector determines a sixth probability score indicating the probability that target 210 belongs to the "car" category.

[0057] If the determined first probability score is lower than the first classification threshold and the determined second to sixth probability scores are all higher than the second classification threshold, then one or more of the following actions are performed: lowering the first classification threshold and increasing the first probability score.

[0058] Then, under the conditions that the increased first probability score is higher than the first classification threshold, the first probability score is higher than the decreased first classification threshold, or the increased first probability score is higher than the decreased first classification threshold, the first target detector determines that the detected target in the first image frame belongs to the specific category "car". Then, the target 210 can be occluded in the first image frame.

[0059] In the first example, both the first and second classification thresholds are 70%. Furthermore, the first probability score is 63%, the second probability score is 72%, the third probability score is 89%, the fourth probability score is 98%, the fifth probability score is 98%, and the sixth probability score is 98%. This means that the determined first probability score is lower than the first classification threshold, and the determined second to sixth probability scores are all higher than the second classification threshold. In a first alternative, the first classification threshold can then be reduced, for example, by 10%, i.e., reduced to 60%. In a second alternative, the first probability score can be increased, for example, by 10%, i.e., increased to 73%. In a third alternative, the first classification threshold can be reduced, for example, by 5%, i.e., reduced to 65%, and the first probability score can be increased, for example, by 5%, i.e., increased to 68%. In all three alternatives, the conditions that the increased first probability score is higher than the first classification threshold, the first probability score is higher than the reduced first classification threshold, or the increased first probability score is higher than the reduced first classification threshold are all satisfied. Therefore, the first target detector determines that the detected target in the first image frame belongs to the specific category "car," and then target 210 can be occluded in the first image frame. If any one of the second to sixth probability scores will not be higher than the second classification threshold, the condition that all of the determined second to sixth probability scores should be higher than the second classification threshold will not be met. In this case, it may be decided not to perform the action of lowering the first classification threshold and increasing one or more of the first probability scores. Alternatively, if each of the other four of the second to sixth probability scores will be higher than the second classification threshold, since the condition is met for most of the second image frames 202 to the sixth image frames 206, it may be decided to perform the action of lowering the first classification threshold and increasing one or more of the first probability scores.

[0060] In the second example, the first classification threshold and the first to sixth probability scores are the same as in the first example, but the second classification threshold is 80%. Therefore, 72% of the second probability scores are below the 80% second classification threshold. Thus, if all second to sixth probability scores need to meet a condition to be above the second classification threshold for performing one or more of lowering the first classification threshold and increasing the first probability score, this operation will not be performed in the second example. Alternatively, if it is sufficient for a majority of the second to sixth probability scores to meet the condition of being above the second classification threshold, since the third to sixth probability scores meet this condition, then in the second example, it will be decided to perform one or more of lowering the first classification threshold and increasing the first probability score.

[0061] Depending on some of the conditions met among the second to sixth probability scores, other alternatives are envisioned. For example, a decision can be based on the average of the second to sixth probability scores, such that if the average is higher than a second classification threshold, the condition is considered met. As another example, a decision can be based on the median of the second to sixth probability scores, such that if the median is higher than a second classification threshold, the condition is considered met. As yet another example, a decision can be based on both the average and the median of the second to sixth probability scores, such that if both the average and the median are higher than a second classification threshold, the condition is considered met. As yet another example, a decision can be based on the maximum of the second to sixth probability scores, such that if the maximum is higher than the second classification threshold by a certain amount, i.e., the second object detector has a very high probability score in one of the second to sixth images, the condition is considered met. As yet another example, a decision can be based on the minimum of the second to sixth probability scores, such that if the minimum is lower than the second classification threshold by a certain amount, i.e., the second object detector has a very low probability score in one of the second to sixth images, the condition is considered not met.

[0062] In the third example, the first classification threshold and the first to sixth probability scores are the same as in the first example, but the second classification threshold is 90%. Therefore, 72% of the second probability scores are below the 90% second classification threshold. Thus, if all second to sixth probability scores need to meet the condition of being above the second classification threshold to perform one or more of lowering the first classification threshold and increasing the first probability score, this operation will not be performed in the third example. Alternatively, if it is sufficient for the average of the second to sixth probability scores to meet the condition of being above the second classification threshold, since (72% + 89% + 98% + 98% + 98%) / 5 = 91%, then in the third example, it will be decided to perform one or more of lowering the first classification threshold and increasing the first probability score.

[0063] The first target detector detected the object within the dashed box 231. Figure 2a The first image frame 201 indicates the target 210, and the second detector detects the target 210 in the second to sixth regions indicated by dashed boxes 232 to 236, respectively. Performing one or more of lowering the first classification threshold and increasing the first probability score can be further conditional on the distance from each of the second to sixth regions 236 to the first region 231 being less than a distance threshold. The distance threshold increases with increasing temporal distance from the first image frame 201, such that it increases across the second to sixth image frames 202 and 206, and this increase depends on the fact that the specific target category is "car" such that the increase corresponds to the expected speed of the car. Typically, this increase is based on the maximum expected speed of the car. The increased distance threshold in Figures 2b to 2f The numbers 242 to 246 are indicated by dashed boxes respectively. For example, they can be found in... Figures 2b to 2f As seen in the diagram, the second to sixth regions, indicated by dashed boxes 232 to 236, are all within the distance threshold indicated by dashed boxes 242 to 246. Therefore, in this example, the further condition that the distance from each of the second to sixth regions 236 to the first region 231 should be below the distance threshold is met.

[0064] If the second target detector does not detect target 230 in one of the second image frames 202 to the sixth image frames 206, but detects another target (not shown) and identifies it as belonging to the "car" category in a different region (not shown) outside the distance threshold indicated by the respective dashed boxes 242 to 246, the further condition that the distance from each of the second region 232 to the sixth region 236 to the first region 231 should be below the distance threshold will not be met. In this case, it may be decided not to perform one or more of lowering the first classification threshold and increasing the first probability score. Alternatively, if the second target detector detects target 230 in each of the other four of the second image frames 202 to the sixth image frames 206, and the further condition that the distance from each of the corresponding regions in the second region 232 to the sixth region 236 to the first region 231 should be below the distance threshold is met, since the further condition is met for most of the second image frames 202 to the sixth image frames 206, it may be decided to perform one or more of lowering the first classification threshold and increasing the first probability score. Depending on whether further conditions are met for some of the second images 202 to the sixth image frames 206, but not for others, other alternatives are envisioned.

[0065] Method 100 has already been described with respect to a detected target and a specific category. However, Method 100 is also applicable to scenarios where there are two or more targets in each image frame. Method 100 is also applicable to scenarios where it should be determined that the detected target belongs to one of two or more specific categories.

[0066] Figure 3 A block diagram is shown relating to an embodiment of a device 300 for determining that a detected target belongs to a specific category. The device 300 may be, for example, a camera, included in a camera, or connected to a camera.

[0067] Device 300 includes circuitry 310. Circuitry 310 is configured to perform the functions of device 300. Circuitry 310 may include processor 312, such as, for example, a central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), microcontroller, or microprocessor. Processor 312 is configured to execute program code. The program code may, for example, be configured to perform the functions of device 300.

[0068] Device 300 may further include memory 320. Memory 320 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical arrangement, memory 320 may include non-volatile memory for long-term data storage and volatile memory serving as device memory for circuit 310. Memory 320 may exchange data with circuit 310 on a data bus. Additional control lines and address buses may also exist between memory 320 and circuit 310.

[0069] The functionality of device 300 can be embodied in the form of an executable logic program (e.g., lines of code, software program, etc.) stored on a non-transitory computer-readable medium (e.g., memory 320) of device 300 and executed by circuitry 310 (e.g., using processor 312). Furthermore, the functionality of device 300 can be a standalone software application or part of a software application that performs additional tasks associated with device 300. The described functionality can be considered as a method configured to be executed by a processing unit (e.g., processor 312 of circuitry 310). Moreover, although the described functionality can be implemented in software, it can also be executed via dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0070] The device 300 further includes a first target detector 321 and a second target detector 321.

[0071] Circuit 310 is configured to perform change function 323, and optionally to perform gain function 324.

[0072] The first target detector 321 is configured to perform a first target detection, which includes: detecting a target in a first image frame of a video sequence; and determining a first probability score indicating the probability that the detected target in the first image frame belongs to a particular category.

[0073] The second target detector 322 is configured to perform a second target detection, which includes: detecting a target in a second image frame of a video sequence; and determining a second probability score indicating the probability that the detected target in the second image frame belongs to a specific category, wherein the second image frame in the video sequence follows the first image frame.

[0074] Compared to the second target detector 322, the first target detector 321 has higher target detection accuracy, and the first target detector 321 requires a first processing time to perform the first target detection, which is longer than the second processing time required by the second target detector 322 to perform the second target detection. Furthermore, the second target detection is performed in parallel with the first target detection.

[0075] The change function 323 is configured to perform one or more of lowering the first classification threshold and increasing the first probability score when the determined first probability score is lower than the first classification threshold and the determined second probability score is higher than the second classification threshold.

[0076] The first target detector 321 is further configured to determine that the detected target in the first image frame belongs to a specific category if the increased first probability score is higher than a first classification threshold, the first probability score is higher than a decreased first classification threshold, or the increased first probability score is higher than a decreased first classification threshold.

[0077] In the first target detector 321, a target can be detected in a first region of a first image frame, and in the second target detection function, a target can be detected in a second region of a second image frame. In the first target detector 321, then, further, based on the condition that the distance from the second region to the first region is less than a distance threshold, one or more of decreasing a first classification threshold and increasing a first probability score are performed.

[0078] The distance threshold can be increased as the temporal distance between the first and second image frames in the video sequence increases.

[0079] The distance threshold can be increased depending on the specific target category.

[0080] Circuit 310 can be further configured to perform an acquisition function 324 configured to obtain the velocity of a detected target in the first image frame. The increase in the distance threshold can then depend on the obtained velocity.

[0081] The second object detector 322 may be further configured to perform a third object detection, which includes: detecting an object in a third image frame of the video sequence; and determining a third probability score indicating the probability that the detected object in the third image frame belongs to a specific category, wherein the third image frame follows the first image frame in the video sequence. The first processing time is longer than the second processing time plus the third processing time required by the second object detector 322 to perform the third object detection. The third object detection is performed sequentially with the second object detection and in parallel with the first object detection. In the change function 323, one or more of lowering the first classification threshold and increasing the first probability score are further performed conditioned on the determined third probability score being higher than the second classification threshold.

[0082] The first target detector 321 can be further configured such that the occlusion in the first image frame is identified as a detected target belonging to a specific category.

[0083] One or both of the first target detector 321 and the second target detector 322 can be implemented as a first target detection function and a second target detection function respectively, which can be embodied in software. This software is stored on a non-transitory computer-readable medium, such as memory 320, within device 300 and executed by circuitry 310, such as processor 312, within device 300. Alternatively, one or both of the first target detector 321 and the second target detector 322 can be implemented in software, which can be stored on a non-transitory computer-readable medium of device 300 other than memory 320 and executed by circuitry of device 300 other than circuitry 310. Alternatively, one or both of the first target detector 321 and the second target detector 322 can be implemented in hardware within device 300. Furthermore, one or both of the first target detector 321 and the second target detector 322 can be implemented in a device separate from device 300.

[0084] In the above text about Figure 1 and Figures 2a to 2f The detailed description of the operation of method 100 also applies to the corresponding function of device 300. Furthermore, where applicable, the above description regarding... Figure 1 and Figures 2a to 2f The optional additional features of the described method 100 also apply to the device 300.

[0085] Those skilled in the art will recognize that the present invention is not limited to the embodiments described above. Instead, many modifications and variations can be made within the scope of the claims. These modifications and variations will be understood and implemented by those skilled in the art in practicing the claimed invention through a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method for determining that a detected target belongs to a specific category, the method comprising: A first target detection is performed using a first target detector, the first target detection comprising: detecting a target in a first image frame of a video sequence; and determining a first probability score indicating the probability that the detected target in the first image frame belongs to a particular category; A second target detection is performed using a second target detector, the second target detection comprising: detecting a target in a second image frame of the video sequence; and determining a second probability score indicating the probability that the detected target in the second image frame belongs to the specific category, wherein the second image frame in the video sequence follows the first image frame, wherein the first target detector has higher target detection accuracy compared to the second target detector, wherein the first target detector requires a first processing time to perform the first target detection, which is longer than the second processing time required by the second target detector to perform the second target detection, and wherein the second target detection is performed in parallel with the first target detection; If the determined first probability score is lower than the first classification threshold and the determined second probability score is higher than the second classification threshold, then one or more of the following actions are performed: decreasing the first classification threshold and increasing the first probability score; and If the increased first probability score is higher than the first classification threshold, the first probability score is higher than the decreased first classification threshold, or the increased first probability score is higher than the decreased first classification threshold, the first target detector determines that the detected target in the first image frame belongs to the specific category.

2. The method according to claim 1, wherein, In the first target detection, the target is detected in a first region of the first image frame, and in the second target detection, the target is detected in a second region of the second image frame, wherein, further, one or more of lowering the first classification threshold and increasing the first probability score are performed, provided that the distance from the second region to the first region is less than a distance threshold.

3. The method according to claim 2, wherein, The distance threshold increases as the temporal distance between the first image frame and the second image frame in the video sequence increases.

4. The method according to claim 3, wherein, The increase in the distance threshold depends on the specific target category.

5. The method of claim 3, further comprising: The speed of the detected target is obtained in the first image frame. The increase in the distance threshold depends on the obtained speed.

6. The method of claim 1, further comprising: A third target detection is performed using the second target detector, the third target detection comprising: detecting a target in a third image frame of the video sequence; and determining a third probability score indicating the probability that the detected target in the third image frame belongs to the specific category, wherein the third image frame in the video sequence follows the first image frame, wherein the first processing time is longer than the second processing time plus the third processing time required by the second target detector to perform the third target detection, and wherein the third target detection is performed sequentially with the second target detection and in parallel with the first target detection; and Furthermore, if the determined third probability score is higher than the second classification threshold, one or more of the following actions are performed: lowering the first classification threshold and increasing the first probability score.

7. The method of claim 1, further comprising: The first target detector is used to mask the detected target that is determined to belong to the specific category in the first image frame.

8. A non-transitory computer-readable storage medium having instructions stored on the non-transitory computer-readable storage medium, which, when executed in an apparatus having a processor, implement the method according to any one of claims 1-7.

9. An apparatus for determining that a detected target belongs to a specific category, the apparatus comprising: A first target detector is configured to perform a first target detection, the first target detection comprising: detecting a target in a first image frame of a video sequence; and determining a first probability score indicating the probability that the detected target in the first image frame belongs to the particular category; A second target detector is configured to perform a second target detection, the second target detection comprising: detecting a target in a second image frame of the video sequence; and determining a second probability score indicating the probability that the detected target in the second image frame belongs to the specific category, wherein the second image frame in the video sequence follows the first image frame, wherein the first target detector has higher target detection accuracy compared to the second target detector, wherein the first target detector requires a first processing time for performing the first target detection, which is longer than the second processing time required by the second target detector for performing the second target detection, and wherein the second target detection is performed in parallel with the first target detection; and The circuit is configured to perform a modification function, which is configured to perform one or more of decreasing the first classification threshold and increasing the first probability score when a determined first probability score is lower than a first classification threshold and a determined second probability score is higher than a second classification threshold. The first target detector is further configured to determine that the detected target in the first image frame belongs to the specific category when the increased first probability score is higher than the first classification threshold, the first probability score is higher than the decreased first classification threshold, or the increased first probability score is higher than the decreased first classification threshold.

10. The apparatus according to claim 9, wherein, In the first target detector, the target is detected in a first region of the first image frame, and in the second target detector, the target is detected in a second region of the second image frame, wherein, in the first target detector, one or more of lowering the first classification threshold and increasing the first probability score are further performed, conditioned that the distance from the second region to the first region is less than a distance threshold.

11. The apparatus according to claim 10, wherein, The distance threshold increases as the temporal distance between the first image frame and the second image frame in the video sequence increases.

12. The apparatus according to claim 11, wherein, The increase in the distance threshold depends on the specific target category.

13. The apparatus according to claim 11, wherein, The circuit is further configured to perform: The acquisition function is configured to acquire the velocity of the detected target in the first image frame, and The increase in the distance threshold depends on the obtained speed.

14. The apparatus according to claim 9, wherein, The second target detector is further configured to perform a third target detection, the third target detection comprising: detecting a target in a third image frame of the video sequence; and determining a third probability score indicating the probability that the detected target in the third image frame belongs to the particular category, wherein the third image frame in the video sequence follows the first image frame, wherein the first processing time is longer than the second processing time plus the third processing time required by the second target detector to perform the third target detection, and wherein the third target detection is performed sequentially with the second target detection and in parallel with the first target detection; Specifically, in the change function, one or more of the following are performed: lowering the first classification threshold and increasing the first probability score, provided that the determined third probability score is higher than the second classification threshold.

15. The apparatus according to claim 9, wherein, The first target detector is further configured to mask the detected target that is determined to belong to the specific category in the first image frame.

Images