Method and Apparatus for Outputting Stabilized Object Detection Result in Edge-based Low-Light Image

Abstract

The present disclosure relates to a method and apparatus for outputting stabilized object detection results in edge-based low-light images. An object detection result output apparatus according to one embodiment of the present disclosure may include: a preprocessing unit for receiving image frames and preprocessing the image frames; a low-light improvement unit for improving the illumination of the image frames based on at least one of the brightness of the image frames, the contrast of the image frames, or the color reproduction of the image frames; an object detection unit for detecting objects within the image frames based on categories; an object determination unit for generating object detection results by category within the image frames using the number of detections of the objects and the reliability of the objects; and an output unit for outputting the object detection results.

Description

Technology Field

[0001] The present disclosure relates to a method and apparatus for outputting stabilized object detection results in edge-based low-light images. More specifically, it relates to a method and apparatus that minimize shaking and misclassification of object detection results to provide stable object detection results, and can secure real-time processing capabilities even in an environment with limited computational performance of edge devices. Background Technology

[0002] The following description merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Deep learning-based object detection models are being widely utilized in various fields recently, and are particularly attracting attention as a technology capable of recognizing and analyzing multiple objects, such as people and vehicles, in real time. This technology has evolved based on high-performance deep learning algorithms, high-resolution cameras, and computing resources. Furthermore, by enabling the accurate identification of object locations and types and processing this data in real time, it opens up possibilities for application in diverse fields. For example, object detection technology plays a core role in autonomous vehicles, smart cities, and security systems.

[0004] However, current object detection technologies often operate on a single-frame basis, which presents a problem in that it is difficult to maintain stable performance in specific environments. In particular, in low-light environments, the outlines of objects may become blurry or difficult to distinguish, potentially degrading detection accuracy. Furthermore, if an object shakes, tilts, or moves rapidly, detection results may fluctuate unstably or flicker. These issues pose a significant obstacle to object detection technologies providing reliable results in real-world environments.

[0005] Furthermore, existing technologies lack information processing mechanisms that account for temporal consistency, which limits their ability to stabilize object detection results across consecutive frames. This exacerbates the problem in situations where objects are misclassified or not detected. Particularly in applications requiring real-time operation, such instability can undermine the reliability of the entire system. Therefore, research is needed on new object detection technologies that can effectively utilize temporal information and stabilize misclassification. The problem to be solved

[0006] The present disclosure is intended to improve the illumination of video frames.

[0007] In addition, according to one embodiment, the purpose is to generate an object detection result using the number of object detections and the reliability of the objects.

[0008] In addition, according to one embodiment, the purpose is to enable real-time processing even in a limited computational performance environment of an edge device.

[0009] The problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0010] In one embodiment of the present disclosure, a device for outputting stabilized object detection results in edge-based low-light images may be provided. The object detection result output device may include a preprocessing unit for receiving image frames and preprocessing the image frames; a low-light improvement unit for improving the illumination of the image frames based on at least one of the brightness of the image frames, the contrast of the image frames, or the color reproduction of the image frames; an object detection unit for detecting objects within the image frames based on categories; an object determination unit for generating object detection results for each category within the image frames using the number of detections of the objects and the reliability of the objects; and an output unit for outputting the object detection results. The number of image frames is a preset number, and the categories may include people, animals, vehicles, and background objects.

[0011] In one embodiment of the present disclosure, a method for outputting stabilized object detection results in edge-based low-light images may be provided. The method may include receiving image frames and preprocessing the image frames; improving the illumination of the image frames based on at least one of the brightness of the image frames, the contrast of the image frames, or the color reproduction of the image frames; detecting objects within the image frames based on categories; generating object detection results for each category within the image frames using the number of detections of the objects and the reliability of the objects; and outputting the object detection results. The number of image frames is a preset number, and the categories may include people, animals, vehicles, and background objects.

[0012] In one embodiment of the present disclosure, a computer-readable recording medium storing instructions may be provided. When the instructions are executed by the computer, the computer may perform the following steps: receiving image frames and preprocessing the image frames; improving the illumination of the image frames based on at least one of the brightness of the image frames, the contrast of the image frames, or the color reproduction of the image frames; detecting objects within the image frames based on categories; generating object detection results for each category within the image frames using the number of object detections and the reliability of the objects; and outputting the object detection results. The number of image frames is a preset number, and the categories may include people, animals, vehicles, and background objects. Effects of the invention

[0013] The present disclosure has the effect of providing stable object detection results even in low illumination environments.

[0014] In addition, according to one embodiment, there is an effect of minimizing shaking and misclassification of object detection results.

[0015] In addition, according to one embodiment, there is an effect of simultaneously achieving high-reliability object detection quality and real-time capability in various surveillance environments.

[0016] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below. Brief explanation of the drawing

[0017] FIG. 1 is a block diagram illustrating an apparatus for outputting stabilized object detection results in edge-based low-light images according to one embodiment of the present disclosure. FIG. 2 is a diagram illustrating the overall processing flow for the collection and preprocessing of image frames, the improvement of image frames' illumination, and the detection and determination of objects within image frames, according to one embodiment of the present disclosure. FIG. 3 is a block diagram illustrating an object determination unit according to one embodiment of the present disclosure. FIG. 4 is a flowchart illustrating a process for generating an object detection result according to one embodiment of the present disclosure. FIG. 5 is a diagram illustrating a process for determining a final object label by comparing reliability for each category according to an embodiment of the present disclosure. FIG. 6 is a flowchart illustrating a method for outputting a stabilized object detection result in an edge-based low-light image according to one embodiment of the present disclosure. FIG. 7 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the method according to the present disclosure. Specific details for implementing the invention

[0018] Some embodiments of the present disclosure are described in detail below with reference to exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known components or functions could obscure the essence of the present disclosure, such detailed description is omitted.

[0019] In describing the components of the embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b), etc., may be used. These symbols are intended only to distinguish the components from other components, and the essence, order, or sequence of the components is not limited by the symbols. When a part in the specification is described as 'comprising' or 'having' a component, this means that, unless explicitly stated otherwise, it does not exclude other components but may include additional components.

[0020] The detailed description set forth below, together with the accompanying drawings, is intended to describe exemplary embodiments of the present disclosure and is not intended to represent the only embodiment in which the present disclosure can be practiced.

[0021] FIG. 1 is a block diagram illustrating an apparatus for outputting stabilized object detection results in edge-based low-light images according to one embodiment of the present disclosure.

[0022] Referring to FIG. 1, a device for outputting stabilized object detection results in edge-based low-light images (hereinafter, object detection result output device, 10) includes all or part of a preprocessing unit (110), a low-light improvement unit (120), an object detection unit (130), an object determination unit (140), and an output unit (150). The object detection result output device (10) and each of its components may be implemented in hardware or software, or in a combination of hardware and software. Additionally, the function of each component may be implemented in software, and one or more processors may be implemented to execute the software function corresponding to each component. The object detection result output device (10) may be implemented in an edge device. The object detection result output device (10) can improve the stability of object detection results within images acquired in a low-light environment.

[0023] The preprocessing unit (110) receives video frames and preprocesses the received video frames. The preprocessing unit (110) can sequentially receive and process a predetermined number of video frames. The preprocessing unit (110) can perform the role of removing noise included in the video frames and converting the video data to be suitable for subsequent processing steps. For example, the preprocessing unit (110) can apply a median filter to remove noise included in the video frames. In addition, the preprocessing unit (110) can increase processing efficiency by normalizing the pixel values ​​of the video frames to a predetermined range.

[0024] The preprocessing unit (110) can effectively remove impulse noise that is prone to occur in low-light environments. By using a median filter, the preprocessing unit (110) can reduce noise while preserving the boundary information of the object. Pixel value normalization can ensure stability in the learning and inference process by maintaining a constant range of input data for the neural network model included in the object detection unit (130).

[0025] Meanwhile, the preprocessing unit (110) may use other noise removal filters in addition to the median filter. For example, the preprocessing unit (110) may remove noise using a Gaussian filter, a bilateral filter, a wavelet transform-based filter, a non-local means filter, a block-matching and 3D filtering (BM3D), a deep learning-based denoising neural network, etc. However, the present disclosure is not limited thereto.

[0026] The low-illumination improvement unit (120) improves the illumination of preprocessed video frames. The low-illumination improvement unit (120) can improve the illumination based on at least one of the brightness, contrast, or color reproduction of the video frames. This can improve object detection performance by increasing the visibility of images captured in dark environments.

[0027] The low-illumination enhancement unit (120) may use the Contrast Limited Adaptive Histogram Equalization (CLAHE) technique to improve the brightness of video frames. The CLAHE technique can effectively restore details in dark areas by enhancing local contrast of the image. Instead of applying uniform brightness enhancement to the entire image, the low-illumination enhancement unit (120) may perform histogram equalization locally by considering the characteristics of each area. Additionally, the low-illumination enhancement unit (120) may improve illumination based on the Retinex theory. The low-illumination enhancement unit (120) may separate an illumination map and a reflectance map from the video frame. Subsequently, the low-illumination enhancement unit (120) may improve the separated illumination map and then re-synthesize it with the reflectance map to finally generate a video frame with improved illumination.

[0028] The object detection unit (130) detects objects based on categories within video frames with improved illumination. For example, categories may be people, animals, vehicles, and background objects. However, the present disclosure is not limited thereto. The object detection unit (130) may detect objects within video frames using an object detection model. For each detected object, the object detection unit (130) may generate a bounding box indicating the location and a class probability indicating the probability of belonging to a specific category.

[0029] The object detection model used in the object detection unit (130) may include a neural network backbone network. For example, the neural network backbone network may be a ResNet (Residual Network) architecture including a plurality of residual blocks. However, the present disclosure is not limited thereto. The object detection model may be a model pre-trained using images taken in a low-light environment, and may improve the accuracy of object detection in a low-light environment.

[0030] The object detection unit (130) may apply a Non-Maximum Suppression (NMS) algorithm to remove bounding boxes that are duplicated for the same object. The NMS algorithm selects the bounding box with the highest confidence level and removes other bounding boxes that have a large overlap area with the selected bounding box. Through this, the object detection unit (130) can output a single refined detection result for each object. The object detection result output device (10) may further include a rolling buffer for storing the detection results of objects within video frames on a frame-by-frame basis. The rolling buffer may include multiple buffers configured independently based on categories.

[0031] The object determination unit (140) generates object detection results by category within video frames using the detection count of objects and the reliability of objects. The object determination unit (140) can analyze the frame-unit detection results received from the object detection unit (130) by integrating them in time. Through this, the object determination unit (140) can correct temporary detection errors or misclassifications and generate stable object detection results. In one embodiment, the object determination unit (140) can calculate a weighted average result value by applying weights to the detection count of each object and the reliability of each object. The object determination unit (140) can confirm that the object is detected if the result value exceeds a preset threshold. The preset threshold can be set independently for each category. In another embodiment, the preset threshold can be dynamically adjusted according to the overall illumination level of the video frames.

[0032] The object determination unit (140) can stabilize the object detection results by considering temporal consistency. For example, the object determination unit (140) can predict the object location in the current frame based on the object locations in previous frames using a Kalman filter. Subsequently, the object determination unit (140) can perform object tracking by associating the predicted location with the newly detected object and maintain consistency of the detection results.

[0033] The object determination unit (140) can manage the persistence of objects by using a state machine model that defines initial state, confirmed state, and lost state for the objects. The object determination unit (140) can include only objects in the confirmed state in the final object detection result. By using this state management, the object determination unit (140) can filter out intermittently detected noisy objects or objects incorrectly detected for a short period of time from the final result.

[0034] The output unit (150) outputs the final object detection result generated by the object determination unit (140). The output unit (150) can transmit the stabilized object detection result to an external device or user interface. The output information may include location information such as the category of each object, bounding box coordinates, and reliability. The output unit (150) can process and provide the object detection result in various forms. For example, the output unit (150) can visually display and output the bounding box and label of the detected object on the original image frame. Additionally, the output unit (150) can generate a list of detected objects in a structured data format, for example, a JSON (JavaScript Object Notation) or XML (eXtensible Markup Language) file, so that other systems can utilize it.

[0035] FIG. 2 is a diagram illustrating the overall processing flow for the collection and preprocessing of image frames, the improvement of image frame illumination, and the detection and determination of objects within image frames, according to one embodiment of the present disclosure.

[0036] Referring to FIG. 2, the overall processing flow for the collection and preprocessing of image frames, the illumination improvement of image frames, and the object detection and object confirmation process within image frames according to one embodiment of the present disclosure may be illustrated. The object detection result output device (10) may sequentially perform the processes of collecting and preprocessing image frames (210), improving the illumination of image frames (220), and detecting and confirming objects within image frames (230). The object detection result output device (10) may output the generated object detection result.

[0037] The collection and preprocessing of video frames (210) may be a process of preparing initial data for object detection. The object detection result output device (10) may collect a preset number of video frames from an image capture device, such as a closed-circuit television (CCTV) installed in a low-light environment. The collected video frames may undergo a preprocessing process to increase the efficiency and accuracy of subsequent processing steps.

[0038] The illumination improvement (220) of the video frames may be a process of improving the visibility of the preprocessed video frames. In video frames taken in a low-light environment, the outline or color information of an object may be unclear. Therefore, the low-light improvement unit (120) can improve at least one of the brightness of the video frames, the contrast of the video frames, or the color reproduction of the video frames so that the object detection unit (130) can recognize the object more clearly.

[0039] Object detection and object determination (230) within video frames may be a process of detecting actual objects from video frames with improved illumination and stabilizing the detection results by considering temporal continuity. This process may include an object detection step of the object detection unit (130) and an object determination step of the object determination unit (140). The object detection unit (130) may detect objects present within video frames using an object detection model such as YOLO (You Only Look Once). In particular, the object detection unit (130) may include a Convolution-Batch Normalization-Activation Block (CBS: Convolution + BatchNorm + SiLU), a C3K2 module, a Simplified Spatial Pyramid Pooling Fast (SPPF) module, and a Channel Partial Self-Attention (C2PSA) module as backbone components. These components can efficiently extract various spatial / channel features to improve object detection accuracy in low-light environments. The object detection unit (130) can perform multi-scale feature fusion through a neck structure and enhance the resolution and information content of the feature map using upsampling and concat structures. Subsequently, it can generate prediction results for categories such as people, animals, vehicles, and background objects by using a Conv2d layer with settings such as K=3, S=2, and P=1 in the head area. The object determination unit (140) can generate a final object detection result based on the detection result received from the object detection unit (130).

[0040] FIG. 3 is a block diagram illustrating an object determination unit according to one embodiment of the present disclosure.

[0041] Referring to FIG. 3, the object determination unit (140) includes, in whole or in part, a detection count calculation unit (310), a reliability calculation unit (320), a weight setting unit (330), and an object confirmation unit (340). The object determination unit (140) and each of its components may be implemented in hardware or software, or in combination of hardware and software. Additionally, the function of each component may be implemented in software, and one or more processors may be implemented to execute the software function corresponding to each component. The object determination unit (140) may perform the role of stabilizing the object detection result received from the object detection unit (130) by considering temporal continuity. The object determination unit (140) may generate a final object detection result by synthesizing the calculation results of each component.

[0042] The object detection unit (130) detects objects within video frames using an object detection model. For each object detected within video frames, the object detection unit (130) can generate a bounding box indicating its location and a class probability indicating the probability of it being one of the categories. The object detection model may include a neural network backbone. Additionally, the object detection model may be a model trained using images captured in a low-light environment. Meanwhile, the object detection unit (130) may apply a Non-Maximum Suppression (NMS) algorithm to remove bounding boxes generated redundantly for the same object.

[0043] The detection count calculation unit (310) can calculate the number of times a specific object is detected within consecutive video frames. The detection count calculation unit (310) can assign a unique identifier to each object and calculate the cumulative number of detections of the same object by performing object tracking between frames. The detection count information calculated by the detection count calculation unit (310) can be used to determine whether the object is a temporary appearance or a continuous presence.

[0044] The reliability calculation unit (320) can calculate the reliability of each object detected within video frames. The reliability calculation unit (320) can determine the reliability based on the class probability generated by the object detection unit (130). The reliability calculation unit (320) can calculate the reliability on a frame-by-frame basis. The reliability calculation unit (320) can calculate the final reliability for each object by summing or averaging the reliability over a specific period.

[0045] The weight setting unit (330) can set weights based on the positions of the video frames. The weight setting unit (330) can assign a greater weight to the results detected in the recent video frame. Additionally, the weight setting unit (330) can assign progressively reduced weights to previous video frames by applying the Exponential Moving Average (EMA) method. This weight setting method can ensure that the latest state of the object is reflected more importantly while reliably considering the past detection history.

[0046] The object confirmation unit (340) finally confirms the objects detected within the video frames by combining the detection count, reliability, and weight. The object confirmation unit (340) can assign the weight set by the weight setting unit (330) to the detection count received from the detection count calculation unit (310) and the reliability received from the reliability calculation unit (320), respectively. The object confirmation unit (340) can generate a final object detection result using the values ​​to which the weights are applied.

[0047] In one embodiment, the object confirmation unit (340) may calculate a weighted average result by applying weights to the detection count and reliability of each object. The object confirmation unit (340) may confirm that the object has been detected only when the calculated result exceeds a preset threshold. The preset threshold may be set independently for each category, such as people, vehicles, etc. Through this, precise object confirmation tailored to the characteristics of each category may be possible.

[0048] In another embodiment, the object confirmation unit (340) may manage the persistence of an object using a State Machine Model. The object confirmation unit (340) may define a Tentative state, a Confirmed state, and a Lost state for each object. For example, if an object in the Tentative state is detected in consecutive preset frames, the object confirmation unit (340) may convert the object to the Confirmed state. Additionally, if the average confidence level of an object in the Tentative state over a predetermined period exceeds a preset threshold, the object confirmation unit (340) may convert the object in the Tentative state to an object in the Confirmed state. Conversely, if an object in the Confirmed state is not detected in consecutive preset frames, the object confirmation unit (340) may convert the object in the Confirmed state to an object in the Tentative state. The object confirmation unit (340) may increase the stability of the output by including only objects in the Confirmed state in the final object detection result.

[0049] Additionally, the object determination unit (340) can perform an object tracking function to determine the association of objects between frames. The object determination unit (340) can calculate the intersection over union (IoU) between the object detected in the current frame and the object detected in previous frames. Based on the calculated intersection over union, the object determination unit (340) can associate objects between frames to determine whether they are the same object.

[0050] FIG. 4 is a flowchart illustrating a process for generating an object detection result according to an embodiment of the present disclosure. FIG. 4 illustrates a process in which the components shown in FIG. 3 operate in conjunction with each other.

[0051] Referring to FIG. 4, the detection count calculation unit (310) calculates the detection count of each object (S410). The detection count calculation unit (310) may refer to the detection results of previous frames stored in a rolling buffer. If the detection count calculation unit (310) determines that an object detected in the current frame is the same object as an object detected in previous frames, it may calculate the detection count of the object by accumulating it. The determination of object identity may be performed using the intersection over union (IoU) of a bounding box or location prediction based on a Kalman filter. For example, the method for determining object identity may be a tracking method based on the intersection over union (IoU) of a bounding box, a Kalman filter, a Hungarian algorithm, a SORT (Simple Online and Realtime Tracking) algorithm, a DeepSORT algorithm, an optical flow, or a correlation filter. However, the present disclosure is not limited thereto.

[0052] The reliability calculation unit (320) calculates the reliability of each object (S420). The reliability calculation unit (320) can calculate the reliability based on the class probability received from the object detection unit (130). The reliability calculation unit (320) can use the class probability of a single frame as the reliability, or calculate the average or weighted average of the class probabilities measured over multiple frames to calculate the stabilized reliability. For example, the reliability calculation unit (320) can calculate the reliability on a frame-by-frame basis. The reliability calculation unit (320) can calculate the reliability of each object by summing or averaging the calculated reliabilitys.

[0053] The weight setting unit (330) sets weights (S430). The weight setting unit (330) may set weights to assign differential importance to the detection results of each frame by considering temporal continuity. The weight setting unit (330) may assign greater weights to recent frames among the video frames. The weight setting unit (330) may adjust the influence of information over time by assigning reduced weights to previous frames using an exponential moving average method. S310, S320, and S330 may be performed in a reversed order, and some steps may be performed simultaneously or omitted.

[0054] The object confirmation unit (340) generates an object detection result (S440). The object confirmation unit (340) may consider the detection count calculated by the detection count calculation unit (310). Additionally, the object confirmation unit (340) may comprehensively consider the reliability calculated by the reliability calculation unit (320) and the weight set by the weight setting unit (330). Through comprehensive consideration, the object confirmation unit (340) can finally confirm the object existing within the video frames. The object confirmation unit (340) can generate an object detection result based on the confirmed object.

[0055] The object confirmation unit (340) can calculate a weighted average result by applying weights to the detection count of each object and the reliability of each object. The object confirmation unit (340) can confirm that an object has been detected if the result exceeds a preset threshold. The preset threshold can be set independently for each category. For example, the threshold for the person category can be set differently from the threshold for the vehicle category. Through differential setting of the threshold, precise detection control tailored to the characteristics of each object can be achieved.

[0056] The object confirmation unit (340) can manage the persistence of objects by using a state machine model that defines an initial state, a confirmed state, and a lost state for the objects. The object confirmation unit (340) can include only objects in the confirmed state in the object detection results. If an object in the confirmed state is not associated with a newly detected object for a predetermined period, the object confirmation unit (340) can convert the object in the confirmed state to an object in the lost state. If an object in the lost state is not re-associated with a newly detected object for a predetermined period, the object confirmation unit (340) can permanently delete the object in the lost state.

[0057] FIG. 5 is a diagram illustrating a process for determining a final object label by comparing reliability for each category according to an embodiment of the present disclosure.

[0058] Referring to FIG. 5, the animal category (510) may provide criteria for determining whether a detected object corresponds to an animal. For example, as illustrated in FIG. 5, the confidence score for a cat for a specific object may be calculated as 4. The confidence score for a misclassification object may be calculated as 3. When the output label threshold for the animal category (510) is set to 4, the object determination unit (140) may finally determine the object as a cat because the confidence score for a cat satisfies the threshold.

[0059] The human category (520) can provide criteria for determining whether a detected object corresponds to a human. As illustrated in FIG. 5, the confidence score for a human for a specific object can be calculated as a value exceeding 5. The confidence score for a misclassification object can be calculated as 2. When the output label threshold for the human category (520) is set to approximately 5, the object determination unit (140) can finally determine that the object is a human because the confidence score of a human exceeds the threshold.

[0060] The vehicle category (530) may provide criteria for determining whether a detected object corresponds to a vehicle. For example, for a specific object, the reliability score of a car may be calculated as 5. The reliability score of a bus may be calculated as 2. If the output label threshold for the vehicle category (530) is set to 5, the object determination unit (140) may finally determine the object as a car because the reliability score of a car has reached the threshold.

[0061] The background object category (540) can provide criteria for determining whether a detected object corresponds to a background element rather than a primary object of interest. As illustrated in FIG. 5, the confidence score for a specific object as a balloon can be calculated as 4. The confidence score for a misclassification object can be calculated as a value exceeding 3. When the output label threshold for the background object category (540) is set to 4, the object determination unit (140) can finally determine the object as a balloon and distinguish it from the primary object. By classifying background objects separately in this way, the system can increase the accuracy of primary object detection.

[0062] FIG. 6 is a flowchart illustrating a method for outputting a stabilized object detection result in an edge-based low-light image according to an embodiment of the present disclosure. FIG. 6 illustrates the process in which the components shown in FIG. 1 operate in conjunction with each other.

[0063] Referring to FIG. 6, the preprocessing unit (110) preprocesses image frames (S610). The preprocessing unit (110) can receive image frames. The preprocessing unit (110) can apply a filter to remove noise included in the image frames. For example, the filter may be a median filter, a Gaussian filter, a bilateral filter, a non-local means filter, a wavelet transform-based filter, a deep learning-based noise removal model, or a frequency domain filter. However, the present disclosure is not limited thereto. The preprocessing unit (110) can normalize the pixel values ​​of the image frames to a preset range.

[0064] The low-illumination improvement unit (120) improves the illumination of video frames (S620). The low-illumination improvement unit (120) can improve the illumination of video frames based on at least one of the brightness of video frames, the contrast of video frames, or the color reproduction of video frames. The low-illumination improvement unit (120) can improve the brightness of video frames using the Contrast Limited Adaptive Histogram Equalization (CLAHE) technique. The low-illumination improvement unit (120) can separate the illumination map and the reflectance map from video frames based on the Retinex theory. The low-illumination improvement unit (120) can improve the illumination by improving the separated illumination map and then re-synthesizing it with the reflectance map. For example, the low-light enhancement unit (120) may use gamma correction, a dehazing algorithm, a deep learning-based low-light image enhancement model, a Fourier transform-based technique, a wavelet transform-based technique, Multi-Scale Retinex (MSR), or an adaptive local tone mapping technique. However, the present disclosure is not limited thereto.

[0065] The object detection unit (130) detects objects within video frames (S630). The object detection unit (130) can detect objects within video frames based on categories. The object detection unit (130) can detect objects within video frames using an object detection model. For example, the object detection model may be YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Faster R-CNN (Region-based Convolutional Neural Networks), RetinaNet, EfficientDet, Transformer-based DETR (DEtection TRansformer), or Centernet. However, the present disclosure is not limited thereto. For each object detected within video frames, the object detection unit (130) can generate a bounding box indicating the location and a class probability indicating the probability of being one of the categories. The object detection model may include a neural network backbone. The object detection model may be a model trained using images captured in a low-light environment.

[0066] The object determination unit (140) generates object detection results by category within the video frames (S640). The object determination unit (140) can generate object detection results by category within the video frames using the number of detections of objects and the reliability of objects. The number of video frames may be a preset number. Categories may include people, animals, vehicles, and background objects. Step S640 may include a step of calculating the number of detections of each object within the video frames. Additionally, Step S640 may include a step of calculating the reliability of each object within the video frames. Additionally, Step S640 may include a step of setting weights based on the positions of the video frames. Additionally, Step S640 may include a step of generating object detection results by determining the objects detected within the video frames by assigning weights to the number of detections of each object and the reliability of each object.

[0067] The object determination unit (140) can manage the persistence of objects by using a state machine model that defines initial state, confirmed state, and lost state for the objects. The object determination unit (140) can include only objects in the confirmed state in the object detection results. The object determination unit (140) can calculate a weighted average result value by applying weights to the detection count of each object and the reliability of each object. The object determination unit (140) can confirm that an object is detected if the result value exceeds a preset threshold. The preset threshold can be set independently for each category. The object determination unit (140) can calculate the bounding box overlap area between the object detected in the current frame and the object detected in previous frames. The object determination unit (140) can perform object tracking that associates objects between frames using the calculated bounding box overlap area. For example, object tracking can be performed using the SORT (Simple Online and Realtime Tracking) algorithm, the DeepSORT algorithm, a Kalman filter, a particle filter, a correlation filter-based tracker, a Siamese network-based tracker, or a Multi-Object Tracking (MOT) algorithm. However, the present disclosure is not limited thereto.

[0068] The object determination unit (140) can convert an object in a confirmed state to an object in a lost state if the object in the confirmed state is not associated with a newly detected object for a predetermined period. The object determination unit (140) can permanently delete an object in a lost state if the object in the lost state is not re-associated with a newly detected object for a predetermined period.

[0069] The output unit (150) outputs an object detection result (S650). The output unit (150) can generate a final object detection result including location, category, and reliability information of the confirmed objects. The output unit (150) can transmit the generated object detection result to an external device or a user interface. For example, the external device may be a server, a monitoring system, a security system, a control unit of an autonomous vehicle, a mobile device, a database system, or cloud storage. However, the present disclosure is not limited thereto.

[0070] FIG. 7 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the method according to the present disclosure.

[0071] Referring to FIG. 7, the computing device (700) may include some or all of a memory (710), a processor (720), a storage (740), an input / output interface (760), and a communication interface (780). The computing device (700) may structurally and / or functionally include at least a part of an object detection result output device (10). The computing device (700) may be a stationary computing device such as a desktop computer, a server, an AI accelerator, etc., as well as a portable computing device such as a laptop computer, a smartphone, etc.

[0072] Memory (710) may store a program that enables the processor (720) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (720), and the method illustrated in FIGS. 4 and FIGS. 6 may be performed by executing the plurality of instructions by the processor (720).

[0073] The memory (710) may be a single memory or multiple memories. In this case, information required to perform a method or operation according to various embodiments of the present disclosure may be stored in a single memory or divided and stored in multiple memories. If the memory (710) is composed of multiple memories, the multiple memories may be physically separated.

[0074] The memory (710) may include at least one of volatile memory and non-volatile memory. Volatile memory includes Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), and non-volatile memory includes flash memory.

[0075] The processor (720) may include at least one core capable of executing at least one instruction. The processor (720) may execute instructions stored in memory (710). The processor (720) may be a single processor or multiple processors.

[0076] The storage (740) retains stored data even when the power supplied to the computing device (700) is cut off. For example, the storage (740) may include non-volatile memory and may include a storage medium such as magnetic tape, optical disk, or magnetic disk.

[0077] A program stored in storage (740) can be loaded into memory (710) before being executed by the processor (720). Storage (740) can store a file written in a programming language, and a program generated from the file by a compiler, etc., can be loaded into memory (710). Storage (740) can store data to be processed by the processor (720) and / or data processed by the processor (720).

[0078] The input / output interface (760) may include input devices such as a keyboard and mouse, and output devices such as a display device and printer. Through the input / output interface, the user can trigger the execution of a program by the processor (720) and / or check the processing results of the processor (720).

[0079] The communication interface (780) can provide access to an external network. For example, the computing device (700) can communicate with other devices through the communication interface (780).

[0080] Each component of the device or method according to the present invention may be implemented in hardware or software, or in a combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the function of the software corresponding to each component.

[0081] Various embodiments of the systems and techniques described herein may be realized as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented as one or more computer programs executable on a programmable system. A programmable system comprises a storage system, at least one input device, and at least one programmable processor (which may be a special-purpose processor or a general-purpose processor) coupled to receive data and instructions from and transmit data and instructions to at least one output device. Computer programs (which are also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0082] Computer-readable recording media include all types of recording devices in which data that can be read by a computer system is stored. Such computer-readable recording media may be non-volatile or non-transitory media such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device, and may also include transitory media such as data transmission media. Additionally, computer-readable recording media may be distributed across networked computer systems, and computer-readable code may be stored and executed in a distributed manner.

[0083] Although the flowcharts and timing diagrams in this specification describe each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present disclosure. In other words, a person skilled in the art to which one embodiment of the present disclosure belongs may modify and adapt the flowcharts and timing diagrams in various ways, such as changing the order described in the flowcharts and timing diagrams or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present disclosure; therefore, the flowcharts and timing diagrams are not limited to a chronological order.

[0084] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

Claims

Claim 1 An object detection result output device comprising: a preprocessing unit for receiving video frames and preprocessing said video frames; a low illumination improvement unit for improving the illumination of said video frames based on at least one of the brightness of said video frames, the contrast of said video frames, or the color reproduction of said video frames; an object detection unit for detecting objects within said video frames based on a category; an object determination unit for generating an object detection result for each category within said video frames using the number of detections of said objects and the reliability of said objects; and an output unit for outputting said object detection result, wherein the number of said video frames is a preset number, and said categories include people, animals, vehicles, and background objects, and said object determination unit performs object tracking that associates objects between frames by calculating the bounding box overlap area between an object detected in the current frame and an object detected in previous frames. Claim 2 An object detection result output device according to claim 1, wherein the object detection unit detects the objects within the image frames using an object detection model, and for each object detected within the image frames, generates a bounding box indicating the location and a class probability indicating the probability of being one of the categories, wherein the object detection model includes a neural network backbone network, and the object detection model is a model learned using images taken in a low-light environment. Claim 3 An object detection result output device according to claim 1, wherein the object determination unit comprises: a detection count calculation unit for calculating the detection count of each of the objects within the image frames; a reliability calculation unit for calculating the reliability of each of the objects within the image frames; a weight setting unit for setting a weight based on the position of the image frames; and an object confirmation unit for determining the objects detected within the image frames by assigning the weight to the detection count of each of the objects and the reliability of each of the objects to generate the object detection result. Claim 4 In paragraph 3, the reliability calculation unit calculates reliability on a frame-by-frame basis and calculates the reliability of each of the objects by summing or averaging the reliabilitys, an object detection result output device. Claim 5 In paragraph 3, the weight setting unit assigns a large weight to recent frames among the image frames and assigns a reduced weight to previous frames using an exponential moving average method, an object detection result output device. Claim 6 In paragraph 3, the object confirmation unit manages the persistence of objects using a state machine model that defines an initial (Tentative), confirmed, and lost state for the objects, and includes only objects in the confirmed state in the object detection result output device. Claim 7 In claim 6, the object determination unit converts the object in the initial state into an object in the confirmed state when the object in the initial state is detected in consecutive preset frames, and converts the object in the confirmed state into an object in the initial state when the object in the confirmed state is not detected in consecutive preset frames. Claim 8 In paragraph 3, the object confirmation unit calculates a weighted average result value by applying the weights to the detection count of each of the objects and the reliability of each of the objects, and confirms the object as detected when the result value exceeds a preset threshold, wherein the preset threshold is independently set for each category, an object detection result output device. Claim 9 An object detection result output device according to claim 1, further comprising a rolling buffer for storing detection results of the objects in the image frames on a frame-by-frame basis, wherein the rolling buffer comprises a plurality of buffers independently configured based on the category. Claim 10 delete Claim 11 In claim 6, the object confirmation unit is an object detection result output device that converts the object in the initial state into an object in the confirmed state when the average reliability of the object in the initial state over a predetermined period exceeds a preset threshold. Claim 12 In claim 11, the object confirmation unit converts the object in the confirmed state into an object in the lost state if the object in the confirmed state is not associated with a newly detected object for a predetermined period, and permanently deletes the object in the lost state if the object in the lost state is not re-associated with the newly detected object for a predetermined period. Claim 13 A method performed by an object detection result output device comprises: receiving image frames and preprocessing the image frames; improving the illumination of the image frames based on at least one of the brightness of the image frames, the contrast of the image frames, or the color reproduction of the image frames; detecting objects within the image frames based on categories; generating object detection results for each category within the image frames using the number of detections of the objects and the reliability of the objects; and outputting the object detection results for outputting the object detection results, wherein the number of image frames is a preset number and the categories include people, animals, vehicles, and background objects, and the process of generating the object detection results includes performing object tracking that associates objects between frames by calculating the bounding box overlap area between an object detected in the current frame and an object detected in previous frames. Claim 14 In claim 13, the process of detecting the objects comprises: a process of detecting the objects within the image frames using an object detection model; and a process of generating a bounding box indicating the location and a class probability indicating the probability of being one of the categories for each of the objects detected within the image frames, wherein the object detection model includes a neural network backbone network and the object detection model is a model trained using images taken in a low-light environment. Claim 15 In claim 13, the process of generating the object detection result comprises: a process of calculating the detection count of each of the objects within the image frames; a process of calculating the reliability of each of the objects within the image frames; a process of setting weights based on the positions of the image frames; and a process of determining the objects detected within the image frames by assigning the weights to the detection count of each of the objects and the reliability of each of the objects to generate the object detection result. Claim 16 In claim 15, the process of generating the object detection result further comprises a process of managing the persistence of the objects using a state machine model that defines an initial state, a confirmed state, and a lost state for the objects, and including only objects in the confirmed state in the object detection result. Claim 17 In claim 15, the process of generating the object detection result further includes the process of calculating a weighted average result by applying the weights to the detection count of each of the objects and the reliability of each of the objects, and determining that the object is detected if the result exceeds a preset threshold, wherein the preset threshold is set independently for each category. Claim 18 delete Claim 19 A computer-readable recording medium having a stored command, wherein the command causes the computer, when executed by the computer, to perform the following steps: receiving video frames and preprocessing the video frames; improving the illumination of the video frames based on at least one of the brightness of the video frames, the contrast of the video frames, or the color reproduction of the video frames; detecting objects within the video frames based on a category; generating object detection results by category within the video frames using the number of object detections and the reliability of the objects; and outputting the object detection results, wherein the number of video frames is a preset number, and the categories include people, animals, vehicles, and background objects, and the step of generating the object detection results includes the step of performing object tracking to associate objects between frames by calculating the bounding box overlap area between an object detected in the current frame and an object detected in previous frames.

Images