Framework, system, device, method and computer program for detecting object

Abstract

A system according to one embodiment of the present disclosure comprises: a detection module that detects at least one object in an input image based on a first reference value and a second reference value and generates at least one first image segment including the detected at least one object; a first classification module that compares at least one first image segment with a second image segment stored in advance and assigns one of a first label and a second label to at least one first image segment; and a second classification module that extracts class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among at least one first image segment.

Description

Technology Field

[0001] The present disclosure relates to an object detection framework, system, device, method, and computer program, and more specifically, to an object detection framework, system, device, method, and computer program capable of detecting, classifying, and tracking objects within an input image. Background Technology

[0002] Recently, interest in technologies such as artificial intelligence, big data, the Internet of Things, and cloud computing has been on the rise. Here, Artificial Intelligence (AI) refers to technology capable of reasoning, learning, and acting using data that requires human intelligence or is larger in scale than humans can analyze. With recent improvements in computing efficiency, AI-based computer vision technology is rapidly advancing.

[0003] Computer vision technology is a technology that extracts meaningful information from images, videos, and other visual inputs using computers and systems, and then performs tasks or makes recommendations based on this information.

[0004] Recently, much research has been conducted on image processing technologies that perform information acquisition, identification, and tracking using images. Depending on the purpose of application, various algorithms such as object detection algorithms and tracking algorithms are used in these image processing technologies. The problem to be solved

[0005] One objective of the present disclosure is to provide an object detection framework, system, device, method, and computer program with improved accuracy in object detection and classification.

[0006] Another objective of the present disclosure is to provide an object detection framework, system, device, method, and computer program that can improve the utilization of data processing.

[0007] The purposes of this disclosure are not limited to those mentioned above, and other unmentioned purposes will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0008] A system according to one embodiment of the present disclosure may include, in a system for detecting objects in an input image, a detection module that detects at least one object in the input image based on a first reference value and a second reference value and generates at least one first image segment including the detected at least one object; a first classification module that compares the at least one first image segment with a previously stored second image segment and assigns one of a first label and a second label to the at least one first image segment; and a second classification module that extracts class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment. The second classification module may, when there is a sub-label among the plurality of sub-labels in which the class score is greater than or equal to the third reference value of the corresponding sub-label, designate the sub-label in which the class score is greater than or equal to the third reference value for the first image segment to which the first label is designated, and when the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, provide the first image segment to which the first label is designated as the third image segment to the first classification module.

[0009] An apparatus according to one embodiment of the present disclosure may include, in an apparatus for detecting an object in an input image, a memory for storing at least one instruction, and a processor for performing at least one process for detecting an object in the input image in response to the instruction stored in the memory. The processor detects at least one object in the input image based on a first reference value and a second reference value, generates at least one first image segment including the detected at least one object, compares the at least one first image segment with a previously stored second image segment to assign one of a first label and a second label to the at least one first image segment, extracts class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment, and if there is a sub-label among the plurality of sub-labels whose class score is greater than or equal to a third reference value of the corresponding sub-label, assigns the sub-label whose class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned, and if the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, the first image segment to which the first label is assigned can be stored as the second image segment.

[0010] A computer program according to one embodiment of the present disclosure is a computer program stored on a computer-readable medium, and when executed by a computer system, the computer system may include instructions to perform a method for detecting an object in an input image, wherein the computer system performs a method comprising: detecting at least one object in the input image based on a first reference value and a second reference value, and generating at least one first image segment including the detected at least one object; comparing the at least one first image segment with a previously stored second image segment to assign one of a first label and a second label to the at least one first image segment; and extracting class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment. The step of extracting the class scores involves, if among the plurality of sub-labels there is a sub-label whose class score is greater than or equal to the third reference value of the corresponding sub-label, assigning the sub-label whose class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned, and if the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, the first image segment to which the first label is assigned can be stored as the second image segment.

[0011] Specific details of other embodiments are included in the description of the invention and the drawings. Effects of the invention

[0012] In an object detection framework, system, device, method, and computer program according to embodiments of the present disclosure, the object detection framework (or, electronic device) can detect, classify (or filter) and track objects in an input image and detect events occurring in the input image by using at least one of each module (e.g., detection module, first classification module, second classification module, tracking module, and event detection module).

[0013] Here, the object detection framework (or electronic device) can detect, classify (or filter) and track objects within an input image and detect events occurring within the input image by sequential operation (or operation) of a detection module, a first classification module, a second classification module, a tracking module, and an event detection module.

[0014] As multiple modules operate sequentially in this manner, the accuracy of object detection, classification, tracking, and event detection operations of the object detection framework (or electronic device) can be improved through the operations of modules to which various conditions are applied. In particular, in the operation of detecting and classifying objects, the object detection framework (or electronic device) can detect and classify objects through the sequential operation (or operation) of multiple modules (e.g., detection module, first classification module, second classification module) driven using different reference values. Accordingly, the accuracy of object detection and classification can be improved.

[0015] In addition, in the object detection framework, system, device, method, and computer program according to the embodiments of the present disclosure, the object detection framework (or electronic device) can increase the utilization of data processing by utilizing an image segment (third image segment) filtered by the operation of a second classification module as a second image segment required for the operation of a first classification module.

[0016] In addition, by utilizing the third image segment as the second image segment, the number of data (second image segment) required for the operation of the first classification module increases, so the classification performance (performance of the first classification module) can be further improved.

[0017] However, the effects of the present disclosure are not limited to those exemplified above, and various other effects are included within the present disclosure and may be extended in various ways without departing from the spirit and scope of the present disclosure. Brief explanation of the drawing

[0018] FIG. 1 is a block diagram showing an electronic device according to various embodiments of the present disclosure. FIG. 2 is a block diagram illustrating an object detection framework according to various embodiments of the present disclosure. Figure 3 is a block diagram showing an example of a detector included in the object detection framework of Figure 2. Figure 4 is a diagram illustrating an example of an input image input to the detector of Figure 3. FIG. 5 is a diagram illustrating an example of a sub-label table used in the operation of the detector of FIG. 3. FIG. 6 is a block diagram showing an example of a detection module, a first classification module, a second classification module, a tracking module, and an event detection module included in the detector of FIG. 3. FIG. 7 is a diagram illustrating an example of the operation of the detection module and the first classification module of FIG. 6. FIG. 8 is a diagram illustrating an example of the operation of the first classification module of FIG. 6. FIG. 9 is a flowchart illustrating an object classification method according to various embodiments of the present disclosure. FIG. 10 is a flowchart illustrating an object classification method according to various embodiments of the present disclosure. FIG. 11a is a drawing for illustrating an example of a first comparator included in the tracking module of FIG. 6. FIG. 11b is a diagram illustrating an example of the operation of the first comparator of FIG. 11a. FIG. 12a is a drawing for illustrating an example of a second comparator included in the tracking module of FIG. 6. FIG. 12b is a diagram illustrating an example of the operation of the second comparator of FIG. 12a. FIG. 13 is a diagram illustrating an example of the operation of the event detection module of FIG. 6. FIG. 14 is a block diagram showing another example of a second classification module included in the detector of FIG. 3. FIG. 15 is a flowchart illustrating an object classification method according to various embodiments of the present disclosure. FIG. 16 is a flowchart illustrating an object filtering method according to various embodiments of the present disclosure. FIG. 17 is a diagram illustrating an example of the operation of the second classification module of FIG. 14. FIG. 18 is a flowchart illustrating an object detection method according to various embodiments of the present disclosure. Specific details for implementing the invention

[0019] In describing each drawing, similar reference numerals have been used for similar components. In the attached drawings, the dimensions of the structures are depicted enlarged from their actual size for clarity of the present disclosure. The area and thickness of each component shown in the drawings are depicted for convenience of explanation and are not necessarily limited to the area and thickness of the components depicted in the present disclosure.

[0020] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0021] In this disclosure, terms such as “comprising,” “having,” and “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should not be understood as precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0022] The features of each of the various embodiments of the present disclosure may be combined or combined with one another, either partially or wholly, and may technically enable various interlocking and operation. Each embodiment may be implemented independently of one another or may be implemented together in an associated relationship.

[0023] Hereinafter, various embodiments of the present disclosure will be described in more detail with reference to the attached drawings. Identical reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0025] <Apparatus according to embodiments of the present disclosure>

[0026] FIG. 1 is a block diagram showing an electronic device (100) according to various embodiments of the present disclosure.

[0027] Referring to FIG. 1, an electronic device (100) according to embodiments of the present disclosure may include a processor (110), a communication module (120), an input unit (130), a display unit (140), and a memory (150).

[0028] The processor (110) can control at least one other component (e.g., a hardware or software component) of the electronic device (100) connected to the processor (110) by executing software (e.g., a program (151)), and can perform various data processing or operations. In one embodiment, as at least part of the data processing or operations, the processor (110) can store commands or data received from other components (e.g., an input unit (130) or a communication module (120)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. For example, memory (150) may include volatile memory or non-volatile memory.

[0029] In one embodiment, the processor (110) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (100) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use less power than the main processor or to be specialized for a designated function. Depending on the embodiment, the auxiliary processor may be implemented separately from the main processor or as part thereof.

[0030] The processor (110) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence model may be generated through machine learning. Machine learning refers to learning that optimizes parameters with given data using a model composed of multiple parameters. Such learning may be performed, for example, on the electronic device (100) itself where the artificial intelligence is performed, or through a separate server. An artificial intelligence learning paradigm refers to various learning methods and approaches, and may include, for example, supervised learning, unsupervised learning, semisupervised learning, reinforcement learning, self-supervised learning, or transfer learning. An artificial intelligence learning algorithm may include, for example, linear regression, logistic regression, decision trees, random forest, support vector machines (SVM), or artificial neural networks, but is not limited to the examples described above.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a deep convolutional neural network (DCNN), a recurrent neural network (RNN), a generative adversarial network (GAN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, a feedforward algorithm, or a combination of two or more of the above, but is not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers. In addition to a hardware structure, an artificial intelligence model may additionally or substantially include a software structure.

[0031] The communication module (120) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (100) and an external electronic device (e.g., an electronic device or server other than the electronic device (100)), and the performance of communication through the established communication channel. The communication module (120) may include one or more communication processors that operate independently of the processor (110) and support direct communication (e.g., wired communication) or wireless communication. In one embodiment, the communication module (120) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module). Among these communication modules, the corresponding communication modules can communicate with external electronic devices through a short-range communication network such as Bluetooth, Wi-Fi Direct, or IrDA, or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0032] The input unit (130) can receive commands or data to be used for a component of the electronic device (100) (e.g., processor (110)) from outside the electronic device (100) (e.g., user). The input unit (130) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The display unit (140) can visually provide information to an external (e.g., user) of the electronic device (100). The display unit (140) may include, for example, a display (e.g., a display panel), a holographic device, or a projector, and may further include a control circuit for controlling said device. Additionally, the display unit (140) may further include a touch screen to recognize touch input from the user.

[0034] In one embodiment, the display unit (140) may display a graphic user interface (GUI) for receiving user input (e.g., touch input, etc.).

[0035] The memory (150) can store various data used by at least one component of the electronic device (100) (e.g., processor (110) or input unit (130)). The data may include, for example, input data or output data for software (e.g., program (151)) and related instructions.

[0036] In one embodiment, memory (150) may store a learned model (e.g., a model learned through a processor (110) or a separate server). For example, at least some of the programs (151) stored in memory (150) may be learned models. Accordingly, the processor (110) may use the learned models by executing the programs (151).

[0037] The program (151) may be stored as software in memory (150) and may include, for example, an operating system, middleware, or an application.

[0039] <Framework according to embodiments of the present disclosure>

[0040] FIG. 2 is a block diagram showing an object detection framework (200) according to various embodiments of the present disclosure.

[0041] FIG. 3 is a block diagram showing an example of a detector (220) included in the object detection framework (200) of FIG. 2.

[0042] FIG. 4 is a diagram illustrating an example of an input image (IMG) input to the detector (220) of FIG. 3.

[0043] FIG. 5 is a drawing for illustrating an example of a sub-label table used in the operation of the detector (220) of FIG. 3.

[0044] Referring to FIG. 2, an object detection framework (200) according to embodiments of the present disclosure may include a learner (210) and a detector (220).

[0045] Meanwhile, in the present disclosure, a framework may refer to a software environment (or a collection of software) that provides the design and implementation of parts corresponding to specific functions of software in a reusable, collaborative form to facilitate software or solution development. A software framework may include various other components that enable project or solution development, such as support programs, compilers, code libraries, toolsets, and Application Programming Interfaces (APIs).

[0046] In addition, frameworks are similar to libraries in that they structure code into a reusable form using clearly defined APIs; however, unlike libraries, frameworks allow users to extend and reuse code by optionally overriding or specializing it into user code that performs specific functions.

[0047] Referring to FIG. 2, to explain more specifically the object detection framework (200) according to embodiments of the present disclosure, the learner (210) is configured to learn an artificial intelligence model, and the learner (210) can learn an artificial intelligence model necessary for the object detection framework (200) to detect, classify (or filter) and track objects within an input image from an input image and to detect events occurring within the input image. For example, the learner (210) can learn at least one of a detection model, a first classification model, a second classification model, a tracking model, and an event detection model.

[0048] Referring further to FIG. 3, the detector (220) of the object detection framework (200) according to embodiments of the present disclosure may include a detection module (221), a first classification module (222), and a second classification module (223). In one embodiment, the object detection framework (200) may further include a tracking module (224) and an event detection module (225).

[0049] In the present disclosure, a module (or software module) may be understood to mean a unit in software design that is broken down into functional units and abstracted to a level where it can be reused and shared. That is, a module (or software module) may be configured to utilize at least one of the aforementioned artificial intelligence models. For example, the detection module (221) of the detector (220) illustrated in FIG. 3 corresponds to a module using a detection model learned (or generated) by a learner (210), the first classification module (222) corresponds to a module using a first classification model learned (or generated) by a learner (210), the second classification module (223) corresponds to a module using a second classification model learned (or generated) by a learner (210), the tracking module (224) corresponds to a module using a tracking model learned (or generated) by a learner (210), and the event detection module (225) corresponds to a module using an event detection model learned (or generated) by a learner (210).

[0050] Referring again to FIG. 2, the detection model may be an artificial intelligence model for detecting at least one object in an input image based on a first reference value and a second reference value, and for generating at least one first image segment including the detected at least one object. For example, a detector (220) (e.g., the detection module (221) of FIG. 3) may use the detection model to extract (or generate) an image segment (or bounding box) as a first image segment that satisfies both the first reference value and the second reference value (e.g., greater than or equal to the first reference value and greater than or equal to the second reference value) within the input image.

[0051] Additionally, the first classification model may be an artificial intelligence model for classifying (or performing first classification, first filtering) the first image segment by comparing the at least one first image segment detected or generated using a detection model with a previously stored second image segment, and assigning one of a first label and a second label to the at least one first image segment. For example, a detector (220) (e.g., the first classification module (222) of FIG. 3) may use the first classification model to calculate (or extract) the similarity between the at least one first image segment and the previously stored second image segment, assign a first label to the first image segment if the calculated similarity is less than a preset value, and assign a second label to the first image segment if the calculated similarity is greater than or equal to the preset value.

[0052] Additionally, the second classification model may be an artificial intelligence model for classifying (or, secondary classification, secondary filtering) the first image segment to which the first label is assigned through the first classification model among the at least one first image segment to which the first label is assigned. For example, a detector (220) (e.g., the second classification module (223) of FIG. 3) can classify the first image segment to which the first label is assigned by using the second classification model, if there is a sub-label among the multiple sub-labels in which the extracted class score is greater than or equal to the third reference value of the corresponding sub-label, by assigning the corresponding sub-label in which the class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned. In one embodiment, a detector (220) (e.g., the second classification module (223) of FIG. 3) can classify (or filter) the first image segment with the first label assigned as the third image segment by using the second classification model when the class scores extracted for each of the plurality of sub-labels are all less than the third reference value of the corresponding sub-label.

[0053] Additionally, the tracking model may be an artificial intelligence model for tracking a first image segment classified (or filtered) using a second classification model among the first image segments with the first label, for example, a first image segment with a sub-label. For example, a detector (220) (e.g., the tracking module (224) of FIG. 3) may track a first image segment with a sub-label by using a tracking model to compare image segments (or bounding boxes) included in at least two consecutive frame images, and / or by comparing feature points of the image segments.

[0054] Additionally, the event detection model may be an artificial intelligence model for detecting events occurring for a sub-labeled first image segment.

[0055] In one embodiment, the learner (210) may learn a detection model, a first classification model, a second classification model, a tracking model, and / or an event detection module using at least one of the artificial intelligence learning algorithms described with reference to FIG. 1 and / or at least one of the artificial neural networks, but is not limited thereto.

[0056] Next, the detector (220) may use at least one of the artificial intelligence models learned by the learner (210), for example, artificial intelligence models required to detect, classify, and track objects in an input image and to detect events occurring in an input image. For example, the detector (220) (e.g., each module included in the detector (220)) may use at least one of a detection model, a first classification model, a second classification model, a tracking model, and an event detection model to detect, classify, and track objects in an input image and to detect events occurring in an input image.

[0057] In one embodiment, a detector (220), for example, a detection module (221), can detect at least one object in an input image based on a first reference value and a second reference value using a detection model learned (or generated) by a learner (210), and can generate at least one first image segment including the detected at least one object.

[0058] For example, the detection module (221) can extract (or generate) an image segment (or, bounding box) that satisfies both a first reference value and a second reference value (e.g., greater than or equal to the first reference value and greater than or equal to the second reference value) as a first image segment.

[0059] The object detection operation of the object detection framework (200), for example, the detector (220) included in the object detection framework (200), for example, the detection module (221), will be explained in more detail with reference to FIGS. 6 and 7.

[0060] Additionally, in one embodiment, a detector (220), for example, a first classification module (222), can use a first classification model learned (or generated) by a learner (210) to compare at least one first image segment generated by a detection module (221) with a second image segment stored in advance, and assign one of a first label and a second label to the at least one first image segment.

[0061] For example, the first classification module (222) may calculate (or extract) the similarity between at least one first image segment generated by the detection module (221) and a stored second image segment, and if the calculated similarity is less than a preset value, assign a first label to the first image segment, and if the calculated similarity is greater than or equal to the preset value, assign a second label to the first image segment. As an example, the first classification module (222) may calculate (or extract) the similarity using the Intersection over Union (IoU) of the at least one first image segment and the stored second image segment, but this is merely illustrative and the embodiments of the present disclosure are not limited thereto.

[0062] The object detection framework (200), for example, the first classification operation (or first filtering operation) of the detector (220) included in the object detection framework (200), for example, the first classification module (222), will be described in more detail with reference to FIGS. 6 to 9.

[0063] Additionally, in one embodiment, the detector (220), for example, the second classification module (223), can extract class scores for each of a plurality of sub-labels for the first image segment, which is labeled first by the first classification module (222) among the at least one first image segment, using a second classification model learned (or generated) by the learner (210).

[0064] For example, if there is a sub-label among a plurality of sub-labels in which the extracted class score is greater than or equal to the third reference value of the sub-label, the second classification module (223) may assign the sub-label in which the class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned. According to an embodiment, the second classification module (223) may output (or generate) the first image segment to which the sub-label is assigned as a fourth image segment or a fifth image segment.

[0065] Additionally, in one embodiment, the second classification module (223) may provide the first image segment with the first label assigned to it as the third image segment when the class scores extracted for each of the plurality of sub-labels are all less than the third reference value of the corresponding sub-label. Here, the first classification module (222) may store the third image segment provided from the second classification module (223) as the second image segment.

[0066] The second classification operation (or second filtering operation) of the object detection framework (200), for example, the detector (220) included in the object detection framework (200), for example, the second classification module (223), will be described in more detail with reference to FIG. 6 and FIG. 10.

[0067] Additionally, in one embodiment, the detector (220), for example, the tracking module (224), can track the fourth image segment and / or the fifth image segment provided by the second classification module (223) using a tracking model learned (or generated) by the learner (210).

[0068] For example, the tracking module (224) can track the fourth image segment and / or the fifth image segment provided by the second classification module (223) by using a method of comparing image segments (or bounding boxes) included in at least two consecutive frame images. As an example, the tracking module (224) can track the fourth image segment and / or the fifth image segment provided by the second classification module (223) by using the IoU of image segments included in at least two consecutive frame images.

[0069] As another example, the tracking module (224) can track the fourth image segment and / or the fifth image segment provided by the second classification module (223) by using a method of comparing feature points of the image segment. As an example, the tracking module (224) can track the fourth image segment and / or the fifth image segment provided by the second classification module (223) by using a re-identification (ReID) method.

[0070] The object tracking operation of the object detection framework (200), for example, the detector (220) included in the object detection framework (200), for example, the tracking module (224), will be explained in more detail with reference to FIG. 6 and FIG. 11a to FIG. 12b.

[0071] Additionally, in one embodiment, the detector (220), for example, the event detection module (225), can detect events occurring on the fourth image segment using an event detection model learned (or generated) by the learner (210). For example, the event detection module (225) can detect events such as collapse occurring on the fourth image segment.

[0072] The event detection operation of the object detection framework (200), for example, the detector (220) included in the object detection framework (200), for example, the event detection module (225), will be explained in more detail with reference to FIG. 6 and FIG. 13.

[0073] In one embodiment, referring to FIGS. 2 and FIGS. 3, an object detection framework (200), e.g., a detector (220), may be operated by sequential operation (or operation) of a detection module (221), a first classification module (222), a second classification module (223), a tracking module (224), and an event detection module (225). For example, after the detection module (221) operates in response to an input image input to the detector (220), the first classification module (222) operates in response to output data (e.g., a first image segment) provided from the detection module (221), the second classification module (223) operates in response to output data (e.g., a first image segment with a first label) provided from the first classification module (222), the tracking module (224) operates in response to output data (e.g., a fourth image segment and / or a fifth image segment) provided from the second classification module (223), and the event detection module (225) operates in response to output data (e.g., a fourth image segment) provided from the tracking module (224).

[0074] However, this is merely illustrative and the embodiments of the present disclosure are not limited thereto. For example, the order of operation of the detection module (221), the first classification module (222), the second classification module (223), the tracking module (224), and the event detection module (225) included in the detector (220) may be changed, and / or at least some of the operations of the detection module (221), the first classification module (222), the second classification module (223), the tracking module (224), and the event detection module (225) may be performed simultaneously. Additionally, as another example, at least some of the operations of the detection module (221), the first classification module (222), the second classification module (223), the tracking module (224), and the event detection module (225) in the detector (220) may be omitted.

[0075] According to an embodiment, an object detection framework (200), for example, a detector (220), can detect, classify, track, and detect events of objects in camera images of a construction site, etc., by using at least one of the aforementioned modules (e.g., detection module (221), first classification module (222), second classification module (223), tracking module (224) and event detection module (225)).

[0076] For example, referring further to FIG. 4, FIG. 4 illustrates an example of an input image (IMG) input to an object detection framework (200) according to embodiments of the present disclosure, for example, a detector (220), such as a camera image of a construction site (e.g., CCTV image, etc.).

[0077] In one embodiment, the detector (220) can detect objects such as a person, fire, and smoke within an input image (IMG) using at least one of the aforementioned modules (e.g., detection module (221), first classification module (222), second classification module (223), tracking module (224) and event detection module (225)), classify (or filter) the detected objects, and assign (or designate) a sub-label (or sub-class) to each of the detected objects.

[0078] For example, referring further to FIG. 5, the detector (220) may assign (or designate) a sub-label (SLB) to each of the detected objects. For example, the detector (220) may designate a first sub-label (SLB1) for a person object (or, first object) among the detected objects, designate a second sub-label (SLB2) for a fire object (or, second object), and designate a third sub-label (SLB3) for a smoke object (or, third object).

[0079] Additionally, in one embodiment, the detector (220) can detect whether a safety helmet (or helmet) is being worn by a person object among the detected objects. For example, the detector (220) can calculate a class score for a fourth sub-label (SLB4) corresponding to a person head object (or a fourth object) and / or a fifth sub-label (SLB5) corresponding to a safety helmet (e.g., a helmet) object (or a fourth object) for a person object among the detected objects, e.g., an object assigned a first sub-label (SLB1), and based on this, detect whether a safety helmet (or helmet) is being worn by a person object (e.g., an object assigned a first sub-label (SLB1).

[0080] Additionally, in one embodiment, the detector (220) can continuously track a person object (e.g., an object assigned a first sub-label (SLB1)) among the detected objects and detect an event (e.g., collapse, etc.) for the person object (e.g., an object assigned a first sub-label (SLB1)).

[0081] However, the embodiments of the present disclosure are not limited thereto, and the input image (IMG) input to the object detection framework (200), for example, the detector (220), may be an image in various environments (e.g., an image inside a building, an image of a fire scene, etc.). For convenience of explanation, the following description will be based on the assumption that the input image (IMG) is a camera image of a construction site, as shown in FIG. 4.

[0082] In this way, an object detection framework (200), for example, a detector (220) according to embodiments of the present disclosure, can detect, classify (or filter) and track objects within an input image (IMG) and detect events occurring within the input image (IMG) by using at least one of each module (e.g., detection module (221), first classification module (222), second classification module (223), tracking module (224) and event detection module (225)). Here, as described above, the object detection framework (200), for example, a detector (220), can detect, classify (or filter) and track objects within the input image (IMG) and detect events occurring within the input image (IMG) by sequential operation (or operation) of the detection module (221), the first classification module (222), the second classification module (223), the tracking module (224) and the event detection module (225). As multiple modules operate sequentially in this manner, the accuracy of object detection, classification, tracking, and event detection operations of the object detection framework (200), e.g., detector (220), can be improved by the operations of the modules to which various conditions are applied. In particular, in the operation of detecting and classifying objects, the object detection framework (200) can detect and classify objects through the sequential operation (or operation) of multiple modules (e.g., detection module (221), first classification module (222), second classification module (223)) driven using different reference values. Accordingly, the accuracy of object detection and classification can be improved.

[0083] According to an embodiment, the object detection framework (200) may include additional configurations in addition to the learner (210) and the detector (220). For example, the object detection framework (200) may further include a communication module (e.g., the communication module (120) of FIG. 1), an input unit (e.g., the input unit (130) of FIG. 1), a display unit (e.g., the display unit (140) of FIG. 1), or a memory (e.g., the memory (150) of FIG. 1). In one example, a model learned through the learner (210) (e.g., a detection model, a first classification model, a second classification model, a tracking model, and / or an event detection model) may be stored in the memory of the object detection framework (200) (e.g., the memory (150) of FIG. 1).

[0084] In one embodiment, with reference to FIGS. 1 and FIGS. 2, the object detection framework (200) of FIGS. 2 may be the electronic device (100) described with reference to FIGS. 1 and / or may be implemented as the electronic device (100).

[0085] For example, at least one operation of the learner (210) and detector (220) shown in FIG. 2 may be performed by the processor (110) shown in FIG. 1. For example, the object detection framework (200) includes a processor (110) (e.g., CPU), and the processor (110) of the object detection framework (200) may detect, classify (or filter) and track objects within the input image (IMG) and detect events within the input image (IMG) based on the input image (IMG) input to the object detection framework (200) using a model learned (or created) by the aforementioned learner (210), for example, a detection model, a first classification model, a second classification model, a tracking model and / or an event detection model stored in memory (150).

[0086] As another example, at least one of the learner (210) and detector (220) shown in FIG. 2 may be at least one processor included in the processor (110) shown in FIG. 1 and / or may be performed by said at least one processor. For example, at least one of the learner (210) and detector (220) may be at least one auxiliary processor (e.g., GPU) included in the processor (110) and / or may be performed by said at least one auxiliary processor.

[0087] Hereinafter, with reference to FIGS. 6 to 17, the operation of an object detection framework (200) (or electronic device (100)) according to embodiments of the present disclosure will be described in more detail. Specifically, below, with reference to FIGS. 6 and 7, the object detection operation of the object detection framework (200) (or electronic device (100)) will be described in more detail; with reference to FIGS. 6 to 9, the first classification operation (or first filtering operation) of the object detection framework (200) (or electronic device (100)) will be described in more detail; with reference to FIGS. 6 and 10, the second classification operation (or second filtering operation) of the object detection framework (200) (or electronic device (100)) will be described in more detail; with reference to FIGS. 6 and FIGS. 11a to 12b, the object tracking operation of the object detection framework (200) (or electronic device (100)) will be described in more detail; and with reference to FIGS. 6 and 13, the event detection operation of the object detection framework (200) (or electronic device (100)) will be described in more detail.

[0089] <One embodiment of the detector>

[0090] FIG. 6 is a block diagram showing an example of a detection module (221), a first classification module (222), a second classification module (223), a tracking module (224), and an event detection module (225) included in the detector (220) of FIG. 3.

[0091] Meanwhile, FIG. 6 shows a block diagram illustrating an example of a detection module (221), a first classification module (222), a second classification module (223), a tracking module (224), and an event detection module (225) included in the detector (220) described with reference to FIG. 3, along with input images (IMG, IMG_2, IMG_3) for convenience of explaining the operation of each module (221 to 225).

[0092] Referring to FIGS. 2 through 6, a detector (220) according to embodiments of the present disclosure may include a detection module (221), a first classification module (222), a second classification module (223), a tracking module (224), and an event detection module (225).

[0093] In one embodiment, the detector (220) can detect, classify (or filter) and track objects within an input image (IMG) and detect events occurring within the input image (IMG) by using at least one of a detection module (221), a first classification module (222), a second classification module (223), a tracking module (224), and an event detection module (225). For example, the detector (220) can detect, classify (or filter) and track objects within the input image (IMG) and detect events occurring within the input image (IMG) through the sequential operation of the detection module (221), the first classification module (222), the second classification module (223), the tracking module (224), and the event detection module (225).

[0094] To explain the operation of each module (221 to 225) in more detail, FIGS. 7 to 13 may be further referenced below.

[0096] <One Example of a Detection Module>

[0097] FIG. 7 is a drawing for explaining an example of the operation of the detection module (221) and the first classification module (222) of FIG. 6.

[0098] Referring to FIGS. 2 through 7, the detection module (221) receives an input image (IMG) and can detect at least one object (e.g., person object, fire object, smoke object, etc.) from the input image (IMG). For example, the detection module (221) can detect at least one object based on a first reference value (REF1) and a second reference value (REF2) stored in the input image (IMG). Additionally, the detection module (221) can generate at least one first image segment containing (or corresponding to) the detected object.

[0099] More specifically, the detection module (221) can extract (or generate) image segments that satisfy both a first reference value (REF1) and a second reference value (REF2) among a plurality of image segments included in the input image (IMG) as a first image segment (IS1). For example, the detection module (221) can detect an object that satisfies both the first reference value (REF1) and the second reference value (REF2) in the input image (IMG) in the form of a bounding box, and generate the first image segment (IS1) by segmenting the image based on the bounding box within the input image (IMG), but is not limited thereto.

[0100] In one embodiment, the first reference value (REF1) is a value representing the probability or confidence that the corresponding image segment belongs to one of a plurality of sub-labels corresponding to a moving object or object of interest (e.g., person, fire, smoke, etc.) within the input image (IMG), for example, the first to third sub-labels (SLB1 to SLB3), and can be expressed as a probability distribution. That is, the first reference value (REF1) can be expressed as the value of a probability distribution that an object included in the corresponding image segment belongs to one of the first to third sub-labels (SLB1 to SLB3) corresponding to a moving object or object of interest (e.g., person, fire, smoke, etc.). Accordingly, the detection module (221) can determine whether the value of the probability distribution of a plurality of image segments included in the input image (IMG) belonging to one of the first to third sub-labels (SLB1 to SLB3) corresponding to a moving object or object of interest (e.g., person, fire, smoke, etc.) is greater than or equal to the first reference value (REF1).

[0101] Additionally, in one embodiment, the second reference value (REF2) may correspond to the value of the size (or width) of the area of ​​the image segment. That is, the second reference value (REF2) may be expressed as the value of the size (or width) of the corresponding image segment. Accordingly, the detection module (221) can determine whether the value of the size (or width) of the corresponding image segment among a plurality of image segments included in the input image (IMG) is greater than or equal to the second reference value (REF2).

[0102] The detection module (221) can extract (or generate) image segments as a first image segment (IS1) among a plurality of image segments included in an input image (IMG), wherein the value of the probability distribution for the image segment is greater than or equal to a preset first reference value (REF1) and the value of the size (or width) of the area of ​​the image segment is greater than or equal to a preset second reference value (REF2).

[0103] For example, as illustrated in the input image (IMG_1) of FIG. 7, the detection module (221) can extract (or generate) image segments 1-1 to 1-16 (IS1-1 to IS1-16) that satisfy both the first reference value (REF1) and the second reference value (REF2) within the input image (IMG) as first image segments (IS1).

[0104] Meanwhile, the first reference value (REF1) and the second reference value (REF2) may be stored in a separate memory (not shown) included in the detection module (221), but are not limited thereto, and the first reference value (REF1) and the second reference value (REF2) may be provided to the detection module (221) from an external memory (e.g., memory (150) of FIG. 1).

[0105] According to an embodiment, the detection module (221) may perform batch processing on the extracted (or generated) first image segments (IS1) and output a plurality of first image segments (IS1) in the form of batch data, but is not limited thereto.

[0106] In this way, the first image segment (IS1) detected (or extracted, generated) by the detection module (221) can be provided to the first classification module (222).

[0108] <One embodiment of the first classification module>

[0109] FIG. 8 is a diagram illustrating an example of the operation of the first classification module (222) of FIG. 6.

[0110] FIG. 9 is a flowchart illustrating an object classification method according to various embodiments of the present disclosure.

[0111] Referring to FIGS. 2 through 6, the first classification module (222) receives at least one first image segment (IS1) from the detection module (221), compares at least one first image segment (IS1) with a previously stored second image segment (IS2), and can assign (or, assign) a label of either a first label (LB1) or a second label (LB2) to at least one first image segment (IS1) based on the comparison result. For example, the first classification module (222) can assign (or, assign) a label of either a first label (LB1) or a second label (LB2) to at least one first image segment (IS1) using label data (LBD) containing data for the first label (LB1) and the second label (LB2).

[0112] Meanwhile, the second image segment (IS2) and label data (LBD) may be stored in a separate memory (not shown) included in the first classification module (222), but are not limited thereto, and the second image segment (IS2) and label data (LBD) may be provided to the first classification module (222) from an external memory (e.g., memory (150) of FIG. 1).

[0113] More specifically, referring further to FIG. 7, the first classification module (222) calculates (or extracts) similarity by comparing each of the first-1 to first-16 image segments (IS1-1 to IS1-16) provided by the detection module (221) with a stored second image segment (IS2), and if the calculated similarity is less than a preset value, a first label (LB1) may be assigned to the corresponding first image segment (IS1). According to the embodiment, the preset value may have a value determined based on experimental data, but is not limited thereto, and may be determined through various methods.

[0114] For example, if the similarity between the first-1 to first-7 image segments (IS1-1 to IS1-7) and the previously stored second image segment (IS2) is less than a preset value, as shown in FIG. 7, the first classification module (222) can assign a first label (LB1) to each of the first-1 to first-7 image segments (IS1-1 to IS1-7).

[0115] Additionally, the first classification module (222) calculates (or extracts) similarity by comparing each of the first-1 to first-16 image segments (IS1-1 to IS1-16) provided by the detection module (221) with a previously stored second image segment (IS2), and if the calculated similarity is greater than or equal to a preset value, it can assign a second label (LB2) to the corresponding first image segment (IS1).

[0116] For example, if the similarity between the first-8 to first-16 image segments (IS1-8 to IS1-16) and the previously stored second image segment (IS2) is greater than or equal to a preset value, as shown in FIG. 7, the first classification module (222) can assign a second label (LB2) to each of the first-8 to first-16 image segments (IS1-8 to IS1-16). For example, referring further to FIG. 8, as illustrated in FIG. 7 and FIG. 8, if the similarity between the first-8 image segment (IS1-8) and the previously stored second-1 image segment (IS2-1) is greater than or equal to a preset value, the similarity between each of the first-9 to first-14 image segments (IS1-9 to IS1-14) and the previously stored second-2 image segment (IS2-2) is greater than or equal to a preset value, the similarity between the first-15 image segment (IS1-15) and the previously stored second-3 image segment (IS2-3) is greater than or equal to a preset value, and the similarity between the first-16 image segment (IS1-16) and the previously stored second-4 image segment (IS2-4) is greater than or equal to a preset value, the first classification module (222) assigns a second label (LB2) to each of the first-8 to first-16 image segments (IS1-8 to IS1-16). It can be designated.

[0117] Here, the second image segments (IS2) may be image segments corresponding to background image segments. For example, as illustrated in FIG. 8, each of the second image segments (IS2), for example, 2-1 to 2-4 image segments (IS2-1 to IS2-4), may be an image segment corresponding to a fixed object (e.g., structure, signboard, etc.) rather than a moving object or object of interest (e.g., person, fire, smoke, etc.) that can move within the input image (IMG) provided through a fixed camera.

[0118] That is, the first classification module (222) can classify (or filter) objects (e.g., fixed objects such as structures, signs, etc.) that are not objects of interest (e.g., moving objects such as people, fire, smoke, etc.) among the first image segments (IS1) extracted (or generated) from the detection module (221) by filtering (e.g., background filtering) the first image segments (IS1) provided from the detection module (221) using a second image segment (IS2) corresponding to a background image segment such as a fixed object. For example, if the similarity between the first image segment (IS1) and the second image segment (IS2) corresponding to the background image segment is greater than or equal to a preset value, the first classification module (222) determines that the first image segment (IS1) is an image segment corresponding to a fixed object (e.g., structure, signboard, etc.) that does not correspond to a moving object or object of interest (e.g., person, fire, smoke, etc.) that can move within the input image (IMG), and can classify (or filter) the first image segment (IS1) as a background image segment (e.g., assign a second label (LB2) to the first image segment (IS1)). In other words, the fact that a first label (LB1) is assigned to a first image segment (IS1) means that the first image segment (IS1) has been classified (or filtered) by the operation of the first classification module (222) as an image segment corresponding to a moving object or object of interest (e.g., person, fire, smoke, etc.) that can move within the input image (IMG), and the fact that a second label (LB2) is assigned to a first image segment (IS1) means that the first image segment (IS1) has been classified (or filtered) as an image segment corresponding to a fixed object (e.g., structure, signboard, etc.) by the operation of the first classification module (222).

[0119] Accordingly, as illustrated in FIG. 6, the first image segment (IS1) designated as the first label (LB1) by the first classification module (222) is classified (or filtered) as an object of interest and output (e.g., output to the second classification module (223)), and the first image segment (IS1) designated as the second label (LB2) by the first classification module (222) may not be classified (or filtered) as an object that is not an object of interest and output.

[0120] In one embodiment, the first classification module (222) may assign a first label (LB1) to the first image segment (IS1) if the Intersection over Union (IoU) value between the first image segment (IS1) provided by the detection module (221) and the previously stored second image segment (IS2) is less than a preset IoU value, and assign a second label (LB2) to the first image segment (IS1) if the IoU value between the first image segment (IS1) provided by the detection module (221) and the previously stored second image segment (IS2) is greater than or equal to the preset IoU value. For example, the preset IoU value may be 0.5, but this is merely illustrative and the embodiments of the present disclosure are not limited thereto.

[0121] To explain the operation of the first classification module (222) more specifically, further refer to FIG. 9. The object classification method of FIG. 9 can be performed within the first classification module (222).

[0122] In one embodiment, the object classification method of FIG. 9 may include the step of loading a second image segment (IS2) (S901) and the step of comparing the IoU of the first image segment (IS1) and the stored second image segment (IS2) (S902).

[0123] Here, as described above, in the step of loading the second image segment (IS2) (S901), the second image segment (IS2) may be stored on a separate memory (not shown) included in the first classification module (222) and loaded from said separate memory (not shown), and / or stored on an external memory (e.g., memory (150) of FIG. 1) and loaded from said external memory (e.g., memory (150) of FIG. 1).

[0124] According to an embodiment, the second image segment (IS2) may be updated at preset times on a separate memory (not shown) and / or external memory (e.g., memory (150) of FIG. 1) included in the first classification module (2212). For example, the second image segment (IS2) may be deleted from the memory after a preset time (e.g., 10 seconds) has elapsed, after being stored on a separate memory (not shown) and / or external memory (e.g., memory (150) of FIG. 1) included in the first classification module (2212). However, this is merely illustrative, and the second image segment (IS2) may be stored on a separate memory (not shown) and / or external memory (e.g., memory (150) of FIG. 1) included in the first classification module (222) and retained without being deleted.

[0125] Additionally, in the step of comparing the IoU of the first image segment (IS1) and the stored second image segment (IS2) (S902), the first classification module (222) can determine whether the IoU value between the first image segment (IS1) and the second image segment (IS2) is greater than or less than a preset value, for example, 0.5.

[0126] Here, the object classification method of FIG. 9 may further include the step of assigning a second label (LB2) to the first image segment (IS1) (S903) when the IoU value between the first image segment (IS1) and the second image segment (IS2) is 0.5 or higher. In this case, as described above, the first image segment (IS1) to which the second label (LB2) is assigned may not be output. For example, the first image segment (IS1) to which the second label (LB2) is assigned may not be output from the first classification module (222).

[0127] Additionally, the object classification method of FIG. 9 may further include the steps of assigning a first label (LB1) to the first image segment (IS1) (S904) when the IoU value between the first image segment (IS1) and the second image segment (IS2) is less than 0.5, and outputting the first image segment (IS1) to which the first label (LB1) is assigned (S905). In this case, as described above, the first image segment (IS1) to which the first label (LB1) is assigned may be output. For example, the first image segment (IS1) to which the first label (LB1) is assigned may be output from the first classification module (222) and provided to the second classification module (223).

[0129] <One embodiment of the second classification module>

[0130] FIG. 10 is a flowchart illustrating an object classification method according to various embodiments of the present disclosure.

[0131] Referring to FIGS. 2 to 6, the second classification module (223) receives a first image segment (IS1) with a first label (LB1) assigned from the first classification module (222), and can extract (or calculate) class scores for each of a plurality of sub-labels (SLB1 to SLB5) for the first image segment (IS1) with the first label (LB1).

[0132] For example, as illustrated in the input image (IMG_2) of FIG. 6, the second classification module (223) can extract class scores for each of the multiple sub-labels (SLB1 to SLB5) included in the sub-label data (SLBD) stored for each of the first image segment (IS1) to which the first label (LB1) is assigned from the first classification module (222), for example, the first-1 to first-7 image segments (IS1-1 to IS1-7).

[0133] Here, a class score is a score representing the probability or confidence that the corresponding image segment (e.g., the first image segment (IS1) assigned the first label (LB1)) belongs to the corresponding sub-label (e.g., any one of the sub-labels (SLB1 to SLB5)), and may be expressed, for example, as a probability distribution, but is not limited thereto.

[0134] In one embodiment, the second classification module (223) can compare the class scores for the sub-labels corresponding to the aforementioned object of interest (e.g., moving object such as person, fire, smoke, etc.) among a plurality of sub-labels (SLB1 to SLB5) with the third reference value (REF3) of the corresponding sub-label. For example, referring further to FIG. 5, the second classification module (223) can compare the class scores for each of the first to third sub-labels (SLB1 to SLB3) corresponding to moving object such as person, fire, smoke, etc. with the third reference value (REF3) of the corresponding sub-label.

[0135] Here, the third reference value (REF3) is a value pre-set for each of the first to third sub-labels (SLB1 to SLB3) corresponding to moving objects such as people, fire, smoke, etc. The third reference values ​​(REF3) corresponding to each of the first to third sub-labels (SLB1 to SLB3) may be pre-stored in a separate memory (not shown) included in the second classification module (223), but are not limited thereto, and the third reference values ​​(REF3) may be provided to the second classification module (223) from an external memory (e.g., memory (150) of FIG. 1).

[0136] In one embodiment, if there is a sub-label among the first to third sub-labels (SLB1 to SLB3) corresponding to an object of interest (e.g., a moving object such as a person, fire, smoke, etc.) where the class score extracted corresponding to the sub-label (e.g., any one of the first to third sub-labels (SLB1 to SLB3)) is greater than or equal to the third reference value (REF3) corresponding to the sub-label (e.g., any one of the first to third sub-labels (SLB1 to SLB3)), the second classification module (223) may assign the sub-label having a class score greater than or equal to the third reference value (REF3) to the corresponding image segment, that is, the first image segment (IS1) to which the first label (LB1) provided by the first classification module (222) is assigned. For example, as shown in the input image (IMG_2) of FIG. 6, based on the class scores extracted corresponding to each of the first to third sub-labels (SLB1 to SLB3), the second classification module (223) may assign a first sub-label (SBL1) to the first-1 image segment (IS1-1) and the first-2 image segment (IS1-2), respectively, and assign a second sub-label (SBL2) to the first-3 image segment (IS1-3).

[0137] According to an embodiment, if there are multiple sub-labels among the first to third sub-labels (SLB1 to SLB3) in which the extracted class score is greater than or equal to the third reference value (REF3) corresponding to the sub-label, the second classification module (223) may assign the sub-label among the multiple sub-labels that has the largest difference between the extracted class score and the third reference value (REF3) to the corresponding image segment (i.e., the first image segment (IS1) to which the first label (LB1) is assigned).

[0138] According to an embodiment, the second classification module (223) may output (or generate) a first image segment (IS1) to which any one of the first to third sub-labels (SLB1 to SLB3) is specified as a fourth image segment (IS4) or a fifth image segment (IS5). Hereinafter, the fourth image segment (IS4) is the first image segment (IS1) to which the first sub-label (SLB1) corresponding to a person object among the first to third sub-labels (SLB1 to SLB3) is specified, and the fifth image segment (IS5) is the first image segment (IS1) to which the second sub-label (SLB2) or third sub-label (SLB3) corresponding to a fire object or a smoke object among the first to third sub-labels (SLB1 to SLB3) is specified. For example, as shown in the input image (IMG_3) of FIG. 6, the first-1 and first-2 image segments (IS1-1, IS1-2) corresponding to a person object are assigned a first sub-label (SLB1) and output from the second classification module (223) as a fourth image segment (IS4), and the first-3 image segment (IS1-3) corresponding to a fire object is assigned a second sub-label (SLB2) and output from the second classification module (223) as a fifth image segment (IS5).

[0139] Additionally, in one embodiment, if each of the class scores extracted for each of the first to third sub-labels (SLB1 to SLB3) corresponding to an object of interest (e.g., a moving object such as a person, fire, smoke, etc.) is less than the third reference value (REF3) of the corresponding sub-label (e.g., any one of the first to third sub-labels (SLB1 to SLB3)), the second classification module (223) may provide (e.g., feedback) the corresponding image segment, that is, the first image segment (IS1) to which the first label (LB1) provided from the first classification module (222) is assigned, to the first classification module (222) as the third image segment (IS3). In other words, if all of the class scores extracted for each of the first to third sub-labels (SLB1 to SLB3) corresponding to the object of interest (e.g., moving object such as person, fire, smoke, etc.) are less than the third reference value (REF3) of the corresponding sub-label (e.g., any one of the first to third sub-labels (SLB1 to SLB3)), the second classification module (223) may provide (e.g., feedback) the corresponding image segment, that is, the first image segment (IS1) to which the first label (LB1) provided from the first classification module (222) is assigned, to the first classification module (222) as the third image segment (IS3). For example, as illustrated in the input image (IMG_2) of FIG. 6, based on class scores extracted corresponding to each of the first to third sub-labels (SLB1 to SLB3), the second classification module (223) can provide (e.g., provide feedback) the first-4 to first-7 image segments (IS1-4 to IS1-7) to the first classification module (222) as the third image segment (IS3), e.g., the third-1 to third-4 image segments (IS3-1 to IS3-4).

[0140] In one embodiment, the first classification module (222) can store the third image segment (IS3) provided from the second classification module (223) as the second image segment (IS2) corresponding to the background image segment.

[0141] For example, in the operation of the first classification module (222) described with reference to FIGS. 6 to 9, the first classification module (222) can classify (or filter) objects (e.g., fixed objects such as structures, signs, etc.) that are not objects of interest (e.g., moving objects such as people, fire, smoke, etc.) among the first image segments (IS1) extracted (or generated) from the detection module (221) by filtering (e.g., background filtering) the first image segment (IS1) using the second image segment (IS2) stored as a background image segment. Here, the third image segment (IS3) provided (e.g., feedback) from the second classification module (223) corresponds to an image segment that is determined by the operation of the second classification module (223) not to correspond to any of the first to third sub-labels (SLB1 to SLB3) corresponding to objects of interest (e.g., moving objects such as people, fire, smoke, etc.), and can be treated as a background image segment. Accordingly, the object detection framework (200) according to the embodiments of the present disclosure, for example, the detector (220), can increase the utilization of data processing by utilizing the image segment filtered by the operation of the second classification module (223), i.e., the third image segment (IS3), as the second image segment (IS2) required for the operation of the first classification module (222). In addition, since the number of data (i.e., the second image segment (IS2)) required for the operation of the first classification module (222) increases by utilizing the third image segment (IS3) as the second image segment (IS2), the performance of the first classification module (222) can be further improved.

[0142] Meanwhile, as described above, according to the embodiment, the third image segment (IS3) may be stored as the second image segment (IS2) in a separate memory (not shown) and / or an external memory (e.g., memory (150) of FIG. 1) included in the first classification module (2212), and then updated at preset times.

[0143] To explain the operation of the second classification module (223) more specifically, refer further to FIG. 10. The object classification method of FIG. 10 can be performed within the second classification module (223).

[0144] In one embodiment, the object classification method of FIG. 10 may include the steps of extracting class scores for each of a plurality of sub-labels (SLB1 to SLB5) for a first image segment (IS1) assigned a first label (LB1) (S1001) and determining whether the class score of any one of the first to third sub-labels (SLB1 to SLB3) is greater than or equal to a third reference value (REF3) of the corresponding sub-label (S1002).

[0145] Here, the object classification method of FIG. 10 may further include the step of outputting (S1003) a first image segment (IS1) assigned to a first label (LB1) as a third image segment (IS3) when there is no sub-label among the first to third sub-labels (SLB1 to SLB3) in which the class score is greater than or equal to the third reference value (REF3) of the corresponding sub-label. For example, when each of the class scores extracted for each of the first to third sub-labels (SLB1 to SLB3) (i.e., all of them) is less than the third reference value (REF3) of the corresponding sub-label (e.g., any one of the first to third sub-labels (SLB1 to SLB3)), the first image segment (IS1) assigned to a first label (LB1) may be output as a third image segment (IS3).

[0146] In one embodiment, as described above, the third image segment (IS3) may be provided from the second classification module (223) to the first classification module (222), and the first classification module (222) may store the third image segment (IS3) as the second image segment (IS2).

[0147] Additionally, the object classification method of FIG. 10 may further include the step of determining whether the class score of the first sub-label (SLB1) is greater than or equal to the third reference value (REF1) when the class score of any one of the first to third sub-labels (SLB1 to SLB3) is greater than or equal to the third reference value (REF3) of the corresponding sub-label (S1004).

[0148] For example, if the class score of any one of the first to third sub-labels (SLB1 to SLB3) is greater than or equal to the third reference value (REF3) of the corresponding sub-label, the second classification module (223) classifies the corresponding image segment (e.g., the first image segment (IS1) assigned to the first label (LB1)) as an image segment corresponding to an object of interest (e.g., a moving object such as a person, fire, smoke, etc.), and in order to determine whether the corresponding image segment (e.g., the first image segment (IS1) assigned to the first label (LB1)) corresponds to an important object (e.g., a person object) in object detection and classification, it can determine whether the class score extracted from the corresponding image segment (e.g., the first image segment (IS1) assigned to the first label (LB1)) corresponding to the first sub-label (SLB1) is greater than or equal to the third reference value (REF3) of the first sub-label (SLB1).

[0149] Here, the object classification method of FIG. 10 may further include the steps of generating a first alarm (S1005) when the class score of the first sub-label (SLB1) is less than the third reference value (REF1), assigning a second sub-label (SLB2) or a third sub-label (SLB3) to the first image segment (IS1) (S1006), and outputting the first image segment (IS1) to which the second sub-label (SLB2) or the third sub-label (SLB3) is assigned (S1007).

[0150] For example, the step of determining whether the class score of the first sub-label (SLB1) is greater than or equal to the third reference value (REF1) (S1004) may be performed after the step of determining whether the class score of any one of the aforementioned first to third sub-labels (SLB1 to SLB3) is greater than or equal to the third reference value (REF3) of the corresponding sub-label (S1002). Accordingly, in the step of determining whether the class score of the first sub-label (SLB1) is greater than or equal to the third reference value (REF1) (S1004), if it is determined that the class score extracted from the corresponding image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned) corresponding to the first sub-label (SLB1) is less than the third reference value (REF3) of the first sub-label (SLB1), the corresponding image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned) may be an image segment corresponding to either the second sub-label (SLB2) or the third sub-label (SLB3).

[0151] Accordingly, the second classification module (223) of the object detection framework (200), for example, the detector (220), can determine through the corresponding image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned) that a fire corresponding to the second sub-label (SLB2) or smoke corresponding to the third sub-label (SLB3) has occurred within the environment (e.g., a construction site) included in the input image (IMG), and can generate a first alarm. That is, the first alarm may be an alarm to warn the user that a fire or smoke has occurred in the environment (e.g., a construction site) included in the input image (IMG).

[0152] Additionally, the second classification module (223) may assign a second sub-label (SLB2) or a third sub-label (SLB3) based on class scores extracted for the corresponding image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned), and output the first image segment (IS1) to which the second sub-label (SLB2) or the third sub-label (SLB3) is assigned. According to an embodiment, if all of the extracted class scores among the second and third sub-labels (SLB2, SLB3) are greater than or equal to the third reference value (REF3) corresponding to the sub-label, the second classification module (223) may assign the sub-label with the largest difference between the extracted class score and the third reference value (REF3) among the second and third sub-labels (SLB2, SLB3) to the corresponding image segment (i.e., the first image segment (IS1) to which the first label (LB1) is assigned).

[0153] Meanwhile, as described above, the second classification module (223) can output the corresponding image segment designated with the second sub-label (SLB2) or the third sub-label (SLB3) as the fifth image segment (IS5). For example, as shown in the input images (IMG_2, IMG_3) of FIG. 6, the first-third image segments (IS1-3) designated with the second sub-label (SLB2) can be output from the second classification module (223) as the fifth image segment (IS5) and provided to the tracking module (224).

[0154] Additionally, the object classification method of FIG. 10 may further include a step (S1008) of determining whether the class score of the fourth sub-label (SLB4) is greater than or equal to the third reference value (REF3) when the class score of the first sub-label (SLB1) is greater than or equal to the third reference value (REF1).

[0155] For example, if the class score for the first sub-label (SLB1) of the corresponding image segment (e.g., the first image segment (IS1) with the first label (LB1)) is greater than or equal to the third criterion value (REF1), the second classification module (223) may determine that the corresponding image segment (e.g., the first image segment (IS1) with the first label (LB1)) corresponds to a person object which is an important object.

[0156] Here, the second classification module (223) can determine whether a person object, which is an important object, is wearing safety equipment in an environment (e.g., a construction site) within the input image (IMG). For example, the second classification module (223) can determine whether the person object, which is an important object, is wearing a safety helmet (e.g., a helmet).

[0157] To this end, the second classification module (223) can determine whether the class score for the fourth sub-label (SLB4) of the corresponding image segment (e.g., the first image segment (IS1) assigned the first label (LB1)) determined to correspond to a person object, which is an important object, is greater than or equal to the third criterion value (REF3). Here, the fourth sub-label (SLB4) may correspond to a sub-label corresponding to a person's head object.

[0158] Here, the object classification method of FIG. 10 may further include the steps of assigning a first sub-label (SLB1) to a first image segment (IS1) (S1009) when the class score of the fourth sub-label (SLB4) is less than the third reference value (REF1), and outputting the first image segment (IS1) to which the first sub-label (SLB1) is assigned (S1010).

[0159] For example, if the class score of the fourth sub-label (SLB4) is less than the third reference value (REF1), with respect to the image segment determined to correspond to the important object, the person object, through the previous step (e.g., step S1004), the second classification module (223) may determine that the image segment does not contain an image segment corresponding to the person's head object corresponding to the fourth sub-label (SLB4). In this case, the second classification module (223) may determine that the image segment does not contain an image segment corresponding to the person's head object because the object (e.g., person) is wearing safety equipment such as a safety helmet (e.g., helmet). Accordingly, if the class score of the fourth sub-label (SLB4) is less than the third reference value (REF1), the second classification module (223) determines that the object (e.g., person) is wearing safety equipment such as a safety helmet (e.g., helmet), and without generating a separate alarm, assigns only the first sub-label (SLB1) corresponding to the person object, which is an important object, to the image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned), and outputs the first image segment (IS1) to which the first sub-label (SLB1) is assigned.

[0160] Meanwhile, as described above, the second classification module (223) can output the corresponding image segment with the first sub-label (SLB1) as the fourth image segment (IS4). For example, as shown in the input images (IMG_2, IMG_3) of FIG. 6, the first-1 image segment (IS1-1) with the first sub-label (SLB1) can be output from the second classification module (223) as the fourth image segment (IS4) (e.g., the fourth-1 image segment (IS4-1)) and provided to the tracking module (224).

[0161] Additionally, the object classification method of FIG. 10 may further include a step (S1011) of determining whether the class score of the fifth sub-label (SLB5) is greater than or equal to the third reference value (REF3) when the class score of the fourth sub-label (SLB4) is greater than or equal to the third reference value (REF1).

[0162] For example, the second classification module (223) can determine whether a person object, which is an important object, is wearing safety equipment in an environment (e.g., a construction site) within the input image (IMG). For example, to directly determine whether the person object, which is an important object, is wearing a safety helmet (e.g., a helmet), the second classification module (223) can determine whether the class score for the fifth sub-label (SLB5) corresponding to the safety helmet (e.g., a helmet) object is greater than or equal to the third reference value (REF3).

[0163] Here, the object classification method of FIG. 10 may further include the steps of assigning a first sub-label (SLB1) to a first image segment (IS1) (S1009) when the class score of the fifth sub-label (SLB5) is greater than or equal to the third reference value (REF1), and outputting the first image segment (IS1) to which the first sub-label (SLB1) is assigned (S1010).

[0164] For example, if the class score of the fifth sub-label (SLB5) is greater than or equal to the third reference value (REF1), the second classification module (223) determines that the object (e.g., person) is wearing safety equipment such as a safety helmet (e.g., helmet), and without generating a separate alarm, assigns only the first sub-label (SLB1) corresponding to the person object, which is an important object, to the image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned), and outputs the first image segment (IS1) to which the first sub-label (SLB1) is assigned.

[0165] Additionally, the object classification method of FIG. 10 may further include the steps of generating a second alarm (S1012) when the class score of the fifth sub-label (SLB5) is less than the third reference value (REF1), assigning a first sub-label (SLB1) and a sixth sub-label (SLB6) to a first image segment (IS1) (S1013), and outputting the first image segment (IS1) to which the first sub-label (SLB1) and the sixth sub-label (SLB6) are assigned (S1014).

[0166] For example, if the class score of the fourth sub-label (SLB4) is greater than or equal to the third reference value (REF3) and the class score of the fifth sub-label (SLB5) is less than the third reference value (REF1), the second classification module (223) may determine that the person object contains an image segment corresponding to a person head object, but does not contain an image segment corresponding to a safety helmet (e.g., helmet) object.

[0167] Accordingly, the second classification module (223) of the object detection framework (200), for example, the detector (220), can determine through the corresponding image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned) that there is an object (e.g., a person) not wearing safety equipment such as a safety helmet (e.g., a helmet) in the environment (e.g., a construction site) included in the input image (IMG), and can generate a second alarm. That is, the second alarm may be an alarm to warn the user that there is an object (e.g., a person) not wearing safety equipment in the environment (e.g., a construction site) included in the input image (IMG).

[0168] Additionally, the second classification module (223) may additionally assign a sixth sub-label (SLB6) along with a first sub-label (SLB1) corresponding to a person object for the corresponding image segment (e.g., the first image segment (IS1) to which the first label (LB1) is assigned). Here, the sixth sub-label (SLB6) may correspond to a sub-label indicating that the object (e.g., a person) is not wearing a safety helmet (e.g., a helmet). In other words, the object corresponding to the first image segment (IS1) to which the first sub-label (SLB1) and the sixth sub-label (SLB6) are assigned may correspond to a person object that is not wearing a safety helmet (e.g., a helmet).

[0169] The second classification module (223) can output a first image segment (IS1) to which a first sub-label (SLB1) and a sixth sub-label (SLB6) are assigned.

[0170] Meanwhile, as described above, the second classification module (223) can output the corresponding image segment with the first sub-label (SLB1) (or the first sub-label (SLB1) and the sixth sub-label (SLB6)) designated as the fourth image segment (IS4). For example, as shown in the input images (IMG_2, IMG_3) of FIG. 6, the first-2 image segment (IS1-2) with the first sub-label (SLB1) and the sixth sub-label (SLB6) designated can be output from the second classification module (223) as the fourth image segment (IS4) (e.g., the fourth-2 image segment (IS4-2)) and provided to the tracking module (224).

[0171] In this way, the second classification module (223) included in the object detection framework (200), for example, the detector (220) according to the embodiments of the present disclosure, can increase the accuracy of object detection and classification by assigning a sub-label to the corresponding image segment (e.g., object) based on the class scores of a plurality of sub-labels (SLB1 to SLB5). In addition, the second classification module (223) included in the object detection framework (200), for example, the detector (220) according to the embodiments of the present disclosure, can generate an alarm to the user in response to emergency situations, such as the occurrence of fire or smoke or failure to wear a safety helmet (e.g., helmet), based on the class scores of a plurality of sub-labels (SLB1 to SLB5). Accordingly, the utility of the object detection framework (200) can be improved.

[0173] <One Example of a Tracking Module>

[0174] FIG. 11a is a drawing for illustrating an example of a first comparator (CP1) included in the tracking module (224) of FIG. 6.

[0175] FIG. 11b is a diagram illustrating an example of the operation of the first comparator (CP1) of FIG. 11a.

[0176] FIG. 12a is a drawing for illustrating an example of a second comparator (CP2) included in the tracking module (224) of FIG. 6.

[0177] FIG. 12b is a diagram illustrating an example of the operation of the second comparator (CP2) of FIG. 12a.

[0178] Referring to FIGS. 2 through 6, the tracking module (224) receives at least one of a fourth image segment (IS4) and a fifth image segment (IS5) from the second classification module (223) and can track at least one of the fourth image segment (IS4) and the fifth image segment (IS5). For example, the tracking module (224) can track the corresponding image segment in a preset frame unit (e.g., one frame unit, two or more frame units, etc.) for an input image (IMG) that is input continuously in frame units, but is not limited thereto.

[0179] In one embodiment, the tracking module (224) can track a fourth image segment (IS4) corresponding to a person object among the image segments (IS4, IS5) provided by the second classification module (223). For example, the tracking module (224) can assign identification numbers to the fourth image segment (IS4) and the fifth image segment (IS5) provided by the second classification module (223). For example, as shown in the input image (IMG_3) of FIG. 6, the tracking module (224) assigns a first identification number (ID1) and a second identification number (ID2) to the fourth image segment (IS4) corresponding to a person object provided by the second classification module (223), for example, the fourth-1 image segment (IS4-1) and the fourth-2 image segment (IS4-2), respectively, and can track the fourth image segment (IS4) to which the identification number (e.g., ID1 or ID2) has been assigned. Additionally, the tracking module (224) may assign a sub-identification number (SID) to the fifth image segment (IS5) corresponding to the fire object and not perform a separate tracking operation. That is, according to the embodiment, the tracking module (224) may perform a tracking operation for the image segment (e.g., the fourth image segment (IS4)) corresponding to the person object among the image segments (IS4, IS5) provided by the second classification module (223).

[0180] However, the embodiments of the present disclosure are not limited thereto, and the tracking module (224) may perform a tracking operation for the fifth image segment (IS5). For convenience of explanation, the following description will be based on the premise that the tracking module (224) performs a tracking operation only for the fourth image segment (IS4).

[0181] In one embodiment, the tracking module (224) can track the fourth image segment (IS4) by using a method of comparing image segments (or bounding boxes) included in at least two consecutive frame images and / or a method of comparing feature points of image segments.

[0182] To this end, in one embodiment, the tracking module (224) may include a first comparator (CP1) and a second comparator (CP2). Here, the first comparator (CP1) can compare image segments (or bounding boxes) included in at least two consecutive frame images, and the second comparator (CP2) can compare feature points of image segments included in the frame images with previously stored feature points.

[0183] According to an embodiment, the tracking module (224) may be operated by the sequential operation (or operation) of the first comparator (CP1) and the second comparator (CP2). For example, a tracking operation for the fourth image segment (IS4) may be performed according to the operation of the first comparator (CP1), and a tracking operation for the fourth image segment (IS4) may be performed according to the operation of the second comparator (CP2). However, the embodiments of the present disclosure are not limited thereto, and the tracking module (224) may be operated according to the sequential operation (or operation) of the second comparator (CP2) and the first comparator (CP1), or the tracking operation of the tracking module (224) may be performed by the first and second comparators (CP1, CP2) operating simultaneously. According to an embodiment, the tracking module (224) may be operated by the operation of either one of the first and second comparators (CP1, CP2).

[0184] Referring further to FIGS. 11a and 11b to describe the first comparator (CP1) more specifically, the first comparator (CP1) may include a first data buffer (DBF1) containing at least one image segment to which each identification number (e.g., ID#1, ID#2, ID#3 in FIG. 11a) is assigned. Here, the at least one image segment stored in the first data buffer (DBF1) may be an image segment containing an object detected in previous frame images (e.g., input images (IMG)) and classified as a human object, that is, a first image segment (IS1) to which a first sub-label (SLB1) is assigned in previous frame images. For example, the first data buffer (DBF1) may be a frame buffer, but is not limited thereto.

[0185] Meanwhile, as illustrated in FIG. 11a, image segments included in the first data buffer (DBF1) may each be assigned an identification number (e.g., ID#1, ID#2, ID#3 of FIG. 11a).

[0186] As illustrated in FIG. 11b, the first comparator (CP1) can track objects (image segments) by comparing image segments included in the previous frame image (IMGa) (e.g., first sub-image segment (ISa), second sub-image segment (ISb)) with image segments included in the current frame image (IMGb) (e.g., third sub-image segment (ISc), fourth sub-image segment (ISd), fifth image segment (ISe)) and assigning identification numbers to image segments similar to those classified and assigned identification numbers in the previous frame image (IMGa) for the image segments included in the current frame image (IMGb) using data stored in the first data buffer (DBF1). For example, the first comparator (CP1) can extract an IoU value between image segments included in the previous frame image (IMGa) and image segments included in the current frame image (IMGb), and compare the image segments included in the previous frame image (IMGa) and the image segments included in the current frame image (IMGb) based on whether the IoU value is greater than or equal to a threshold value (TH). As an example, the first comparator (CP1) may be implemented as an IoU-based Tracker, but is not limited thereto.

[0187] Specifically, as illustrated in FIG. 11b, if the IoU value between the first sub-image segment (ISa) of the previous frame image (IMGa) and the third sub-image segment (ISc) of the current frame image (IMGb) is greater than or equal to the threshold value (TH), the first comparator (CP1) may assign the same identification number (e.g., ID#1) to the third sub-image segment (ISc) of the current frame image (IMGb) as the first sub-image segment (ISa) of the previous frame image (IMGa).

[0188] In contrast, if it is determined that there are no image segments in the current frame image (IMGb) whose IoU value is greater than or equal to the threshold value (TH) and the second sub-image segment (ISb) of the previous frame image (IMGa), the first comparator (CP1) may not assign the identification number (e.g., ID#2) of the second sub-image segment (ISb) of the previous frame image (IMGa) to the image segments included in the current frame image (IMGb).

[0189] Accordingly, if there is an image segment within the current frame image (IMGb) in which the IoU value between the previous frame image (IMGa) and the current frame image (IMGb) is greater than or equal to the threshold value (TH), an identification number may be assigned by the operation of the first comparator (CP1).

[0190] However, in the case of the first comparator (CP1), identification numbers may be assigned only to image segments whose IoU value is greater than or equal to the threshold value (TH) with any one of the image segments included in the previous frame image (IMGa), and identification numbers may not be assigned to other image segments (e.g., the third and fourth sub-image segments (ISd, ISe) included in the current frame image (IMGb)).

[0191] To compensate for the limitations of such a first comparator (CP1), the tracking module (224) may further include a second comparator (CP2).

[0192] Referring further to FIGS. 12a and 12b to describe the second comparator (CP2) more specifically, the second comparator (CP2) may include a second data buffer (DBF2) containing information on feature points (e.g., FTR#1, FTR#2, FTR#3 in FIG. 12a) for at least one image segment assigned each identification number (e.g., ID#1, ID#2, ID#3 in FIG. 12a). Here, the information on feature points stored in the second data buffer (DBF2) may include information on feature points for an image segment containing an object detected in previous frame images (e.g., input images (IMG)) and classified as a human object, that is, a first image segment (IS1) for which a first sub-label (SLB1) is assigned in previous frame images. For example, the information on feature points may include information on a feature vector extracted from the corresponding image segment, but is not limited thereto. For example, the second comparator (CP2) may be implemented as a ReID-based Tracker using a pre-learned Re-Identification (ReID) algorithm, but is not limited thereto.

[0193] As illustrated in FIG. 11b, the second comparator (CP2) extracts feature points for each of the image segments (e.g., third sub-image segment (ISc), fourth sub-image segment (ISd), fifth image segment (ISe)) included in the current frame image (IMGb), and determines similarity by comparing the extracted feature points with the feature points stored in the second data buffer (DBF2) (e.g., FTR#1, FTR#2, FTR#3 of FIG. 12a), and if the similarity is greater than or equal to a preset value, it can assign an identification number of the image segment whose feature point similarity is greater than or equal to the preset value to the corresponding image segment.

[0194] According to an embodiment, when the first comparator (CP1) and the second comparator (CP2) are driven in a sequential operation as described above, the second comparator (CP2) can perform a feature point comparison operation for image segments not identified by the first comparator (CP1), for example, the fourth and fifth sub-image segments (ISd, ISe) of the current frame image (IMGb).

[0195] More specifically, as illustrated in FIG. 11b, the second comparator (CP2) can extract a first feature point (FTR1) from the fourth sub-image segment (ISd) and determine similarity by comparing it with feature points stored in the second data buffer (DBF2) (e.g., FTR#1, FTR#2, FTR#3 of FIG. 12a). Here, if there is a feature point similar to the first feature point (FTR1) in the second data buffer (DBF2) (e.g., if there is a feature point in the second data buffer (DBF2) whose similarity to the first feature point (FTR1) is greater than or equal to a preset value), an identification number (ID#2) of the image segment corresponding to the feature point (e.g., FTR#2) can be assigned to the fourth sub-image segment (ISd).

[0196] In contrast, the second comparator (CP2) can extract a second feature point (FTR2) from the fifth sub-image segment (ISe) and determine similarity by comparing it with feature points stored in the second data buffer (DBF2) (e.g., FTR#1, FTR#2, FTR#3 of FIG. 12a). Here, if there are no feature points similar to the second feature point (FTR2) in the second data buffer (DBF2) (e.g., if there are no feature points in the second data buffer (DBF2) whose similarity to the second feature point (FTR2) is greater than or equal to a preset value), the second comparator (CP2) can determine that an object that was not included in previous frame images (e.g., IMGb) in the case of the fifth sub-image segment (ISe) has newly appeared in the current frame image (IMGa), and can assign a new identification number (e.g., ID#4) to the corresponding fifth sub-image segment (ISe).

[0197] Meanwhile, although the above description is based on the assumption that the first data buffer (DBF1) and the second data buffer (DBF2) are configured separately, the embodiments of the present disclosure are not limited thereto, and the first data buffer (DBF1) and the second data buffer (DBF2) can be implemented as a single data buffer.

[0199] <One Example of an Event Detection Module>

[0200] FIG. 13 is a diagram illustrating an example of the operation of the event detection module (225) of FIG. 6.

[0201] Referring to FIGS. 2 through 6, the event detection module (225) receives a fourth image segment (IS4) from the tracking module (224) and can detect an event occurring on the fourth image segment (IS4). For example, the event detection module (225) can detect an event such as a collapse occurring on the fourth image segment (IS4).

[0202] In one embodiment, the event detection module (225) may include a storage unit (STR) and an event detection unit (IVT).

[0203] Referring further to FIG. 13 to describe the event detection module (225) more specifically, the event detection module (225) can detect events such as collapse occurring in the image segments by comparing the width (e.g., a1, a2) and height (e.g., b1, b2) of the image segments (e.g., 6th to 8th sub-image segments (ISf, ISg, ISh)) of the current frame image (IMGd) with the width (e.g., a3, a4) and height (e.g., b3, b4) of the image segments (e.g., 9th to 11th sub-image segments (ISi, ISj, ISk)) of the previous frame image (IMGc).

[0204] In one embodiment, the event detection unit (IVT) of the event detection module (225) detects at least one image segment among the image segments included in the current frame image (IMGd) in which the horizontal width is greater than the vertical width, and can detect that a collapse event has occurred for the image segment among at least one image segment of the previous frame image (IMGc) corresponding to the at least one image segment in which the vertical width is greater than the horizontal width.

[0205] For example, the event detection unit (IVT) of the event detection module (225) can detect image segments among the image segments included in the current frame image (IMGd) in which the horizontal width is greater than the vertical width. For example, the event detection unit (IVT) can extract 7th and 8th sub-image segments (ISg, ISh) in the current frame image (IMGd) in which the horizontal width (a1, a2) is greater than the vertical width (b1, b2).

[0206] Additionally, the event detection unit (IVT) can compare the width (a3, a4) and height (b1, b2) of image segments of the previous frame image (IMGc) corresponding to the extracted 7th and 8th sub-image segments (ISg, ISh), for example, the 10th and 11th sub-image segments (ISj, ISk). According to an embodiment, the image segments of the previous frame image (IMGc) (e.g., 9th to 11th sub-image segments (ISi, ISj, ISk)) may be provided from the storage unit (STR).

[0207] In one embodiment, the event detection unit (IVT) detects a 11th sub-image segment (ISk) in which the vertical width (b4) of the 10th and 11th sub-image segments (ISj, ISk) of the previous frame image (IMGc) corresponding to the 7th and 8th sub-image segments (ISg, ISh) extracted in correspondence with the current frame image (IMGd) is greater than the horizontal width (a4), and detects that a collapse event has occurred for the corresponding image segment.

[0208] According to an embodiment, the event detection module (225) may provide this to the user in the form of an alarm when a fall event is detected, but is not limited thereto.

[0210] <Another example of the second classification module>

[0211] FIG. 14 is a block diagram showing another example of a second classification module (223_1) included in the detector (220) of FIG. 3.

[0212] FIG. 15 is a flowchart illustrating an object classification method according to various embodiments of the present disclosure.

[0213] FIG. 16 is a flowchart illustrating an object filtering method according to various embodiments of the present disclosure.

[0214] FIG. 17 is a diagram illustrating an example of the operation of the second classification module (223_1) of FIG. 14.

[0215] Referring to FIGS. 14 to 17, the second classification module (223_1) receives a first image segment (IS1) with a first label (LB1) assigned from the first classification module (222), and can extract (or calculate) class scores for each of a plurality of sub-labels (SLB1 to SLB5) for the first image segment (IS1) with the first label (LB1).

[0216] In one embodiment, the second classification module (223_1) can compare the class scores for sub-labels corresponding to the aforementioned object of interest (e.g., moving objects such as people, fire, smoke, etc.) among a plurality of sub-labels (SLB1 to SLB5) with the third reference value (REF3) of the corresponding sub-label. Additionally, the second classification module (223_1) can receive a first image segment (e.g., the first image segment (IS1_1) of FIG. 7) to which a first label (LB1) is assigned from the first classification module (222), and can extract (or calculate) class scores for each of the plurality of sub-labels (SLB1 to SLB5) for the first image segment to which the first label (LB1) is assigned. For example, the second classification module (223_1) can extract class scores for each of the multiple sub-labels (SLB1 to SLB5) included in the sub-label data (SLBD) stored for each of the first image segments assigned a first label (LB1) from the first classification module (222). The operation of the second classification module (223_1) comparing the class scores for the sub-labels corresponding to the object of interest with the third reference value (REF3) of the corresponding sub-label, and the operation of extracting class scores for each of the multiple sub-labels (SLB1 to SLB5) for the first image segment (IS1) assigned a first label (LB1), are substantially the same or similar to the operation of the second classification module (223_1) described with reference to FIGS. 2 to 13, so a detailed description is omitted.

[0217] In one embodiment, the second classification module (223_1) can filter (or transform the image) the first image segment (IS1_1) for which it is not possible to determine whether a person (or person object) among the objects of interest is wearing a safety helmet (or helmet) even when comparing the class scores for sub-labels corresponding to the objects of interest with the third reference value (REF3) of the corresponding sub-labels. For example, the second classification module (223_1) can extract (or generate) a classified image segment (DIS) or an unclassified image segment (UDIS) from the first image segment (IS1) using a fourth reference value (REF4). The operation of the second classification module (223_1) extracting (or generating) a classified image segment (DIS) or an unclassified image segment (UDIS) from the first image segment (IS1) will be explained in detail with reference to FIGS. 15 to 17.

[0218] In one embodiment, referring to FIG. 15, the second classification module (223_1) can perform a classification operation on the first image segment (e.g., the first image segment (IS1_1) of FIG. 17) through steps S1501 to S1515. Since steps S1501 to S1514 illustrated in FIG. 15 are substantially the same or similar to steps S1001 to S1014 illustrated in FIG. 10, a detailed description of steps S1501 to S1514 is omitted.

[0219] In one embodiment, the second classification module (223_1) can determine (step S1515) whether the first image segment (e.g., the first image segment (IS1_1) of FIG. 17) corresponds to a classified image segment (DIS) if, in step S1504, the class score of the first sub-label is greater than or equal to the third criterion value (REF3). For example, the second classification module (223_1) can convert the first image segment into a classified image segment (DIS) or an unclassified image segment (NDIS) depending on whether the first image segment satisfies a specified condition (i.e., extract (or create) a classified image segment (DIS) or an unclassified image segment (NDIS) from the first image segment).

[0220] Referring further to FIG. 16 to explain in more detail the operation of the second classification module (223_1) extracting (or generating) a classified image segment (DIS) or an unclassified image segment (NDIS) from the first image segment (IS1_1) between step S1504 and step S1515, the second classification module (223_1) may extract a first mean value and a first standard deviation value of the binary mask extracted from the first image segment (IS1_1) (step S1516) if the class score of the first sub-label in step S1504 is greater than or equal to the third reference value (REF3).

[0221] In one embodiment, the second classification module (223_1) can generate (or extract) a binary mask (e.g., the binary mask (BMSK) of FIG. 17) by converting the first image segment (IS1_1), which is determined (or determined) that the class score of the first sub-label is the third reference value (REF3), into a grayscale, applying a Gaussian blur to the first image segment (IS1_1) converted into a grayscale, and setting (or specifying) half the value of the maximum pixel value of the first image segment (IS1_1) to which the Gaussian blur is applied (or the maximum pixel value x 0.5) as a threshold value.

[0222] In one embodiment, the second classification module (223_1) may extract (or generate) a first average value, which is the average of the pixel values ​​included in each pixel of the binary mask, and a first standard deviation value, which is the standard deviation of the pixel values ​​included in each pixel of the binary mask. For example, referring further to FIG. 17 to explain step S1516 in more detail, if the class score of the first sub-label of the first image segment (IS1_1) is greater than or equal to the third reference value (REF3), a binary mask (BMSK) containing pixels identical to the first image segment (IS1_1) may be generated (or extracted) from the first image segment (IS1_1), and a first average value (MV1), which is the average of the pixel values ​​included in each pixel of the binary mask (BMSK), and a first standard deviation value (SDV1), which is the standard deviation of the pixel values ​​included in each pixel of the binary mask (BMSK), may be extracted (or generated).

[0223] In one embodiment, referring again to FIG. 16, when the second classification module (223_1) extracts (or generates) the first average value (MV1) and the first standard deviation value (SDV1) of the binary mask (BMSK) from the first image segment (IS1_1) in step S1516, it can determine (step S1517) whether the size of the first image segment (IS1_1) (i.e., the value obtained by multiplying the row size and the column size of the first image segment (IS1_1)) is smaller than the fourth reference value. For example, the second classification module (223_1) can perform the operation of step S1517, which involves extracting the first average value (MV1) and the first standard deviation value (SDV1) from the binary mask (BMSK) and determining whether the value obtained by multiplying the row and column sizes of the first image segment (IS1_1) is less than or equal to the fourth reference value in order to determine whether filtering (or image transformation) is required for the binary mask (BMSK).

[0224] In one embodiment, the second classification module (223_1) can determine (step S1518) whether the sum of the first average value (MV1) and the first standard deviation value (SDV1) of the binary mask (BMSK) is less than or equal to the fifth standard value when the size of the first image segment (IS1_1) in step S1517 is less than or equal to the fourth standard value. For example, when the size of the first image segment (IS1_1) is less than or equal to the fourth standard value, the second classification module (223_1) can determine whether the sum of the first average value (MV1) and the first standard deviation value (SDV1) of the binary mask (BMSK) extracted in step S1516 is less than or equal to the fifth standard value.

[0225] In one embodiment, the second classification module (223_1) can extract a classification image segment (DIS) from the first image segment (IS1_1) (step S1519) if the sum of the first average value (MV1) and the first standard deviation value (SDV1) of the binary mask (BMSK) is less than or equal to the fifth reference value. For example, the second classification module (223_1) can extract (or generate) a classification image segment (DIS) from the first image segment (IS1_1) if the size of the first image segment (IS1_1) is less than or equal to the fourth reference value and the sum of the first average value (MV1) and the first standard deviation value (SDV1) is less than or equal to the fifth reference value. Here, the classification image segment (DIS) may be an image segment containing a person who can determine whether a safety helmet (or helmet) is being worn within the region of interest.

[0226] In one embodiment, the second classification module (223_1) may extract an unclassified image segment (NDIS) from the first image segment (IS1_1) (step S1520) if, in step S1517, the size of the first image segment (IS1_1) exceeds a fourth reference value. Additionally, the second classification module (223_1) may extract an unclassified image segment (NDIS) from the first image segment (IS1_1) (step S1520) if, in step S1518, the sum of the first average value (MV1) and the first standard deviation value (SDV1) exceeds a fifth reference value. For example, the second classification module (223_1) may extract an unclassified image segment (NDIS) from the first image segment (IS1_1) (step S1520) if the size of the first image segment (IS1_1) exceeds a fourth reference value or if the sum of the first average value (MV1) and the first standard deviation value (SDV1) exceeds a fifth reference value. Here, the unclassified image segment (NDIS) may be an image segment containing a person in the region of interest for whom it is not possible to determine whether a safety helmet (or helmet) is being worn.

[0228] <Method according to embodiments of the present disclosure>

[0229] FIG. 18 is a flowchart illustrating an object detection method according to various embodiments of the present disclosure.

[0230] Referring to FIGS. 1 through 18, in one embodiment, the object detection method of FIG. 18 may be performed on the object detection framework (200) (or electronic device (100)) described with reference to FIGS. 1 through 17. For example, the object detection method of FIG. 18 may be substantially identical to the operation of the object detection framework (200) (or electronic device (100)) described with reference to FIGS. 1 through 17. Accordingly, redundant descriptions will not be repeated.

[0231] Referring to FIG. 18, the object detection method of FIG. 18 may include the step of detecting at least one object in an input image based on a first reference value and a second reference value, and generating at least one first image segment including the detected at least one object (S1801). In one embodiment, the step of generating at least one first image segment (S1801) is substantially identical or similar to the configuration in which a detection module (e.g., detection module (221)) described with reference to FIG. 1 to FIG. 17 generates a first image segment (e.g., first image segment (IS1)), so redundant descriptions will not be repeated.

[0232] Additionally, the object detection method of FIG. 18 may further include the step of comparing at least one first image segment with a previously stored second image segment to assign one of a first label and a second label to at least one first image segment (S1802). In one embodiment, the step of assigning one of a first label and a second label to at least one first image segment (S1802) is substantially the same or similar to the configuration in which a first classification module (e.g., first classification module (222)) described with reference to FIG. 1 to FIG. 17 assigns one of a first label (e.g., first label (LB1)) and a second label (e.g., second label (LB2)) to at least one first image segment (e.g., first image segment (IS1)), so redundant descriptions will not be repeated.

[0233] Additionally, the object detection method of FIG. 18 may include the step of extracting class scores for each of a plurality of sub-labels for a first image segment with a first label assigned among at least one first image segment (S1803). In one embodiment, the step of extracting class scores (S1803) is substantially identical or similar to the configuration in which the second classification module (e.g., second classification module (223, 223_1)) described with reference to FIG. 1 to 17 extracts class scores, so redundant descriptions will not be repeated.

[0235] <Computer program according to embodiments of the present disclosure>

[0236] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0237] Meanwhile, at least some of the configurations or methods described above (e.g., the method of FIG. 9, the method of FIG. 10, the method of FIG. 15, the method of FIG. 16, the method of FIG. 18) may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or a combination of two or more of the above. Where at least some of the configurations described above are implemented by software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks may be stored on a machine or computer-readable medium, such as a storage medium, thereby enabling a processor to perform the necessary tasks.

[0239] <Various effects according to embodiments of the present disclosure>

[0240] As described above, in the object detection framework, system, device, method, and computer program according to the embodiments of the present disclosure, the object detection framework (or, electronic device) can detect, classify (or filter) and track objects in an input image and detect events occurring in the input image by using at least one of each module (e.g., detection module, first classification module, second classification module, tracking module, and event detection module).

[0241] Here, the object detection framework (or electronic device) can detect, classify (or filter) and track objects within an input image and detect events occurring within the input image by sequential operation (or operation) of a detection module, a first classification module, a second classification module, a tracking module, and an event detection module.

[0242] As multiple modules operate sequentially in this manner, the accuracy of object detection, classification, tracking, and event detection operations of the object detection framework (or electronic device) can be improved through the operations of modules to which various conditions are applied. In particular, in the operation of detecting and classifying objects, the object detection framework (or electronic device) can detect and classify objects through the sequential operation (or operation) of multiple modules (e.g., detection module, first classification module, second classification module) driven using different reference values. Accordingly, the accuracy of object detection and classification can be improved.

[0243] In addition, in the object detection framework, system, device, method, and computer program according to the embodiments of the present disclosure, the object detection framework (or electronic device) can increase the utilization of data processing by utilizing an image segment (third image segment) filtered by the operation of a second classification module as a second image segment required for the operation of a first classification module.

[0244] In addition, by utilizing the third image segment as the second image segment, the number of data (second image segment) required for the operation of the first classification module increases, so the classification performance (performance of the first classification module) can be further improved.

[0245] However, the effects of the present disclosure are not limited to those exemplified above, and various other effects are included within the present disclosure and may be extended in various ways without departing from the spirit and scope of the present disclosure.

[0247] <Various embodiments of the present disclosure>

[0248] A system according to various embodiments of the present disclosure can be described as follows.

[0249] A system according to one embodiment of the present disclosure may include, in a system for detecting objects in an input image, a detection module that detects at least one object in the input image based on a first reference value and a second reference value and generates at least one first image segment including the detected at least one object; a first classification module that compares the at least one first image segment with a previously stored second image segment and assigns one of a first label and a second label to the at least one first image segment; and a second classification module that extracts class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment. The second classification module may, when there is a sub-label among the plurality of sub-labels in which the class score is greater than or equal to the third reference value of the corresponding sub-label, designate the sub-label in which the class score is greater than or equal to the third reference value for the first image segment to which the first label is designated, and when the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, provide the first image segment to which the first label is designated as the third image segment to the first classification module.

[0250] In one embodiment, the first classification module may store the third image segment provided from the second classification module as the second image segment.

[0251] In one embodiment, the first classification module may assign a first label to the at least one first image segment when the Intersection over Union (IoU) value between the at least one first image segment and the second image segment is less than a preset value, and assign a second label to the at least one first image segment when the IoU value between the at least one first image segment and the second image segment is greater than or equal to the preset value.

[0252] In one embodiment, the preset value may be 0.5.

[0253] In one embodiment, the plurality of sub-labels may include a first sub-label corresponding to a first object, a second sub-label corresponding to a second object different from the first object, a third sub-label corresponding to a third object different from the first and second objects, a fourth sub-label corresponding to a fourth object different from the first to third objects, and a fifth sub-label corresponding to a fifth object different from the first to fourth objects. The second classification module may extract class scores for each of the first to fifth sub-labels for a first image segment to which the first label is assigned among the at least one first image segment.

[0254] In one embodiment, the second classification module may provide the first image segment to which the first label is assigned as the third image segment to the first classification module when the class score of each of the first to third sub-labels is less than the third reference value of the corresponding sub-label.

[0255] In one embodiment, the second classification module can extract a binary mask corresponding to the first image segment with the first label assigned, and extract a first average value which is the average of the pixel values ​​included in each pixel of the binary mask and a first standard deviation value which is the standard deviation of the pixel values ​​included in each pixel of the binary mask.

[0256] In one embodiment, the second classification module can extract a classified image segment from the first image segment with the first label if the size of the first image segment is less than or equal to a fourth reference value and the sum of the first average value and the first standard deviation value is less than or equal to a fifth reference value.

[0257] An apparatus according to various embodiments of the present disclosure can be described as follows.

[0258] An apparatus according to one embodiment of the present disclosure may include, in an apparatus for detecting an object in an input image, a memory for storing at least one instruction, and a processor for performing at least one process for detecting an object in the input image in response to the instruction stored in the memory. The processor detects at least one object in the input image based on a first reference value and a second reference value, generates at least one first image segment including the detected at least one object, compares the at least one first image segment with a previously stored second image segment to assign one of a first label and a second label to the at least one first image segment, extracts class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment, and if there is a sub-label among the plurality of sub-labels whose class score is greater than or equal to a third reference value of the corresponding sub-label, assigns the sub-label whose class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned, and if the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, the first image segment to which the first label is assigned can be stored as the second image segment.

[0259] A computer program according to one embodiment of the present disclosure is a computer program stored on a computer-readable medium, and when executed by a computer system, the computer system may include instructions to perform a method for detecting an object in an input image, wherein the computer system performs a method comprising: detecting at least one object in the input image based on a first reference value and a second reference value, and generating at least one first image segment including the detected at least one object; comparing the at least one first image segment with a previously stored second image segment to assign one of a first label and a second label to the at least one first image segment; and extracting class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment. The step of extracting the class scores involves, if among the plurality of sub-labels there is a sub-label whose class score is greater than or equal to the third reference value of the corresponding sub-label, assigning the sub-label whose class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned, and if the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, the first image segment to which the first label is assigned can be stored as the second image segment.

[0260] Although the present disclosure has been described with reference to embodiments thereof, those skilled in the art will understand that various modifications and changes can be made to the present disclosure without departing from the spirit and scope of the disclosure as set forth in the claims. Explanation of the symbols

[0261] 100: Electronic device 110: Processor 120: Communication module 130: Input section 140: Display unit 150: Memory 151: Program 200: Object Detection Framework 210: Learning device 220: Detector 221: Detection Module 222: Classification Module 1 223: Classification Module 2 224: Trace Module 225: Event detection module

Claims

Claim 1 A system for detecting objects in an input image, wherein the system comprises: a detection module that detects at least one object in the input image based on a first reference value and a second reference value, and generates at least one first image segment including the detected at least one object; and a first classification module that compares the at least one first image segment with a previously stored second image segment and assigns one of a first label and a second label to the at least one first image segment. A system comprising: a second classification module for extracting class scores for each of a plurality of sub-labels for a first image segment with a first label assigned among at least one first image segment; wherein the second classification module assigns a sub-label with a class score greater than or equal to a third reference value for the first image segment with a first label assigned if there is a sub-label among the plurality of sub-labels whose class score is greater than or equal to a third reference value for the corresponding sub-label, and provides the first image segment with a first label assigned as a third image segment to the first classification module if the class score of each of the plurality of sub-labels is less than the third reference value for the corresponding sub-label; and wherein the second classification module extracts a binary mask corresponding to the first image segment with a first label assigned, and extracts a first average value which is the average of pixel values ​​included in each pixel of the binary mask and a first standard deviation value which is the standard deviation of pixel values ​​included in each pixel of the binary mask. Claim 2 A system according to claim 1, wherein the first classification module stores the third image segment provided from the second classification module as the second image segment. Claim 3 A system according to claim 1, wherein the first classification module assigns the first label to the at least one first image segment when the Intersection over Union (IoU) value between the at least one first image segment and the second image segment is less than a preset value, and assigns the second label to the at least one first image segment when the IoU value between the at least one first image segment and the second image segment is greater than or equal to the preset value. Claim 4 In claim 3, the system, wherein the preset value is 0.

5. Claim 5 In claim 1, the plurality of sub-labels includes a first sub-label corresponding to a first object, a second sub-label corresponding to a second object different from the first object, a third sub-label corresponding to a third object different from the first and second objects, a fourth sub-label corresponding to a fourth object different from the first to third objects, and a fifth sub-label corresponding to a fifth object different from the first to fourth objects, and the second classification module extracts class scores for each of the first to fifth sub-labels for a first image segment to which the first label is assigned among the at least one first image segment. Claim 6 A system according to claim 5, wherein the second classification module provides the first image segment to which the first label is assigned as the third image segment to the first classification module when the class score of each of the first to third sub-labels is less than the third reference value of the corresponding sub-label. Claim 7 delete Claim 8 A system according to claim 1, wherein the second classification module extracts a classified image segment from the first image segment with the first label assigned when the size of the first image segment is less than or equal to a fourth reference value and the sum of the first average value and the first standard deviation value is less than or equal to a fifth reference value. Claim 9 A device for detecting objects in an input image, a memory for storing at least one instruction; The apparatus includes a processor that performs at least one process for detecting an object in the input image in response to the instruction stored in the memory, wherein the processor detects at least one object in the input image based on a first reference value and a second reference value, generates at least one first image segment including the detected at least one object, compares the at least one first image segment with a previously stored second image segment to assign one of a first label and a second label to the at least one first image segment, extracts class scores for each of a plurality of sub-labels for the first image segment to which the first label is assigned among the at least one first image segment, and if there is a sub-label among the plurality of sub-labels whose class score is greater than or equal to a third reference value of the corresponding sub-label, assigns the sub-label whose class score is greater than or equal to the third reference value to the first image segment to which the first label is assigned, and if the class score of each of the plurality of sub-labels is less than the third reference value of the corresponding sub-label, the first image segment to which the first label is assigned is the second An apparatus that stores as an image segment, extracts a binary mask corresponding to the first image segment with the first label assigned, and extracts a first average value which is the average of pixel values ​​included in each pixel of the binary mask and a first standard deviation value which is the standard deviation of pixel values ​​included in each pixel of the binary mask. Claim 10 A computer program stored on a computer-readable medium, wherein when executed by a computer system, the computer system, in a method for detecting an object in an input image, comprises the steps of: detecting at least one object in the input image based on a first reference value and a second reference value, and generating at least one first image segment including the detected at least one object; comparing the at least one first image segment with a previously stored second image segment and assigning one of a first label and a second label to the at least one first image segment; A computer program comprising instructions for performing a method including the step of extracting class scores for each of a plurality of sub-labels for a first image segment with a first label assigned among at least one first image segment, wherein the step of extracting class scores comprises: assigning a sub-label with a class score greater than or equal to a third reference value for the first image segment with a first label assigned if there is a sub-label among the plurality of sub-labels for which the class score is greater than or equal to a third reference value for the corresponding sub-label; storing the first image segment with a first label assigned as the second image segment if the class score of each of the plurality of sub-labels is less than the third reference value for the corresponding sub-label; and extracting a binary mask corresponding to the first image segment with a first label assigned if the class score of the first sub-label among the plurality of sub-labels is greater than or equal to the third reference value, and extracting a first average value which is the average of pixel values ​​included in each pixel of the binary mask and a first standard deviation value which is the standard deviation of pixel values ​​included in each pixel of the binary mask.

Images