Device and Method for adding target class thereon

Abstract

As a preferred embodiment of the present invention, the terminal comprises: an MSE buffer that stores a portion of a received image; a first interface for selecting a target class in an image displayed on a display; and a second interface for selecting at least one specific point in time in an image displayed on the display; and the display is characterized in that, after the target class is selected, the target class can be estimated and displayed on a captured image received in real time from at least one image capturing device.

Description

Technology Field

[0001] The present invention relates to a method for analyzing images by applying deep learning to a network surveillance camera, and more specifically, to a method for performing transfer learning to add a new class to a network surveillance camera by a user. Background Technology

[0002] Conventional network surveillance cameras are implemented as embedded systems; after training using a pre-existing dataset, the learned model parameters and labels are fixed, and only the task of object inferencing is performed within the embedded system.

[0003] However, deep learning using existing methods that classify and analyze each object in video acquired from network surveillance cameras into fixed classes classified through prior training has limitations in producing limited results. Prior art literature

[0004] KR 10-2019-0062030 A The problem to be solved

[0005] In a preferred embodiment of the present invention, a user monitors video received from a network surveillance camera, selects a new target class through the video, and performs deep learning inference by reflecting the update of the target class added by the user through transfer learning on the network surveillance camera. means of solving the problem

[0006] As a preferred embodiment of the present invention, the terminal comprises: a receiving unit that receives a video captured by at least one video capturing device in real time; a Media Source Extensions (MSE) buffer that stores a portion of the received video; a display that displays the received video; a first interface that selects a target class from the video displayed on the display; and a second interface that selects at least one specific point in time from the video displayed on the display; wherein the display is characterized in that, after the target class is selected, the target class can be estimated and displayed on the captured video received in real time from the at least one video capturing device.

[0007] As a preferred embodiment of the present invention, the deep learning unit learns the target class based on frames within a predetermined time period relative to the selected specific time point, and transmits the learned target class and related deep learning variables to the at least one image capturing device through the transmission unit. In this case, the MSE buffer is characterized by decoding a portion of the video received during the live monitoring process, rendering it, displaying it on the display, and storing (holding) it within the MSE buffer for a predetermined time period. The deep learning unit is characterized by performing transfer learning on the target class using a data augmentation method by utilizing all frames within a predetermined time period relative to the selected specific time point.

[0008] As another embodiment of the present invention, the terminal comprises: a display; a memory for storing one or more instructions; a buffer for storing a portion of a received live video; and a processor for executing the one or more instructions stored in the memory; a first interface for selecting a target class in the live video; and a second interface for selecting at least one specific point in time in the live video; wherein the processor controls the display to output a live video received in real time by executing the one or more instructions, and further controls the display to estimate and display the target class in the live video received in real time after the target class is selected, and the memory stores a learning result of learning the target class from frames before and after the at least one specific point in time through deep learning.

[0009] As another preferred embodiment of the present invention, a method for adding and displaying a target class at a terminal comprises: receiving a live video captured by at least one video capturing device; storing a portion of the live video in a Media Source Extensions (MSE) buffer; displaying the received live video on a display; specifying a target class in the live video displayed on the display; selecting at least one specific point in time in the live video displayed on the display using an instant playback interface; and estimating and displaying the target class in the live video received from the at least one video capturing device after the target class has been selected; and is characterized by performing transfer learning on the target class using a plurality of frames arranged in a time series within a predetermined time period based on the selected at least one specific point in time. Effects of the invention

[0010] In a preferred embodiment of the present invention, image analysis is not performed only on a fixed class, but image analysis of a target desired by the user can also be performed on a class additionally selected by the user through real-time transfer learning. Brief explanation of the drawing

[0011] FIG. 1 illustrates a video surveillance system that performs real-time transfer learning in a CCTV as a preferred embodiment of the present invention. FIG. 2 illustrates an internal configuration diagram of a terminal as a preferred embodiment of the present invention. FIG. 3 illustrates an example of storing a portion of an image in an MSE buffer using an instant playback interface as a preferred embodiment of the present invention. FIG. 4 illustrates an example of performing real-time transfer learning as a preferred embodiment of the present invention. FIG. 5 illustrates a flowchart showing the addition and display of a target class in a terminal within a video surveillance system as a preferred embodiment of the present invention. Specific details for implementing the invention

[0012] Embodiments of the present disclosure are described below in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Additionally, parts unrelated to the description have been omitted from the drawings to clearly explain the present disclosure.

[0013] The terms “above” and similar designations used in this specification, particularly in the claims, may indicate both singular and plural forms. Furthermore, unless there is a description explicitly specifying the order of the steps describing the method according to this disclosure, the described steps may be performed in a suitable order. This disclosure is not limited by the order in which the described steps are described.

[0014] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0015] The present disclosure will be described in detail below with reference to the attached drawings.

[0016] FIG. 1 illustrates a video surveillance system that performs real-time transfer learning in a CCTV as a preferred embodiment of the present invention.

[0017] The video surveillance system (100) includes a user terminal (120) and a video recording device (110). The video recording device (110) is capable of wired or wireless communication with the terminal (120), and examples of the video recording device (110) include a CCTV, a camera, a terminal capable of recording video, etc.

[0018] The terminal (120) can be implemented as various electronic devices such as a mobile phone, tablet PC, digital camera, camcorder, laptop, computer, tablet PC, desktop, e-book reader, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, wearable device, etc. Additionally, the terminal (120) also includes all physical devices of the form including the components of FIG. 2.

[0019] As a preferred embodiment of the present invention, the image capturing device (110) includes a second deep learning unit (112), and the second deep learning unit (112) is based on the premise that deep learning is performed using a pre-trained data set. The second deep learning unit (112) can analyze a pre-trained class among the images captured by the image capturing device (110) and provide the results in the form of metadata.

[0020] In a preferred embodiment of the present invention, the second deep learning unit (112) can also perform transfer learning for a new target class. The process of performing transfer learning is as follows.

[0021] While monitoring video transmitted from the CCTV (110), the user selects a video at time t containing a new target class using the first interface (S121). An example of a target class is a man wearing a hat.

[0022] Afterwards, the user implements to collect all images from time t+x seconds to ty containing the selected target class using the second interface (S122). In a preferred embodiment of the present invention, when adding a new target class in live video, the same target class can be learned in a data augmentation format by utilizing the feature that the same target class exists before and after a specific time t.

[0023] The terminal (120) performs data augmentation (S123) using all of the series of time-series images collected through the second interface (S122) via the first deep learning unit (122), and performs transfer learning and updates (S124) for a new target class. The terminal (120) transmits the updated deep learning parameters and new target class information to the CCTV (110).

[0024] CCTV (110) performs object inference (deep learning inferencing) on ​​the image after updating the received deep learning variables and new target class information.

[0025] Referring to FIG. 2, the terminal (200) includes a processor (210), memory (220), buffer (230), and display (240), and the display (240) supports a first interface (250) and a second interface (260).

[0026] The memory (220) may be in the form of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM, ROM, SRAM, EEPROM, PROM, magnetic memory, a magnetic disk, an optical disk, etc., and may store a program for processing and controlling the processor (210). The processor (210) of the terminal (200)

[0027] The memory (220) can store one or more instructions, and the processor (210) can control the display (240) to output live video received in real time by executing one or more instructions. The memory (220) can store a learning result of learning a target class from frames before and after at least one specific point in time through deep learning.

[0028] The processor (210) can control the live video received in real time after the target class is selected by the user, as in one embodiment of FIG. 3 to 4, so as to estimate and display the selected target class.

[0029] The buffer (230) can store a portion of the received live video. Referring to FIG. 3, the buffer (230) can use an MSE buffer (330) and can store a portion (320) of the live video. Additionally, the buffer (230) can store multiple frames (331~334) within a certain time interval based on a specific point in time selected by the user through the second interface (260) in chronological order.

[0030] The first interface (250) is implemented to select a target class from a live video displayed on a display (240). The second interface (260) is implemented to select at least one specific point in time from a live video displayed on a display (240). In a preferred embodiment of the present invention, the second interface (260) may be an instant playback interface.

[0031] FIGS. 3 and 4 illustrate an example of storing an image using an MSE buffer while monitoring a live image at a terminal (310a, 310b) as a preferred embodiment of the present invention.

[0032] In a preferred embodiment of the present invention, the live video is decoded in the decoder (332), rendered through the rendering unit (334), and displayed on the display (310b) of the user terminal. The MSE buffer (330) used by the user terminal (310b) does not immediately remove the video shown to the user, but holds some frames (331~334) for a certain period of time.

[0033] In a preferred embodiment of the present invention, when a user selects a specific object as a new target class on a terminal (310a), the user plays back time-series video corresponding to t-1 second, t second, t+1 second, and t+2 second using an interface such as instant playback and uses it as training data for the deep learning unit (350). The data augmentation unit (340) increases the training data through methods such as scaling, translation, rotation, flipping, and adding noise, and then provides the collected data to the deep learning unit (350).

[0034] FIG. 5 illustrates a method of adding and displaying a target class at a terminal as a preferred embodiment of the present invention.

[0035] In a video surveillance system including a CCTV and a user terminal, the user terminal receives live video captured by at least one CCTV (S510). The terminal decodes and renders the received live video and displays it on a display, and retains the displayed video in an MSE buffer without immediately removing it (S520, S530). It should be noted that steps S520 and S530 may be performed sequentially, step S520 may be performed after step S530, or steps S520 and S530 may be performed simultaneously.

[0036] The user monitors at least one live video displayed on the terminal's display and, if there is a new target class to be added, can select the target class using a finger, a pen, or various other input interfaces (S540). Afterwards, the user selects at least one specific time point t in the live video displayed on the display using an instant playback interface (S550).

[0037] The first deep learning unit within the terminal collects a series of time-series images held in the MSE buffer, updates deep learning variables and classes of a new target class through transfer learning via data augmentation, and transmits information about this to at least one CCTV.

[0038] At least one CCTV performs deep learning inference based on the deep learning variables and class of the received new target class. At the terminal, after the new target class is selected, the selected target class is estimated and displayed in the live video received from the at least one video recording device (S560).

[0039] The device according to the present invention may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, and user interface devices such as a touch panel, a key, a button, etc. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable code or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs: Digital Versatile Discs). The computer-readable recording medium may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium is readable by a computer, stored in the memory, and can be executed on the processor.

[0040] In the specification of the present invention (particularly in the claims), the use of the term “the above” and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in the present invention, it is to include the invention to which individual values ​​belonging to said range are applied (unless otherwise stated), and is equivalent to describing each individual value constituting said range in the detailed description of the invention. Finally, regarding the steps constituting the method according to the present invention, unless explicitly stated or otherwise stated, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0041] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

Claim 1 A terminal for displaying an added target class, comprising: a receiver that receives video captured in real time from a video recording device including at least one CCTV or network surveillance camera; an MSE (Media Source Extensions) buffer that stores a portion of the received video; a display that displays the received video in real time; a first interface for selecting a new target class from the video displayed on the display; and a second interface for selecting a specific point in time containing the target class from the real-time video displayed on the display. A terminal comprising, wherein the image capturing device includes a second deep learning unit capable of analyzing a plurality of pre-trained classes using a pre-trained dataset, and the second deep learning unit is configured to provide analysis results for the pre-trained classes in the form of metadata, and the terminal collects a plurality of frames within a predetermined time based on a specific point in time selected through the second interface via the first deep learning unit to perform data augmentation, performs transfer learning for the new target class to update deep learning variables, transmits the updated deep learning variables and the new target class information to the image capturing device, and the image capturing device performs object inference on an image captured in real time after the target class is selected by reflecting the transmitted deep learning variables and target class information. Claim 2 delete Claim 3 A terminal according to claim 1, wherein the display supports live monitoring by displaying video received in real time from the video capturing device, and selects the new target class during the live monitoring process using the first interface. Claim 4 A terminal according to claim 3, wherein the MSE buffer decodes a portion of the video received during the live monitoring process, renders it, displays it on the display, and holds it within the MSE buffer for a certain period of time. Claim 5 A terminal according to claim 1, characterized in that when a specific point in time is selected through the second interface, a plurality of frames within a predetermined time based on the selected specific point in time are selected from among the frames stored in the MSE buffer and provided as learning data for transfer learning. Claim 6 A terminal according to claim 5, characterized by generating training data of a data augmentation method based on time-series features containing the same target class by utilizing all multiple frames within a preset time period based on the selected specific point in time, and performing transfer learning on the target class using the same. Claim 7 A terminal according to claim 1, characterized by performing transfer learning on the target class using a plurality of frames arranged in a time series within a predetermined time period based on the selected specific point in time. Claim 8 A terminal according to claim 3, wherein the second interface is an instant playback interface that allows immediate playback and selection of a specific point in time for the video being live monitored. Claim 9 A terminal that adds and displays a target class, comprising: a receiver unit that performs a receiving function configured to receive video captured in real time from a video recording device including at least one CCTV or network surveillance camera; an MSE buffer that stores a portion of the live video received in real time; a display that displays the received live video in real time; and a memory that stores one or more instructions. A processor for executing one or more instructions stored in the memory; a first interface for selecting a new target class in the live video displayed on the display; and a second interface for selecting a specific point in time containing the new target class in the live video displayed on the display; wherein the processor controls the display to output the live video received in real time from the video capturing device by executing the one or more instructions, collects multiple frames within a preset time based on the specific point in time selected through the second interface from the MSE buffer and sorts them in chronological order, updates deep learning parameters by performing transfer learning for the new target class using the chronologically sorted multiple frames, stores the updated deep learning parameters and the new target class information in the memory, transmits the stored deep learning parameters and the new target class information to at least one video capturing device, and the video capturing device reflects the transmitted deep learning parameters and target class information through a deep learning unit including a pre-trained deep learning model so that the new target class is selected A terminal characterized by controlling the performance of object inference on video captured in real time thereafter. Claim 10 A terminal according to claim 9, wherein the processor constructs training data for a single object using multiple frames before and after a specific point in time selected through the second interface, and learns the new target class using the training data. Claim 11 A method for adding and displaying a target class at a terminal, comprising: receiving live video captured in real time by a video recording device including at least one CCTV or network surveillance camera; storing a portion of the live video received in real time in an MSE (Media Source Extensions) buffer; providing live monitoring by displaying the received live video in real time on a display; selecting a new target class from the live video displayed on the display during the live monitoring; selecting a specific point in time containing the new target class using an instant playback interface; collecting multiple frames within a preset time period from the MSE buffer based on the selected specific point in time and arranging them in chronological order; updating deep learning parameters by performing deep learning-based transfer learning for the new target class using the chronologically arranged multiple frames; and transmitting the updated deep learning parameters and the new target class information to the video recording device. A method characterized by including the step of: the image capturing device reflecting the updated deep learning parameters and the new target class information through a deep learning unit including a pre-trained deep learning model, and performing object inference on an image captured in real time after the target class is selected.

Images