Method, apparatus, and system for screen leackage prevention based on normal status detection

Abstract

The present invention relates to a method for preventing screen leakage of an information processing device such as a computer, an apparatus thereof, and a system corresponding to such a method or apparatus. A method for preventing information leakage from a screen connected to an information processing device and outputting secret content according to one embodiment of the present invention may include the steps of: acquiring a surveillance image from a camera device connected to the information processing device and configured to photograph a surveillance space including at least a portion of a display space in which the screen can be viewed; analyzing the surveillance image to determine whether the interior of the surveillance space is in a normal state; and, as a result of the determination, if the interior of the surveillance space is not in a normal state, executing interference control for the purpose of preventing leakage of the secret content through the screen.

Description

Technology Field

[0001] The present invention relates to a method for preventing screen leakage of an information processing device such as a computer, an apparatus such as such, and a system corresponding to such a method or apparatus. Background Technology

[0002] The present invention has as background technology a technology for preventing screen leakage using a camera connected to an information processing device such as a computer.

[0003] When information processing devices, such as computers, are utilized for security purposes, the content displayed on the screen may include secret content that must be treated as secret. Such secret content should only be permitted to be displayed on the screen by authorized handlers (e.g., designated viewers or operators), and the displayed information must not be leaked unlawfully.

[0004] Examples of known methods of unauthorized leakage include screen capture by software methods, screen signal theft by hardware methods, and the act of capturing a screen using separate recording equipment. Prior art also exists to address each of the aforementioned methods of unauthorized leakage. Software technology has been disclosed prior to prevent screen capture by blocking the capture operation or by inserting a watermark on the captured screen that has a function to indicate or track leakage. Additionally, hardware technology has been disclosed prior to encrypt a display signal so that the display signal from a designated output device to a designated screen cannot be intercepted midway. However, methods based on the device itself cannot prevent the act of leaking a screen by capturing it using a separate recording device without any contact with the device.

[0005] According to the prior art, a technology is disclosed in which a surveillance camera is installed in the direction of the display of an information processing device (such as a computer for security purposes, and / or a screen connected to such a device) on which secret content is displayed, in order to prevent leakage through filming, and security actions, such as interfering with the display of the screen, are taken when an act such as filming is detected by said surveillance camera. In particular, among the prior art, a technology is disclosed in which an attempt at leakage is determined when the appearance of the filming act or the shape of the filming means is captured, and among these, there are those that utilize advanced image processing methods such as artificial intelligence.

[0006] For example, according to Registered Patent Publication No. 2336165, prior art is disclosed in which a surveillance camera installed adjacent to the screen of the security device can be used to photograph the front of the screen, and the shape of the camera, lens, flash, and flash flash attempting illegal leakage are identified from the captured image through artificial intelligence image processing based on a deep learning model, and the occurrence of an illegal leakage attempt is confirmed as a result of the identification, thereby causing the security device to take countermeasures such as stopping the display of the screen. In the invention of the above Registered Patent Publication, a configuration is also disclosed in which a surveillance microphone is attached to detect the shutter sound generated by the camera during shooting.

[0007] As another example, according to Registered Patent Publication No. 1925799, prior art is disclosed in which a surveillance camera installed adjacent to the screen of the security device can be used to photograph the front of the screen, and whether the shapes of various image acquisition tools are detected can be learned in advance through machine learning, and if the shape of the image acquisition tool learned in advance is identified in the photographed screen, the occurrence of an attempt at illegal leakage is confirmed, and the security device is instructed to take countermeasures such as stopping the display of the screen. The problem to be solved

[0008] The present invention aims to overcome the fundamental limitations of conventional camera-based screen leakage prevention technology to enable response to a wider variety of threats.

[0009] Among the aforementioned prior art, the technology of Registered Patent Publication No. 2336165 and the technology of Registered Patent Publication No. 1925799 are both configured to determine whether such means and / or acts occur and to respond accordingly using methods such as video analysis or voice analysis, based on the premise that content corresponding to a predetermined shooting means or shooting act is captured by a surveillance device such as a surveillance camera or microphone.

[0010] However, as described above, since it is known that certain tools or actions are subject to surveillance, a person attempting illegal leakage may attempt to capture the screen in a manner that is not recognized by such surveillance devices. For example, screen leakage may occur through various previously unknown anomalous methods, such as utilizing a previously unknown means of recording, concealing the means of recording itself, its operation, or its sound, or having a third party view the content on the screen instead of attempting to record.

[0011] Therefore, in order to further enhance the screen leakage prevention performance of security devices handling secret content, a method capable of detecting previously unknown leakage attempts is required, and the provision of such a method constitutes the technical problem to be solved by the present invention. means of solving the problem

[0012] A method for preventing information leakage from a screen that outputs secret content connected to an information processing device according to an embodiment of the present invention for solving the aforementioned technical problem may include: a step of acquiring a surveillance image from a camera device connected to the information processing device and configured to photograph a surveillance space including at least a portion of a display space in which the screen can be viewed; a step of analyzing the surveillance image to determine whether the interior of the surveillance space is in a normal state; and a step of executing interference control for the purpose of preventing the leakage of the secret content through the screen if, as a result of the determination, the interior of the surveillance space is not in a normal state.

[0013] The analysis of the above surveillance video is performed by a device including artificial intelligence, and the method may further include the step of training the artificial intelligence based on normal state learning data, and the step of the artificial intelligence analyzing the surveillance video and determining whether the interior of the surveillance space is in a normal state.

[0014] The above normal state learning material may include normal state video information for learning, and the normal state video information may be characterized in that it does not include video of persons, actions, and devices related to the leakage of secret content.

[0015] The above normal state learning data may be characterized by including normal state video information of the above surveillance space, wherein the normal state video information is captured within the above surveillance space and does not include video of persons, actions, and devices related to the leakage of the above secret content.

[0016] The above method may further include the step of receiving a message indicating that the monitoring space is determined to be in a normal state, and the step of acquiring new normal state image information based on the monitoring image upon receiving the message, and updating the normal state learning data to include the new normal state image information.

[0017] The above method may further include the step of generating the message when the artificial intelligence analyzes the surveillance video and determines that the interior of the surveillance space is in a normal state.

[0018] The step of receiving the above message may be a step of receiving a command that determines that the interior of the monitoring space is in a normal state through at least one of the input means and communication means connected to the device including the artificial intelligence.

[0019] The above normal state learning data includes person learning information related to at least one person, and the person learning information includes feature values ​​generated based on image information representing the person, and the above normal state learning data may include normal state image information that has been de-identified by applying at least one method among blur, replace, masking, and remove to the image information representing the person.

[0020] The above method may further include the step of the artificial intelligence determining whether a novelty is detected in the surveillance video, and the step of the artificial intelligence determining the analysis result of the surveillance video in a one-class classification method indicating a normal state according to whether the novelty is detected.

[0022] The above method may further include a step of stopping the interference control if, as a result of the above judgment, the interior of the monitoring space is normal and the interference control is being executed.

[0023] An information processing device having a function to prevent leakage of secret content according to an embodiment of the present invention for solving the aforementioned technical problem may include: a processor having a computational function; a memory connected to the processor; a screen connected to enable the display of secret content; a camera configured to photograph a surveillance space including at least a portion of a display space where the screen can be viewed; a normal state determination unit that acquires a surveillance image from the camera and analyzes the surveillance image to determine whether the interior of the surveillance space is in a normal state; and a content control unit that performs an interference control operation for the purpose of preventing leakage of the secret content through the screen when, as a result of the determination by the normal state determination unit, the interior of the surveillance space is not in a normal state.

[0024] The above normal state determination unit includes an artificial intelligence unit that analyzes the surveillance video and determines whether the interior of the surveillance space is in a normal state, and the device may further include an artificial intelligence learning unit that trains the artificial intelligence based on normal state learning data.

[0025] The above normal state learning material may include normal state video information for learning, and the normal state video information may be characterized in that it does not include video of persons, actions, and devices related to the leakage of secret content.

[0026] The above normal state learning data may be characterized by including normal state video information of the above surveillance space, wherein the normal state video information is captured within the above surveillance space and does not include video of persons, actions, and devices related to the leakage of the above secret content.

[0027] The artificial intelligence learning unit may be configured to acquire new normal state image information based on the surveillance image and update the normal state learning data to include the new normal state image information when a message is input indicating that the surveillance space is determined to be in a normal state.

[0028] The artificial intelligence unit may be configured to generate the message and provide it to the artificial intelligence learning unit when it analyzes the surveillance video and determines that the interior of the surveillance space is in a normal state.

[0029] The above device further includes an input unit that receives a command determining that the interior of the monitoring space is in a normal state through at least one of an input means and a communication means connected to the information processing device, and the message can be input through the input unit.

[0030] The artificial intelligence unit may be configured to perform an operation of determining whether a novelty is detected in the surveillance video, and an operation of determining the analysis result of the surveillance video in a one-class classification method that indicates a normal state depending on whether the novelty is detected.

[0031] The above content control unit may be configured to stop the interference control if, as a result of the judgment by the above normal state judgment unit, the interior of the monitoring space is normal and the interference control is being executed.

[0032] A system for preventing leakage of secret content according to an embodiment of the present invention for solving the aforementioned technical problem comprises: an information processing device that handles secret content and executes an operation to prevent leakage of said secret content; a screen device connected to said information processing device and configured to enable display of said secret content; and a camera device connected to said information processing device and configured to photograph a surveillance space including at least a portion of a display space in which said screen device can be viewed. The information processing device may be configured to acquire a surveillance image from said camera device, analyze said surveillance image to determine whether the interior of said surveillance space is in a normal state, and if, as a result of said determination, the interior of said surveillance space is not in a normal state, to perform an interference control operation for the purpose of preventing leakage of said secret content through said screen device. Effects of the invention

[0033] A method, device, and system for preventing information leakage from a screen connected to an information processing device that outputs secret content may be provided. In particular, the present invention may provide a method, device, and system capable of responding to a wider variety of threats by overcoming the fundamental limitations of conventional camera-based screen leakage prevention technology and being configured to analyze and determine a normal state and perform interference control in all other abnormal situations. Brief explanation of the drawing

[0034] FIG. 1 is a conceptual diagram of the act of leaking secret content to be prevented by the present invention, FIG. 2 is a conceptual diagram showing the operating principle of a method for preventing leakage of secret content according to the present invention. FIG. 3 is a conceptual diagram of an interference control execution state according to an embodiment of the present invention, FIG. 4 is an execution flowchart of a leakage prevention method according to an embodiment of the present invention, FIG. 5 is a simplified conceptual diagram of the abnormal detection method utilized by the present invention. FIG. 6 is a conceptual diagram illustrating learning based on normal state image information for learning according to an embodiment of the present invention, FIG. 7 is a conceptual diagram illustrating learning by acquiring normal state image information for learning according to another embodiment of the present invention, FIG. 8 is an exemplary diagram of a process for determining a normal state monitoring image by an external command in one embodiment of the present invention. FIG. 9 is a block diagram relating to the configuration of a device for executing a method for preventing leakage of secret content according to an embodiment of the present invention. FIGS. 10a and 10b are exemplary diagrams of a disturbance control method according to some embodiments of the present invention, FIG. 11 is a conceptual diagram of a system for preventing leakage of secret content composed of a plurality of devices according to an embodiment of the present invention, and FIG. 12 is a conceptual diagram of a system for preventing leakage of secret content composed of a plurality of devices according to another embodiment of the present invention. Specific details for implementing the invention

[0035] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0036] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related described items or any one of the multiple related described items, and is non-exclusive unless otherwise indicated. When items are listed in this application, they are merely illustrative descriptions intended to facilitate the explanation of the spirit of the present invention and possible methods of implementation, and are therefore not intended to limit the scope of the embodiments of the present invention.

[0037] In this specification, "A or B" may mean "only A," "only B," or "both A and B." Alternatively, in this specification, "A or B" may be interpreted as "A and / or B." For example, in this specification, "A, B or C" may mean "only A," "only B," "only C," or "any combination of A, B and C."

[0038] A slash ( / ) or a comma used in this specification may mean "and / or." For example, "A / B" may mean "A and / or B." Accordingly, "A / B" may mean "only A," "only B," or "both A and B." For example, "A, B, C" may mean "A, B or C."

[0039] In this specification, "at least one of A and B" may mean "only A," "only B," or "both A and B." Additionally, in this specification, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as synonymous with "at least one of A and B."

[0040] Additionally, in this specification, "at least one of A, B and C" may mean "only A," "only B," "only C," or "any combination of A, B and C." Also, "at least one of A, B or C" or "at least one of A, B and / or C" may mean "at least one of A, B and C."

[0041] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0042] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical or scientific terms, are used with the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0044] In describing the invention in this application, embodiments may be described or illustrated in terms of the described functions or unit blocks that perform the functions. The blocks may be expressed in this application as one or more devices, units, modules, parts, etc. The blocks may be implemented in hardware by a method of implementing one or more logic gates, integrated circuits, processors, controllers, memory, electronic components, or information processing hardware, which are not limited thereto. Alternatively, the blocks may be implemented in software by a method of implementing application software, operating system software, firmware, or information processing software, which are not limited thereto. A single block may be implemented by being separated into multiple blocks that perform the same function, or conversely, a single block may be implemented to simultaneously perform the functions of multiple blocks. The blocks may also be implemented by being physically separated or combined according to any criteria. The blocks may be implemented to operate in an environment where their physical locations are not specified and they are spaced apart from each other by a communication network, the Internet, a cloud service, a distributed processing network, a blockchain network, or a communication method not limited thereto. Since all of the above-mentioned methods of implementation fall within the scope of various embodiments that a person skilled in the art familiar with the field of information and communication technology can adopt to realize the same technical concept, any detailed methods of implementation should be interpreted as being included within the scope of the technical concept of the invention in this application.

[0045] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In describing the present invention, to facilitate overall understanding, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted. Furthermore, it is assumed that multiple embodiments are not mutually exclusive and that some embodiments may be combined with one or more other embodiments to form new embodiments.

[0047] Basic principles of the present invention

[0048] FIG. 1 is a conceptual diagram of the act of leaking secret content to be prevented by the present invention. For example, an information processing device (110) may be connected (121) to a screen (120). Also, for example, the information processing device (110) may be configured to handle secret content (115), and the secret content (115) may be configured to be displayed on the screen (120) by the information processing device (110). FIG. 1 assumes the existence of an authorized worker (140) capable of handling secret content through the information processing device (110). As shown in FIG. 1 (a), the act of the authorized worker (140) viewing or handling the secret content by operating the information processing device (110) in an authorized manner may not typically be a problem.

[0049] However, as shown in FIG. 1 (b), the authorized worker (140) may decide to leak the secret content (115) outside the scope of viewing and handling authorized to him. In this case, the authorized worker (140) may use a leaking means (151) to obtain (150) the contents of the secret content (115) displayed on the screen (120). The obtaining means (151) may include, for example, a camera, a smartphone, or other portable electronic device having a shooting function. In another case, as shown in FIG. 1 (c), the contents of the secret content (115) displayed on the screen (120) may be obtained (150) and leaked by a leaking means (152) installed in any manner outside the scope of awareness of the authorized worker (140). In another case, as shown in FIG. 1 (d), leakage may occur by an unauthorized person (153) other than the authorized worker (140) observing the secret content (115) displayed on the screen (120). In particular, as exemplified in FIG. 1 (d), the unauthorized person (153) may access the screen (120) by taking advantage of the absence of the authorized worker (140). Alternatively, regardless of the identification of identity, such as whether it is the authorized worker (140) or the unauthorized person (153), if two or more people are simultaneously viewing the secret content (115), it may be considered an abnormal state in which leakage occurs.

[0050] The various examples illustrated in FIG. 1 (b) to (d) imply that any suspicious behavior other than the act of the authorized worker (140) normally operating the information processing device (110) can be considered a potential leakage act. In other words, if the authorized worker (140) is able to recognize the state of normally handling the secret content on the screen (120), all other variant states can be recognized as abnormal states, and actions can be taken to prevent the leakage of the secret content.

[0051] FIG. 2 is a conceptual diagram illustrating the operating principle of a method for preventing leakage of secret content according to the present invention. For example, an information processing device (110) to which the leakage prevention method is applied may be connected to a screen (120) and a camera (130). The camera (130) may be installed to photograph an area belonging to at least a part of a spatial area, i.e., a display area (125), where it is possible to observe the secret content appearing on the screen (120). The area photographed by the camera is referred to as the surveillance area (135).

[0052] Ideally, the monitoring area (135) should include the entire display area (125). However, if this is not possible, it may include a major portion of the display area (125). Preferably, the monitoring area (135) may include at least a spatial area where the authorized worker (140) can normally be located and / or the monitoring area (135) may include a spatial area where the discharge means (151, 152) can be used or installed.

[0053] In addition to the method of the above-described embodiment, even if the practitioner of the present invention designates a suitable spatial area determined to achieve the purpose of preventing the above-described leakage act and includes it in the surveillance area (135), there will be no hindrance to achieving the purpose of the present invention. Furthermore, it is easy to see that the display area (125) can be conceptually expanded. The display area (125) should not be interpreted as being limited to an area where the screen (120) can be directly observed optically, and it is easy to understand that it includes all expanded spatial areas where it is determined that observation of the screen (120) and / or leakage act may occur. That is, as long as the purpose is achieved according to the operating principle of the present invention described below, it is reasonable to consider that at least a part of the display area (125) is included in the surveillance area (135) captured by the camera (130).

[0054] The camera (130) may capture the surveillance area (135) to generate a surveillance image (137). As described with respect to the surveillance area (135), preferably, the surveillance image (137) may include an image (141) of the authorized worker (140). The surveillance image (137) may, for example, mean a single digital image composed of two-dimensional image frames captured in the visible light region. However, other methods may also be used in the practice of the present invention. More modified embodiments of the camera (130) and the surveillance image (137) will be described later.

[0055] According to the present invention, the surveillance image (137) can be analyzed to determine whether the interior of the surveillance space (135) is in a normal state. The analysis and determination can be performed automatically by a digital image analysis method. The digital image analysis can be performed by any one of the information processing device (110), the screen (120), and the camera (130) as the executing entity according to the method of implementation, and preferably, the information processing device (110) can acquire the surveillance image (137) through a connection (122) from the camera (130) and perform the analysis. However, it can be performed in various other ways, and more modified embodiments will be described later.

[0056] The normal state of the surveillance space (135), determined through the analysis of the surveillance video (137), may mean a state in which there are no abnormal elements in the person or background object captured in the surveillance video (137). That is, through the surveillance video (137), it can be confirmed that the secret content (115) is normally displayed only to the authorized worker (140) as in (a) of FIG. 1, and that the authorized worker (140) is not seen taking any exceptional actions that are not permitted. In other words, regarding the prevention of leakage of the secret content (115), which is one of the main objectives of the present invention, if leakage risk elements including a person, action, and device related to the leakage of the secret content are not captured in the surveillance video (137), it can be considered to be in the normal state. However, the present invention is not limited to the aforementioned leakage risk factors, and aims to further reduce the risk of leakage of the secret content (115) by determining that it is not in a normal state when an unknown abnormal element is included in the surveillance video (137). The method of such analysis and determination will be described later.

[0057] When the above monitoring space (135) is determined to be in a normal state, preferably, the implementer of the present invention may implement the present invention in such a way that the display space (125) is also recognized as being in a normal state through the fact that the above monitoring space (135) is in a normal state. Accordingly, in such a case, an authorized worker (140) can view the secret content (115) through the screen (120) without restriction and perform necessary tasks.

[0058] On the other hand, if the above-mentioned monitoring space (135) is determined not to be in a normal state, it may be considered that there is a potential risk factor for the leakage of the above-mentioned secret content (115). Therefore, observation of the above-mentioned secret content (115) may need to be immediately obstructed. Various controls implemented for the purpose of such obstruction are collectively referred to as "obstruction controls" in the present invention.

[0059] FIG. 3 is a conceptual diagram of the interference control execution state according to an embodiment of the present invention. When the surveillance image (137) includes an image of an unknown object (in the example of FIG. 3, the shape (155) in which the leakage means (151) is captured), such an image is not determined to be an image in a normal state as a result of analysis, and therefore, an interference control indicator (315) can be displayed on the screen (120) together with the secret content (115). Accordingly, even if the authorized operator (140) uses the leakage means (151) to acquire (350) the content of the secret content (115) displayed on the screen (120), the identification of the secret content (115) is obstructed by the interference control indicator (315), and a visual trace (316) of the interference control indicator (315) remains on the content acquired for the leakage, so that the effect of preventing the leakage of the secret content (115) can be obtained.

[0060] According to another embodiment of the present invention, the interference control may be performed by means other than the display of the interference control indicator (315). The interference control may include a control operation understood as a full or partial blocking of the screen (120), or any control operation that substantially interferes with or does not interfere with the display of the secret content (115), provided that it is understood to include any additional control operation that interferes with the occurrence of an abnormal state (i.e., an attempt to leak the secret content (115)) such as the capture of the secret content (115), or that may cause disadvantage to the party concerned. Furthermore, the interference control may be applied with varying parameters that may include strength, luminousity, transparency, or opacity depending on the case, even when implementing the same interference control method. Additionally, one or more of the interference controls may be performed in combination, simultaneously, sequentially, or randomly. Various embodiments illustrating some of the specific implementation methods of interference control will be described later.

[0061] The above interference control may be stopped when the monitoring space (135) is restored to a normal state. It can be easily seen that whether the monitoring space is in a normal state can be determined by a procedure using a camera (130), etc., as described above. For example, referring to FIG. 3, if it is confirmed through analysis of the monitoring video (137) that the authorized worker (140) has abandoned the use of the leakage means (151) and returned to a normal state, the interference control indicator (315) may be removed, allowing normal viewing of the secret content (115). Various methods of implementation regarding the cessation of interference control may also exist, and various embodiments regarding such will be described later.

[0062] FIG. 4 is an execution flowchart of a leakage prevention method according to an embodiment of the present invention. When a surveillance image is input from a camera (S410), the surveillance image is analyzed (S420), and then, based on the result of the analysis, it is determined whether the surveillance space is in a normal state and subsequent operations are determined (S425). If it is in a normal state, normal surveillance is continued, but if interference control was previously executed (S430), it can be stopped (S435). On the other hand, if it is not in a normal state, interference control can be executed (S440).

[0064] Video analysis and judgment by artificial intelligence

[0065] In the implementation of the present invention, analysis of surveillance images is required as described above. According to one embodiment of the present invention, the analysis of surveillance images may be performed by a device including artificial intelligence. The term "artificial intelligence" may include judgment procedures based on artificial neural networks, machine learning, and / or other computer methods. The term "device including artificial intelligence" may preferably be the information processing device (110) shown in FIGS. 1 to 3, but may also refer to other devices not shown or illustrated in the drawings, and various such embodiments will be described separately later.

[0066] As described above, one of the primary objectives of the present invention is to determine all states other than normal as abnormal states where potential danger exists, by using a method to identify whether the state inside a surveillance space is normal through surveillance images. Since this method of determination has the advantage of being able to respond to threats of unknown form, unlike conventional technology, it is possible to provide a differentiated effect compared to conventional technology. Accordingly, the artificial intelligence for surveillance image analysis of the present invention configured for the above-mentioned purpose may preferably be an artificial intelligence configured to make a determination by an anomaly detection method in which criteria for determining whether an input image (i.e., surveillance image) is in a normal state are learned in advance, and the determination is made based on whether a novel element is detected in the input image.

[0067] The above anomaly detection method may refer to one of the information analysis techniques that can be utilized in image analysis processing by artificial intelligence. FIG. 5 is a simplified conceptual diagram of the anomaly detection method utilized by the present invention. Each point shown in FIG. 5 is an example of an analysis result value of various image data that can be defined on a virtual multidimensional information vector hyperplane (500) as being on a two-dimensional plane; that is, it is an example of an extremely simplified classification method of information that is actually more complex. It will be obvious to a person skilled in the art in the field of image information processing and / or artificial intelligence that the content can teach the operating principle of any artificial intelligence or other information processing method that can be utilized to implement the present invention, not limited to the form depicted in the drawing.

[0068] The artificial intelligence above learns in advance at least one image data from which a normal state, that is, a "positive" value, must be derived, and stores the analysis result value (510) as a normal category)530). The method of storage above may be, for example, in a form that can be represented as a vector point in an N-dimensional space including multiple semantic vector dimensions, and may be a form that is theoretically considered to be located on an N-dimensional hyperplane (500).

[0069] Subsequently, the artificial intelligence may be configured to determine that when the result of analyzing the image given as input is recognized within the category (530) of the learned image data (520-1), it is existing data and inlier, that is, belongs to a normal state. On the other hand, any input recognized in an area (535) that deviates from the category in any form (e.g., recognized as code 520-2) is considered novelty data, that is, it may be configured to determine that it belongs to an abnormal state containing anomaly elements.

[0070] Therefore, the above artificial intelligence may be configured to determine the analysis results of the surveillance video using a one-class classification method. As described above, the one-class classification method based on novelty data is distinguished from the method of detecting outlier data as an analysis method for abnormal elements. Although the categories of "normal" and "abnormal" are described in the description of the present invention, the artificial intelligence using the one-class classification method can be utilized in such a way that the surveillance video is substantially analyzed only as to whether it belongs to the "normal" category. Meanwhile, the category of "abnormal" refers to all novelty data included in the complement of the normal category, and may not be a limited category separately defined according to specific attributes. Therefore, in the implementation of the present invention, it is not necessary for the artificial intelligence to determine whether the surveillance video belongs to the "abnormal" category, that is, whether it belongs to the outlier category.

[0071] It is obvious that, as an implementation method for implementing the above-mentioned single-type classification method by artificial intelligence, any single-type classification method widely known in the field of artificial intelligence, such as the One-Class SVM (OC-SVM), the Isolation Forest method, the Local Outlier Factor (LOF), or the Deep Support Vector Data Description (D-SVDD), can be used.

[0072] The above single-type support vector device may, for example, mean scattering information vector data on an N-dimensional hyperplane with the origin as a reference point and determining a steady state based on the distance from the origin, but is not limited to the above example.

[0073] The above-mentioned forest of isolation method may mean, for example, arranging multidimensional data in a binary tree in an arbitrary order according to a predetermined threshold and then determining a steady state based on the depth required to reach specific data on the binary tree, but is not limited to the above example.

[0074] The above-mentioned local externality index method may mean, for example, scattering information vector data on an N-dimensional hyperplane, calculating the distance between each data point on the hyperplane to calculate the local density around each data point on the hyperplane, and determining a steady state based on the value of the local externality index (LOF) that indirectly represents the density, but is not limited to the above example.

[0075] The above deep support vector descriptor may refer to, for example, scattering information vector data on an N-dimensional hyperplane, setting a decision boundary for a minimum-sized N-dimensional shape (e.g., an N-dimensional hypersphere) based on training data representing a steady state, and performing a procedure by a deep neural network to determine a steady state based on whether it is inside or outside the decision boundary, but is not limited to the above example.

[0076] Of course, in addition to the exemplary methods described above, any method for artificial intelligence analysis and / or judgment developed prior to the filing date of this invention, or newly developed or modified after the filing date of this invention for the achievement of a single type classification method, is also not limited to being applied to this invention and may be implemented in combination without departing from the spirit of this invention.

[0077] In order for the above artificial intelligence to perform analysis of the surveillance video by a single-type classification method, the artificial intelligence needs to be trained on a normal state in advance. Accordingly, the artificial intelligence may be configured to machine learn from normal state learning data containing normal state video information. The machine learning method may be any of supervised learning, semi-supervised learning, or unsupervised learning. As exemplary embodiments for disclosing the present invention, embodiments equivalent to supervised learning or semi-supervised learning are described below.

[0078] FIG. 6 is a conceptual diagram illustrating learning using normal state image information for learning according to an embodiment of the present invention. According to an embodiment of the present invention, the normal state image information may include normal state image information for learning. The normal state image information for learning may mean at least one image information collected and / or generated to enable the artificial intelligence to learn a normal state within the surveillance space that can be captured by the camera in a general implementation environment intended by the implementer.

[0079] For example, if an embodiment of the present invention intends to apply the present invention to a state in which an authorized worker in an office space handles secret content using a personal information processing device, the normal state image information for learning may refer to at least one normal state image information (610) captured by a camera at a normal angle in a state in which a normal worker performs normal handling of the secret content within a normal office environment. For instance, in an embodiment of the present invention, if the surveillance area of ​​the camera takes a direction substantially identical to the display area of ​​the screen, it may refer to various image information showing the shape of a person looking straight ahead (i.e., toward the screen and camera) and operating the information processing device (e.g., using a keyboard or mouse). That is, in the operation process according to one embodiment of the present invention, the artificial intelligence may be configured to consider the image information input as the normal state image information (610) for learning as data (510) belonging to the category of the inherent type (530) of FIG. 5, and to distinguish whether the surveillance image is a normal state image (620-1) or a non-normal image (620-2) through analysis of the surveillance image.

[0080] The above-mentioned normal state image information for training may be pre-selected for the training. The selection may utilize a procedure comprising the collection, generation, and selection of training data consisting of "Positive" or "Healthy" samples among widely known artificial intelligence training data, and may be manual or automatic. The method may be selected by the practitioner for a preferred embodiment of the present invention, and it is evident that even if the detailed method of the selection is modified, it will not affect the technical concept of the present invention.

[0081] The above-mentioned normal state video information for learning may include video information captured at an angle that the camera can capture in the operating environment or at an angle close thereto, but it may also include video information captured at a different angle. Furthermore, according to the main embodiment of the present invention, the above-mentioned normal state video information for learning may be characterized by not including abnormal elements, such as images of persons, actions, and devices related to the leakage of secret content, but the abnormal elements are not limited to those listed above. This is because the above-mentioned normal state video information for learning is intended for learning general normal elements, not for learning specific abnormal elements.

[0082] FIG. 7 is a conceptual diagram illustrating learning by acquiring normal state image information for learning according to another embodiment of the present invention. For the implementation of the present invention, in addition to, and / or in place of, the normal state image information for learning of FIG. 6, learning based on a normal state surveillance image acquired in an actual surveillance area (i.e., a display area) may be performed, and subsequent surveillance image analysis may be performed based on this. That is, new normal state image information (711) may be acquired based on a surveillance image (715) recognized as normal state among at least one surveillance image (710) acquired in the implementation environment, and the normal state image information for learning (610a) may be updated (712) to include the new normal state image information (711), and the artificial intelligence may be configured to (re)learn based on the updated normal state image information. The above new steady-state learning data may be updated in a form that appends to the conventional steady-state image information for learning, as exemplified in FIG. 8, or the steady-state image information for learning may be composed solely of the new steady-state image information obtained in the implementation environment.

[0083] According to one embodiment of the present invention, in order to select a surveillance image (715) recognized as being in a normal state among the various surveillance images (715), a message may be input indicating that the appearance of the surveillance space captured in a specific surveillance image is recognized as being in a normal state. Similar to the method of implementing normal state image information for learning described above, the message may utilize a procedure including the collection, generation, and selection of learning data consisting of "Positive" or "Healthy" samples among widely known artificial intelligence learning data.

[0084] Alternatively, if the artificial intelligence according to the present invention determines that the interior of the corresponding surveillance space is in a normal state for a specific surveillance image (720) through predetermined learning that has already been performed, the normal state image information (610a) for learning may be updated (713) to include image information of the corresponding surveillance image by generating the message for the surveillance image, and so-called 're-learning' (815) of the artificial intelligence (810) may be performed. Such re-learning does not need to be performed for all surveillance images. For example, it may be performed by acquiring surveillance images in a normal state collected at regular frame intervals or time intervals and performing limited (re)learning.

[0085] Alternatively, if a command is input through at least one of any input means and communication means connected to the device including the artificial intelligence to determine that the interior of the surveillance space corresponding to a specific surveillance video is in a normal state, the device may be configured to generate the message for the said surveillance video.

[0086] FIG. 8 is an example diagram of a process for determining a normal state monitoring image by an external command in an embodiment of the present invention. In the exemplary state of FIG. 8, an administrator (160) observes that an authorized operator (140) is operating an information processing device (110) normally to display secret content (115) on a screen (120). As a result of the observation, the administrator (860) recognizes the current state as a normal state and can input the command (867) to the artificial intelligence (810) through an arbitrary input means (865). In FIG. 8, the artificial intelligence (810) is considered to be embedded in the information processing device (110). The input means (865) can input a command to the artificial intelligence (810) embedded in the information processing device (110) to consider the monitoring image (137) currently obtained from the camera (130) as a normal state image and to (re)learn it through a signal connection (867) including wired or wireless. Of course, Figure 8 is intended to explain the concept of the operation intended by the present invention by citing one of the various embodiments of the present invention as an example, and it is obvious that the detailed embodiments of the present invention are not limited by the contents shown in the figure.

[0088] Examples

[0089] FIG. 9 is a block diagram relating to the configuration of a device for executing a method for preventing leakage of secret content according to an embodiment of the present invention. An information processing device (910) having a function for preventing leakage of secret content according to an embodiment of the present invention may include a processor (911) having a computational function, a memory (912) connected to the processor, a screen (920) configured to enable the display of secret content, a camera (930) configured to photograph a surveillance space including at least a part of a display space where the screen can be viewed, a normal state determination unit (913) that acquires a surveillance image from the camera and analyzes the surveillance image to determine whether the inside of the surveillance space is in a normal state, and a content control unit (914) that performs a interference control operation for the purpose of preventing leakage of the secret content through the screen when, as a result of the determination by the normal state determination unit, the inside of the surveillance space is not in a normal state.

[0090] Additionally, according to an embodiment, the normal state determination unit (913) may be configured to include an artificial intelligence unit (915) that analyzes the surveillance image and determines whether the interior of the surveillance space is in a normal state, and an artificial intelligence learning unit (916) that trains the artificial intelligence unit (915) based on normal state learning data.

[0091] In addition, according to an embodiment, in order to implement an embodiment such as illustrated in FIG. 8, the device may be configured to further include an input unit (940) capable of receiving a command indicating that the inside of the monitoring space is in a normal state through at least one of the input means and communication means connected to the information processing device.

[0092] The above processor (911) may mean one or more general-purpose computers or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or any other device capable of executing and responding to instructions.

[0093] The processor (911) may be configured to execute an operating system (OS) and one or more software programs executed on the operating system. Additionally, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, even if the processor (911) is expressed in the singular, a person of ordinary knowledge in the art will know that the processor (911) may include multiple processing elements and / or multiple types of processing elements. For example, the information processing device (910) may include multiple processors or one processor and one controller. Additionally, the processor (911) may be implemented by various processing configurations, such as a parallel processor or a multi-core processor.

[0094] The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. The software may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processor or to provide instructions or data to the processor. The software may be distributed over networked computer systems and may be stored or executed in a distributed manner.

[0095] The software may also be implemented in the form of program instructions that can be executed through various computer means and may be recorded or stored in the memory (912). The memory (912) may be a computer-readable recording medium, and program instructions, data files, data structures, etc. may be recorded in the computer-readable recording medium alone or in combination. The program instructions stored in the memory may be based on a command system specifically designed and configured for embodiments of the present invention, or may follow a command system known and available to those skilled in the art of computer software, such as Assembly, C, C++, Java, Python, etc. It should be understood that the command system and the program instructions thereunder include not only machine code such as that produced by a compiler, but also high-level language code that can be executed by the information processing device (910) and / or the processor (911) using an interpreter, etc.

[0096] It is obvious to a person skilled in the art that the computer-readable recording medium constituting the memory (912) may include a temporary or volatile recording medium that is maintained only while the processor is operating, such as a processor cache, RAM, or flash memory, or may include a relatively non-volatile or long-term recording medium such as a magnetic media such as a hard disk, floppy disk, and magnetic tape, an optical recording medium such as a CD-ROM or DVD, a magneto-optical media such as a floptical disk, or a solid state memory, or may include a read-only recording medium such as a ROM placed on the hardware, and furthermore, the hardware itself configured to perform operations equivalent to a series of program instructions by a hard-wired structure by circuit wiring can also be seen as substantially equivalent to the memory (912) as a type of recording medium on which program instructions implementing an embodiment of the present invention are recorded.

[0097] The embodiments described above with respect to the processor (911) and the memory (912) are not mutually exclusive and may be selected or combined as needed. For example, a single hardware device may be configured to operate as one or more software modules to perform the operation of an embodiment of the present invention, and vice versa.

[0098] According to an embodiment of the present invention, each part included in the information processing device (910) and its operation may be implemented in a form that is a combination of software means, hardware means, or such means as described above. For example, the operation of blocks such as the normal state determination unit (913), the content control unit (914), and the input unit (940) may be implemented in whole or in part by one or more software stored in the information processing device (910) (preferably in a recording medium belonging to the memory (912)) and configured to be executed by the processor (911). That is, the present invention may be implemented in a form such as a background program or agent software that operates when secret content is displayed in the information processing device (910).

[0099] The information processing device (910) may be configured to acquire surveillance images through the camera (930) and analyze and determine them through the normal state determination unit (913), or the information processing device (910) may be configured to acquire surveillance images from the camera (930), transmit them to another information processing device to entrust part or all of the operation of the normal state determination unit (913), and receive the results to process and operate. Alternatively, the information processing device (910) may be configured to acquire surveillance images of another information processing device from the camera (930), analyze them through the normal state determination unit (913), and, if necessary, instruct the other information processing device to execute interference control.

[0100] In a preferred embodiment of the present invention, the camera (930) may be a device capable of acquiring a surveillance image consisting of a video by capturing image information in a continuous frame in the visible light region. However, in another embodiment of the present invention, the camera (930) may be configured to capture a light region other than visible light (which may include infrared, ultraviolet, and X-rays), and the surveillance image may include information captured in such a light region. In yet another embodiment, the camera (930) may be configured to acquire three-dimensional information of the surveillance area (135) through a stereoscopic image (stereograph) method and / or a depth recognition method, and the surveillance image may include such three-dimensional information. The camera (930) may be implemented in the form of a camera array that simultaneously captures the same, overlapping, or different parts of the display area where the screen (920) can be observed by two or more independent camera devices, and the surveillance image may mean a set of two or more independent image information obtained from the camera array. It is obvious that various embodiments of the camera (930) may be implemented simultaneously or combined with one another, and furthermore, it is reasonable to consider any configuration that contributes to achieving the purpose of the present invention by being substantially identical or continuous to the technical concept intended by the embodiments as a modified embodiment of the present invention.

[0101] The above normal state determination unit (913) may be configured to include an artificial intelligence unit (915) for determination as described above and an artificial intelligence learning unit (916) for learning and / or relearning the artificial intelligence unit (915).

[0102] According to one embodiment of the present invention, the artificial intelligence unit (915) may be implemented in a form including a hardware device, logic, or software program or code for executing image analysis artificial intelligence, preferably based on the processor (911) and / or the memory (120), but may also operate using other independent resources.

[0103] According to one embodiment of the present invention, the artificial intelligence unit (915) may be configured to include artificial intelligence based on a deep neural network capable of machine learning. In order to realize the identification operation to be performed by the artificial intelligence unit (915), the artificial intelligence unit (915) may include an image analysis artificial intelligence model trained in advance by supervised learning or unsupervised learning based on the normal state image information for learning. According to an embodiment of the present invention, the image analysis artificial intelligence model may be implemented as a convolutional neural network (CNN).

[0104] The method of configuring and operating the artificial intelligence unit (915) and / or the artificial intelligence and the neural network may, in accordance with an embodiment of the present invention, be a method of implementing a single-type classification method as described above with reference to FIG. 5, and it is obvious that a single-type classification method or a modified form thereof that is widely known in the field of artificial intelligence, such as a One-Class SVM (OC-SVM), an Isolation Forest method, a Local Outlier Factor (LOF), or a Deep Support Vector Data Description (D-SVDD), may be used.

[0105] In one embodiment of the present invention, the artificial intelligence unit may be configured to learn an image of a first person's normal state as normal state image information for learning. In addition, the learning of such normal state may include normal state image information for learning that includes an unspecified number of people, such as a second person, a third person, etc., in addition to the first person.

[0106] In one embodiment of the present invention, the image of the normal state of the first person may be configured to include a first normal image of the normal state of the first person and / or at least one first augmented normal image resulting from applying an operation to the first normal image that is determined to be a deformation within a normal range. The operation for deriving the first augmented normal image may include, for example, an operation to change the shape of the image of the first person within the normal range, which may include an expression and / or an object worn, with respect to the first normal image.

[0107] The first augmented image may be generated by an adaptive method. The adaptive method may include a method for adaptively adjusting the location, scale, method, and randomness determination method of the operation to match the shape or composition of the first normal image. The adaptive method may, for example, utilize the learning data augmentation method described in Patent Application No. 10-2020-0185355 filed by the same applicant as the present invention, but is not necessarily limited to the above method.

[0108] In one embodiment of the present invention, the artificial intelligence unit may be configured to learn an image of a first person's normal state as normal state image information for learning. Additionally, reference learning for a normal state may be performed by learning an image of a specific plurality of people, including a second person and a third person in addition to the first person, as learning data. In the above embodiment, the specific plurality of people may refer to one or more of at least one person that may be included in the image when determining the image as a normal state.

[0109] The above training and / or guidance may be performed by the artificial intelligence learning unit (916). The artificial intelligence learning unit (916) may be configured to enable, modify, update, or correct the operation of the artificial intelligence unit (915) by an artificial intelligence learning and / or re-learning method including the method described above with reference to FIGS. 5 to 7.

[0110] According to one embodiment of the present invention, normal state learning data provided to the artificial intelligence learning unit (916) for learning purposes includes person learning information related to at least one person, and the person learning information may include feature values ​​generated based on image information representing the person. The feature values ​​may refer to values ​​obtained by processing image information to enable machine learning on data corresponding to the image information, while not including the image information itself or including it only partially. For example, they may include statistical figures for the image information representing the person, values ​​processed to be impossible to decrypt through hashing or encryption, or results of recognizing the person based on abstract geometry using facial points or joint points, but are not limited thereto.

[0111] The above feature value may be a value processed so that it can be input to the artificial intelligence unit (915) for machine learning, or a value derived by the artificial intelligence unit (915) through machine learning. A separate artificial intelligence model may be used for processing or deriving the above feature value.

[0112] When feature values ​​are used as described above, the normal state learning data may be configured to include normal state image information that has been de-identified by applying at least one method among blur, replace, masking, and remove to image information representing the person. The configuration of de-identifying image information after using feature values ​​as described above has the beneficial effect of suppressing the disclosure of personal identification information that is unnecessarily generated during the process of generating artificial intelligence learning data by making it difficult for humans to identify the person exposed in the normal state image included in the normal state learning data, but it is not essential.

[0113] The configuration for extracting and using the above feature values ​​and / or de-identifying image information is not limited to the above normal state learning data, but may also be applied to the surveillance video input through the camera (930) before the normal state determination unit (913) processes it. In this case, it may have the effect of suppressing the disclosure of personal identification information of a person located in the surveillance area by the camera (930). As a method of implementing improvements to ensure the desirable operation of the artificial intelligence and to achieve a corresponding purpose, a configuration for extracting the above feature values ​​from objects other than people (e.g., may include arbitrary objects or background objects) may also be implemented, and it may be configured so that only the above feature values ​​are used for the operation of the normal state determination unit (913) and the learning of the artificial intelligence unit (915), and the actual image information is used only in the feature value generation stage and then discarded or deleted.

[0114] Meanwhile, one of the new technical advancements provided by the present invention is to enable the artificial intelligence unit (915) to learn normal state learning data and then provide a function to determine whether it is in a normal state, thereby enabling the determination of all abnormal states. However, this configuration does not necessarily mean that it must be used mutually exclusively with the surveillance video-based method, device, and / or system (in particular, using artificial intelligence) used in the prior art to recognize the presence or absence of an abnormal state or a person at risk of leakage and to implement measures to prevent leakage. A person skilled in the art can use the method, device, and system of the present invention, which is differentiated from the prior art, in combination with the prior art method, device, and / or system. However, even in such cases, it is obvious that the scope of application of the present invention extends to a certain range in which the present invention operates according to the technical concept presented in this disclosure.

[0115] The content control unit (914) may be configured to execute interference control when the inside of the monitoring area is determined to be in a non-normal state by the normal state determination unit (913), and any control method is acceptable as long as it is intended to prevent secret content from leaking through the screen (920). Depending on the embodiment, the screen (920) may be turned off, secret content appearing on the screen (920) may be removed, the screen (920) may be blacked out, part or all of the screen (920) may not be displayed, or any characters, letters, shapes, lines, surfaces, dots and / or other markings may be additionally included on the screen (920) along with the secret content.

[0116] The above other markings may include a visible watermark, an invisible watermark, and an almost-invisible watermark depending on the embodiment. The watermark may be, for example, a forensic watermark. The forensic watermark may include the name or identification code of the information processing device (910), the screen (920), and / or the camera (930), the name or identification code of the authorized operator of the information processing device (910), the current date and time and the content, name, or identification code of the displayed secret content, and tracking information such that the source can be identified even if the information is encoded and / or encrypted.

[0117] FIGS. 10a and 10b are exemplary diagrams of a interference control method according to some embodiments of the present invention. As shown in FIG. 10a, an example of the interference control may further include displaying a message (1091) as an interference control indicator on the screen (1020) where the secret content (1015) is displayed, indicating that an abnormal state has been detected. The message (1091) may be intended to perform interference control by warning of an abnormal state while keeping the secret content (1015) in a viewable state, thereby achieving the objective of preventing leakage according to the present invention.

[0118] An example of the above interference control may include displaying a message (1092) warning against the leakage of the secret content as an interference control indicator. The message (1092) may be intended to perform interference control by blocking the display of the secret content and warning of an abnormal state, thereby achieving the objective of preventing leakage according to the present invention.

[0119] An example of the above interference control may include further displaying an overlay content (1093) as an interference control display that interferes with the visual or electronic identification of the secret content by being displayed over the secret content (1015). This may be the same implementation method as illustrated in FIG. 3. That is, the visual identification of the secret content is interfered with through interference control, and even if the secret content is obtained for leakage, traces of the overlay content (1093) remain, thus achieving the objective of preventing leakage according to the present invention.

[0120] An example of the above interference control may include further displaying an overlay content (1094) containing tracking information of the secret content (1015) as an interference control indicator. The overlay content (1094) may be a visible, invisible, or semi-visible traceable watermark, and in the case of the traceable watermark, an example of the tracking information included in the watermark is as described above. The overlay content (1094) may also enable subsequent disposition of the leakage act by identifying the information processing device (1010) where the leakage occurred and the authorized worker, etc., when the leakage result of the secret content is detected, thereby reducing the motivation to commit the leakage act and thus achieving the objective of preventing leakage according to the present invention.

[0121] Additionally, as shown in FIG. 10b, an example of the interference control may include displaying a surveillance image (1095) obtained through the camera (1030) on the screen (1020) as an interference control indicator. Since the interference control operates when the surveillance image (1095) is analyzed as not being in a normal state, it may be intended to perform interference control by blocking the display of the secret content and warning about the abnormal state, thereby achieving the purpose of preventing leakage according to the present invention. Furthermore, if the surveillance image (1095) is analyzed incorrectly even though it substantially displays a surveillance space in a normal state, it may have an additional purpose of correcting the operation of the normal state determination unit and / or the artificial intelligence unit, etc., that may be included therein, which operates according to the present invention.

[0122] An example of the above interference control may include performing a control (1096) to stop the operation of the screen (1020) in order to prevent the leakage of the secret content. The control (1096) may mean an interference control that modulates or darkens the content of the image signal transmitted from the information processing device (1010) to the screen (1020), stops the transmission of the image signal to the screen (1020) by the information processing device (1010), cuts off the signal connection between the information processing device (1010) and the screen (1020), turns off the power to the screen (1020), cuts off the power to the screen (1020), or causes the operation of the screen (1020) to be substantially stopped (1096-1) by any method other than those described above. Accordingly, the display of the secret content itself may be immediately stopped, and this may be for the purpose of preventing leakage according to the present invention.

[0123] An example of the above interference control may include performing a control (1097) to stop the operation of the information processing device (1010) in order to prevent the leakage of the above secret content. The above control (1097) may mean a control that stops the execution of at least one application program running on the information processing device (1010), controls the operating system (OS) of the information processing device (1010) to transition to a lock state, a power saving state, or a hibernation state, stops the operation of major components such as the processor or memory of the information processing device (1010), or damages the major components (e.g., may include control that causes self-destructive actions such as applying an overcurrent to the processor or main board, or damaging the contents of the memory), turns off the power of the information processing device (1010), cuts off the power of the information processing device (1010), or causes the operation of the information processing device (1010) to be substantially stopped (1097-1) by any method other than those described above. Therefore, the display of the secret content can be immediately stopped, and since it is also impossible to attempt a new display of such content, it may be for the purpose of preventing leakage according to the present invention.

[0124] An example of the above interference control may include a control operation that logs, such as collecting a surveillance image (1098-1) obtained by the camera (1030) as evidence, transmitting a warning to another device (1098-2) (e.g., a manager's terminal device) that the surveillance image (1098-1) is determined to be in an abnormal state, and storing the collected surveillance image in a database (1098-3). The above-described interference control can support an administrator, etc., who has confirmed a warning through the other device (1098-2), in visiting the location of the information processing device (1010) where an abnormal state has occurred, or the location and display space of the screen (1020), or the surveillance space captured by the camera (1030), or the person (e.g., an authorized worker or other person) captured in the surveillance video (1098-1) based on the surveillance video (1098-1) to verify the abnormal state or, in some cases, to crack down on leakage activities. It can also support subsequent response through the log. Thus, it may be intended to achieve the objective of preventing leakage according to the present invention.

[0125] The examples of interference control described above are merely a part of various modified implementation methods that can be anticipated within the technical scope of the present invention. The present invention is configured to execute interference control for the purpose of preventing the leakage of secret content by a normal state determination unit according to the present invention. The method of interference control may be diverse in addition to that illustrated in the above embodiments, and furthermore, it may be configured as control by any new device, person, or other object that may be further included in the leakage prevention system according to the present invention.

[0126] Furthermore, the above interference control does not necessarily have to be implemented using a specific single method; as previously described, multiple different interference controls may be implemented simultaneously, sequentially, or randomly in a combined or overlapping form.

[0127] As described above in the description with reference to FIG. 1, the interference control may be stopped when the monitoring space is restored to a normal state. The cessation of such interference control may be immediate or delayed, and depending on the embodiment, the objective of the present invention may be achieved in the same way even without being carried out by automatic means. That is, in one embodiment, the interference control may be stopped immediately when a return to a normal state is confirmed through the analysis of the monitoring image. For example, measures including the removal of the interference control indicator (1091, 1092, 1093, 1094, 1095), the restoration of the state of function suspension (1096-1, 1097-1) caused by the interference control, or the cessation of evidence collection and / or logging (1098-1, -2, -3) for the purpose of the interference control may be carried out automatically.

[0128] However, in other embodiments, the release of the interference control may be configured to occur after a predetermined time delay or only after obtaining approval from an administrator. For example, an authorized operator who has experienced the interference control improperly may explain to a designated administrator that they were in a normal usage state, and the designated administrator may be configured to stop the interference control by controlling the method, device, or system according to the present invention (e.g., by implementing an input including a release command, code, password, etc., to stop the interference removal).

[0129] In addition, the objective of the present invention is to prevent the possibility of leakage, and embodiments of the present invention may also include implementing interference control that is unrecoverable or substantially irreversible. For example, if an abnormal state is determined, interference control is performed in the form of causing mechanical damage to the information processing device (1010) or screen (1020), or damaging or deleting secret content stored in a form that can be displayed on the information processing device (1010), such interference control cannot be restored even if the monitoring area returns to a normal state, but such implementation methods are also not excluded from the intent of the present invention.

[0130] In addition, when a plurality of different interference controls are executed simultaneously, sequentially, or randomly in a combined or overlapping form, the plurality of interference controls may be partially and / or sequentially released according to different methods or steps. In addition, a new interference control may be implemented during the process of releasing sequential interference controls. For example, in one embodiment of the present invention, when a first interference control that cuts off the power to the screen (e.g., shown by reference numeral 1096 in FIG. 10b) is executed and then the first interference control is released, instead of the power to the screen being restored, a second interference control is newly executed so that a warning screen (e.g., shown by reference numeral 1092 in FIG. 10a) is displayed for a certain period of time, and then all interference controls are released.

[0131] According to one embodiment of the present invention, the present invention may be configured as a system composed of a plurality of devices, rather than being composed of a single device such as the information processing device (910).

[0132] FIG. 11 is a conceptual diagram of a system for preventing leakage of secret content composed of a plurality of devices according to an embodiment of the present invention. In the embodiment of FIG. 11, the system (1100) according to the present invention may be configured to include an information processing device (1110), a screen device (1120), a camera device (1120), a normal state determination device (1150), and an administrator terminal (1160). Additionally, the information processing device (1110) may be configured to include a processor (1111), a memory (1112), a content control unit (1114), and an input unit (1140). Additionally, the normal state determination device (1150) may be configured to include an artificial intelligence unit (1155) and an artificial intelligence learning unit (1156).

[0133] In the leakage prevention system (1100) according to one embodiment of the present invention, the operation, configuration, and implementation method of the information processing device (1110) may be based on the embodiment of the information processing device (910) of FIG. 9, but are not limited thereto. In addition, the operation, configuration, and implementation method of the processor (1111) may be based on the embodiment of the processor (914) of FIG. 9, but are not limited thereto. In addition, the operation, configuration, and implementation method of the memory (1112) may be based on the embodiment of the memory (912) of FIG. 9 described above, but are not limited thereto. In addition, the operation, configuration, and implementation method of the content control unit (1114) may be based on the embodiment of the content control unit (914) of FIG. 9 described above, but are not limited thereto. In addition, the operation, configuration, and implementation method of the input unit (1140) may be based on the embodiment of the input unit (940) of FIG. 9 described above, but are not limited thereto.

[0134] The operation, configuration, and implementation method of the above-described normal state determination device (1150) may be based on the embodiment of the normal state determination device (950) of FIG. 9 described above, but are not limited thereto. The above-described normal state determination device (1150) may be an information processing device independent of the above-described information processing device (1110). That is, it may be included in the "other information processing device" shown in the above-described embodiment of the above-described information processing device (910) and the above-described normal state determination unit (913) of FIG. 9. Depending on the embodiment, the above-described normal state determination device (1150) may be implemented in various forms including a service computer, a server computer, a central system, a cloud server, or a distributed ledger-based application program (so-called D-APP). In addition, the operation, configuration, and implementation method of the above-described artificial intelligence unit (1155) may be based on the embodiment of the above-described artificial intelligence unit (955) of FIG. 9, but are not limited thereto. In addition, the operation, configuration, and implementation method of the artificial intelligence learning unit (1156) described above may be based on the embodiment of the artificial intelligence learning unit (956) of FIG. 9 described above, but are not limited thereto.

[0135] The operation, configuration, and implementation method of the above-described screen device (1120) may be based on the embodiment of the screen device (920) of FIG. 9 described above, but are not limited thereto. Additionally, the operation, configuration, and implementation method of the above-described camera device (1130) may be based on the embodiment of the camera device (930) of FIG. 9 described above, but are not limited thereto.

[0136] The operation, configuration, and implementation method of the above-described administrator terminal (1160) may mean any input means (865) possessed by the administrator (860) of FIG. 8 described above, or may mean another device (1098-2) that receives surveillance images in an example related to interference control mentioned above, or may mean any method, means, device, and / or system used by a designated administrator to stop interference control by controlling the method, device, or system according to the present invention in order to configure the interference control to be performed after obtaining approval from the administrator in the description of an example related to interference control mentioned above, but is not limited to such examples.

[0137] FIG. 12 is a conceptual diagram of a system for preventing leakage of secret content composed of a plurality of devices according to another embodiment of the present invention. In the embodiment of FIG. 12, the system (1200) according to the present invention may be configured to include an information processing device (1210), a screen device (1220), a camera device (1220), a normal state determination device (1250), and an administrator terminal (1260). However, unlike the embodiment of FIG. 11, the information processing device (1210) may be configured to include a processor (1211) and a memory (1212), the normal state determination device (1250) may be configured to include an artificial intelligence unit (1255), an artificial intelligence learning unit (1256), a content control unit (1214), and an input unit (1240), and the camera device (1230) and the administrator terminal (1260) may be configured to be connected to the normal state determination device (1250).

[0138] The system (1200) according to the embodiment of FIG. 12 may correspond to the case described above through the embodiment of FIG. 9, in which "the information processing device (910) acquires a surveillance image of another information processing device from the camera (930), analyzes it through the normal state determination unit (913), and, if necessary, instructs the other information processing device to execute interference control." That is, the normal state determination unit (1250) may be interpreted as the information processing device (910), and the information processing device (1210) may be interpreted as the other information processing device (not shown in FIG. 9).

[0139] In addition to the configurations shown in FIGS. 11 and 12, it is also readily apparent that the system (1100, 1200) can be modified and configured in various other forms as long as it is implemented as a leakage prevention system according to the present invention and achieves its purpose. Each device and / or part performing a function may be integrated, separated, exist in multiples in series or parallel, or configured to operate in different sequences by the practitioner of the present invention. Such forms of modification are not limited and are sufficient as long as they enable the achievement of the purpose based on the technical concept of the present invention; therefore, it is also obvious that all such modifications fall under the category of variations of the present invention.

[0141] Although the present invention has been described above with reference to the drawings and embodiments, as previously stated, the scope of protection of the present invention is not limited by the drawings or embodiments presented above, and those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the invention as described in the following claims. Furthermore, it will be readily apparent that the present invention may be implemented by software methods or hardware computing devices provided for various purposes or functions in addition to the specific embodiments described above, and that the present invention may be modified and utilized in various ways to support such methods or devices.

Claims

Claim 1 A method for preventing information leakage from a screen connected to an information processing device that outputs secret content, comprising: a step of acquiring a surveillance image from a camera device connected to the information processing device and configured to photograph a surveillance space including at least a portion of a display space in which the screen can be viewed; a step of analyzing the surveillance image to determine whether the interior of the surveillance space falls within a predefined category of normal state; and a step of executing interference control for the purpose of preventing the leakage of the secret content through the screen when, as a result of the determination, the interior of the surveillance space does not fall within the category of normal state; wherein the analysis of the surveillance image is performed by a device including artificial intelligence, and when the artificial intelligence analyzes the surveillance image and determines that the interior of the surveillance space is in a normal state, the method further comprises a step of generating a message indicating that the interior of the surveillance space is in a normal state. Claim 2 A method according to claim 1, further comprising: a step of training the artificial intelligence based on normal state learning data; and a step in which the artificial intelligence analyzes the surveillance video and determines whether the interior of the surveillance space is in a normal state. Claim 3 A method according to claim 2, wherein the normal state learning material includes normal state image information for learning, and the normal state image information does not include images of persons, actions, and devices related to the leakage of secret content. Claim 4 A method according to claim 2, wherein the normal state learning data includes normal state video information of the surveillance space, and the normal state video information is captured within the surveillance space and does not include images of persons, actions, and devices related to the leakage of the secret content. Claim 5 A method according to claim 4, further comprising: a step of receiving a message indicating that the monitoring space is determined to be in a normal state; and a step of acquiring new normal state image information based on the monitoring image upon receiving the message, and updating the normal state learning data to include the new normal state image information. Claim 6 delete Claim 7 In claim 5, the step of receiving the message is a method in which a command is received through at least one of an input means and a communication means connected to a device including the artificial intelligence, which determines that the interior of the monitoring space is in a normal state. Claim 8 In claim 2, the normal state learning data comprises person learning information related to at least one person, the person learning information comprises feature values ​​generated based on image information representing the person, and the normal state learning data comprises normal state image information that has been de-identified by applying at least one method among blur, replace, masking, and remove to the image information representing the person. Claim 9 A method according to claim 2, further comprising: a step in which the artificial intelligence determines whether a novelty is detected in the surveillance image; and a step in which the artificial intelligence determines the analysis result of the surveillance image in a one-class classification method indicating a normal state according to whether the novelty is detected. Claim 10 A method according to claim 1, further comprising the step of stopping the interference control when, as a result of the above judgment, the interior of the monitoring space is normal and the interference control is being executed. Claim 11 An information processing device having a function to prevent leakage of secret content, comprising: a processor having a computational function; a memory connected to the processor; a screen configured to enable the display of secret content; a camera configured to photograph a surveillance space including at least a portion of a display space in which the screen can be viewed; a normal state determination unit that acquires a surveillance image from the camera and analyzes the surveillance image to determine whether the interior of the surveillance space falls within a predefined category of a normal state; and a content control unit that performs a interference control operation for the purpose of preventing leakage of the secret content through the screen when, as a result of the determination by the normal state determination unit, the interior of the surveillance space is not in a normal state; wherein the normal state determination unit includes an artificial intelligence unit that analyzes the surveillance image and determines whether the interior of the surveillance space is in a normal state; and wherein the artificial intelligence unit is configured to generate a message indicating that the interior of the surveillance space is in a normal state when it analyzes the surveillance image and determines that the interior of the surveillance space is in a normal state. Claim 12 In claim 11, the device comprises: an artificial intelligence learning unit that trains the artificial intelligence unit based on normal state learning data, wherein the normal state determination unit is an artificial intelligence learning unit. Claim 13 A device according to claim 12, wherein the normal state learning material includes normal state image information for learning, and the normal state image information does not include images of persons, actions, and devices related to the leakage of secret content. Claim 14 A device according to claim 12, wherein the normal state learning data includes normal state video information of the surveillance space, and the normal state video information is captured within the surveillance space and does not include images of persons, actions, and devices related to the leakage of the secret content. Claim 15 In claim 14, the device is configured such that when a message is input indicating that the monitoring space is in a normal state, the artificial intelligence learning unit acquires new normal state image information based on the monitoring image and updates the normal state learning data to include the new normal state image information. Claim 16 delete Claim 17 In claim 15, the device further comprises an input unit that receives a command determining that the interior of the monitoring space is in a normal state through at least one of an input means and a communication means connected to the information processing device, wherein the message is the command input through the input unit. Claim 18 In claim 12, the normal state determination unit is configured to perform the operation of determining whether a novelty is detected in the surveillance image, and the operation of determining the analysis result of the surveillance image in a one-class classification method indicating a normal state according to whether the novelty is detected. Claim 19 In claim 11, the content control unit is configured to stop the interference control when, as a result of the judgment by the normal state judgment unit, the interior of the monitoring space is normal and the interference control is being executed. Claim 20 A system for preventing leakage of secret content comprises: an information processing device that handles secret content and executes an operation to prevent leakage of said secret content; a screen device connected to said information processing device and configured to enable display of said secret content; and a camera device connected to said information processing device and configured to photograph a surveillance space including at least a portion of a display space in which said screen device can be viewed; wherein the information processing device acquires a surveillance image from said camera device and analyzes said surveillance image based on artificial intelligence to determine whether the interior of said surveillance space falls within a predefined category of normal state, and if the result of said determination determines that the interior of said surveillance space is in a normal state, generates a message indicating that the interior of said surveillance space is in a normal state, and if the result of said determination determines that the interior of said surveillance space is not in a normal state, performs a interference control operation for the purpose of preventing leakage of said secret content through said screen device.

Images