Method for providing image analysis and management, image management agent device for this purpose

Abstract

The present invention relates to an image analysis and management technology performed in an image management agent device, wherein the image management agent device is included within a server device of a content service system and may include the steps of: acquiring an image that is uploaded and stored in a database of the server device; analyzing a pattern of the image; separating and processing the image into a separate storage area within the content service system based on the analysis result of the pattern; and, if there is confirmation from a user terminal after performing the separation processing, moving the image separated and processed into the storage area to the database.

Description

Technology Field

[0001] The present invention relates to a technology for analyzing and managing images in real time. Background Technology

[0003] With the advancement of mobile and internet technologies, access to environments where various content, especially various types of images, can be transmitted to others or shared with one another has become easier, and users can easily view or upload various types of images using user interface (UI) environments such as web surfing or browsing.

[0004] While these images contain essential or useful information, they may also contain information that expresses profanity or offensive language due to social issues such as race or gender.

[0005] Information containing social issues or offensive language can be defined as controversial content, but there may be limitations in terms of accuracy or time for users to identify such content in real time.

[0006] In particular, as the potential for the redistribution and spread of controversial content through social network services (SNS) increases, countermeasures are necessary.

[0007] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as publicly known technology disclosed to the general public prior to the filing of the present invention. Prior art literature

[0009] Published Patent Application No. 10-2018-0077846 (July 9, 2018) The problem to be solved

[0010] In an embodiment of the present invention, we propose a technology that enables real-time analysis and management of images by installing an image management agent on a server of a company providing community services or on a server that generates or records content in a web surfing or browsing environment.

[0011] The problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned problems to be solved will be clearly understood by those skilled in the art to which the present invention pertains from the descriptions below. means of solving the problem

[0013] According to an embodiment of the present invention, an image management agent device may be provided, comprising: an acquisition unit that acquires an image uploaded and stored in the server unit, wherein the image management agent device is included within a server unit; a storage unit that includes a command for outputting a result of analyzing a pattern of the image using a previously trained artificial neural network; and a processing unit that outputs the analyzed result as a result of determining whether controversial content is included in the image by executing the command, and if the controversial content is included in the image as a result of determining, separates or modifies the image.

[0014] Here, the processing unit may block or isolate the image by storing it in a separate storage area of ​​the server device when processing the image separately, and generate a modified image by inserting a warning message indicating the controversial content within the image when processing the image modified.

[0015] According to an embodiment of the present invention, an image analysis and management method performed by an image management agent device can be provided, wherein the image management agent device is included within a server device of a content service system and the image is uploaded or stored in a database of the server device; a step of acquiring an image that is uploaded or stored in a database of the server device; a step of inputting a pattern of the image into a pre-trained artificial neural network to analyze whether the image contains controversial content; and a step of determining whether to distribute the image in the content service system according to the result of the analysis.

[0016] Here, the determining step may include: a step of separating and processing the image into a separate storage area within the content service system based on the results of the analysis; and a step of moving the image separated and processed into the storage area to the database when there is confirmation from the user terminal after performing the separation processing.

[0017] Additionally, the step of separating the image may include: a step of separating the image in the storage area if the analysis result determines that the image contains controversial content; and a step of maintaining the image so that it is uploaded and stored in the database if the analysis result determines that the image does not contain controversial content.

[0018] In addition, the above method may further include a step of processing to cause an alarm to occur on the user terminal after performing the separation process.

[0019] In addition, the above method may further include a step of deleting the image if there is no confirmation from the user terminal after performing the separation process and a preset time has elapsed.

[0020] Additionally, the method may further include the step of updating the results of the separation process and the deletion process; and the step of generating statistical data for the results.

[0021] According to an embodiment of the present invention, an image analysis and management method performed by an image management agent device can be provided, wherein the image management agent device is included within a server device of a content service system and comprises the steps of: acquiring an image that is uploaded and stored in a database of the server device; analyzing a pattern of the image; processing the image to be modified based on the analysis result of the pattern; and moving the modified image generated by the modification processing to the database.

[0022] Here, the step of processing the modification may include the step of generating the modified image by inserting a warning phrase into the image if the analysis result determines that controversial content is included within the image.

[0023] According to an embodiment of the present invention, a computer-readable recording medium storing a computer program, wherein the computer program includes instructions for a processor to perform an image analysis and management method performed by an image management agent device, and the image management agent device is included within a server device of a content service system, and the method may include: a step of acquiring an image uploaded and stored in a database of the server device; a step of analyzing a pattern of the image; a step of separating and processing the image into a separate storage area within the content service system based on the result of the analysis of the pattern; and a step of moving the image separated and processed into the storage area to the database when there is confirmation from a user terminal after performing the separation processing.

[0024] According to an embodiment of the present invention, a computer program stored on a computer-readable recording medium, wherein the computer program includes instructions for a processor to perform an image analysis and management method performed by an image management agent device, and wherein the image management agent device is included within a server device of a content service system, and the method may include: a step of acquiring an image uploaded and stored in a database of the server device; a step of analyzing a pattern of the image; a step of processing the image to be modified based on the analysis result of the pattern; and a step of moving a modified image generated by the modification processing to the database. Effects of the invention

[0026] According to an embodiment of the present invention, by installing an AI-based social issue verification model on a server based on an agent, an environment is provided in which a server operator can easily identify and manage content uploaded to the server without separate monitoring work. This significantly reduces the possibility of redistribution and spread of controversial content.

[0027] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description of the invention or the claims. Brief explanation of the drawing

[0029] FIG. 1 is a schematic block diagram of a content service system including an image management agent device for image analysis and management according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating the functions of an image management agent device for image analysis and management according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating the function of an artificial neural network included in a storage unit within the image management agent device of FIG. 2, and is a diagram illustrating an exemplary case in which the result of pattern analysis of an image (the result of determining whether controversial content is included) is output through the artificial neural network. FIG. 4 is a block diagram illustrating the function of an artificial neural network included in a storage unit within the image management agent device of FIG. 2, and is a diagram illustrating an exemplary case in which an artificial neural network is trained to output an image pattern analysis result (a result of determining whether controversial content is included) by inputting an image pattern as training data. FIG. 5 is a block diagram that exemplarily illustrates the function of the processing unit within the image management agent device of FIG. 2. FIG. 6 is a flowchart illustrating an image analysis and management method performed in an image management agent device according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating an image analysis and management method performed in an image management agent device according to another embodiment of the present invention. Specific details for implementing the invention

[0030] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the scope of the present invention is defined only by the claims.

[0031] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted unless actually necessary for describing the embodiments of the present invention. Furthermore, the terms described below are defined in consideration of the functions in the embodiments of the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0032] There are limitations to users identifying controversial content containing social issues or offensive language in real time. In particular, since the potential for the redistribution and spread of controversial content through social media is high, countermeasures are necessary.

[0033] Accordingly, in an embodiment of the present invention, we propose a technology that enables real-time analysis and management of images by directly installing an image management agent on a server that generates or records content or on a company that services a community in a web surfing or browsing environment.

[0034] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0035] FIG. 1 is a schematic block diagram of a content service system (1) including an image management agent device (100) for image analysis and management according to an embodiment of the present invention. In an embodiment of the present invention, the content service system (1) is a system that creates and stores content and provides a community environment related to the content, and may, for example, provide a content service platform environment.

[0036] As illustrated in FIG. 1, the content service system (1) includes a server device (10) and a database (20), and an image management agent device (100) according to an embodiment of the present invention may be installed within the server device (10). Although only one server device (10) is illustrated in FIG. 1, this is merely an example to explain the embodiment, and it should be understood that the content service system (1) may include a plurality of server devices (10).

[0037] A database (20) is connected to a server device (10), and the database (20) can upload and store various contents, such as specific images, received through the server device (10). Images uploaded and stored in the database (20) can be delivered to the server device (10) at the request of the server device (10). Such a database (20) can be implemented for the purposes of the present invention using a relational database management system (RDBMS) such as Oracle, Informix, Sybase, DB2, or an object-oriented database management system (OODBMS) such as Gemston, Orion, O2, and may have appropriate fields to achieve its functions.

[0038] The image management agent device (100) acquires an image that is uploaded and stored in a database (20), analyzes the pattern of the acquired image, and can separate and process the image into a separate storage area within the content service system (1) based on the result of the pattern analysis. Alternatively, the image management agent device (100) can modify the image based on the result of the pattern analysis. The specific functions and operation processes of such an image management agent device (100) will be described in detail in FIGS. 2 to 7 below.

[0039] The image management agent device (100) is an electronic device installed within the server device (10). In an embodiment of the present invention, the electronic device may include, for example, at least one of a desktop computer, a tablet personal computer, an e-book reader, a laptop personal computer, and a netbook computer. In another embodiment, the electronic device may include at least one of network security equipment, navigation equipment, marine electronic equipment, avionics, a vehicle head unit, a point of sales (POS) for a store, and an Internet of Things device, and is not limited to a specific computing device or equipment. Additionally, the image management agent device (100) is an application installed within the server device (10). In an embodiment of the present invention, the application may include, for example, a smartphone application, a PC (personal computer) application, a set-top box (STB) application, a web application, an instant application, etc., and is not limited to a specific application.

[0040] FIG. 2 is a block diagram illustrating the functions of an image management agent device (100) for image analysis and management according to an embodiment of the present invention.

[0041] As illustrated in FIG. 2, the image management agent device (100) may include an acquisition unit (110), a storage unit (120), and a processing unit (130).

[0042] The acquisition unit (110) can acquire images that are uploaded and stored in the database (20). This acquisition unit (110) may include, for example, a data interface means, a user interface means, etc.

[0043] The storage unit (120) may include a command for analyzing the pattern of an image acquired through the acquisition unit (110) using a pre-trained artificial neural network and outputting the result of the pattern analysis. To this end, the artificial neural network may be pre-trained to use an issue image containing an issue of social controversy as label data, input the pattern of an image acquired through the acquisition unit (110) as training data, and output the result of the pattern analysis. Any command within the storage unit (120) may be stored in the form of an application, a program, etc., and any stored command may be selected and executed by the processing unit (130). Such a storage unit (120) may include a recording medium such as memory such as RAM (random access memory) or ROM (read only memory), a local disk connected via a network, or storage, for example, and there is no need to be limited to a specific recording medium in implementing the embodiment of the present invention.

[0044] The processing unit (130) can process to output a pattern analysis result through an artificial neural network within the storage unit (120) by executing a command within the storage unit (120). This pattern analysis result may be, for example, a result of determining whether controversial content is included in an image. This processing unit (130) may include, for example, a microprocessor-based processing device.

[0045] Here, the processing unit (130) can separate or modify the image if it determines, based on the pattern analysis result output through the artificial neural network of the storage unit (120), that controversial content is included within the image. Specifically, when the processing unit (130) separates the image, it can block or isolate the image by storing it in a separate storage area within the server device (10) or the content service system (1), and when the processing unit (130) modifies the image, it can generate a modified image by inserting a warning message indicating the controversial content within the image. The separate storage area may include, for example, an internal cache memory area in the case of the server device (10), or a remote memory area utilizing a cache server such as a CDN (content delivery network).

[0046] In an embodiment of the present invention, the processing unit (130) can execute at least one command to analyze whether the image acquired through the acquisition unit (110) contains controversial content containing social issues through an artificial neural network that has been trained in the storage unit (120). Controversial content containing social issues may be, for example, content containing keywords related to race, gender, profanity, etc.

[0047] In an embodiment of the present invention, the processing unit (130) can monitor whether an attempt to distribute content has been made through an attempt to distribute content. In an embodiment of the present invention, an attempt to distribute content is a user terminal control action performed for the purpose of distributing content. In an embodiment of the present invention, an attempt to distribute content may be any one of the following: copying content and saving it to the clipboard of the user terminal, downloading content to the user terminal, copying the address where the content is stored, copying the address of a post containing content, or creating a new post containing content, but is not limited thereto. In an embodiment of the present invention, if at least one of the aforementioned attempts to distribute content is recognized, the processing unit (130) monitors that an attempt to distribute content has been made.

[0048] Meanwhile, in an embodiment of the present invention, the image management agent device (100) can collect issue content and evaluations or comments of issue content through web crawling to select controversial content among the issue content, and train an artificial neural network with the selected controversial content. In an embodiment of the present invention, the image management agent device (100) can immediately and quickly collect controversial content that changes over time and then train an artificial neural network.

[0049] In an embodiment of the present invention, an image management agent device (100) collects at least one of issue content, an evaluation of the issue content, and comments on the issue content, and can select controversial content among the collected issue content using at least one of the collected issue content, the evaluation of the collected issue content, and comments on the collected issue content. Subsequently, the image management agent device (100) can train an artificial neural network to verify whether a certain social controversial issue is included within a certain content using the selected controversial content.

[0050] In an embodiment of the present invention, the image management agent device (100) can continuously train an artificial neural network with training data for each classification through a learning engine. Additionally, in an embodiment of the present invention, the image management agent device (100) can update training data daily through a crawling engine. Since controversial content changes over time, it must be possible to inspect social issues of content requested for inspection by reflecting the latest trends; therefore, the image management agent device (100) needs to collect issue content daily. Subsequently, controversial content is selected from the collected issue content to enable training of an artificial neural network using training data updated with the latest controversial content. Furthermore, the image management agent device (100) can inspect content requested for inspection through the trained artificial neural network to detect controversial content included in the content requested for inspection, and generate a report based on the detected controversial content.

[0051] Additionally, the image management agent device (100) transmits the report to the inspection request terminal and collects evaluation information of the report from the inspection request terminal, and can retrain or provide feedback to the artificial neural network according to the evaluation information. In an embodiment of the present invention, the evaluation information may include, for example, a rating, satisfaction, or survey for the report, and if the report contains errors, it may include explanatory text regarding the errors. The inspection request terminal is a terminal of a client that transmits inspection request content to the image management agent device (100). In an embodiment of the present invention, the inspection request content may include, for example, an image.

[0052] FIG. 3 is a block diagram illustrating the function of an artificial neural network (122) included in a storage unit (120) within an image management agent device (100) of FIG. 2, and is a diagram illustrating an exemplary case in which a pattern analysis result of an image (a result of determining whether controversial content is included) is output through the artificial neural network (122).

[0053] As illustrated in FIG. 3, when an image uploaded and stored in a database (20) is acquired through the acquisition unit (110), the processing unit (130) can input the image pattern of the image into an artificial neural network (122) to determine whether the image pattern is an image pattern containing controversial content and output a result (pattern analysis result). Here, the controversial content may include issues of social controversy, and the issues of social controversy may include keywords related to, for example, race, gender, age, politics, etc.

[0054] In one embodiment, the artificial neural network (122) can output a pattern analysis result that determines that the image pattern contains controversial content when it determines that the input image pattern contains keywords related to race, such as yellow.

[0055] In one embodiment, the artificial neural network (122) can output a pattern analysis result in which it determines that the image pattern contains controversial content when it determines that the input image pattern contains at least two keywords related to race, such as yellow.

[0056] In one embodiment, the artificial neural network (122) can output a pattern analysis result that determines that the image pattern contains controversial content when it determines that the input image pattern contains keywords related to gender, such as femi, etc.

[0057] In one embodiment, the artificial neural network (122) can output a pattern analysis result that determines that the image pattern contains controversial content when it determines that the input image pattern contains at least two keywords related to gender, such as femi, etc.

[0058] In one embodiment, the artificial neural network (122) can calculate an argument index for the issue content using the similarity that a common feature extracted from a plurality of argument texts or argument images regarding gender has with respect to at least one of the issue content, the evaluation, and the comment, and can select the issue content according to the argument index. Additionally, the artificial neural network (122) can select the issue content as argument content if keywords, profanity, or images of expressions that demean men or women are detected a certain number of times or more in the issue content.

[0059] In one embodiment, the artificial neural network (122) selects age-related controversial content based on the features of the age-related controversial content stored in advance. For example, the artificial neural network (122) can calculate a controversy index for the issue content using the similarity that a common feature extracted from a plurality of age-related controversial texts or controversial images has with respect to at least one of the issue content, the evaluation, and the comment.

[0060] In one embodiment, the artificial neural network (122) can classify the issue content as controversial content when keywords and images of expressions that disparage the elderly or children are detected more than a certain number of times in the issue content.

[0061] In one embodiment, the artificial neural network (122) can select controversial content by calculating a controversy index. That is, the processing unit (130) inputs collected issue content into the artificial neural network (122), and the artificial neural network (122) can calculate the similarity between the input issue content and the characteristics of the previously stored controversial content as a controversy index. Subsequently, through the artificial neural network (122), issues with a calculated controversy index above a certain level can be selected as controversial content.

[0062] In one embodiment, similarity can be calculated based on the ratio of words among the words included in the issue content that match the characteristics of the controversial content, and the ratio of objects among the number of objects in the image included in the issue content that match the characteristics of the controversial content.

[0063] In one embodiment, the artificial neural network (122) recognizes text and images included in the collected issue content and can input topics related to the issue content according to the recognition result of the issue content. For example, the artificial neural network (122) can select the topic with the highest relevance among race, age, and gender through the analysis of the text and images of the issue content and input issue content for the selected topic.

[0064] In one embodiment, the artificial neural network (122) can calculate an argument index based on the degree of negative nuance in the evaluation or comment by analyzing the learned results of the evaluation or comment on the issue content. For example, the artificial neural network (122) can select the issue content using the similarity that the features of the collected issue content have with respect to common features from multiple contents that have previously been selected as issue content, the negative nuance measured for the comments on the collected issue content, and the negative nuance measured for the evaluation of the collected issue content.

[0065] To this end, the processing unit (130) receives evaluation or comment data related to issue content from the acquisition unit (110) and inputs it into the artificial neural network (122). In an embodiment of the present invention, the artificial neural network (122) processes text data using natural language processing technology and machine learning algorithms and analyzes whether the text is associated with a controversial issue. Additionally, in an embodiment of the present invention, the artificial neural network (122) performs sentiment analysis of the comments to measure the sentiment or nuance of each comment. In an embodiment of the present invention, the artificial neural network (122) classifies comments into positive, neutral, and negative nuances and measures how strong the negative nuance is. Subsequently, based on the analysis of sentiment and negative nuance, the controversy index of the comments is calculated. Since negative evaluations or comments are generally likely to be highly controversial, the controversy index can be assigned to issue content that has evaluations or comments with high measured negative nuances. Subsequently, the artificial neural network (122) selects issues with a controversy index above a certain level as controversial content.

[0066] In one embodiment, the artificial neural network (122) can select controversial content based on the similarity between the controversial content characteristics and the issue content, and the negative nuances of comments and evaluations regarding the issue content. For example, the artificial neural network (122) provides the issue content to each social issue verification model to measure the similarity between the controversial content characteristics and the issue content. Additionally, the artificial neural network (122) measures the negative nuances of comments and evaluations based on the comments and evaluations regarding each issue content. Subsequently, the artificial neural network (122) can calculate an argument index based on the measured similarity and negative nuances, according to the weights set for similarity and nuance. In an embodiment of the present invention, the artificial neural network (122) calculates the weights of similarity and negative nuance according to Equation 1 and sets the calculated weights to each of the similarity and negative nuances.

[0067] Mathematical formula 1

[0068] Weight of similarity = G1X + b

[0069] (X: Similarity between issue content and features, G1: Preset slope based on similarity weights, b: Preset default similarity weights)

[0070] In Equation 1, G1 is a slope value preset according to the similarity weight. If the similarity falls within a first range (e.g., 0 percent or more and less than 20 percent), it is matched to the first value; if it falls within a second range (e.g., 20 percent or more and less than 50 percent), it is matched to the second value; and if it falls within a third range (e.g., 50 percent or more), it is matched to the third range. In the embodiment, among the first, second, and third values, the first value is the smallest value and the third value is the largest value. In the embodiment of the present invention, through Equation 1, the similarity weight can be calculated to be higher as the similarity increases.

[0071] In addition, in an embodiment of the present invention, the processing unit (130) calculates the weight of the negative nuance through Equation 2.

[0072] Mathematical formula 2

[0073] Weight of negative nuance = G2X2 + c

[0074] (X2: Negative nuance measurement result of issue content, G2: Pre-set slope based on negative nuance measurement result, c: Pre-set default weight of negative nuance)

[0075] In mathematical formula 2, G2 is a pre-set slope value according to the negative nuance of the issue content. If the negative nuance falls within the first range (e.g., 0 percent or more and less than 30 percent), it is matched to the first value; if it falls within the second range (e.g., 30 percent or more and less than 60 percent), it is matched to the second value; and if it falls within the third range (e.g., 60 percent or more), it is matched to the third range. In the embodiment, among the first value, the second value, and the third value, the first value is the smallest value and the third value is the largest value. In the embodiment of the present invention, the slopes (G1, G2) used for calculating weights can be corrected according to the learning and update results of the artificial neural network (122). In the embodiment of the present invention, the processing unit (130) calculates each weight calculated according to similarity and negative nuance through mathematical formulas 1 and 2 according to the measurement results of similarity and nuance information.

[0076] In one embodiment, the artificial neural network (122) can calculate weights based on the difference between the similarity and negative nuance of the issue content. For example, if the similarity is below a certain level but the negative nuance of comments and evaluations is above a certain level, the content can be selected as controversial content containing controversial elements. To this end, the artificial neural network (122) determines that the difference between the similarity and negative nuance is above a certain level, and if the negative nuance is greater than the similarity, sets a weight of more than half to the negative nuance and calculates a controversy index by reflecting the set weight. Through this, the artificial neural network (122) enables the collected issue content to be selected as controversial content that causes a negative reaction, even if the similarity to the characteristics of controversial content is not high.

[0077] In one embodiment, the artificial neural network (122) adjusts the weight of negative nuances according to the number of comments and replies attached to the issue content. For example, if the number of comments and replies attached to the issue content exceeds a certain number, the weight of negative nuance information can be adjusted to 50 percent or more, and the weight of negative nuances can be increased and adjusted according to the growth rate of comments and replies. In the embodiment, the weight of negative nuances can be calculated through Equation 3.

[0078] Mathematical formula 3

[0079] Negative nuance weight (W) adjustment value = Previous weight + (I*Gn)

[0080] (I: Comment growth rate, Gn: Gradient matched to previous weights)

[0081] In one embodiment, when an artificial neural network (122) measures an argument index for each issue content, it corrects the argument index according to the upload time of each issue content. This is intended to reflect the possibility that the argument index may increase in the future even if it is low at the current time. Accordingly, the argument index can be corrected higher as the difference between the current time and the content upload time is smaller. For example, if the difference between the upload time of the issue content and the current time is less than a certain amount of time (e.g., 3 hours), the argument index calculated in the first step can be corrected upward by a certain percentage.

[0082] In one embodiment, the artificial neural network (122) corrects the controversy index of the issue content by increasing it by a certain percentage when the issue content is crawled from a pre-stored controversy-inducing site. In the embodiment, the controversy-inducing site is a portal or site where controversies and social issues occur, including Bobaedream, Megalia, Ilganbest, etc., and may be set through web crawling or set and changed by an administrator. For example, when the issue content is crawled from a controversy-inducing site, the controversy index can be corrected by adding a correction constant matched to each site or portal to the first calculated controversy index.

[0083] In one embodiment, the artificial neural network (122) can correct the controversy index based on the results of the metadata analysis of comments and evaluations of collected issue content. For example, the bias of the metadata is identified by analyzing the metadata of comments and evaluations of collected issue content. In the embodiment, the bias of the metadata can be identified through the ratio of gender, age, and nationality of commenters or evaluators of the issue content. Specifically, if the gender ratio of commenters or evaluators is divided by a ratio of half, if the gender ratio of commenters exceeds the maximum value (e.g., 85 percent), or if the nationality ratio of commenters exceeds a certain level, the metadata is determined to be biased, and the controversy index can be corrected accordingly.

[0084] In one embodiment, the artificial neural network (122) can correct the controversy index by adding a set controversy index correction value to the first calculated controversy index when the gender ratio of commenters exceeds the maximum value (e.g., 85 percent) or the nationality ratio of commenters exceeds a certain level. Additionally, the artificial neural network (122) can calculate a controversy index correction value according to the degree of bias of the metadata and correct the controversy index by adding the calculated controversy index correction value to the first calculated controversy index.

[0085] FIG. 4 is a block diagram illustrating the function of an artificial neural network (122) included in a storage unit (120) within an image management agent device (100) of FIG. 2, and is a diagram illustrating an exemplary case in which an image pattern is input as training data to train the artificial neural network (122) to output an image pattern analysis result (a result of determining whether controversial content is included).

[0086] As illustrated in FIG. 4, when an image uploaded and stored in the database (20) is acquired through the acquisition unit (110), the processing unit (130) can set the image pattern of the image as input data for training the artificial neural network (122) and set an issue image containing an issue with social controversy as label data, thereby training the artificial neural network (122) to output a result (pattern analysis result) of determining whether the image pattern contains controversial content. Here, the issue image used as label data may be an image containing keywords related to, for example, race, gender, age, politics, etc.

[0087] Meanwhile, the term "model" in this specification may refer to any form of computer program that operates based on a network function, an artificial neural network, and / or a neural network. Throughout this specification, the terms "model," "neural network," "network function," and "neural network" may be used interchangeably. A neural network consists of one or more nodes interconnected through one or more links to form input and output node relationships within the neural network. The characteristics of a neural network may be determined by the number of nodes and links within the neural network, the relationships between the nodes and links, and the values ​​of the weights assigned to each link. A neural network may consist of a set of one or more nodes. A subset of nodes constituting a neural network may form a layer.

[0088] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, Generative Adversarial Networks (GAN), restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), transformers, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0089] Neural networks can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. The training of a neural network may be a process of applying knowledge to the neural network to perform a specific action.

[0090] Neural networks can be trained to minimize output errors. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, labeled data with correct answers is used for each training point, whereas in unsupervised learning, unlabeled data can be used. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a training cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's training cycle. In addition, to prevent overfitting, methods such as increasing training data, regularization, dropout (which disables some nodes), and batch normalization layers can be applied.

[0091] In one embodiment, the model may borrow at least a part of a transformer. The transformer may be composed of an encoder that encodes embedded data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of data and outputs a series of data of different types after undergoing encoding and decoding steps. In one embodiment, the series of data may be processed into a form that the transformer can compute. The process of processing the series of data into a form that the transformer can compute may include an embedding process. Expressions such as data token, embedding vector, embedding token, etc., may refer to data embedded in a form that the transformer can process.

[0092] To encode and decode a series of data, the encoders and decoders within the transformer can be processed using an attention algorithm. An attention algorithm can refer to an algorithm that calculates the similarity between one or more keys for a given query, applies this similarity to the values ​​corresponding to each key, and then calculates an attention value by performing a weighted sum of the similarity-applied values.

[0093] Various types of attention algorithms can be classified depending on how the query, key, and value are configured. For example, if attention is calculated by setting the query, key, and value identically, this can be referred to as a self-attention algorithm. If attention is calculated by reducing the dimensionality of embedding vectors to process a series of input data in parallel and determining an individual attention head for each partitioned embedding vector, this can be referred to as a multi-head attention algorithm.

[0094] In one embodiment, the transformer may be composed of modules that perform a plurality of multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional components other than attention algorithms, such as embeddings, normalization, and softmax. A method for constructing the transformer using an attention algorithm may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0095] A transformer can be applied to various data domains, such as embedded natural language, segmented image data, and audio waveforms, to convert a series of input data into a series of output data. To convert data with various data domains into a series of data that can be input to the transformer, the transformer can embed the data. The transformer can process additional data that represents the relative positional or phase relationships between the series of input data. Alternatively, the series of input data may be embedded by additionally reflecting vectors that represent the relative positional or phase relationships between the input data. In one example, the relative positional relationships between the series of input data may include, but are not limited to, word order within a natural language sentence, the relative positional relationships of each segmented image, and the temporal order of segmented audio waveforms. The process of adding information that represents the relative positional or phase relationships between the series of input data may be referred to as positional encoding.

[0096] In one embodiment, the model may include, but is not limited to, at least one of a Recurrent Neural Network (RNN), a Long Short Term Memory (LSTM) network, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).

[0097] In one embodiment, the model may be a model trained using a transfer learning method. Here, transfer learning refers to a learning method in which a pre-trained model having a first task is obtained by pre-training a large amount of unlabeled training data using a semi-supervised or self-learning method, and the pre-trained model is fine-tuned to be suitable for a second task, and a target model is implemented by training the labeled training data using a supervised learning method.

[0098] FIG. 5 is a block diagram that exemplarily illustrates the function of a processing unit (130) within the image management agent device (100) of FIG. 2.

[0099] As illustrated in FIG. 5, the processing unit (130) may include an image processing unit (132), an alarm processing unit (134), an update processing unit (136), and a statistics processing unit (138).

[0100] The image processing unit (132) can separate and process the image acquired through the acquisition unit (110) into a separate storage area based on the pattern analysis results output through the artificial neural network (122) of the storage unit (120). Additionally, the image processing unit (132) can modify the image acquired through the acquisition unit (110) based on the pattern analysis results output through the artificial neural network (122) of the storage unit (120). The specific processing process of the image processing unit (132) will be explained in detail in the flowcharts of FIGS. 6 and FIGS. 7 described later.

[0101] The alarm processing unit (134) can generate alarm information based on the separation processing result or transformation processing result of the image processing unit (132). The alarm information may be provided in the form of a voice message or a text message, for example, through an image management agent device (100), and, if necessary, the alarm information may be transmitted to a separate manager terminal so that the alarm information is displayed or output as voice through the manager terminal.

[0102] The update processing unit (136) can update the separation processing result or the transformation processing result of the image processing unit (132). For example, the update processing unit (136) can update the separation processing result or the transformation processing result in real time on the dashboard of the server device (10).

[0103] The statistical processing unit (138) can statistically process the separated or modified results of the image processing unit (132), the results of generating alarm information of the alarm processing unit (134), the updated results of the update processing unit (136), etc. The results statistically processed by the statistical processing unit (138) can be converted into a report form and stored in the database (20).

[0104] Hereinafter, along with the configuration described above, an image analysis and management method according to an embodiment of the present invention will be explained in detail with reference to the flowcharts of FIGS. 6 and FIGS. 7 attached.

[0105] First, FIG. 6 is a flowchart illustrating an image analysis and management method performed in an image management agent device (100) according to one embodiment of the present invention.

[0106] As illustrated in FIG. 6, the image management agent device (100) can acquire an image that is uploaded and stored in a database (20) and analyze the image pattern of the acquired image (S100, S102).

[0107] The analysis of image patterns can be performed through the artificial neural network (122) of FIG. 3, and the image management agent device (100) can obtain the image pattern analysis results through this artificial neural network (122).

[0108] Afterwards, the image management agent device (100) can process the image separately in a separate storage area based on the image pattern analysis result obtained through the artificial neural network (122) (S104). For example, if the image management agent device (100) determines that the pattern analysis result of the artificial neural network (122) indicates that the image contains controversial content, it can store the image separately in a separate storage area, for example, a cache memory or a cache server, and if it determines that the image does not contain controversial content, it can maintain the image in the database (20).

[0109] After separating and processing the image in this way, the image management agent device (100) can generate alarm information and provide the alarm information to the image management agent device (100) or a separate manager terminal (S106).

[0110] Afterward, the image management agent device (100) determines whether there is user verification after separating and processing the image (S108), and if there is user verification, proceeds to step (S110), and if there is no user verification, proceeds to step (S116).

[0111] In step (S110), the image management agent device (100) can move the image processed separately in a separate storage area to the database (20).

[0112] Afterwards, the image management agent device (100) can update the separated processed results and the results moved to the database (20), and generate statistical data from the updated processed results and store them in the storage unit (120) (S112, S114).

[0113] On the other hand, in step (S116), the image management agent device (100) can determine whether a preset time has elapsed without user confirmation after the separation processing of the image.

[0114] If, as a result of the judgment in step (S116), a preset time has elapsed, the image management agent device (100) can delete the image separated and processed in a separate storage area (S118). The result of this deletion process can also be updated in the image management agent device (100), and the updated result can be generated as statistical data.

[0115] FIG. 7 is a flowchart illustrating an image analysis and management method performed in an image management agent device (100) according to another embodiment of the present invention.

[0116] As illustrated in FIG. 7, the image management agent device (100) can acquire an image that is uploaded and stored in a database (20) and analyze the image pattern of the acquired image (S200, S202).

[0117] The analysis of image patterns can be performed through the artificial neural network (122) of FIG. 3, and the image management agent device (100) can obtain the image pattern analysis results through this artificial neural network (122).

[0118] Subsequently, the image management agent device (100) can process the image by transforming it based on the image pattern analysis results obtained through the artificial neural network (122) (S204). For example, if the pattern analysis result of the artificial neural network (122) determines that the image contains controversial content, the image management agent device (100) can insert information or a warning message related to the controversial content into the image and generate a transformed image as a result of the insertion; if the result determines that the image does not contain controversial content, the image can be kept stored in the database (20). At this time, the information related to the controversial content may include, for example, a sentence such as "The image contains content related to racism." Also, the warning message may include, for example, a sentence such as "Caution is required as the image contains profanity."

[0119] After separating and processing the image in this way, the image management agent device (100) can generate alarm information and provide the alarm information to the image management agent device (100) or a separate manager terminal (S206).

[0120] Afterwards, the image management agent device (100) can move the image processed in this manner to the database (20) (S208).

[0121] Afterwards, the image management agent device (100) can update the modified processed result and the result moved to the database (20), and generate the updated processed result as statistical data and store it in the storage unit (120) (S210, S212).

[0122] According to an embodiment of the present invention as described above, an artificial intelligence-based social issue verification model is provided, and such a model is implemented in the form of a plugin for a browser such as Chrome. As a result, anyone using a terminal such as a smartphone or PC can receive a service environment in which they can easily identify and manage controversial content while web surfing or browsing. Through this, it is expected that the possibility of redistribution and spread of controversial content can be significantly reduced.

[0123] Meanwhile, combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a specialized computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create a means to perform the functions described in each block of the block diagram.

[0124] Since these computer program instructions may be stored in a computer-available or computer-readable recording medium (or memory), etc., which can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, the instructions stored in the computer-available or computer-readable recording medium (or memory) may also be used to produce a manufactured item containing instruction means that perform the function described in each block of the block diagram.

[0125] And, since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on a computer or other programmable data processing equipment to create a process executed by a computer and perform the computer or other programmable data processing equipment may also provide steps for executing the functions described in each block of the block diagram.

[0126] Additionally, each block may represent a module, segment, or part of code containing at least one executable instruction for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks may occur out of order. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to the corresponding function. Explanation of the symbols

[0128] 1: Content Service System 10: Server device 20: Database 100: Image management agent device 110: Acquisition Department 120: Storage section 122: Artificial Neural Network 130: Processing unit 132: Image processing unit 134: Alarm handling unit 136: Update Processor 138: Statistical Processing Department

Claims

Claim 1 In an image management agent device, the image management agent device comprises: an acquisition unit that acquires an image uploaded and stored in the server device, wherein the image management agent device is included within a server device; and a storage unit that includes a command for outputting a result of analyzing the pattern of the image using a previously trained artificial neural network. and by executing the above command, the analysis result is output as a result of determining whether controversial content is included in the image, and if the image contains the controversial content as a result of the determination, the image is separated and processed; wherein the processing unit stores the image in a separate storage area within the server device to block or isolate the image and separates and processes the image, and if there is confirmation from the user terminal after performing the separation processing, the image separated and processed in the storage area is moved to a database connected to the server device, and if there is no confirmation from the user terminal after performing the separation processing and a preset time has elapsed, the image separated and processed in the storage area is deleted, and using a pre-trained social issue verification model, common features are calculated from a plurality of images that have been selected as controversial content, and the similarity that the common features have with respect to the image is calculated, and a controversy index for the image is calculated based on the similarity, the negative nuance regarding the evaluation or comments on the image, and a weight calculated according to the difference between the similarity and the negative nuance, and an image with a controversy index above a certain level An image management agent device that selects the above-mentioned controversial content, and the above-mentioned storage area includes a cache memory area within the server device. Claim 2 delete Claim 3 A method for image analysis and management performed by an image management agent device comprises: a step of acquiring an image that is uploaded or stored in a database connected to the server device, wherein the image management agent device is included within a server device of a content service system; a step of analyzing whether controversial content is included in the image by inputting the pattern of the image into a pre-trained artificial neural network; and a step of determining whether to distribute the image in the content service system according to the result of the analysis; wherein the determining step comprises: a step of separating and processing the image into a separate storage area within the content service system based on the result of the analysis; and a step of moving the image separated and processed in the storage area to the database when there is confirmation from a user terminal after performing the separation processing. The method for image analysis and management includes the step of deleting the separated image in the storage area if there is no confirmation from the user terminal after performing the separation process and a preset time has elapsed; wherein the analysis step includes the step of calculating common features from a plurality of images that have been selected as controversial content using a previously trained social issue verification model, calculating the similarity that the common features have with respect to the images, calculating a controversy index for the images based on the similarity, negative nuances regarding evaluations or comments on the images, and weights calculated based on the difference between the similarity and the negative nuances, and selecting images with a controversy index above a certain level as controversial content; and wherein the storage area includes a cache memory area within the server device. Claim 4 delete Claim 5 In claim 3, the step of separating and processing comprises: a step of separating and processing the image in the storage area if the analysis result determines that controversial content is included in the image; and a step of maintaining the image so that it is uploaded and stored in the database if the analysis result determines that controversial content is not included in the image. Claim 6 An image analysis and management method according to claim 5, further comprising the step of processing to cause an alarm to occur on the user terminal after performing the separation processing. Claim 7 delete Claim 8 An image analysis and management method according to claim 3, further comprising: a step of updating the results for the separation processing and the deletion processing; and a step of generating statistical data for the results. Claim 9 delete Claim 10 delete

Images