Method and system for detecting macrobot on basis of artificial intelligence

Abstract

According to an embodiment of the present disclosure, a method for detecting a macrobot in a macrobot detection system, the method being performed by at least one processor, may comprise the steps of: collecting input data including gesture information; extracting a feature vector from the input data; and calculating self-similarity between feature vectors. Accordingly, a code constituting a detection module is generated on the basis of a machine learning model trained to detect an input through a macro program, so that an input through a macrobot can be more accurately detected.

Description

AI-based macrobot detection method and system

[0001] The present disclosure relates to an artificial intelligence-based macrobot detection method and system, and more particularly, to an artificial intelligence-based macrobot detection method and system utilizing gesture information.

[0002] Macro generators that automatically generate input data are being distributed and used online. Macro generators are widely used in gaming applications, online ticketing sites, and online voting sites.

[0003] A macro bot with a specific algorithm is executed through a macro generation program, and the macro bot automatically generates input data based on the specific algorithm. The use of macro bots can create an unfair environment. For example, if a macro bot is applied to a game application, it can create an unfair gaming environment. Specifically, in a game application environment that does not support automatic hunting, users can use macro bots to automatically play the game, acquire items, and level up. Furthermore, users can easily implement skills requiring high proficiency using macro bots.

[0004] Although some techniques based on heuristic algorithms are used to detect these macrobots, the problem is that detection is difficult in the case of macros that contain anomalous, intelligent, and / or random elements.

[0005] To solve the above-described problems, the present disclosure relates to an artificial intelligence-based macrobot detection method and system using gesture information.

[0006] The present disclosure can be implemented in various ways, including as a method, a computer program stored on a computer-readable recording medium, and / or a system.

[0007] According to one embodiment of the present disclosure, a method for detecting a macrobot in a macrobot detection system performed by at least one processor may include the steps of collecting input data including gesture information; extracting a feature vector from the input data; and calculating self-similarity between the feature vectors.

[0008] Additionally, the gesture information may include a touch gesture or a mouse gesture.

[0009] In addition, the step of extracting the feature vector may include grouping data having the same start input event occurrence time from the input data to obtain data grouped by gesture unit, and generating vector data converted into multidimensional feature vectors at a certain time interval from the grouped data.

[0010] Additionally, the multidimensional feature vectors may include an event vector, a position vector, and a touch count vector.

[0011] In addition, the step of calculating the self-similarity may include obtaining a divided feature vector by applying a specific number of divisions to each feature vector, and calculating the self-similarity between the divided feature vectors and the remaining feature vectors.

[0012] Additionally, the self-similarity can be calculated as a distance value between the divided feature vectors.

[0013] Additionally, the macrobot detection method may further include a step of generating a similarity image using the self-similarity.

[0014] In addition, the method for detecting macrobots may further include, in the information processing device of the macrobot detection system, the step of collecting the input data may further include a step of collecting a plurality of input data, and learning each similarity image related to each input data using an artificial neural network model to create a macrobot detection model.

[0015] In addition, the macrobot detection method may further include a step of executing an application equipped with a detection module including the macrobot detection model in a user terminal of the macrobot detection system; and a step of detecting the macrobot using the similarity image in the equipped module.

[0016] Additionally, the step of detecting the macrobot may include at least three of a blocking response, a warning response, a suspicious response, and a normal response performed according to the detection value.

[0017] Additionally, the above warning response or the above suspicion response may display a CAPTCHA screen, and if the CAPTCHA information displayed on the CAPTCHA screen is not entered, the running application may be blocked.

[0018] A computer program stored in a computer-readable recording medium may be stored to execute the above-described macrobot detection method on a computer.

[0019] According to another embodiment of the present disclosure, a macrobot detection system includes an information processing device, a device memory; and at least one device processor connected to the device memory and configured to execute at least one computer-readable device program included in the device memory, wherein the at least one device program may include instructions for collecting a plurality of input data including gesture information, extracting a feature vector from each input data, calculating self-similarity between the feature vectors, and learning the self-similarity using an artificial neural network model to generate a macrobot detection model.

[0020] According to some embodiments of the present disclosure, macrobots can be detected even when the macros contain anomalous, intelligent, and / or random elements.

[0021] According to some embodiments of the present disclosure, a detection module capable of blocking macrobots can be incorporated into various applications.

[0022] According to some embodiments of the present disclosure, code for constructing a detection module can be generated based on a machine learning model trained to detect inputs via a macro program. Accordingly, inputs via the macrobot can be detected more accurately.

[0023] According to some embodiments of the present disclosure, log data is collected based on input data from a user terminal, and a machine learning model can be further trained based on the log data. Furthermore, the detection module can be updated based on the additionally trained machine learning model. Accordingly, even if a new macrobot is developed, inputs from the new macrobot can be detected through the updated detection module.

[0024] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure belongs (referred to as “one skilled in the art”) from the description of the claims.

[0025] Embodiments of the present disclosure will be described below with reference to the accompanying drawings, wherein like reference numerals represent similar elements, but are not limited thereto.

[0026] FIG. 1 is a diagram illustrating a method for loading a detection module for detecting macrobots into an application file according to one embodiment of the present disclosure.

[0027] FIG. 2 is a diagram showing a configuration in which an information processing device is connected to enable communication with a plurality of user terminals according to one embodiment of the present disclosure.

[0028] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing device according to one embodiment of the present disclosure.

[0029] FIG. 4 is a block diagram illustrating a method for generating a macrobot detection model in an information processing device according to one embodiment of the present disclosure.

[0030] FIG. 5 is a diagram showing an example of an artificial neural network model according to one embodiment of the present disclosure.

[0031] FIG. 6 is a diagram illustrating an example of feature vector extraction according to one embodiment of the present disclosure.

[0032] FIG. 7 is a block diagram illustrating macrobot detection in a user terminal according to one embodiment of the present disclosure.

[0033] FIG. 8 is a diagram illustrating a flowchart of a macrobot detection method according to another embodiment of the present disclosure.

[0034] FIG. 9 is a diagram illustrating a flowchart of detection response according to a detection value according to another embodiment of the present disclosure.

[0035] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions of widely known functions or configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0036] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, redundant descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0037] The advantages and features of the disclosed embodiments, and the methods for achieving them, will become clearer with reference to the embodiments described below, along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure the completeness of the disclosure and to fully inform those skilled in the art of the scope of the invention.

[0038] The terms used in this specification will be briefly explained, followed by a detailed description of the disclosed embodiments. The terms used in this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0039] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, plural expressions include singular expressions unless the context clearly indicates otherwise. When a part of the specification is said to include a component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

[0040] Also, the term 'module' or 'part' used in the specification means a software or hardware component, and the 'module' or 'part' performs certain roles. However, the 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the 'module' or 'part' may include at least one of components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. The functionality provided within the components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0041] According to one embodiment of the present disclosure, a 'module' or 'unit' may be implemented as a processor and a memory. 'Processor' should be broadly construed to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a 'processor' may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. A 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination of configurations. In addition, 'memory' should be broadly construed to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or marking data storage, registers, etc. Memory is said to be in electronic communication with the processor if the processor can read information from, and / or write information to, the memory. Memory integrated in a processor is in electronic communication with the processor.

[0042] In the present disclosure, the "system" may include, but is not limited to, at least one of a server device and a cloud device. For example, the system may be comprised of one or more server devices. As another example, the system may be comprised of one or more cloud devices. As yet another example, the system may be configured and operated by a combination of a server device and a cloud device.

[0043] In addition, the terms first, second, etc. used in the following embodiments are only used to distinguish certain components from other components, and the nature, order, or sequence of the components are not limited by the terms.

[0044] Additionally, in the embodiments below, when it is described that a component is 'connected', 'coupled' or 'connected' to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be 'connected', 'coupled' or 'connected' between each component.

[0045] Additionally, the terms 'comprises' and / or 'comprising' used in the following embodiments do not exclude the presence or addition of one or more other components, steps, operations and / or elements.

[0046] Before describing various embodiments of the present disclosure, the terminology used will be explained.

[0047] In embodiments of the present disclosure, 'application code' is program code for implementing an application, and may include source code and / or binary code. Here, the source code may be programming code before compilation.

[0048] In the embodiments of the present disclosure, a "macrobot" may be a macro program that automatically generates input data based on an algorithm. In other words, a macrobot may be referred to as a macro program generated to automatically perform a specific task through a macro generation program.

[0049] In embodiments of the present disclosure, the 'detection code' is a program code for implementing a detection module in software, and may include source code and / or binary code.

[0050] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0051] FIG. 1 is a diagram illustrating a method for loading a detection module for detecting macrobots into an application file according to one embodiment of the present disclosure.

[0052] As illustrated in FIG. 1, the information processing device (100) can receive an application file (10) from a customer system and mount a detection module (20) on the received application file (10). In addition, the application file (10) equipped with the detection module (20) can be provided to the customer system. That is, through the information processing device (100) that provides a detection module (20) that detects a macrobot on a network basis, the application file (10) can be equipped with the detection module (20) and provided to the customer system. Here, the customer system may be a system related to the company that developed the application.

[0053] According to one embodiment, the information processing device (100) may insert detection code associated with the detection module (20) into the application code so that the detection module (20) can be mounted on the application file (10). At this time, the information processing device (100) may analyze the code included in the application file (10), determine the location where the detection code is to be inserted among the entire area of ​​the code, and insert the detection code into the determined location. Here, the insertion of the detection code may mean that all or part of the detection code is inserted into the application code, or that a command code for calling a function that performs a procedure associated with the detection code is inserted. In addition, if the detection code is configured as a separate library, the code associated with this library may be inserted into the application code.

[0054] According to one embodiment, the detection code constituting the detection module (20) may be generated based on a trained machine learning model. According to one embodiment, the machine learning model may be trained to detect inputs through a macro program (macrobot) based on a training set including a plurality of learning input data. A macrobot detection algorithm associated with the machine learning model may be applied to the detection module, and the detection module may detect inputs by the macrobot based on the macrobot detection algorithm. The learning method performed in the machine learning model will be described later with reference to FIGS. 4 to 6 and 8.

[0055] The information processing device (100) can transmit an application file (10) equipped with a detection module (20) to a customer system. The customer system can upload the application file (10) equipped with the detection module (20) to an app store. Accordingly, a user can download the application file (10) equipped with the detection module.

[0056] In response to an application being executed on a user terminal, a detection module (20) may be activated. The activated detection module (20) may monitor input data. When an input via a macrobot is detected, the detection module (20) may perform subsequent procedures based on a predetermined policy. For example, when the number of inputs detected by the macrobot reaches a first number, the detection module (20) may temporarily suspend the execution of the application, output CAPTCHA information to the user terminal, and control the re-execution of the suspended application when the CAPTCHA screen is passed. As another example, the detection module (20) may control the execution of the application to be terminated when the number of inputs detected by the macrobot reaches a second number. Here, the second number may be greater than the first number.

[0057] FIG. 2 is a diagram showing a configuration in which an information processing device is connected to enable communication with a plurality of user terminals according to one embodiment of the present disclosure.

[0058] As illustrated in FIG. 2, a plurality of user terminals (210_1, 210_2, 210_3) may be connected to an information processing device (230) capable of providing a detection module (20) for detecting macrobots via a network (220). In addition, a plurality of user terminals (210_1, 210_2, 210_3) may be connected to a customer system (240) capable of providing game services, portal services, banking services, ticket reservation services, etc. via the network (220). The information processing device (100) of FIG. 1 may correspond to the information processing device (230).

[0059] In one embodiment, the information processing device (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to a macrobot detection method, or one or more distributed computing devices and / or distributed databases based on a cloud computing service. For example, the information processing device (230) may include a cloud computing device for loading a detection module into an application file and a server for performing training on a machine learning model. The cloud computing device may be equipped with a virtual machine for loading a detection module into an application file. As another example, a cloud computing device included in the information processing device (230) may be equipped with a first virtual machine for loading one detection module into an application file and a second virtual machine for performing training on a machine learning model.

[0060] A detection module for detecting a macrobot learned by an information processing device (230) can be provided to a customer system (240). For example, the information processing device (230) can receive an application including an application code from a customer system (240), install a detection module in the received application file, and then transmit the application file with the detection module installed to the customer system (240).

[0061] The customer system (240) may include at least one server and / or terminal that provides web services, such as game services, portal services, banking services, and ticket reservation services. The customer system (240) may transmit an application file for providing the web service to the information processing device (230) and receive an application file equipped with a detection module from the information processing device (230). The customer system (240) may upload the application file equipped with the detection module to a downloadable storage, such as an app store.

[0062] A plurality of user terminals (210_1, 210_2, 210_3) can communicate with each of the information processing device (230) and the customer system (240) via the network (220). The network (220) can be configured to enable communication among the plurality of user terminals (210_1, 210_2, 210_3), the information processing device (230), and the customer system (240). Depending on the installation environment, the network (220) can be configured as a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a wireless network such as a mobile communication network, WLAN (Wireless LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network (220) may include, but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0063] According to one embodiment, the user terminal (210_1, 210_2, 210_3) can install the application by downloading the application file uploaded from the customer system (240) and executing the installation file included in the downloaded application file. When the application is executed on the user terminal (210_1, 210_2, 210_3), a detection module associated with the application can be executed. The detection module can collect input data including gesture information and transmit the collected input data to the information processing device (230).

[0064] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto, and the user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and executing applications or web browsers. For example, the user terminals may include smartphones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, although FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with an information processing device (230) via a network (220), the present invention is not limited thereto, and a different number of user terminals may be configured to communicate with an information processing device (230) via a network (220).

[0065] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing device according to one embodiment of the present disclosure.

[0066] The user terminal (210) may refer to any computing device capable of executing applications related to a customer system and capable of wired / wireless communication, and may include, for example, a mobile phone terminal (210_1), a tablet terminal (210_2), a PC terminal (210_3), etc. of FIG. 2. As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing device (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing device (230) may be configured to communicate information and / or data via a network (220) using their respective communication modules (316, 336). Additionally, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) via the input / output interface (318).

[0067] The memory (312, 332) may include any non-transitory computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a permanent mass storage device such as a read-only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the user terminal (210) or the information processing device (230) as a separate permanent storage device distinct from the memory. In addition, the memory (312, 332) may store an operating system and at least one program code (e.g., a detection code for detecting a macrobot installed and run on the user terminal (210).

[0068] These software components may be loaded from a computer-readable recording medium separate from the memory (312, 332). This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal (210) and the information processing device (230), and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. As another example, the software components may be loaded into the memory (312, 332) through a communication module other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (312, 332) based on a computer program that is installed by files provided by developers or a file distribution system that distributes installation files of applications through a network (220).

[0069] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by a memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a storage device such as the memory (312, 332).

[0070] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing device (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing device (230) to communicate with another user terminal or another system (e.g., a separate cloud system, etc.). For example, a request or data generated by the processor (314) of the user terminal (210) according to a program code stored in a recording device such as a memory (312) may be transmitted to the information processing device (230) via the network (220) under the control of the communication module (316). Conversely, at least one of the control signals, commands or data provided under the control of the processor (334) of the information processing device (230) may be received by the user terminal (210) through the communication module (316) of the user terminal (210) via the communication module (336) and the network (220).

[0071] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera, keyboard, microphone, mouse, etc., including an audio sensor and / or an image sensor, and the output device may include a device such as a display, a speaker, a haptic feedback device, etc. As another example, the input / output interface (318) may be a means for interfacing with a device that has a configuration or function integrated into one for performing input and output, such as a touch screen. For example, when the processor (314) of the user terminal (210) processes a command of a computer program loaded in the memory (312), a service screen configured using information and / or data provided by the information processing device (230) or another user terminal may be displayed on the display through the input / output interface (318). In FIG. 3, the input / output device (320) is illustrated as not being included in the user terminal (210), but is not limited thereto, and may be configured as a single device with the user terminal (210). In addition, the input / output interface (338) of the information processing device (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing device (230) or that the information processing device (230) may include. In FIG. 3, the input / output interfaces (318, 338) are illustrated as elements configured separately from the processors (314, 334), but are not limited thereto, and the input / output interfaces (318, 338) may be configured to be included in the processors (314, 334).

[0072] The user terminal (210) and information processing device (230) may include more components than those illustrated in FIG. 3. However, it is not necessary to explicitly illustrate most conventional components. In one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. In addition, the user terminal (210) may further include other components, such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, and the like.

[0073] While the program for the detection module (20) is running, the processor (314) can receive text, images, videos, voices and / or actions, etc. input or selected through input devices such as a camera, microphone, including a touch screen, keyboard, audio sensor and / or image sensor connected to an input / output interface (318), and can store the received text, images, videos, voices and / or actions, etc. in the memory (312) or provide them to the information processing device (230) through the communication module (316) and the network (220).

[0074] The processor (314) of the user terminal (210) may be configured to manage, process, and / or store information and / or data received from an input / output device (320), another user terminal, an information processing device (230), and / or a plurality of external systems. The information and / or data processed by the processor (314) may be provided to the information processing device (230) via a communication module (316) and a network (220). The processor (314) of the user terminal (210) may transmit the information and / or data to the input / output device (320) via an input / output interface (318) and output the information and / or data. For example, the processor (314) may control a display device included in or connected to the user terminal (210) so that the received information and / or data is displayed on the screen of the user terminal (210) by outputting the received information and / or data.

[0075] The processor (334) of the information processing device (230) may be configured to manage, process, and / or store information and / or data received from multiple user terminals (210) and / or multiple external systems. Information and / or data processed by the processor (334) may be provided to the user terminal (210) via a communication module (336) and a network (220).

[0076] FIG. 4 is a block diagram illustrating a method for generating a macrobot detection model in an information processing device according to an embodiment of the present disclosure, and FIG. 5 is a diagram illustrating an example of an artificial neural network model according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of feature vector extraction according to an embodiment of the present disclosure.

[0077] As illustrated in FIG. 4, the information processing device (100) may include a data collection unit (410), a feature vectorization unit (420), a self-similarity calculation unit (430), an image conversion unit (440), and an artificial intelligence learning unit (450) to generate a detection model of a macrobot.

[0078] The data collection unit (410) can collect or arbitrarily generate multiple input data. For example, each input data may include gesture information actually input from a user terminal (210_1, 210_2, 210_3). The gesture information may include a touch gesture generated from a mobile phone terminal and a mouse gesture generated from a PC terminal. As another example, each input data may include gesture information arbitrarily generated by an information processing device (230) or another system. Here, one input data may include multiple gesture information input from a user terminal (210_1, 210_2, 210_3) during a unit time. The unit time may be a predetermined time, and for example, may be 1 minute. Here, the gesture information may include, for example, a touch-on time, a touch movement time, a touch-off time, a touch location, etc. Here, the touch-on time is the time when the mouse button is pressed.

[0079] Click Assistant, a mobile macro program, supports macros for touches, random intervals, and even gestures. These touches and gestures are detected by the MotionEvent class, generating motion event data. This motion event data can then be detected by the GestureDetector class to generate gesture information. The GestureDetector class is used as an example; the gesture information can include gesture information obtained using other classes.

[0080] The feature vectorization unit (420) can extract feature vectors from the raw data, which is the input data collected by the data collection unit (410). The extraction of feature vectors may include grouping and vectorization of features. The grouping of features may group data having the same start input event occurrence time in the input data, which is the raw data. Here, the same start input event occurrence time may be based on the touch-on time. Accordingly, data may be grouped by gesture unit. Feature vectorization may mean generating multidimensional feature vectors at a certain time interval from the data grouped by gesture unit. Feature vectorization is related to the flow of time, and the certain time interval here may be 100 ms, for example. Such feature vectorization may be composed of multidimensional feature vectors. An example of such multidimensional feature vectors is illustrated in FIG. 6.

[0081] In Fig. 6, 3D feature vectors of an Event vector, a Position vector, and a Touch Count vector are illustrated as examples. Here, an Event represents a type of touch, and the number of this event can distinguish whether there is a touch in only one area or in two areas. A Position is a value indicating where a touch is located, and the number of this position can be a value obtained by sorting the entire touch area into, for example, 4 X 6 or 8 X 12 and then assigning an order. A Touch Count is intended to indicate how many touches occurred at a specific time, and the number of this Touch Counter can be the number of touches that occurred. Although the feature vectors in Fig. 6 are illustrated in 3D, the feature vectors can be additionally configured as 4-dimensional or 5-dimensional feature vectors.

[0082] The self-similarity calculation unit (430) can calculate self-similarity between feature vectors using the feature vectors extracted from the feature vectorization unit (420). Calculating self-similarity between feature vectors may include performing vector segmentation and calculating self-similarity. The vector segmentation may mean performing segmentation by applying a specific number of segments to the feature vectorized data. Here, the interval for a specific segmentation may be the same as or larger than a certain time interval in feature vectorization. However, it may be necessary to divide the input data, which is the source data, evenly. Calculating self-similarity may mean calculating self-similarity, i.e., similarity distance, between feature vectors excluding one's own feature vectors. Here, the self-similarity calculation may be calculated based on the Euclidean distance between the segmented feature vectors or based on DTW (Dynamic Time Warping).

[0083] The image conversion unit (440) can generate a similarity image using the self-similarity calculated by the self-similarity calculation unit (430). Specifically, the self-similarity calculated using a computer vision library (OpenCV) can be converted into a similarity image to generate a similarity image. In addition, the maximum self-similarity distance information that can be possessed by each feature is separately calculated from the self-similarity, and then the self-similarity table normalization process is performed, and the normalized self-similarity table can be converted into a similarity image using a computer vision library (OpenCV).

[0084] The artificial intelligence learning unit (450) can perform learning using the similarity images generated by the image conversion unit (440). An example of an artificial neural network model according to one embodiment of the present disclosure is illustrated in FIG. 5. The artificial neural network model (500), as an example of a machine learning model, may be a statistical learning algorithm implemented based on the structure of a biological neural network in machine learning technology and cognitive science, or a structure that executes the algorithm.

[0085] According to one embodiment, the artificial neural network model (500) may represent a machine learning model that has problem-solving capabilities by learning to reduce the error between the correct output corresponding to a specific input and the inferred output by repeatedly adjusting the weights of synapses, which are artificial neurons that form a network by combining synapses, as in a biological neural network. For example, the artificial neural network model (500) may include any probability model, neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning. The artificial neural network model (500) may be implemented as a multilayer perceptron (MLP) composed of multiple layers of nodes and connections between them. The artificial neural network model (500) according to the present embodiment may be implemented using one of various artificial neural network model structures including an MLP.

[0086] As illustrated in FIG. 5, the artificial neural network model (500) may be composed of an input layer (520) that receives an input signal or data (510) from the outside, an output layer (540) that outputs an output signal or data (550) corresponding to the input data, and n hidden layers (530_1 to 530_n) located between the input layer (520) and the output layer (540) that receive a signal from the input layer (520), extract a characteristic, and transmit it to the output layer (540) (where n is a positive integer). Here, the output layer (540) may receive a signal from the hidden layers (530_1 to 530_n) and output data to the outside.

[0087] The learning method of the artificial neural network model (500) includes a supervised learning method that learns to optimize problem solving through input of a teacher signal (correct answer), and an unsupervised learning method that does not require a teacher signal. In one embodiment, the artificial neural network model (500) may be unsupervised learning to classify whether a gesture is input by a human or a gesture input by a macrobot based on similarity images. Additionally or alternatively, the artificial neural network model (500) may be supervised learning to infer whether input data is data input by a macrobot. Code related to the operation of the artificial neural network model (500) trained in this way may be included in the detection code.

[0088] In this way, a plurality of input data and a plurality of corresponding output data are respectively matched to the input layer (520) and the output layer (540) of the artificial neural network model (500), and the synapse values ​​between the nodes included in the input layer (520), the hidden layers (530_1 to 530_n), and the output layer (540) are adjusted, so that learning can be performed so that the correct output corresponding to a specific input can be extracted. Through this learning process, the characteristics hidden in the input data of the artificial neural network model (500) can be identified, and the synapse values ​​(or weights) between the nodes of the artificial neural network model (500) can be adjusted so that the error between the output data calculated based on the input data and the target output is reduced. The artificial neural network model (500) trained in this way can output a macrobot detection result for the input data in response to the input data.

[0089] Meanwhile, this artificial neural network model (500) may be a deep learning model based on CNN (Convolutional Neural Networks) in the present disclosure.

[0090] FIG. 7 is a block diagram illustrating macrobot detection in a user terminal according to one embodiment of the present disclosure.

[0091] As illustrated in FIG. 7, the user terminal (210) may include a data collection unit (410), a feature vectorization unit (420), a self-similarity calculation unit (430), an image conversion unit (440), a macrobot detection unit (750), and a detection response unit (760) to detect a macrobot. The data collection unit (410), the feature vectorization unit (420), the self-similarity calculation unit (430), the image conversion unit (440), the macrobot detection unit (750), and the detection response unit (760) may be included in a detection module (20). In response to an application being executed on the user terminal, the detection module (20) may be activated. The activated detection module (20) may monitor input data. Here, the data collection unit (410), feature vectorization unit (420), self-similarity calculation unit (430), and image conversion unit (440) are the same except that they process multiple input data for learning and process input data one by one to detect macrobots, so their description is omitted.

[0092] The macrobot detection unit (750) can detect whether a gesture is made by a macrobot by using an artificial neural network model (500) trained by the artificial intelligence learning unit (450), for example, a deep learning model based on CNN (Convolutional Neural Networks), on a similarity image generated by the image conversion unit (440). The macrobot detection unit (750) can output the detection result as a value.

[0093] The detection response unit (760) may terminate the execution of the application based on the value detected by the macrobot detection unit (750), or may temporarily suspend the execution of the application and output CAPTCHA information to the user terminal (210), and control the suspended application to be executed again when the CAPTCHA screen is passed.

[0094] FIG. 8 is a diagram illustrating a flowchart of a macrobot detection method according to another embodiment of the present disclosure.

[0095] The data collection unit (410) of the user terminal (210) can collect input data including gesture information (S802). The data collection unit (410) of the information processing device (230) can read input data stored in the memory of the user terminal (210) and collect the input data.

[0096] The feature vectorization unit (420) can group features from the raw data, which is the input data collected by the data collection unit (410) (S804). The features can be grouped into features having the same start input event occurrence time in the input data, which is the raw data. Here, the same start input event occurrence time can be based on the touch-on time. Accordingly, features can be grouped by gesture unit. The feature vectorization unit (420) can use the data grouped by gesture unit to vectorize features at a certain time interval and convert them into multidimensional feature vectors (S806).

[0097] The self-similarity calculation unit (430) can calculate the self-similarity between feature vectors by using the self-similarity of the feature vectors extracted from the feature vectorization unit (420) (S808). Calculating the self-similarity between feature vectors may include performing vector division and calculating self-similarity. The vector division may mean performing division by applying a specific number of divisions to the feature vectorized data. Calculating self-similarity may mean calculating the self-similarity, i.e., the similarity distance, between the remaining feature vectors excluding the own feature vectors.

[0098] The image conversion unit (440) can generate a similarity image using the self-similarity calculated by the self-similarity calculation unit (430) (S810). Specifically, the similarity image can be generated by converting the self-similarity calculated using a computer vision library (OpenCV) into a similarity image.

[0099] The artificial intelligence learning unit (450) can perform artificial intelligence learning using the similarity images generated by the image conversion unit (440) (S812). The artificial neural network model may be a deep learning model based on CNN (Convolutional Neural Networks).

[0100] The artificial intelligence learning unit (450) can create a detection model using the values ​​of the learned artificial neural network model (S814). This detection model can be installed in the detection module (20).

[0101] Meanwhile, the macrobot detection unit (750) of the user terminal (210) can detect whether a gesture is made by a macrobot by using an artificial neural network model (500) trained by the artificial intelligence learning unit (450), for example, a deep learning model based on CNN (Convolutional Neural Networks), on the similarity image generated by the image conversion unit (440) (S820). The macrobot detection unit (750) can output the detection result as a value.

[0102] The detection response unit (760) can perform different responses depending on the values ​​detected by the macrobot detection unit (750) (S822). A flowchart of detection responses based on detection values ​​according to another embodiment of the present disclosure is illustrated in FIG. 9.

[0103] The detection response unit (760) can block the application (S904) if the probability detected by the macrobot detection unit (750) is greater than or equal to the first threshold value, for example, 95% (S902). Meanwhile, the detection response unit (760) can increase the warning count (S912) if the probability detected by the macrobot detection unit (750) is greater than or equal to the second threshold value, for example, 80% (S910), and can perform a warning response to block the application if the warning count (Count1) is greater than or equal to the warning threshold value (TH1) (S914) (S916). Meanwhile, the detection response unit (760) can perform a suspicion response by increasing the suspicion count (S922) if the probability detected by the macrobot detection unit (750) is greater than or equal to the third threshold value, for example, 60% (S920), displaying CAPTCHA information on the display (S926) if the suspicion count (Count1) is greater than or equal to the suspicion threshold value (TH2) (S924), and blocking the application if CAPTCHA information is not input (S928). In addition, the detection response unit (760) can perform a normal response by determining that it is a gesture input by a human if the probability detected by the macrobot detection unit (750) is less than or equal to the third threshold value, for example, 60% (S920).

[0104] The above flowchart and description are merely examples, and some embodiments may implement the system differently. For example, in some embodiments, the order of each step may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0105] The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0106] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software will depend on the particular application and the design requirements imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0107] In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.

[0108] Accordingly, the various exemplary logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0109] In a firmware and / or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, a compact disc (CD), a magnetic or marking data storage device, etc. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

[0110] When implemented in software, the techniques described above may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

[0111] For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of media. Disk or disc, as used herein, includes compact discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks usually reproduce data magnetically, whereas discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0112] A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in the user terminal.

[0113] While the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the present disclosure may be implemented in multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include personal computers, network servers, and portable devices.

[0114] While the present disclosure has been described in connection with certain embodiments herein, various modifications and variations may be made without departing from the scope of the present disclosure, as would be understood by those skilled in the art. Furthermore, such modifications and variations are intended to fall within the scope of the claims appended to this specification.

Claims

1. A method for detecting a macrobot in a macrobot detection system performed by at least one processor, A step of collecting input data including gesture information; A step of extracting a feature vector from the above input data; and A method for detecting a macrobot, comprising a step of calculating self-similarity between the above feature vectors.

2. In paragraph 1, A method for detecting a macrobot, wherein the above gesture information includes a touch gesture or a mouse gesture.

3. In paragraph 1, A method for detecting a macrobot, wherein the step of extracting the feature vector comprises: obtaining data grouped by gesture unit by grouping data having the same start input event occurrence time from the input data, and generating vector data converted into multidimensional feature vectors at a certain time interval from the grouped data.

4. In paragraph 3, The above multidimensional feature vectors include an event vector, a position vector, and a touch count vector, and are a macrobot detection method.

5. In paragraph 3, A method for detecting a macrobot, wherein the step of calculating the self-similarity includes obtaining a divided feature vector by applying a specific number of divisions to each feature vector, and calculating the self-similarity between the divided feature vectors and the remaining feature vectors.

6. In paragraph 5, A macrobot detection method in which the above self-similarity is calculated as a distance value between the divided feature vectors.

7. In paragraph 5, A method for detecting macrobots, further comprising a step of generating a similarity image using the self-similarity.

8. In paragraph 7, In the information processing device of the above macrobot detection system, The step of collecting the above input data collects multiple input data, A method for detecting macrobots, further comprising a step of generating a macrobot detection model by learning each similarity image related to each input data using an artificial neural network model.

9. In paragraph 8, In the user terminal of the above macrobot detection system, A step of executing an application equipped with a detection module including the above macrobot detection model; and A method for detecting a macrobot, further comprising a step of detecting the macrobot using the similarity image in the above-mentioned mounting module.

10. In paragraph 9, A method for detecting a macrobot, wherein the step of detecting the above macrobot includes at least three of a blocking response, a warning response, a suspicious response, and a normal response performed according to a detection value.

11. In paragraph 10, A macrobot detection method in which the above warning response or the above suspicion response displays a CAPTCHA screen and blocks the running application if the CAPTCHA information displayed on the CAPTCHA screen is not entered.

12. A computer program stored on a computer-readable recording medium for executing the method according to any one of claims 1 to 11 on a computer.

13. In the macrobot detection system, The information processing device is, device memory; and At least one device processor coupled to said device memory and configured to execute at least one computer-readable device program contained in said device memory; At least one device program of the above, Collect multiple input data including gesture information, Extract feature vectors from each input data, Calculate the self-similarity between the above feature vectors, and A macrobot detection system comprising commands for generating a macrobot detection model by learning the self-similarity using an artificial neural network model.

14. In paragraph 13, The user terminal is, terminal memory; and At least one terminal processor connected to the terminal memory and configured to execute at least one computer-readable terminal program contained in the terminal memory, At least one terminal program of the above, Collect input data containing gesture information, Extract feature vectors from the above input data, Calculate the self-similarity between the above feature vectors, and A macrobot detection system, comprising commands for detecting macrobots using the self-similarity in the above macrobot detection model.

Images