Artificial intelligence-based behavior monitoring method, program, and device

Abstract

According to an embodiment of the present disclosure, an AI-based face recognition method, a program, and an apparatus are disclosed, which may include synchronizing first image data captured in a first direction and second image data captured in a second direction different from the first direction based on a face of a person who is a target of behavior monitoring, generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the synchronized first image data and second image data using a pre-trained neural network model, and estimating the behavior of the person by combining the first analysis result based on the synchronized first image data and the second analysis result based on the synchronized second image data.

Description

[Technical Field]

[0001] The present disclosure relates to data analysis technology, and more particularly to a method and apparatus for predicting and monitoring human behavior based on complex judgment results based on artificial intelligence. [Background technology]

[0002] In environments built for specific purposes, there are situations where it is necessary to monitor the actions taken by people and analyze the consequences of those actions. For example, in an educational environment where exams are taken, it is necessary to monitor the actions taken by test takers during the exam. In particular, unlike offline exams, it is difficult to effectively monitor the actions of test takers and their surrounding environment in online exams. Therefore, in online exam environments, it is even more important for administrators to accurately analyze the actions taken by test takers in real time to determine whether cheating has occurred.

[0003] As can be seen from the above examples, it is not easy to effectively monitor a person's behavior and the surrounding environment in an online environment. While there are conventional technologies that analyze a person's specific behavior using a sensing device such as a camera, most of them analyze a person's specific behavior based only on fragmentary information acquired in a specific situation. However, such an analysis based only on fragmentary information makes it difficult to accurately interpret whether a person is engaging in behavior that requires judgment in a specific environment. For example, if cheating is detected by analyzing only a frontal image of a person in an online exam environment, the limited information that can be acquired from the frontal image increases the likelihood of failing to determine cheating even when cheating is suspected, or of misjudging cheating even when it is not. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been devised in response to the above-mentioned background art, and aims to provide a method and apparatus that can comprehensively determine and accurately monitor what actions a person will take in a specific environment based on various sensing results.

[0005] However, the problems to be solved by the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood from the following description. [Means for solving the problem]

[0006] To achieve the above object, one embodiment of the present disclosure discloses a behavior monitoring method based on artificial intelligence, which is performed by a computing device. The method includes: synchronizing first image data captured in a first direction and second image data captured in a second direction different from the first direction based on a face of a person who is a target of behavior monitoring; generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the synchronized first image data and second image data using a pre-trained neural network model; and estimating the behavior of the person by combining the first analysis result based on the synchronized first image data and the second analysis result based on the synchronized second image data.

[0007] Alternatively, the sensing item may be status information identified based on a subclass of the sensing object, and the status information may be changeable depending on the person's actions.

[0008] Alternatively, the plurality of sensing objects may include, in addition to the body parts of the person, at least one of an object other than the person, a sound of an object related to the person's actions, or a time of an object related to the person's actions.

[0009] Alternatively, the neural network model may include a first model that tracks a person's gaze based on video data captured in the first direction, and a second model that estimates a person's pose based on video data captured in the second direction.

[0010] Alternatively, the step of generating an analysis result for at least one sensing item included in each of the plurality of sensing objects based on each of the synchronized first and second image data using the pre-trained neural network model may include a step of correcting the feature points extracted from the synchronized second image data based on a coordinate system of the feature points extracted by inputting the synchronized first image data into the first model when the feature points on the human face extracted by inputting the synchronized second image data into the second model deviate by more than a predetermined angle from the feature points on the human body.

[0011] Alternatively, the step of determining judgment conditions for estimating the person's behavior by combining the first analysis result based on the synchronized first video data and the second analysis result based on the synchronized second video data may include the steps of determining whether a first predicted behavior based on the first analysis result and a second predicted behavior based on the second analysis result match, estimating the reliability of at least one of the first analysis result or the second analysis result, and combining the first analysis result and the second analysis result based on the determined match and the estimated reliability.

[0012] Alternatively, the step of combining the first analysis result and the second analysis result to estimate the person's behavior based on the determined match and the estimated reliability may include a step of, if the first predicted behavior and the second predicted behavior match, estimating the matched predicted behavior as the person's behavior.

[0013] Alternatively, the step of combining the first analysis result and the second analysis result to estimate the person's behavior based on the determined match and the estimated reliability may include the step of estimating the first predicted behavior as the person's behavior if the first predicted behavior and the second predicted behavior do not match and the reliability of the second analysis result is less than a threshold value.

[0014] Alternatively, the step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability may include, when the first predicted behavior and the second predicted behavior do not match and the reliability of the second analysis result is equal to or greater than a threshold, estimating the person's behavior based on a judgment condition derived by combining the first analysis result and the second analysis result.

[0015] Alternatively, the first direction may be a frontal direction of the face and the second direction may be a lateral direction of the face.

[0016] According to one embodiment of the present disclosure, there is disclosed a computer program stored on a computer-readable storage medium. When executed by one or more processors, the computer program performs operations for monitoring behavior based on artificial intelligence. Here, the operations may include: synchronizing first image data captured in a first direction with respect to a face of a person who is a target of behavior monitoring and second image data captured in a second direction different from the first direction; generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the synchronized first image data and second image data using a pre-trained neural network model; and estimating the behavior of the person by combining the first analysis result based on the synchronized first image data and the second analysis result based on the synchronized second image data.

[0017] According to one embodiment of the present disclosure, there is disclosed a computing device for monitoring behavior based on artificial intelligence. The device may include a processor including at least one core, a memory including program code executable by the processor, and a network unit for acquiring video data. The processor may synchronize first video data captured in a first direction and second video data captured in a second direction different from the first direction based on the face of a person who is a target of behavior monitoring, generate an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the synchronized first video data and second video data using a pre-trained neural network model, and estimate the person's behavior by combining the first analysis result based on the synchronized first video data and the second analysis result based on the synchronized second video data. [Effects of the Invention]

[0018] The present disclosure can provide a method and apparatus that can comprehensively determine and accurately monitor what actions a person will take in a specific environment based on various sensing results. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a process for performing behavioral monitoring of a computing device according to one embodiment of the present disclosure. [Figure 3] 1 is a conceptual diagram illustrating an inference process broken down into different behaviors of a computing device according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a conceptual diagram illustrating a calculation process for correcting features of a neural network model according to an embodiment of the present disclosure. [Figure 5]1 is a flowchart illustrating an artificial intelligence-based activity monitoring method according to one embodiment of the present disclosure. [Figure 6] 1 is a flowchart illustrating a method for monitoring behavior in an online testing environment according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. The embodiments presented in this disclosure are provided to enable those skilled in the art to use or practice the contents of the present disclosure. Therefore, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the following embodiments.

[0021] Throughout the specification of the present disclosure, the same or similar reference numerals refer to the same or similar components. In addition, in order to clearly explain the present disclosure, reference numerals of parts that are not relevant to the explanation of the present disclosure may be omitted from the drawings.

[0022] The term "or" as used in this disclosure is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or otherwise clear from the context in this disclosure, "X uses A or B" should be understood to mean one of the natural inclusive permutations. For example, unless otherwise specified or otherwise clear from the context in this disclosure, "X uses A or B" can be interpreted as either X uses A, X uses B, or X uses both A and B.

[0023] The term "and / or" as used in this disclosure must be understood to indicate and include all possible combinations of one or more of the associated listed concepts.

[0024] The terms "comprises" and / or "comprising" as used in this disclosure should be understood to mean that the specified features and / or components are present. However, the terms "comprises" and / or "comprising" should not be understood to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0025] In this disclosure, unless otherwise specified or clear from the context as referring to the singular form, the singular should generally be construed as including "one or more."

[0026] The term "nth (n is a natural number)" used in this disclosure can be understood as an expression used to distinguish components of the present disclosure from one another based on a predetermined criterion, such as functional, structural, or convenience of description. For example, in this disclosure, components that perform different functional roles can be classified as a first component or a second component. However, components that are substantially identical within the technical concept of the present disclosure but must be distinguished for convenience of description can also be classified as a first component or a second component.

[0027] The term "acquire" as used in this disclosure may be understood to refer to generating or receiving data in an on-device form, as well as receiving data from an external device or system via a wireless communication network.

[0028] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood to refer to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a portion thereof, hardware or a portion thereof, or a combination of software and hardware. Here, a "module" or "unit" may refer to a unit composed of a single element or a unit expressed as a combination or collection of multiple elements. For example, as a concept of connotation, a "module" or "unit" may refer to a hardware element or a collection of hardware elements of a computing device, an application program that achieves a specific software function, a processing procedure implemented by the execution of software, or a collection of instructions for executing a program. Furthermore, as a broad concept, a "module" or "unit" may refer to a computing device itself that constitutes a system, or an application executed on a computing device. However, the above concepts are merely examples, and the concepts of "module" and "unit" may be defined in various ways within the scope of understanding of those skilled in the art based on the contents of this disclosure.

[0029] The term "model" as used in this disclosure may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a collection of software units for solving a specific problem, or an abstract model of a processing process for solving a specific problem. For example, a deep learning "model" may refer to a system generally implemented as a neural network that has problem-solving capabilities through learning. Here, a neural network may have problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A deep learning "model" may include a single neural network or a neural network ensemble in which multiple neural networks are combined.

[0030] As used in this disclosure, the term "image" refers to multidimensional data composed of discrete image elements. In other words, "image" can be understood as a term referring to a digital representation of an object that can be viewed by the human eye. For example, "image" can refer to multidimensional data composed of elements that correspond to pixels in a two-dimensional image. "Image" can refer to multidimensional data composed of elements that correspond to voxels in a three-dimensional image.

[0031] The explanations of the above terms are intended to facilitate understanding of the present disclosure. Therefore, unless the above terms are explicitly stated as matters limiting the contents of the present disclosure, care should be taken not to use them in a way that limits the technical ideas of the contents of the present disclosure.

[0032] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0033] The computing device 100 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive data processing and calculations, or may be a software-based computing environment connected via a communication network. For example, the computing device 100 may be a server that performs intensive data processing functions and shares resources, or a client that shares resources by interacting with the server. The computing device 100 may also be a cloud system that enables multiple servers and clients to interact with each other to comprehensively process data. The above description is merely an example of a type of computing device 100, and various types of computing device 100 may be configured within the scope that can be understood by those skilled in the art based on the contents of the present disclosure.

[0034] 1, a computing device 100 according to an embodiment of the present disclosure may include a processor 110, a memory 120, and a network unit 130. However, since FIG. 1 is merely an example, the computing device 100 may include other components for implementing a computer environment. Also, the computing device 100 may include only some of the disclosed components.

[0035] The processor 110 according to an embodiment of the present disclosure may be understood as a component including hardware and / or software for performing computing operations. For example, the processor 110 may read a computer program to perform data processing for machine learning. The processor 110 may process operations such as input data processing for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor 110 for performing such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The types of processor 110 described above are merely examples, and various types of processor 110 may be configured within the scope of what one skilled in the art would understand based on the present disclosure.

[0036] The processor 110 may perform preprocessing on multiple pieces of video data acquired for activity monitoring. Even when the same object or environment is captured, information expressed in the video data may vary depending on the capture time, capture direction, etc. Therefore, the processor 110, which uses multiple pieces of video data for activity monitoring, may perform synchronization to ensure the accuracy of combined analysis and judgment of the multiple pieces of video data. Here, synchronization may be understood as adjusting or aligning the capture times of multiple pieces of video data captured in the same environment based on the object to be detected. For example, unlike video data captured from the front of a person's face, video data captured from the side of a person's face may not contain facial information due to the movement of the person's head. Therefore, the processor 110 may perform preprocessing to synchronize the capture times of the frontal and side-view video data so that the two pieces of data can be used complementarily. Through this synchronization preprocessing, the processor 110 may stably combine the analysis results of both pieces of video data and improve the accuracy of activity estimation results for activity monitoring.

[0037] The processor 110 may use a pre-trained neural network model to generate an analysis result for each of a plurality of sensing objects based on video data captured by a person who is a target of activity monitoring. Here, the sensing object may be understood as a component of the video data that serves as a reference for estimating a person's activity. The analysis result for the sensing object may be information indicating what action a person will take based on the sensing object present in the video data. Specifically, the sensing object may be any one of a human body part, an object other than a person, the sound of an object related to a human action, or the time of an object related to a human action. The object related to a human action may be a human body part or an object that can change depending on a human action. The analysis result for the sensing object may be information about the person's action based on the human body part, object, object sound, or object time present in the video data. That is, the processor 110 may input the video data into the pre-trained neural network model and generate an analysis result for each sensing object present in the video data as basic data for detecting a specific human action performed in a specific environment for activity monitoring.

[0038] The processor 110 can infer a specific human behavior for activity monitoring based on the analysis results of each of the plurality of video data performed based on the above-described neural network model. The processor 110 can interpret the behavior of a person who is the subject of activity monitoring by combining the analysis results of the neural network model for each of the plurality of synchronized video data. Here, the processor 110 can use a ruleset to infer a human behavior based on the combined analysis results. The ruleset may be a set of behavior classes that are candidates for detection in a specific environment for activity monitoring and judgment conditions for each behavior class. The ruleset can be created, changed, or modified by an administrator who has built a specific environment for activity monitoring. That is, the processor 110 can infer a human behavior present in the video data by comprehensively determining the analysis results of each detection target present in the video data based on a ruleset that can be customized for a specific environment for activity monitoring. For example, assuming that the behavior monitoring environment is an environment for an online exam, the rule set generated by the client of the test proctor may be a set of judgment conditions for each of exam cheating and / or abnormal behavior that may be suspected of cheating, and cheating and / or abnormal behavior. The processor 110 may combine the analysis results of the neural network model for each of the frontal image data and the side image data to identify a combination result that matches the judgment conditions included in the rule set. Here, the combination of the analysis results may be understood as a computational process of comparing behavior predicted from the analysis results of the frontal image data with behavior predicted from the analysis results of the side image data, and combining the analysis results into a single underlying behavior based on the comparison results. Then, the processor 110 may determine whether the test taker engaged in cheating or abnormal behavior included in the rule set under the circumstances confirmed in the observation data based on the combination result that matches the judgment conditions.

[0039] In this way, the processor 110 can derive information about the basis behavior for determining what behavior a person will take based on the behavior predicted by each of the multiple video data, and can monitor the specific behavior of a person based on a rule set generated according to a specific environment. Thus, the processor 110 can perform detection and analysis with higher precision, accuracy, and stability than detection and analysis based on a single video data.

[0040] The memory 120 according to an embodiment of the present disclosure may be understood as a component including hardware and / or software for storing and managing data processed by the computing device 100. That is, the memory 120 may store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130. For example, the memory 120 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, multimedia card micro, card-type memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The memory 120 may also include a database system that manages data in a predetermined manner. The types of memory 130 described above are merely examples, and various configurations of the memory 120 are possible within the scope of what would be understood by one skilled in the art based on the present disclosure.

[0041] The memory 120 may structure and organize and manage data, a combination of data, and program code executable by the processor 110 required for the processor 110 to perform calculations. For example, the memory 120 may store video data acquired via the network unit 130 (described later). The memory 120 may store program code that causes the processor 110 to learn a deep learning model, program code that causes the processor 110 to estimate human behavior using the learned deep learning model, and various data calculated by executing the program code.

[0042] The network unit 130 according to an embodiment of the present disclosure may be understood as a component that transmits and receives data via any type of known wired or wireless communication system. For example, the network unit 130 may transmit and receive data using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultra wideband, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity (WiFi), near field communication (NFC), or Bluetooth. The above-described communication systems are merely examples, and various other wired or wireless communication systems for transmitting and receiving data by the network unit 130 may be used.

[0043] The network unit 130 may receive data necessary for the processor 110 to perform calculations via wired or wireless communication with any system, server, client, etc. The network unit 130 may also transmit data generated by calculations of the processor 110 via wired or wireless communication with any system, server, client, etc. The network unit 130 may also transmit data generated by calculations of the processor 110 via wired or wireless communication with any system, server, client, etc. For example, the network unit 130 may receive video data of a person who is a target of activity monitoring via wired or wireless communication with a sensing device such as a camera or a client equipped with a sensing device. The network unit 130 may also receive user input via a user interface implemented in the sensing device or the client equipped with a sensing device. The network unit 130 may transmit various data generated by calculations of the processor 110 based on the video data via wired or wireless communication with the sensing device or the client equipped with a sensing device.

[0044] 2 is a block diagram illustrating a process of performing activity monitoring of a computing device according to an embodiment of the present disclosure, and FIG. 3 is a conceptual diagram illustrating a detailed process of estimating activity-specific behaviors of a computing device according to an embodiment of the present disclosure.

[0045] 2, a computing device 100 according to an embodiment of the present disclosure may perform pre-processing on a plurality of video data of a person who is a target of activity monitoring. Here, the video data may include first video data 11 captured in a first direction based on the face of the person who is a target of activity monitoring, and second video data 13 captured in a second direction different from the first direction. That is, the computing device 100 may synchronize the first video data 11 captured in the first direction with the second video data 13 captured in the second direction different from the first direction based on the face.

[0046] For example, the first image data 11 and the second image data 13 may be images captured by a camera installed in a space for the online test. Here, a sensing device such as a camera may be a component of a client held by the test taker. A monitoring device may be installed in the test space to capture the test taker's face from the front and side, respectively. When the online test begins, the sensing device included in the test taker's client may generate first image data 11 capturing the test taker's face from the front and second image data 13 capturing the test taker's face from the side. The computing device 100 may acquire the first image data 11 and the second image data 13 generated by the test taker's client via wired or wireless communication with the test taker's client. The computing device 100 may synchronize the first image data 11 and the second image data 13 to use them together in an analysis by the neural network model 200, which will be described later. Because the second image data 13 is captured from the side of a person, it may not be possible to properly represent the person's face depending on the person's movement or the shooting angle. Therefore, in order to accurately and stably perform analysis of the face included in the body part of the person being sensed, the computing device 100 can synchronize the first image data 11 and the second image data 13.

[0047] The computing device 100 may input the synchronized first image data and the synchronized second image data to the neural network model 200. Here, the neural network model 200 may include a first model 210 that tracks a person's gaze based on image data captured in a first direction and a second model 220 that estimates a person's pose based on image data captured in a second direction. That is, the computing device 100 may input the synchronized first image data to the first model 210 optimized for a task of gaze tracking. Then, the computing device 100 may input the synchronized second image data to the second model 220 optimized for a task of pose estimation.

[0048] For example, the first model 210 may receive video data captured from the front of a person's face and track the gaze of the person present in the video data. Specifically, the first model 210 may extract a facial region from video data of the person and generate a cropped image. The first model 210 may extract features based on the cropped image to recognize the person's eyes. The first model 210 may then track the person's gaze by analyzing the movement and changes of the pupils included in the recognized eyes. For such gaze tracking, the first model 210 may include a neural network optimized for image processing. The first model 210 may be trained based on supervised learning, semi-supervised learning, unsupervised learning, self-supervised learning, etc.

[0049] The second model 220 can receive video data captured from the side of a person's face and detect the pose of the person in the video data. Specifically, the second model 220 can receive the video data, classify body parts and background based on a plurality of feature points for identifying the person's pose, and generate a mask for the body parts. The second model 220 can then analyze the mask for the body parts to estimate the pose the person is assuming. For this pose estimation, the second model 220 can include a neural network optimized for image processing. The second model 220 can be trained based on supervised learning, semi-supervised learning, unsupervised learning, self-supervised learning, etc.

[0050] The computing device 100 may generate an analysis result for at least one sensing item included in each of a plurality of sensing objects through a neural network model 200 to which a plurality of synchronized image data are input. The computing device 100 may generate an analysis result for each sensing item through a first model 210 to which synchronized first image data is input. The computing device 100 may generate an analysis result for each sensing item through a second model 220 to which synchronized second image data is input. Here, the sensing item of the first model 210 and the sensing item of the second model 220 may be the same or different.

[0051] For example, if the sensing object is a human body part, the subclass of the sensing object can be divided into a face, an arm, etc. The face can be further divided into eyes, a nose, a mouth, and ears. The arm can be further divided into a hand, a palm, a finger, etc. The sensing item is state information that can be sensed based on each of the subclasses of the sensing object, and can be a gaze direction, whether or not there is speech, a hand position, a palm direction, etc. In other words, the sensing item can indicate a specific state or appearance that can appear when the subclass of the sensing object moves or changes due to a human action.

[0052] In other words, the computing device 100 may input the synchronized first video data to the pre-trained first model 210 to generate a first analysis result 15 including analysis results for various sensing items. The computing device 100 may input the synchronized second video data to the pre-trained second model 220 to generate a second analysis result 17 including analysis results for various sensing items. Here, each of the first analysis result 15 and the second analysis result 17 may include analysis results matching each of sensing items such as gaze direction, presence or absence of speech, hand position, palm direction, etc.

[0053] For example, assuming that the behavior monitoring environment is an environment for an online exam, an analysis result matching a gaze direction may indicate a detection result of whether the examinee is gazing at a display where they are checking a test paper. An analysis result matching whether the examinee is speaking may indicate a detection result of whether the examinee's mouth shape has changed. An analysis result matching a hand position may indicate a detection result of whether the examinee's left or right hand is moving within a reference space determined by the examinee's body and the desk arrangement. The analysis results matching the above-mentioned sensed items may be included in the first analysis result 15 and the second analysis result 17, respectively. In this manner, the computing device 100 may individually generate an analysis result for at least one sensed item included in each of the plurality of sensed objects using the first model 210 and the second model 220. Through this process, the computing device 100 may acquire various information from the video data captured at various angles to be used for calculations for behavior estimation, which will be described later.

[0054] Referring to FIG. 2 , the computing device 100 can combine multiple analysis results generated via a neural network model 200 to estimate the behavior of a monitored person. The computing device 100 can combine a first analysis result 15 generated via a first model 210 and a second analysis result 17 generated via a second model 220 to generate a behavior estimation result 19. Here, the computing device 100 can utilize a first predicted behavior inferred from the first analysis result 15 and a second predicted behavior inferred from the second analysis result 19 to combine the first analysis result 15 and the second analysis result 17. The computing device 100 can also utilize the reliability of each of the first analysis result 15 and the second analysis result 17 to combine the first analysis result 15 and the second analysis result 17.

[0055] Specifically, the computing device 100 may estimate a first predicted behavior based on the first analysis result 15 and a second predicted behavior based on the second analysis result 17 based on a predetermined rule set. The computing device 100 may determine whether the estimated first predicted behavior and the estimated second predicted behavior match. The computing device 100 may also analyze at least one of the first model 210 and the second model 220 to estimate at least one of the reliability of the first analysis result 15 and the reliability of the second analysis result 17. The computing device 100 may generate basis information corresponding to a determination condition for combining the first analysis result 15 and the second analysis result 17 to predict behavior based on the determination of whether the first predicted behavior and the second predicted behavior match and the reliability of at least one of the first analysis result 15 and the second analysis result 17.

[0056] For example, referring to FIG. 3 , if the first analysis result 15 is derived based on the sensing item of gaze direction, ie, “the examinee is looking away from the display to check the test paper,” the computing device 100 can infer the first expected behavior as cheating from the first analysis result 15 based on a predetermined rule set. That is, the computing device 100 can infer the first expected behavior as cheating by confirming that the first analysis result 15 matches the criteria for determining cheating included in the predetermined rule set. If the second analysis result 17 is derived based on the sensing item of body part, ie, “the lower end of the examinee's elbow is outside the reference area,” the computing device 100 can infer the second expected behavior as abnormal behavior from the second analysis result 17 based on a predetermined rule set. That is, the computing device 100 can infer the second expected behavior as abnormal behavior by confirming that the second analysis result 17 matches the criteria for determining abnormal behavior included in the predetermined rule set. In this way, when the first expected behavior and the second expected behavior do not match, the computing device 100 can combine the first analysis result 15 and the second analysis result 17 as "the lower end of the examinee's elbow is outside the reference area, and the examinee is looking away from the display." Here, the result of the above-mentioned combination may vary depending on the reliability of the first analysis result 15 and the second analysis result 17. For example, when the reliability of the second analysis result 17 is less than a critical value, the computing device 100 can derive the first analysis result, "the examinee is looking away from the display," as the result of the combination. When the reliability of the second analysis result 17 is equal to or greater than the critical value, the computing device 100 can derive the above-mentioned combination, "the lower end of the examinee's elbow is outside the reference area, and the examinee is looking away from the display." However, the computing device 100 may also derive the accuracy of the combination result based on a weighted sum of the accuracy of the first analysis result 15 and the accuracy of the second analysis result 17 so as to give a greater influence to the first analysis result 15 in the final behavioral estimation.The computing device 100 may also derive a correlation between the combination result and misconduct or abnormal behavior based on a weighted sum of the correlation between the first analysis result 15 and misconduct or abnormal behavior and the correlation between the second analysis result 17 and misconduct or abnormal behavior. Here, the correlation may be understood as a quantitative indicator indicating the degree to which a particular analysis result influences the determination of misconduct or abnormal behavior. That is, the content of the combination result may be derived as the above-mentioned "the lower end of the examinee's elbow is outside the reference area, and the examinee is looking away from the display," while the result of the weighted sum may be reflected in the accuracy or correlation of the combination result. The accuracy or correlation of the combination result may then be reflected in the computing process of the computing device 100 for final behavior estimation. In this way, the computing device 100 may enhance the accuracy of behavior estimation by using a complementary combination of analysis results of data captured in different directions (or angles) to monitor behavior.

[0057] The computing device 100 may use a rule set determined in advance according to a specific environment for activity monitoring to derive the activity inference result 19. Specifically, the computing device 100 may identify judgment conditions for each activity class included in the predetermined rule set that match the combined result of the first analysis result 15 and the second analysis result 17. Here, when the first analysis result 15 and the second analysis result 17 are combined by a weighted sum, the computing device 100 may derive the activity inference result 19 corresponding to the activity class by filtering the judgment conditions for each activity class included in the predetermined rule set, reflecting the accuracy or correlation with the combined result of the first analysis result 15 and the second analysis result 17. The computing device 100 may infer a human activity that should be detected for activity monitoring in a specific environment from the activity class corresponding to the identified judgment conditions.

[0058] For example, assuming that the environment for behavior monitoring is an environment for an online test, the computing device 100 can identify a judgment condition of a first behavioral class for fraudulent behavior or a judgment condition of a second behavioral class for anomalous behavior that matches the combined result of the first analysis result 15 and the second analysis result 17 by screening a predetermined rule set. The computing device 100 can derive the first behavioral class for fraudulent behavior or the second behavioral class for anomalous behavior that corresponds to the identified judgment condition as the behavior estimation result 19.

[0059] FIG. 4 is a conceptual diagram illustrating a calculation process for correcting the features of a neural network model according to an embodiment of the present disclosure.

[0060] 4, a computing device 100 according to an embodiment of the present disclosure may input synchronized first image data 21 to a first model to extract feature points for each of a person's face and body. The computing device 100 may input synchronized second image data 25 to a second model to extract feature points for each of a person's face and body. Here, if the feature points for the person's face extracted from the synchronized second image data 25 deviate from the feature points for the person's body by more than a predetermined angle, the computing device 100 may correct the feature points extracted from the synchronized second image data 25 based on the feature points extracted from the synchronized first image data 21.

[0061] For example, it is assumed that the synchronized first image data 21 is an image captured from the front of a person, and the synchronized second image data 25 is an image captured from the side of a person. Unlike the first image data 21, in which face capture is less affected by human movement, face capture in the second image data 25 is more affected by human movement. That is, if a person moves, such as when the angle of the person's face rotates more than 45 degrees from the body, the second model cannot correctly extract feature points for the person's face from the second image data 25, resulting in unstable analysis. Therefore, if the feature points for the face extracted from the second image data 25 and the feature points for the body are misaligned by more than 45 degrees, the computing device 100 may adjust the coordinates of the feature points extracted from the second image data 25 based on the coordinate system 30 of the feature points extracted from the first image data 21, which is less affected by human movement. The computing device 100 may generate an output of the second model using the feature points of the second image data 25 whose coordinates have been adjusted. That is, the second model may generate an analysis result for each sensing item using feature points corrected by the feature points extracted by the first model. Through such correction, the computing device 100 may ensure the security of data analysis.

[0062] FIG. 5 is a flowchart illustrating an artificial intelligence-based activity monitoring method according to one embodiment of the present disclosure.

[0063] Referring to FIG. 5, a computing device 100 according to an embodiment of the present disclosure may synchronize first image data captured in a first direction and second image data captured in a second direction different from the first direction based on the face of a person who is a target of behavior monitoring (S110). For example, the computing device 100 may synchronize first image data generated through a camera installed in front of the person's face with second image data generated through a camera installed to the side of the person's face. Here, synchronization of the first image data and the second image data may be understood as an operation of aligning the time points at which the person's face was captured. Through synchronization, the computing device 100 may effectively use the first image data and the second image data together as data for behavior estimation.

[0064] The computing device 100 may generate an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the first and second video data synchronized in step S110 using a pre-trained neural network model (S120). Here, the plurality of sensing objects may include at least one of a human body part, an object other than the human, an object sound related to the human's action, or an object time related to the human's action. The sensing item may be status information identified based on a subclass of the sensing object and may be status information that can change depending on the human's action. For example, the computing device 100 may input the first video data synchronized with the second video data to a first model that tracks gaze based on video data captured from the front of the face, and generate an analysis result for at least one sensing item included in at least one of a body part, an object, an object sound, and an object time. The computing device 100 may input second image data synchronized with the first image data to a second model that estimates a person's pose based on image data captured from the side of the face, and generate an analysis result for at least one sensing item included in at least one of a body part, an object, an object sound, and an object time. The sensing object and sensing item that serve as analysis criteria for each of the first model and the second model may be the same or different.

[0065] The computing device 100 may infer a human behavior by combining the first analysis result generated in step S120 and the second analysis result generated in step S120 (S130). Specifically, the computing device 100 may determine whether a first predicted behavior based on the first analysis result and a second predicted behavior based on the second analysis result match. Here, the predicted behavior may correspond to a predicted behavior included in a rule set predetermined by the behavior monitoring environment. The computing device 100 may then estimate the reliability of at least one of the first analysis result and the second analysis result. The computing device 100 may infer a human behavior by combining the first analysis result and the second analysis result based on the previously determined match and the estimated reliability. For example, if the first predicted behavior and the second predicted behavior match, the computing device 100 may infer the matched predicted behavior as the human behavior. If the first predicted behavior and the second predicted behavior do not match and the reliability of the second analysis result is less than a threshold value, the computing device 100 may infer the first predicted behavior as the human behavior. If the first predicted behavior and the second predicted behavior do not match and the reliability of the second analysis result is equal to or greater than a threshold value, the computing device 100 may predict the person's behavior based on a judgment condition derived by combining the first analysis result and the second analysis result. Here, the person's behavior predicted based on each predicted behavior or judgment condition may be determined based on a rule set predetermined by the behavior monitoring environment. Here, the threshold value may be a value predetermined by an administrator for behavior monitoring.

[0066] FIG. 6 is a flow chart illustrating a method for monitoring behavior in an online testing environment according to one embodiment of the present disclosure.

[0067] 6, a computing device 100 according to an embodiment of the present disclosure may generate an online exam based on a user request input via a promoter client for the online exam (S210). Here, environmental conditions for the online exam, a rule set for monitoring examinee behavior, etc. may be determined based on the user request input via the promoter client. For example, the computing device 100 may determine a rule set including a period for acquiring observation data, definitions and judgment conditions for cheating 61 or abnormal behavior, etc., based on the user request input via the promoter client. After the rule set is generated based on the user request, it may be dynamically updated as the computing device 100 repeatedly performs behavior estimation.

[0068] When an online exam is generated (S210), the computing device 100 can acquire images of the examinee at a predetermined interval (S220). For example, the computing device 100 can acquire front and side images of the examinee's face at intervals of 100 ms to 1 s via wired or wireless communication with a sensing device installed in the exam space. Here, the sensing device may be a component installed in the examinee's client or may be a component of the computing device 100. The observation data acquisition interval can be determined in advance in step S210 depending on the environmental conditions of the online exam.

[0069] The computing device 100 may synchronize the front image and the side image (S230). Even though the front image and the side image are images generated by photographing the same subject (i.e., the examinee), the computing device 100 may perform a pre-processing operation to align the photographing time points of the front image and the side image based on the examinee's face for stable analysis.

[0070] The computing device 100 may input the synchronized front image 51 and the synchronized side image 52 into a neural network model to perform an analysis for each sensing item (S240). The computing device 100 may input the synchronized front image 51 into a pre-trained first model 210 to generate a first analysis result 53 for a sensing item matched by the first model 210. The computing device 100 may input the synchronized side image 52 into a pre-trained second model 220 to generate a second analysis result 54 for a sensing item matched by the second model 220. The computing device 100 may estimate a first predicted behavior 55 based on the first analysis result 53. The first predicted behavior 55 may be understood as a behavior determined when fraud or abnormal behavior is predicted based on the first analysis result 53. The computing device 100 may estimate a second predicted behavior 56 based on the second analysis result 54. The second predicted behavior 56 can be understood as a behavior determined when fraudulent behavior or abnormal behavior is predicted based on the second analysis result 54.

[0071] The computing device 100 may determine whether the first predicted behavior 55 and the second predicted behavior 56 match (S251). If the first predicted behavior 55 and the second predicted behavior 56 match, the computing device 100 may determine the test taker's behavior based on the matched predicted behavior (S255). If the first predicted behavior 55 and the second predicted behavior 56 do not match, the computing device 100 may determine whether the reliability of the second analysis result 54 is greater than or equal to a critical value (S261). If the reliability of the second analysis result 54 is less than the critical value, the computing device 100 may determine the test taker's behavior based on the first predicted behavior 55 (S265). If the reliability of the second analysis result 54 is greater than or equal to the critical value, the computing device 100 may combine the first analysis result 53 and the second analysis result 54 (S270). The computing device 100 may then estimate the test taker's behavior based on the combination result (S280).

[0072] For example, if the first predicted behavior 55 and the second predicted behavior 56 match as cheating or abnormal behavior, the computing device 100 may determine the test taker's behavior as the matching behavior. If the first predicted behavior 55 is cheating and the second predicted behavior 56 is abnormal behavior, and they do not match, the computing device 100 may determine whether the reliability of the second analysis result 54 is equal to or greater than a threshold value. If the reliability of the second analysis result 54 is less than the threshold value, the computing device 100 may determine the test taker's behavior as cheating, which is the first predicted behavior 55. If the reliability of the second analysis result 54 is equal to or greater than the threshold value, the computing device 100 may combine the first analysis result 53 and the second analysis result 54 and determine whether the result of the combination exists in a predetermined rule set. If the computing device 100 determines that the result of the combination exists as a judgment condition included in the predetermined rule set, it may determine the test taker's behavior as cheating or abnormal behavior, which is the behavior according to the determined judgment condition.

[0073] The various embodiments of the present disclosure described above can be combined with additional embodiments and can be modified within the scope that can be understood by those skilled in the art based on the above detailed description. It should be understood that the embodiments of the present disclosure are illustrative in all respects and are not limiting. For example, each component described as a single type can also be implemented in a distributed form, and similarly, components described as distributed can be implemented in a combined form. Therefore, all modifications and variations derived from the meaning, scope, and equivalent concepts of the claims of the present disclosure should be construed as being within the scope of the present disclosure.

Claims

1. 1. An artificial intelligence based behavioral monitoring method executed by a computing device including at least one processor, comprising: synchronizing first image data captured in a first direction and second image data captured in a second direction different from the first direction based on a face of a person who is a target of behavior monitoring; generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on each of the synchronized first and second image data using a pre-trained neural network model; estimating the person's behavior by combining a first analysis result based on the synchronized first video data and a second analysis result based on the synchronized second video data; A method comprising:

2. The sensing item is status information identified based on a subclass of the sensing object, The state information is changeable by the person's actions. The method of claim 1.

3. The plurality of sensing targets may include, in addition to the body parts of the person, at least one of an object other than the person, a sound of an object related to the person's actions, or a time of an object related to the person's actions. The method of claim 1.

4. The neural network model is a first model that tracks a person's gaze based on video data captured in the first direction; a second model for estimating a pose of a person based on the image data captured in the second direction; Including, The method of claim 1.

5. generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the synchronized first image data and the synchronized second image data using the pre-trained neural network model, correcting the feature points extracted from the synchronized second image data based on a coordinate system of the feature points extracted by inputting the synchronized second image data into the second model when the feature points of the human face extracted by inputting the synchronized second image data into the second model deviate by more than a predetermined angle from the feature points of the human body; The method of claim 4, comprising:

6. The step of determining a judgment condition for estimating the human behavior by combining a first analysis result based on the synchronized first video data and a second analysis result based on the synchronized second video data includes: determining whether a first predicted behavior based on the first analysis result and a second predicted behavior based on the second analysis result match; estimating a reliability of at least one of the first analysis result or the second analysis result; combining the first analysis result and the second analysis result based on the determined match and the estimated reliability, and estimating the human behavior; The method of claim 1 , comprising:

7. Inferring the human behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability, If the first predicted behavior and the second predicted behavior match, estimating the matched predicted behavior as the behavior of the person; The method of claim 6, comprising:

8. Inferring the human behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability, If the first predicted behavior and the second predicted behavior do not match and the reliability of the second analysis result is less than a threshold value, estimating the first predicted behavior as the behavior of the person; The method of claim 6, comprising:

9. Inferring the human behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability, When the first predicted behavior and the second predicted behavior do not match and the reliability of the second analysis result is equal to or greater than a threshold, estimating the person's behavior based on a judgment condition derived by combining the first analysis result and the second analysis result; The method of claim 6, comprising:

10. the first direction is a front direction of the face, The second direction is a lateral direction of the face. The method of claim 1.

11. A computer program stored on a computer-readable storage medium, the computer program, when executed by one or more processors, performing operations for monitoring behavior based on artificial intelligence; The operation is an operation of synchronizing first image data captured in a first direction based on a face of a person who is a target of behavior monitoring and second image data captured in a second direction different from the first direction; generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on the synchronized first image data and the synchronized second image data using a pre-trained neural network model; an operation of estimating the behavior of the person by combining a first analysis result based on the synchronized first video data and a second analysis result based on the synchronized second video data; a computer program comprising:

12. 1. A computing device for monitoring behavior based on artificial intelligence, comprising: a processor including at least one core; a memory containing program code executable by the processor; a network unit for acquiring video data; Including, The processor: Based on the face of a person who is a target of behavior monitoring, first image data captured in a first direction and second image data captured in a second direction different from the first direction are synchronized; generating an analysis result for at least one sensing item included in each of a plurality of sensing objects based on each of the synchronized first image data and the second image data using a pre-trained neural network model; estimating the person's behavior by combining a first analysis result based on the synchronized first video data and a second analysis result based on the synchronized second video data; Device.

Images