Artificial intelligence-based object behavior analysis method and apparatus

Abstract

Disclosed are an artificial intelligence-based object behavior analysis method and behavior analysis apparatus. The object behavior analysis method may comprise the steps of: obtaining an image in which an object appears; converting a color space of an object area in the image into another color space; extracting one or more body parts of the object from the object area converted into the other color space; obtaining a frequency feature for each body part on the basis of movement of the extracted one or more body parts according to time; and determining analysis result data for a behavior of the object by using a neural network-based behavior classifier using the frequency feature for each body part as an input.

Description

Method and device for analyzing object behavior based on artificial intelligence

[0001] The present disclosure relates to an object behavior analysis technology based on artificial intelligence.

[0002] For medical, veterinary, and behavioral research, it is important to observe and accurately analyze animal behavior. Conventional techniques exist for observing and analyzing animal behavior. According to these conventional techniques, feature points are extracted from the animal's movements observed at predetermined intervals, and the animal's behavior is analyzed based on these extracted feature points.

[0003] A method for analyzing the behavior of an object based on artificial intelligence according to one embodiment may include: acquiring an image in which the object appears; converting the color space of an object region within the image to another color space; extracting one or more body parts of the object from the object region converted to the other color space; acquiring frequency features for each body part based on the movement over time of the extracted one or more body parts; and determining analysis result data for the behavior of the object using a neural network-based behavior classifier that takes the frequency features for each body part as input.

[0004] The step of converting to another color space may include converting the RGB (red, green, blue) color space, which is the color space of the object area, to the HSV (hue saturation value) color space, which is the other color space.

[0005] The above extraction step may include the step of extracting one or more body parts of the object by masking one or more body parts of the object from the object region converted to the HSV color space based on the color, saturation, and brightness of the object region converted to the HSV color space.

[0006] The step of acquiring the frequency features may include: a step of measuring positional movement data of coordinates over time of the main feature values ​​corresponding to the one or more body parts based on the main feature values ​​corresponding to the one or more body parts; a step of tracking the movement over time of the one or more body parts based on the positional movement data of coordinates over time of the main feature values ​​corresponding to the one or more body parts; and a step of acquiring frequency features for each body part based on the tracked movement over time of the one or more body parts.

[0007] The above main feature values ​​may include the x-coordinate of the object's ear, the y-coordinate of the object's ear, the width of the object's body, the height of the object's body, the x-coordinate of the object's body, the y-coordinate of the object's body, the center coordinate of the object's body, and the center coordinate of the object's ear.

[0008] The step of acquiring the above frequency features may include the step of acquiring frequency features for each body part by performing a continuous time wavelet transform on the frequency measured based on the movement over time of one or more extracted body parts.

[0009] The above behavior classifier may be trained to receive as training data frequency features for each body part corresponding to a predetermined behavior and positional movement data of coordinates over time of key feature values ​​corresponding to the predetermined behavior, and to output a probability value that the behavior of the object corresponds to each of the different predetermined behaviors based on the frequency features.

[0010] The above behavior classifier may be a neural network including a convolutional layer and a fully connected layer.

[0011] The above-mentioned determining step may include determining analysis result data for the behavior of the object based on probability values ​​that the behavior of the object corresponds to each of the predefined different behaviors.

[0012] An object behavior analysis device based on artificial intelligence according to one embodiment includes a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor can control the behavior analysis device to determine analysis result data for the object's behavior by using a neural network-based behavior classifier that takes as input the frequency features of each body part, and the behavior analysis device acquires an image in which the object appears, converts the color space of the object region within the image to another color space, extracts one or more body parts of the object from the object region converted to the other color space, acquires frequency features for each body part based on the movement over time of the extracted one or more body parts.

[0013] The processor can control the behavior analysis device so that the behavior analysis device converts the RGB (red, green, blue) color space, which is the color space of the object area, into the HSV (hue saturation value) color space, which is another color space.

[0014] The processor can control the behavior analysis device to extract one or more body parts of the object by masking one or more body parts of the object from the object region converted into the HSV color space based on the color, saturation, and brightness of the object region converted into the HSV color space.

[0015] The processor can control the behavior analysis device so that it measures positional movement data of coordinates over time of a key feature value corresponding to one or more body parts based on a key feature value corresponding to one or more body parts, tracks the movement over time of one or more body parts based on the positional movement data of coordinates over time of a key feature value corresponding to one or more body parts, and acquires frequency features for each body part based on the tracked movement over time of one or more body parts.

[0016] The above main feature values ​​may include the x-coordinate of the object's ear, the y-coordinate of the object's ear, the width of the object's body, the height of the object's body, the x-coordinate of the object's body, the y-coordinate of the object's body, the center coordinate of the object's body, and the center coordinate of the object's ear.

[0017] The processor can control the behavior analysis device to acquire frequency features for each body part by performing a continuous time wavelet transform on the frequency measured based on the movement over time of one or more extracted body parts.

[0018] The above behavior classifier may be trained to receive as training data frequency features for each body part corresponding to a predetermined behavior and positional movement data of coordinates over time of key feature values ​​corresponding to the predetermined behavior, and to output a probability value that the behavior of the object corresponds to each of the different predetermined behaviors based on the frequency features.

[0019] The above behavior classifier may be a neural network including a convolutional layer and a fully connected layer.

[0020] The processor can control the behavior analysis device so that the behavior analysis device determines analysis result data for the behavior of the object based on probability values ​​that the behavior of the object corresponds to each of the different predefined behaviors.

[0021] According to one embodiment, animal behavior analysis technology can be provided that can be applied to drug or gene therapy classification, drug or gene therapy screening, developmental disorder classification, autism model behavior classification, disease research including early detection of disease, toxicology research, side effect research, learning and memory process research, anxiety research, and various behavior analyses.

[0022] According to one embodiment, reliability and accuracy can be improved and costs can be reduced compared to conventional human-dependent passive animal behavior analysis.

[0023] According to one embodiment, the limited accessibility of conventional animal behavior analysis technology, which requires complex and high hardware costs based on deep learning, can be alleviated.

[0024] According to one embodiment, the limitations of rigid behavioral analysis criteria based on characteristic points of movement measured over a fixed period of time can be overcome.

[0025] According to one embodiment, the limitations of conventional techniques for tracking the center point of a body or tracking the boundaries of a body, such as the fact that learning takes a long time and the performance of analysis deteriorates when the environment changes, can be overcome.

[0026] According to one embodiment, it is possible to overcome the limitations of behavioral analysis using key body parts, which are difficult to measure various body parts with a limited imaging device, and to improve the accuracy of classification.

[0027] According to one embodiment, animal behavior can be accurately analyzed without the excessive training data and high-spec computers required for conventional deep learning-based animal behavior analysis.

[0028] FIG. 1 is a diagram illustrating an overview of an object behavior analysis system according to one embodiment.

[0029] FIG. 2 is a flowchart illustrating a method for analyzing the behavior of an object based on artificial intelligence according to one embodiment.

[0030] FIG. 3 is a drawing illustrating an example for obtaining an image in which an object appears according to one embodiment.

[0031] FIG. 4 is a drawing illustrating an example of a user interface related to extracting body parts of an object according to one embodiment.

[0032] FIG. 5 is a drawing for explaining an extracted body part of an object according to one embodiment.

[0033] FIG. 6 is a diagram illustrating key feature values ​​corresponding to body parts of an object according to one embodiment.

[0034] FIG. 7 is a diagram illustrating the main feature values ​​and object regions corresponding to the body parts of an object according to one embodiment.

[0035] FIGS. 8 to 10 are drawings illustrating graphs showing positional movement data of coordinates over time of key feature values ​​corresponding to body parts of an object according to one embodiment.

[0036] FIGS. 11 and 12 are drawings illustrating graphs showing the positional movement data of coordinates over time of key feature values ​​corresponding to body parts of an object according to one embodiment, analysis result data, and frequencies measured based on the movement of body parts over time.

[0037] FIG. 13 is a graph illustrating the positional movement data of coordinates over time of a major feature value corresponding to a body part of an object according to one embodiment, and the frequency measured based on the movement of the body part over time.

[0038] FIG. 14 is a drawing illustrating a behavior classifier according to one embodiment.

[0039] FIG. 15 is a diagram illustrating the configuration of an artificial intelligence-based behavior analysis device according to one embodiment.

[0040] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0041] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0042] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0043] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0044] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0045] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0046] As used in this document, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, that performs certain roles. However, "part" is not limited to software or hardware. "Part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. For example, "part" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. Additionally, '~part' may include one or more processors.

[0047] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0048] FIG. 1 is a diagram illustrating an overview of an object behavior analysis system according to one embodiment.

[0049] The behavior analysis system described herein may provide a behavior analysis method for extracting body parts from an image of an object based on the color of the image in which the object is captured, and for analyzing the behavior of an object based on at least one of artificial intelligence technology, deep learning technology, and machine learning. It may provide an end-to-end or terminal-type automated behavior analysis method for an object that enables insight into the nature of changes in the object's behavior and can objectively discover subtle or minute changes in patterned behavior. The behavior analysis system described herein may provide a lightweight behavior analysis method by eliminating the need for learning in the process of extracting at least one of an object region and an object's body part from an image in which the object appears. Accordingly, the behavior analysis method described herein can accurately analyze animal behavior using only a computer with much lower specifications than that required by conventional technology (e.g., without a GPU). Furthermore, the behavior analysis method described herein can accurately analyze animal behavior using only a smaller amount of input values ​​than that required by the model of conventional technology. The behavior analysis method described in this specification can provide a method for classifying whether an object has a brain disease, a method for classifying whether an object has a developmental disorder, or a method for classifying autistic behavior by analyzing the object's behavior.

[0050] Referring to FIG. 1, the behavior analysis device (120) can acquire an image (110) in which an object appears. The behavior analysis device (120) can extract colors based on the RGB color space from an object region within the image and can convert the RGB color space to the HSV color space. The behavior analysis device (120) can extract one or more body parts of the object from the object region converted to a different color space and acquire frequency features for each body part based on the movement over time of the extracted one or more body parts. The behavior analysis device (120) can determine analysis result data (140) for the object's behavior by using a neural network-based behavior classifier (130) that takes the frequency features for each body part as input.

[0051] FIG. 2 is a flowchart illustrating a method for analyzing the behavior of an object based on artificial intelligence according to one embodiment.

[0052] Referring to FIG. 2, in step (210), the behavior analysis device can acquire an image in which an object appears. For example, the behavior analysis device can acquire an image in which an object appears using a camera. The camera used here may be a general camera capable of capturing color images without needing to be high-performance. The image in which an object appears acquired by the behavior analysis device using a camera may be in formats such as mp4, avi, cam, etc., but is not limited thereto. The behavior analysis device can determine analysis result data regarding the object's behavior in real time while acquiring images by continuously acquiring images in which an object appears even while performing the behavior analysis method. After acquiring images, the behavior analysis device may determine analysis result data regarding the object's behavior in a pipeline form with a delay of about 1 frame (30 seconds), but is not limited to the description in this specification.

[0053] The behavior analysis device can extract a representative image (or reference image) from an image in which an object appears. By extracting a representative image (or reference image) from an image in which an object appears, the accuracy of extracting body parts can be improved. The image described below may also be referred to as a representative image (or reference image). The behavior analysis device can extract colors from an object region within the image. For example, the behavior analysis device can extract colors from an object region within the image based on the RGB color space. In step (220), the behavior analysis device can convert the color space of the object region within the image to another color space. For example, the behavior analysis device can convert the RGB (red, green, blue) color space, which is the color space of the object region, to another color space, the HSV (hue saturation value) color space. The RGB color space has the limitation of not being able to distinguish subtle differences in color, whereas the HSV color space can distinguish subtle differences in color better than the RGB color space or grayscale, can distinguish wider dimensions and a wider range of colors than the RGB color space or grayscale, and can provide better body part contour performance. The behavior analysis device can convert the RGB color space, which is the color space of the object area, into another color space, the HSV color space. By converting the RGB color space, which is the color space of the object area, into another color space, the HSV color space, the behavior analysis method described herein can improve the accuracy of extracting one or more body parts of an object from an image in which the object appears. Through this, the behavior analysis method described herein can distinguish a wider dimension and a wider range of colors than conventional techniques, and thus can extract body parts of an object from an image more accurately.

[0054] Converting the RGB color space to the HSV color space is as follows. H, representing the hue, can be calculated as follows.

[0055]

[0056] S, representing saturation, can be calculated as follows.

[0057]

[0058] V, representing brightness, can be calculated based on the maximum value among R, G, and B as follows.

[0059]

[0060] In step (230), the behavior analysis device may extract one or more body parts of an object from an object area converted to a different color space. The behavior analysis device may extract one or more body parts from the object area based, for example, at least one of a preset color of the object's skin and fur, but is not limited thereto. The behavior analysis device may include the step of extracting one or more body parts of an object by masking one or more body parts of an object from an object area converted to the HSV color space based on the hue, saturation, and brightness of the object area converted to the HSV color space. The behavior analysis device may extract one or more body parts from the object area based, for example, at least one of the hue, saturation, and brightness of at least one of a preset color of the object's skin and fur. Here, one or more body parts of an object may include at least one of the object's ears, nose, paws, tail, and body, but are not limited to those described in this specification. The behavior analysis device can generate an image in which the background is represented in black and one or more extracted body parts of the object are represented in white by masking one or more body parts of the object from the object area.

[0061] In another embodiment, one or more body parts of an object can be extracted by masking one or more body parts of an object from an object region converted to the HSV color space based on the hue, saturation, and brightness of the object region converted to the HSV color space, and by redefining the contours of the mask based on a contouring technique. The behavior analysis device can accurately extract the contours of the mask by applying a contouring technique to the pixels that are missing during the process of masking one or more body parts of the object by setting the range of the HSV color space to a relatively small size. The contouring technique can define the contours of the object's body using cv2.findContours + cv2.fillPoly, which uses an additional algorithm from the OpenCV library, and as a result, the mask can be optimized to approximate the actual values. Additionally, for example, the behavior analysis device can automatically mark the largest contour as the body and apply a Gaussian blur filter to reduce noise and smooth edges. The behavior analysis method described in this specification extracts one or more body parts of an object from an image in which the object appears using the HSV color space, so that body parts can be accurately classified and behavior analysis of the object can be accurately performed even when there are other objects around the object or when the object is equipped with a device for experimentation (e.g., an optic probe).

[0062] In step (240), the behavior analysis device can acquire frequency features for each body part based on the movement over time of one or more extracted body parts. The behavior analysis device can measure positional movement data of coordinates over time of the main feature values ​​corresponding to one or more body parts based on the main feature values ​​corresponding to one or more body parts. The main feature values ​​may include, but are not limited to, at least one of the x-coordinate (EX) of the object's ear, the y-coordinate (EY) of the object's ear, the width (BW) of the object's body, the height (BH) of the object's body, the x-coordinate (BX) of the object's body, the y-coordinate (BY) of the object's body, the center coordinates of the object's body, and the center coordinates of the object's ear. At least one of the center coordinates of the body and the center coordinates of the object's ear may be used to track the movement of the object in real time. The main feature values ​​may have a format such as CSV, for example. Intermediate outputs during the performance of the behavior analysis method may be stored in a format such as CSV, for example. For example, the coordinates of the ear may be used for subsequent pose estimation of the object.

[0063] The behavior analysis device can track the movement over time of one or more body parts based on positional displacement data of coordinates of key feature values ​​corresponding to one or more body parts over time. The behavior analysis device can acquire frequency features for each body part based on the movement over time of the tracked one or more body parts. The behavior analysis device can acquire frequency features for each body part, for example, by performing a continuous time wavelet transform (CWT) on the frequencies measured based on the movement over time of the extracted one or more body parts. The frequency features for each body part obtained by performing a continuous time wavelet transform on the frequencies measured based on the movement over time of the extracted one or more body parts may have a format, for example, CSV. The continuous time wavelet transform may be a time-frequency transform suitable for analyzing abnormal signals. Here, an abnormal signal may mean that the frequency domain representation of a signal changes over time. A behavior analysis device can recognize frequency features (or high-frequency features) associated with subtle behaviors, such as grooming or sniffing, by decomposing key feature values ​​using variable-length wavelets (short for high frequencies and long for low frequencies). In a continuous-time wavelet transform, the number of frequency bands can be determined based on the time length of the image and the inference accuracy indicating the number of frequency bands.

[0064] In step (250), the behavior analysis device may determine analysis result data for the object's behavior by using a neural network-based behavior classifier that takes frequency features for each body part as input. The behavior classifier may be trained to determine analysis result data for the object's behavior based on frequency features by receiving frequency features for each body part corresponding to a predetermined behavior and data on the positional movement of coordinates over time of key feature values ​​corresponding to the predetermined behavior as training data. Here, the predetermined behavior may include at least one of grooming (G), walking (W), rehearing (R), and sniffing (S), but is not limited thereto. The frequency features for each body part corresponding to the predetermined behavior used as training data may be labeled based on the data on the positional movement of coordinates over time of key feature values ​​corresponding to the predetermined behavior. The frequency features for each body part used as training data may each have a predetermined behavior labeled. The behavior classifier may be a neural network including a convolution layer and a fully connected layer. The behavior classifier can output probability values ​​that the object's behavior corresponds to each of predefined different behaviors. The predefined different behaviors may include, but are not limited to, at least one of, for example, grooming (G), walking (W), rearing (R), and sniffing (S). The behavior classifier can output probability values ​​that the object's behavior corresponds to, for example, grooming (G), walking (W), rearing (R), and sniffing (S). The principal feature values ​​of the behavior classifier based on the object's position and velocity may be derived from body segment tracking. At least one of the probability values ​​that the object's behavior corresponds to each of the predefined different behaviors and the analysis result data regarding the object's behavior may be stored in a predefined directory.The simplicity and modularity of the behavior analysis device make debugging easy and transparent by allowing access to all intermediate results, which can significantly improve the reproducibility of the behavior analysis method.

[0065] The behavior analysis device can determine analysis result data for an object's behavior based on probability values ​​corresponding to each of the different predefined behaviors. For example, the behavior analysis device can determine analysis result data for an object's behavior based on the behavior with the highest probability value. For example, the behavior analysis device can determine the analysis result data for an object's behavior as grooming (G) if, among the probability values ​​corresponding to grooming (G), walking (W), rearing (R), and sniffing (S), the probability value corresponding to grooming (G) is the highest.

[0066] FIG. 3 is a drawing illustrating an example for obtaining an image in which an object appears according to one embodiment.

[0067] Referring to FIG. 3, the object (310) may be, for example, a mouse, but is not limited thereto. The camera (320) may capture an image of the object (310). Here, the camera (320) may be a general and universal camera capable of capturing a color image or a color video. Since the behavior analysis device extracts one or more body parts of the object based on the object's color, the camera (320) needs to be capable of capturing a color image or a color video, but there may be no other restrictions.

[0068] FIG. 4 is a drawing illustrating an example of a user interface related to extracting body parts of an object according to one embodiment.

[0069] The process of extracting body parts of an object from an image in which the object appears can be provided to the user in real time through a user interface related to extracting body parts of the object. Referring to FIG. 4, reference number (410) may represent the body color of the object. That is, the behavior analysis device can extract the body color of the object and provide it through the user interface. The behavior analysis device can extract the body, which is a body part of the object, based on the body color of the object. The behavior analysis device can extract the body of the object by masking the body of the object and provide it to the user interface. Reference number (420) may be a visualization of the respective hue, saturation, and brightness values ​​of HSV for the body color of the object. Additionally, reference number (430) may be a visualization of the respective hue, saturation, and brightness values ​​of HSV for the ear color of the object. In one embodiment, the user may arbitrarily modify the respective HSV values ​​for each body part calculated by the behavior analysis device through reference number (420) and reference number (430).

[0070] FIG. 5 is a drawing for explaining an extracted body part of an object according to one embodiment.

[0071] Referring to FIG. 5, reference number (510) may be a masked image from which the body of an object is extracted by masking the body of the object, and reference number (520) may be the skin color of the object. The skin color of the object (520) may be pre-set or may be recognized by a behavior analysis device. The behavior analysis device may extract one or more body parts (530, 540, 550, 560) of an object by masking one or more body parts of the object from the object area. In the embodiment of FIG. 5, the behavior analysis device may extract an ear (530), a nose (540), a foot (550), and a tail (560) from the object area, but is not limited thereto.

[0072] FIG. 6 is a diagram illustrating key feature values ​​corresponding to body parts of an object according to one embodiment.

[0073] Referring to FIG. 6, reference number (610) may be a mask defined by HSV segmentation of the object's body, and reference number (630) may be the object's Region of Interest (ROI) or object region. The object's Region of Interest may be referred to as the object region or the body's Region of Interest. Reference numbers (620) and (625) may be the extracted object's ear, and reference number (650) may be the center coordinates of the object's ear. Reference number (640) may be the center coordinates of the object's body. The object's Region of Interest (630) may be defined, for example, through the cv2.boundingrect(contour) function.

[0074] FIG. 7 is a diagram illustrating the main feature values ​​and object regions corresponding to the body parts of an object according to one embodiment.

[0075] Referring to FIG. 7, the reference number (710) may be the center coordinates of the object's ear, and the reference number (730) may be the center coordinates of the object's body. The reference number (720) may represent the distance between the center coordinates of the object's ear (710) and the center coordinates of the object's body (730). The reference number (740) may be the height of the body, and the reference number (750) may be the width of the body. Additionally, the area formed by the height of the body (740) and the width of the body (750) may be the area of ​​interest of the body, the area of ​​interest of the object, or the object area.

[0076] The process of extracting one or more body parts of an object from the object region described herein can be performed via R2C2 (Real-time Rodent Behaviour Classifier Using Colour-based Body Segmentation). The regions of the object's fur color and skin color are both processed almost simultaneously based on user-defined color ranges, and the largest continuous skin color region can first be defined as the body's region of interest. R2C2 corresponds to the behavior analysis device or behavior analysis method described herein. Since the ears, nose, feet, and tail have similar HSV profiles, the behavior tracking device can simultaneously capture and track at least one of the object's ears, nose, feet, and tail based on the HSV color space once the HSV color space of the skin is established. In particular, the behavior tracking device may utilize an additional algorithm to extract the ear region from a region containing other body parts. Additionally, based on the object region, an extended region of interest of the circular body / ear can be drawn and divided into four quadrants, including the quadrant (left or top right) containing the selected largest tag region. Through this approach, R2C2 can specifically define the estimated ear region and further extract location information. The resulting ear region of interest (e-ROI) may be indicated by reference number (710) in FIG. 7. The behavior analysis device may include a computation module that removes other skin-colored body parts and the background using the position of the ear relative to the center of the body coordinates to define the ear region of interest. The diameter of the center of the body ( ) can be calculated as shown in the equation below.

[0077]

[0078] Here, α is an empirically defined coefficient that can be, for example, 0.75. The behavior analysis device Based on this, the region of interest for the ear can be defined based on a circle centered on the body's center coordinates. The behavioral analysis device can inspect the paws and tail within the ear's region of interest. To detect the paws and tail, the behavioral analysis device may recognize that the ears and tail tend to be partially positioned above the body's center coordinates. To distinguish between the tail and the ears, the behavioral analysis device can select the left or right side of the body and utilize the fact that the ears have a higher number of skin-colored pixels than the tail. Ear extraction can be completed by calculating the pixel center of mass of the extracted ears.

[0079] FIGS. 8 to 10 are drawings illustrating graphs showing positional movement data of coordinates over time of key feature values ​​corresponding to body parts of an object according to one embodiment.

[0080] FIG. 8 may be a graph representing the position movement data of the center coordinate of the body and the position movement data of the ear over time as a 2D trajectory. Referring to FIG. 8, reference number (810) may represent the position movement data of the center coordinate of the body over time, and reference number (820) may represent the position movement data of the ear over time. Reference number (820) may represent the position movement data of the x-coordinate of the ear over time, the position movement data of the y-coordinate of the ear over time, or the position movement data of the center coordinate of the ear over time. The position movement data described in this specification may also be referred to as position trajectory data.

[0081] Figure 9 may be a graph representing the positional movement data of coordinates over time of BH / BW based on the height and width of the object's body among the main feature values, expressed as a 3D trajectory through interpolation. Here, G may represent grooming, W may represent walking, R may represent rearing, and S may represent sniffing behavior. The accuracy of determining the analysis result data of an object's behavior based on the positional movement data of coordinates over time of main feature values ​​expressed as a 3D trajectory may be higher than determining the analysis result data of an object's behavior based on the positional movement data of coordinates over time of main feature values ​​expressed as a 2D trajectory.

[0082] Figure 10 shows the position movement data of the BH / BW coordinates over time and the x-coordinate of the ear over time ( The position movement data of ) may be graphs expressed as 3D trajectories through interpolation. Among the graphs shown in FIG. 10, the upper graphs (1010, 1030, 1050, and 1070) may represent position movement data of BH / BW coordinates over time according to the object's action, and the lower graphs (1020, 1040, 1060, and 1080) represent the x-coordinates of the ear ( It can represent the position movement data of ).

[0083] Graph (1010) may represent data on the positional movement of the BH / BW coordinates over time when the object grooms, and graph (1020) may represent data on the positional movement of the x-coordinate of the ear over time when the object grooms. Graph (1030) may represent data on the positional movement of the BH / BW coordinates over time when the object rears, and graph (1040) may represent data on the positional movement of the x-coordinate of the ear over time when the object rears. Graph (1050) may represent data on the positional movement of the BH / BW coordinates over time when the object walks, and graph (1060) may represent data on the positional movement of the x-coordinate of the ear over time when the object walks. Graph (1070) may represent data on the positional movement of the BH / BW coordinates over time when the object smells, and graph (1080) may represent data on the positional movement of the x-coordinate of the ear over time when the object smells.

[0084] FIGS. 11 and 12 are drawings illustrating graphs showing the positional movement data of coordinates over time of key feature values ​​corresponding to body parts of an object according to one embodiment, analysis result data, and frequencies measured based on the movement of body parts over time.

[0085] FIG. 11 may be a graph (1110) representing positional movement data of the coordinates of the body height over time for rearing, and a scalogram (1120) visualizing the frequency measured based on the movement of the body height over time for rearing, which has been transformed into a continuous-time wavelet. In the scalogram (1120), the section (1130) marked with R at the top may correspond to the time interval during which the object performs rearing.

[0086] FIG. 12 may be a graph (1210) showing positional movement data of the y-coordinate of the ear over time for grooming, and a scalogram (1220) visualizing the frequency measured based on the movement of the y-coordinate of the ear over time for grooming, which has been transformed into a continuous-time wavelet. In the scalogram (1220), the section (1230) marked with G at the top may correspond to the time interval during which the object grooms.

[0087] FIG. 13 is a graph illustrating the positional movement data of coordinates over time of a major feature value corresponding to a body part of an object according to one embodiment, and the frequency measured based on the movement of the body part over time.

[0088] In one embodiment, the position of the ear and the coordinates of the center of the body can be calculated as shown in the following equations.

[0089]

[0090]

[0091]

[0092]

[0093] x ear is the x-coordinate of the ear, and y ear can be the y-coordinate of the ear. x contour can represent the outline of the ear's x-coordinate, and Area contour can represent the area of ​​a contour. Areaallcontour can represent the area of ​​all contours. x c can represent the x-coordinate of the center of the body, and y c can represent the y-coordinate of the center of the body. M 10 , M 00 , M 01 The contour point (M ij It can correspond to ). Contour point (M ijThe moment of ) is the center of the body (x) calculated by points having pixel intensities c , y c It can be. Here, the coordinates of the ear can be calculated based on the center of mass. The coordinates of the ear and the body can be used to evaluate the quality of extracting the ear.

[0094] In another embodiment, if some body parts of an object are obscured in an image showing the object due to other objects around the object, the behavior analysis device may extract the ear from the object area and determine analysis result data regarding the object's behavior based on the coordinates of the ear.

[0095] For the contours of all masks generated for the selected HSV range, the center of the i-th contour can be calculated as follows.

[0096]

[0097] As shown in the mathematical formula below, the center coordinates of the body (x c , y c from ) Anything exceeding may be excluded.

[0098]

[0099] The behavior analysis device can divide the body's region of interest into four equal parts based on the body's center coordinates, and here, by first selecting the upper two quadrants, the contour is the y-axis of the body's center. c Rather than coordinates It can be reduced to a large outline. On the other hand, if the ear is not properly extracted due to the object performing grooming or walking behaviors, the behavior analysis device can select the quadrant containing the largest tag area (top-left or top-right) of the two quadrants and define all outlines contained within it as the ear area.

[0100]

[0101] Since the contours of the ears are often not adjacent (e.g., the left ear and the right ear), the behavioral analysis device uses weights proportional to the area of ​​each contour ( The final center coordinates of the ear can be calculated by multiplying by and then adding.

[0102]

[0103]

[0104] Referring to FIG. 13, the top color bar (1305) may indicate the type of behavior of the object. For example, yellow may indicate grooming, blue may indicate rearing, green may indicate walking, and red may indicate sniffing. Reference numbers (1310, 1320, 1330) may be graphs visualizing newly identified normalized key feature values ​​over time from images showing the object captured in 4000 frames. Reference number (1310) may represent data on the positional shift of the coordinates of the normalized body height over time, and reference number (1320) may represent data on the positional shift of the coordinates of the normalized body width over time. Reference number (1330) may represent data on the positional shift of the coordinates of the normalized ear y-coordinate over time. Reference number (1340) may be a scalogram in which the coordinate displacement data of the normalized body height over time is continuously wavelet transformed, and reference number (1350) may be a scalogram in which the coordinate displacement data of the normalized body width over time is continuously wavelet transformed. Additionally, reference number (1360) may be a scalogram in which the coordinate displacement data of the normalized ear y-coordinate over time is continuously wavelet transformed. In the scalograms (1340, 1350, 1360), the absolute value of the power per frequency band may be expressed as a function of time and frequency.

[0105] FIG. 14 is a drawing illustrating a behavior classifier according to one embodiment.

[0106] Referring to FIG. 14, the behavior classifier may be a neural network including a plurality of 1D-convolution layers and a fully connected layer. For example, the behavior classifier may include five 1D-convolution layers (or five 1D-convolution blocks) and a fully connected layer to output probability values ​​that the behavior of an object corresponds to each of the predefined different behaviors. The behavior classifier may input frequency features for each body part obtained by performing a continuous-time wavelet transform on frequencies measured based on the movement over time of one or more extracted body parts into the first layer or block, and perform convolution for the next layer or block.

[0107] The behavior classifier is based on a neural network and receives as training data frequency features for each body part corresponding to a predetermined behavior and position movement data (1410) of coordinates over time of key feature values ​​corresponding to the predetermined behavior, and can be trained to output a probability value that the object's behavior corresponds to each of the different predetermined behaviors based on the frequency features. When the trained behavior classifier receives frequency features for each body part, it can output a probability value that the object's behavior corresponds to each of the different predetermined behaviors based on the received frequency features for each body part.

[0108] The behavior classifier described in this specification may also be referred to as at least one of a behavior classification model, a behavior analysis model, or a behavior analyzer.

[0109] FIG. 15 is a diagram illustrating the configuration of an artificial intelligence-based behavior analysis device according to one embodiment.

[0110] Referring to FIG. 15, the behavior analysis device (1500) may include a processor (1510) and a memory (1520), a user input interface (1530), a display (1540), and a camera (1550). The behavior analysis device (1500) may correspond to a behavior analysis device (1500) that performs the behavior analysis method described herein.

[0111] The memory (1520) is connected to the processor (1510) and can store instructions executable by the processor (1510), data to be computed by the processor (1510), or data processed by the processor (1510). The memory (1520) may include a non-transient computer-readable medium, such as high-speed random access memory and / or a non-volatile computer-readable storage medium (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0112] The user input interface (1530) can receive user input entered by the user. In one embodiment, the user input interface (1530) can receive user input related to analyzing the behavior of an object. The display (1540) can output an interface related to the behavior analysis method. In one example, the display (1540) may be a monitor. The camera (1550) can capture an image of the object appearing.

[0113] The processor (1510) may perform one or more operations related to the operation of the behavior analysis device (1500) described in this specification. For example, the processor (1510) may control the behavior analysis device (1500) so that the behavior analysis device (1500) acquires an image in which an object appears through a camera (1550), and may control the behavior analysis device (1500) so that the color space of the object area within the image is converted to another color space. The processor (1510) may control the behavior analysis device (1500) so that the behavior analysis device (1500) converts the RGB color space, which is the color space of the object area, to another color space, the HSV color space. The processor (1510) may control the behavior analysis device (1500) so that the behavior analysis device (1500) extracts one or more body parts of an object by masking one or more body parts of an object from the object area converted to the HSV color space based on the hue, saturation, and brightness of the object area converted to the HSV color space. The behavior analysis device (1500) can distinguish a wider dimension and a wider range of colors than the RGB color space by converting the RGB color space to the HSV color space. The formula for converting the RGB color space to the HSV color space may be the same as Equation 1, Equation 2, and Equation 3.

[0114] The processor (1510) can control the behavior analysis device (1500) so that the behavior analysis device (1500) extracts one or more body parts of an object in an object area converted to a different color space, acquires frequency features for each body part based on the movement over time of the extracted one or more body parts, measures positional movement data of the coordinates over time of the main feature values ​​corresponding to one or more body parts based on the main feature values ​​corresponding to one or more body parts, tracks the movement over time of one or more body parts based on the positional movement data of the coordinates over time of the main feature values ​​corresponding to one or more body parts, and acquires frequency features for each body part based on the movement over time of the tracked one or more body parts.

[0115] The processor (1510) can control the behavior analysis device (1500) to obtain frequency features for each body part by performing a continuous time wavelet transform on the frequency measured based on the movement over time of one or more extracted body parts. The processor (1510) can control the behavior analysis device (1500) to determine analysis result data for the object's behavior by using a neural network-based behavior classifier that takes frequency features for each body part as input. The behavior classifier may correspond to the behavior classifier described in FIG. 2. The processor (1510) can control the behavior analysis device (1500) to determine analysis result data for the object's behavior based on probability values ​​that the object's behavior corresponds to each of the different predefined behaviors.

[0116] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0117] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0118] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0119] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0120] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0121] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In a method for analyzing the behavior of an object based on artificial intelligence, A step of acquiring an image in which the above object appears; A step of converting the color space of an object region within the above image to a different color space; A step of extracting one or more body parts of the object from the object region converted to the other color space; A step of acquiring frequency characteristics for each body part based on the movement over time of one or more extracted body parts; and A step of determining analysis result data for the behavior of the object using a neural network-based behavior classifier that takes frequency characteristics for each body part as input. including, Behavioral analysis methods.

2. In Paragraph 1, The step of converting to the above-mentioned different color space is, A step comprising converting the RGB (red, green, blue) color space, which is the color space of the object area, into the HSV (hue saturation value) color space, which is another color space. Behavioral analysis methods.

3. In Paragraph 1, The above extraction step is, A method comprising the step of extracting one or more body parts of an object by masking one or more body parts of the object from an object region converted to an HSV color space based on the hue, saturation, and brightness of the object region converted to an HSV color space. Behavioral analysis methods.

4. In Paragraph 1, The step of acquiring the above frequency characteristics is, A step of measuring positional movement data of coordinates over time of a major feature value corresponding to one or more body parts based on a major feature value corresponding to one or more body parts; A step of tracking the movement over time of one or more body parts based on positional movement data of coordinates over time of key feature values ​​corresponding to one or more body parts; and A step of obtaining frequency characteristics for each body part based on the movement over time of the one or more body parts tracked above. including, Behavioral analysis methods.

5. In Paragraph 4, The above key feature values ​​are, Including the x-coordinate of the ear of the object, the y-coordinate of the ear of the object, the width of the body of the object, the height of the body of the object, the x-coordinate of the body of the object, the y-coordinate of the body of the object, the center coordinate of the body of the object, and the center coordinate of the ear of the object, Behavioral analysis methods.

6. In Paragraph 1, The step of acquiring the above frequency characteristics is, A method comprising the step of obtaining frequency characteristics for each body part by performing a continuous time wavelet transform on the frequency measured based on the movement over time of one or more extracted body parts, Behavioral analysis methods.

7. In Paragraph 1, The above behavior classifier is, A learning model that receives as input as learning data frequency characteristics for each body part corresponding to a predetermined action and positional movement data of coordinates over time of key feature values ​​corresponding to the said predetermined action, and is trained to output a probability value that the action of the object corresponds to each of the different predetermined actions based on the frequency characteristics. Behavioral analysis methods.

8. In Paragraph 1, The above behavior classifier is, A neural network that includes convolutional layers and fully connected layers, Behavioral analysis methods.

9. In Paragraph 1, The above-mentioned determining step is, A method comprising the step of determining analysis result data for the behavior of the object based on probability values ​​that the behavior of the object corresponds to each of predefined different behaviors. Behavioral analysis methods.

10. A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of claim 1.

11. In an artificial intelligence-based object behavior analysis device, Includes memory and processor, The above memory stores instructions executable by the processor, and When the above instructions are executed by the processor, the processor, the behavior analysis device, Acquire an image showing the above object, Convert the color space of the object region within the above image to another color space, and One or more body parts of the object are extracted from the object region converted to the other color space, and Based on the movement over time of one or more of the extracted body parts, frequency characteristics for each body part are obtained, and Controlling the behavior analysis device to determine analysis result data for the behavior of the object using a neural network-based behavior classifier that takes frequency characteristics for each of the above body parts as input. Behavior analysis device.

12. In Paragraph 11, The above processor is the behavior analysis device, Controlling the behavior analysis device to convert the RGB (red, green, blue) color space, which is the color space of the object area, into the HSV (hue saturation value) color space, which is another color space. Behavior analysis device.

13. In Paragraph 11, The above processor is the behavior analysis device, Controlling the behavior analysis device to extract one or more body parts of an object by masking one or more body parts of the object from the object region converted to the HSV color space based on the hue, saturation, and brightness of the object region converted to the HSV color space, Behavior analysis device.

14. In Paragraph 11, The above processor is the behavior analysis device, Based on the main feature values ​​corresponding to the one or more body parts, the positional movement data of the coordinates over time of the main feature values ​​corresponding to the one or more body parts is measured, and Tracking the movement over time of one or more body parts based on positional movement data of coordinates over time of key feature values ​​corresponding to one or more body parts, and Controlling the behavior analysis device to acquire frequency characteristics for each body part based on the movement over time of the one or more body parts tracked above, Behavior analysis device.

15. In Paragraph 14, The above key feature values ​​are, Including the x-coordinate of the ear of the object, the y-coordinate of the ear of the object, the width of the body of the object, the height of the body of the object, the x-coordinate of the body of the object, the y-coordinate of the body of the object, the center coordinate of the body of the object, and the center coordinate of the ear of the object, Behavior analysis device.

16. In Paragraph 11, The above processor is the behavior analysis device, Controlling the behavior analysis device to acquire frequency characteristics for each body part by performing a continuous time wavelet transform on the frequency measured based on the movement over time of one or more extracted body parts, Behavior analysis device.

17. In Paragraph 11, The above behavior classifier is, A learning model that receives as input as learning data frequency characteristics for each body part corresponding to a predetermined action and positional movement data of coordinates over time of key feature values ​​corresponding to the said predetermined action, and is trained to output a probability value that the action of the object corresponds to each of the different predetermined actions based on the frequency characteristics. Behavior analysis device.

18. In Paragraph 11, The above behavior classifier is, A neural network that includes convolutional layers and fully connected layers, Behavior analysis device.

19. In Paragraph 11, The above processor is the behavior analysis device, Controlling the behavior analysis device to determine analysis result data for the behavior of the object based on probability values ​​that the behavior of the object corresponds to each of the predefined different behaviors. Behavior analysis device.

Images