Apparatus and method for analyzing pattern of behavior based on three-dimensional

Abstract

A three-dimensional based behavioral pattern analysis device according to one embodiment of the present invention includes a data transmission and reception module; a memory in which a behavioral pattern analysis program is stored; and a processor that executes the program stored in the memory. The program collects input images through at least one camera positioned above a space in which clustered objects are accommodated, inputs the input images into a detection model to generate a three-dimensional bounding box for an object, detects identification information for an object, and inputs the image and identification information of an object into a behavioral pattern analysis model to detect a behavioral pattern. The three-dimensional bounding box is generated in a form that encloses all or part of the object, and the identification information is generated based on the feature information of the three-dimensional bounding box.

Description

Technology Field

[0001] The present invention relates to a three-dimensional based behavioral pattern analysis device and method. Background Technology

[0002] Camera-based video analysis technology is a useful tool for analyzing the behavioral patterns of humans or animals. It operates by capturing video collected from a camera and applying algorithms that identify specific behavioral patterns. Such algorithms can be used to process large-scale datasets, analyze video in real time, and provide real-time information about occurring events. As such, camera-based video analysis technology is a useful tool for managing and improving populations, including humans or animals.

[0003] For example, animal ecology, welfare, health, and disease have traditionally been observed using camera systems. Particularly in the event of recent outbreaks of new animal infectious diseases, assessments were primarily made by visually observing abnormal behavior, posture, or external changes to rapidly confirm suspected infections in livestock or animals. However, due to the development of the livestock industry and the frequent practice of raising livestock in large numbers, it is difficult to check the behavior or posture of each animal individually with limited personnel. Furthermore, since symptoms can manifest differently depending on each individual animal's free will and personality, it is also challenging to observe animals according to standardized observation criteria.

[0004] In this regard, as an existing method for analyzing the movement patterns of livestock, Korean Patent Registration No. 10-1318716 (Title of Invention: System for Analyzing Movement Patterns of Livestock) discloses a configuration that analyzes movement patterns for each individual livestock, and calculates the actual amount of movement by reflecting not only all movements that an individual livestock can take but also changes in the movement patterns.

[0005] However, health, welfare, and various diseases, particularly the spread of novel infectious diseases, are time-sensitive issues that require early suspicion of infection. Furthermore, it is necessary to implement prevention and response measures through accurate and precise data and analysis regarding suspected diseases. For instance, in the case of zoonotic diseases such as the novel coronavirus, which originate in animals and are transmitted to and mutate in humans, failure to respond promptly with accurate and precise judgment and prediction regarding animal diseases could lead to the accumulation of massive socio-economic damage not only for us but for the entire world. It is not easy to eradicate the vicious cycle of recurring infectious disease problems, and given the current situation where disaster-type infectious diseases are prevalent for which effective vaccines or treatments are lacking, there is a problem of continuously accumulating damage. The problem to be solved

[0006] The present invention aims to solve the aforementioned problems and proposes a three-dimensional based behavioral pattern analysis device and method for detecting behavioral patterns capable of predicting the state of individual entities or groups including humans and animals.

[0007] However, the technical problem that this embodiment aims to solve is not limited to the technical problem described above, and other technical problems may exist. means of solving the problem

[0008] As a technical means for solving the aforementioned technical problem, a three-dimensional based behavioral pattern analysis device according to one embodiment of the present invention comprises a data transmission and reception module; a memory in which a behavioral pattern analysis program is stored; and a processor that executes the program stored in the memory, wherein the program collects input images through at least one camera positioned above a space in which clustered objects are accommodated, inputs the input images into a detection model to generate a three-dimensional bounding box for an object, detects identification information for an object, and inputs the image and identification information of an object into a behavioral pattern analysis model to detect a behavioral pattern, wherein the three-dimensional bounding box is generated in a form that encloses all or part of the object, and the identification information is generated based on the feature information of the three-dimensional bounding box.

[0009] A method for analyzing behavioral patterns using a three-dimensional based behavioral pattern analysis device according to another embodiment of the present invention comprises: (a) collecting input images through at least one camera positioned above a space in which clustered entities are accommodated; (b) inputting the input images into a detection model to generate a three-dimensional bounding box for the entities and detecting identification information for the entities; and (c) inputting the images and identification information of the entities into a behavioral pattern analysis model to detect behavioral patterns, wherein the three-dimensional bounding box is generated in a form that encloses all or part of the entities, and the identification information is generated based on feature information of the three-dimensional bounding box. Effects of the invention

[0010] According to any one of the means for solving the problem of the present invention described above, the behavioral patterns of individuals and groups can be detected to identify the health or risk status of each individual and group.

[0011] In addition, the health, welfare, and presence of diseases, especially infectious diseases, of individuals can be quickly, accurately, and precisely verified using only a 3D depth imaging device and analysis software, resulting in cost savings.

[0012] In particular, the present invention allows for the simultaneous observation of multiple entities, enabling rapid identification and response to infectious disease outbreaks.

[0013] In addition, since the present invention is a non-face-to-face / non-contact inspection method based on three dimensions, it is much more accurate and precise than conventional methods, while also having high safety and reliability.

[0014] In addition, real-time remote monitoring can detect infectious diseases early and prevent their spread.

[0015] In addition, since it detects the behavioral patterns of clusters occurring within a defined space, dangerous situations such as crowd density can be checked in real time. Brief explanation of the drawing

[0016] FIG. 1 is a configuration diagram of a three-dimensional based behavioral pattern analysis device according to one embodiment of the present invention. FIG. 2 is a diagram illustrating a detailed module of a three-dimensional based behavioral pattern analysis device according to an embodiment of the present invention. FIGS. 3 and 4 are drawings for explaining a 3D depth camera and a 3D bounding box for photographing objects according to an embodiment of the present invention. Figure 5 is a diagram illustrating a comparison of two different detection models for detecting objects. FIG. 6 is a diagram illustrating an example of detecting identification information for an object according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of detecting identification information of each image for a target image according to an embodiment of the present invention. FIG. 8 is a diagram illustrating an example of detecting behavioral pattern information for a cluster image according to an embodiment of the present invention. FIGS. 9 and FIGS. 10 are drawings for explaining behavioral pattern information regarding the normal state of a cluster of entities according to an embodiment of the present invention. FIGS. 11 and 12 are drawings for explaining behavioral pattern information regarding an abnormal state of a cluster of entities according to an embodiment of the present invention. FIG. 13 is a drawing showing a user interface according to one embodiment of the present invention. FIG. 14 is a flowchart illustrating a three-dimensional based behavioral pattern analysis method according to another embodiment of the present invention. Specific details for implementing the invention

[0017] Embodiments of the present invention are described below with reference to the attached drawings to enable those skilled in the art to easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0018] Throughout this specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other elements interposed between them.

[0019] Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0020] FIG. 1 is a configuration diagram of a three-dimensional based behavioral pattern analysis device according to one embodiment of the present invention.

[0021] Hereinafter, the individuals mentioned in the present invention include humans, animals, insects, etc., and a community refers to a state in which each individual is gathered in a group within a predetermined space.

[0022] As illustrated in FIG. 1, the behavior pattern analysis device (100) may include at least one camera (10), a data transmission and reception module (120), a processor (130), a memory (140), and a database (150).

[0023] The behavior pattern analysis device (100) can be implemented as a computer or portable terminal that can connect to a network. Here, the computer includes, for example, a notebook, a desktop, a laptop, etc., and the portable terminal is, for example, a wireless communication device that ensures portability and mobility, and can include all kinds of handheld-based wireless communication devices such as various smartphones, tablet PCs, smartwatches, etc.

[0024] Additionally, the behavior pattern analysis device (100) can function as a server that provides the prediction results of the behavior patterns of objects and clusters to an external computing device. In this case, the server may operate in a cloud computing service model such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), or may be built in a form such as a private cloud, a public cloud, or a hybrid cloud.

[0025] At least one camera (10) is assigned above a predetermined space in which a group of individuals is accommodated, and can monitor each individual. Additionally, the camera (10) can transmit images captured at a predetermined angle of view within the search area to a data transmission / reception module (120). Here, the predetermined space may be a space in which animals or insects are accommodated, such as a livestock room, a pen, or a breeding facility, but is not limited thereto; it may also be a space in which people are densely concentrated, such as an entrance or passageway where a bottleneck occurs.

[0026] The data transmission and reception module (120) can receive an image captured by the camera (10) at a predetermined angle of view and transmit it to the processor (130). Here, the data transmission and reception module (120) may be a device including hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.

[0027] The memory (140) may contain a behavior pattern analysis program. Various types of data generated during the execution of the behavior pattern analysis program or the operating system for operating the behavior pattern analysis device (100) are stored in this memory (140). At this time, the memory (140) is a general term for a non-volatile storage device that continues to maintain stored information even when power is not supplied, and a volatile storage device that requires power to maintain stored information.

[0028] Additionally, the memory (140) can perform the function of temporarily or permanently storing data processed by the processor (130). Here, the memory (140) may include magnetic storage media or flash storage media in addition to volatile storage devices that require power to maintain stored information, but the scope of the present invention is not limited thereto.

[0029] The processor (130) executes a behavior pattern analysis program (hereinafter, program) stored in memory (140) and provides the function of controlling the hardware of the behavior pattern analysis device (100) according to the execution of the program. That is, the processor (130) can perform hardware control functions such as necessary file systems, memory allocation, networks, basic libraries, timers, device control (display, media, input devices, 3D, etc.), and other utilities as the program is executed.

[0030] The processor (130) collects input images through at least one camera (10) positioned above the space in which the clustered objects are accommodated, inputs the input images into a detection model (210) to generate a three-dimensional bounding box for the objects, detects identification information for the objects, and inputs the images and identification information of the objects into a behavior pattern analysis model (220) to detect behavior patterns. Additionally, specific steps of the process of predicting behavior patterns of the clustered objects according to the execution of the program will be described later with reference to FIGS. 2 to 8.

[0031] The processor (130) may include all types of devices capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of such data processing devices embedded in hardware may include a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0032] The database (150) stores or provides data required by the behavior pattern analysis device (100) under the control of the processor (130). For example, the database (150) may store images and identification information of each object and cluster, and behavior pattern information labeled for each identification information, based on three-dimensional input images received from the camera (10). This database (150) may be included as a separate component from the memory (140), or may be built in a part of the memory (140).

[0033] FIG. 2 is a drawing for explaining a detailed module of a three-dimensional based behavioral pattern analysis device according to one embodiment of the present invention, and FIG. 3 and FIG. 4 are drawings for explaining a 3D depth camera and a 3D bounding box for photographing objects according to one embodiment of the present invention.

[0034] Referring to FIG. 2, the processor (130) may include detailed modules that perform various functions according to the execution of a behavior pattern analysis program. The processor (130) may include a detection model (210), a behavior pattern analysis model (220), and a discrimination unit (230).

[0035] For example, the detection model (210) can receive an input image through at least one camera (10) positioned above the space in which the cluster of objects are accommodated, generate a three-dimensional bounding box for the object, and detect identification information for the object. As an example, as illustrated in FIG. 6 described below, the object means that a single object (animal) is identified as the target object.

[0036] Additionally, the detection model (210) can generate a three-dimensional bounding box for a cluster containing more than a preset number of objects from the input image and detect identification information for the cluster. As another example, as illustrated in FIG. 7 described later, a cluster refers to a cluster consisting of a set of objects (animals) that is identified as a single target object.

[0037] Referring to FIG. 3, the camera (10) includes a 3D depth camera that captures real-image images of clustered objects and measures and analyzes the Time of Flight (TOF) of light to calculate and display the distance to the object.

[0038] For example, a 3D depth camera (10) is installed above the space containing the object and can take pictures of each object. The series of images taken at this time is a video, and each individual image in the video is a frame. For example, a video with 15 images per second is called 15fps (frames per second).

[0039] The detection model (210) can detect a target image in which an object is identified for an input image including a single frame and a video, and can detect identification information for the object. At this time, the target object may include both an individual or a cluster. For example, in the case of a single frame, a target object is detected, a non-moving 3D bounding box (20) is created around the detected object, and the 3D bounding box (20) and identification information can be overlaid on the target image. For another example, in the case of a video, a target object is detected, a moving 3D bounding box (20) is created around the detected object, and the 3D bounding box (20) and identification information can be overlaid on the target image. That is, the detection model (210) can detect an object in the input image, create a 3D bounding box (20) of the detected target object, detect identification information for the target object, and overlay the 3D bounding box (20) and identification information for the target object on the target image. For example, by sequencing the target image, the output can be made to appear as if the 3D bounding box (20) overlaid on the target object is moving. Thus, the output of the 3D bounding box (20) for object detection and tracking can all appear the same.

[0040] In another embodiment, the program may apply AR technology to a target image captured by a 3D depth camera (10). In this case, the AR technology includes overlaying visual, auditory, or other sensory information onto the real world to enhance the user experience. For example, as illustrated in FIG. 3, the program inputs an input image into a detection model (210) and uses the image of the detected target object and identification information (including 3D location information) to detect (track and determine spatial coordinates) a target object (animal) in the real world and generate a 3D bounding box (20) on the target object (animal). Additionally, the program may apply immersive technologies such as MR and XR to provide a 3D model that visualizes an animal cage in the real world. Accordingly, the present invention may be utilized for monitoring the condition of animals, for simulation learning, for educational purposes, or for practical training.

[0041] Specifically, referring to FIG. 3, the three-dimensional bounding box (20) is created in a form that encloses all or part of an object and can be overlaid on a target image containing at least one object. For example, the three-dimensional bounding box (20) may include a hexahedron or an ellipsoid, but is not limited thereto and may include various shapes.

[0042] Also, referring to FIG. 4, the identification information may include at least one reference point (P: x, y, z) located on one surface of the three-dimensional bounding box (20), the planar area of ​​the three-dimensional bounding box (20), the volume of the three-dimensional bounding box (20), a straight-line distance (A) to the reference point of the three-dimensional bounding box (20) relative to the camera (10), a vertical-line distance (C) relative to the bottom surface relative to the camera (10), or an angle (θ) between the straight line and the vertical line.

[0043] Accordingly, the detection model (210) can detect a target object in an input image, generate a three-dimensional bounding box (20) for the target object, and detect the vertical distance (z) from the floor surface to the reference point of the three-dimensional bounding box (20) using identification information for each detected target object (ID). That is, in addition to the reference point (x, y, z) of each target object location (P), the program can calculate information regarding the actual height (z) of the target object location (P) as identification information. As a result, the program can extract features for each posture position of the object, including whether the target object (animal) is actually lying down or standing, by using accurate vertical distance (z) information for the target object. As another example, the program can detect width information by using the planar area of ​​the three-dimensional bounding box (20). That is, the program can extract features for each change in the horizontal width of the object, including the horizontal position or movement of the animal's legs, by using the width information of the target object.

[0044] Figure 5 is a diagram illustrating a comparison of two different detection models for detecting objects.

[0045] Referring to FIG. 5, the object detection results of two different AI models are shown: a first comparison model (A, B, C; hereinafter referred to as the 'first comparison model') focused on speed and a second comparison model (D, E, F; hereinafter referred to as the 'second comparison model') focused on tracking consistency. This is an experimental comparison example to explain the performance characteristics suitable for the detection model (210) of the present invention.

[0046] The possibility of object tracking was explored by simulating two people crossing multiple times while recording a scene with an AI-based 3D depth camera. The direction of intersection is perpendicular to the camera's field of view.

[0047] First, the detection model receives frames from the video to visualize crossing and object detection. 'Before' represents a frame where two people are standing still to the left and right of the camera. 'After' represents a frame where the two people are standing still in the state after the crossing has occurred. 'Crossing' represents the frame immediately before the crossing occurs. In this case, frame collection for the crossing begins with each person at a stationary position to the right and left of the camera. While the two people walk towards each other simultaneously, the first crossing point aligns the camera with the first person and then with the second person. After the crossing, the two people stand still for 5 seconds. Then, the second crossing point aligns the camera with the second person and the first person. After both people have stood still for 5 seconds, this procedure is repeated.

[0048] As such, the detection model using a 3D depth camera successfully identified people, and both the first comparison model and the second comparison model can detect objects.

[0049] Both the first comparison model and the second comparison model can be implemented based on architectures such as R-CNN and YOLO described below, and can detect objects based on 3D bounding boxes, IDs, and estimated straight-line distance information from a camera (identification information) for each object. In addition, identification information including 3D bounding boxes, IDs, and straight-line distances can be displayed consistently for each object. For example, the bounding boxes and text corresponding to each object can be distinguished by various colors, patterns, and dashes.

[0050] As shown in FIG. 5, both the first comparison model and the second comparison model have the same ID value for each person (object) crossing in front of the frame. That is, both models distinguished people identified at a close distance to the camera with a consistent ID. For example, in the first comparison model, a person with ID 0 was detected identically in (A), (B), and (C). Similarly, in the accurate detection model, a person with ID 1 was detected identically in (D), (E), and (F).

[0051] The results for people who are behind the camera and not visible during the crossover vary from model to model. The first comparison model failed to provide the same ID for the same object. That is, in Fig. 5 (A), the person identified as ID number 5 was identified as ID number 8 in Fig. 5 (C). On the other hand, the accurate detection model showed consistency in providing the same ID. That is, in Figs. 5 (D) and (F), both people were consistently identified as ID numbers 1 and 0.

[0052] That is, the person in front (the person closer to the camera) can completely hide the person behind in every multiple intersection. Therefore, both the first comparison model focused on speed and the second comparison model focused on tracking consistency can provide 3D bounding boxes, IDs, and distance measurements for the two people, and it was found that the second comparison model performed object tracking better than the first comparison model. Accordingly, this indicates that the second comparison model has greater potential. Therefore, the detection model (210) of the present invention can be implemented by applying an AI model (e.g., R-CNN, YOLO, etc.) trained to have excellent object tracking performance, as shown in the experimental results of the second comparison model described above.

[0053] Hereinafter, an example of applying the behavioral pattern analysis device (100) of the present invention to a livestock farm is described.

[0054] FIG. 6 is a diagram illustrating an example of detecting a three-dimensional bounding box and identification information of an object according to an embodiment of the present invention, FIG. 7 is a diagram illustrating an example of detecting a three-dimensional bounding box and identification information of an object and a cluster according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating an example of displaying a three-dimensional bounding box and identification information for a cluster according to an embodiment of the present invention.

[0055] A 3D depth camera (10) is installed in an experimental animal cage by positioning it at an upper location 2.5 m away from the center of the livestock cage, providing a field of view that covers the entire cage area. At this time, the position where the 3D depth camera is installed is for convenience of explanation, and the height of the distance can be set in various ways. As a result, the target images shown in FIGS. 6 to 8 show a plan view of the animal cage.

[0056] The program can identify a target object using a detection model (210), generate a three-dimensional bounding box (20) for an animal and an animal group (target object), and detect identification information (201) of the animal and the animal group. At this time, the identification information (201) may include an ID identifying the animal and the animal group and a straight-line distance (A) to a reference point of the three-dimensional bounding box (20) for the camera, but is not limited thereto, and other location information of the three-dimensional bounding box other than the ID and the straight-line distance (A) may be overlaid on the target image as identification information (201). For example, the detection model (210) can identify an object in cell units and output a three-dimensional bounding box (20) corresponding to the object.

[0057] In addition, the program can provide a user interface that can monitor the behavior or state of objects and clusters based on a three-dimensional bounding box (20).

[0058] Meanwhile, a 3D bounding box (20) is significantly different from a 2D bounding box. Adding another dimension to a 2D bounding box allows for additional analysis, such as the size, orientation, or pose of a given object. For example, a 2D bounding box only requires finding and fitting a rectangle around a given object. However, for a 3D bounding box (20), the software must estimate the size and depth so that it can fit a multidimensional cube around a given object. Most software aimed at generating a 3D bounding box (20) starts with 2D bounding box prediction. Then, the software can identify key points inside and around the 2D bounding box through machine learning such as a Convolutional Neural Network (CNN) or region(R)-CNN and use them to generate a 3D bounding box. As such, 3D bounding boxes are frequently used in the fields of robotics and augmented reality, but recently they are being used in various other fields.

[0059] Referring to FIGS. 6 and 7, an example of detecting an object when the detection model (210) according to the present invention is integrated into a livestock farm is described.

[0060] In one embodiment, FIGS. 6(a) and FIGS. 6(b), a plurality of animals (individuals) can be detected as individual target objects by a detection model (210). Referring to FIGS. 6(a), one animal is identified (ID: 13), a 3D bounding box (20) corresponding to the animal (ID: 13) is generated, and a straight distance (2.78m) from the 3D depth camera (10) to a point on the 3D bounding box (20) can be displayed on the target image as identification information (201). Referring to FIGS. 6(b), another animal is identified (ID: 0), a 3D bounding box (20) for the animal (ID: 0) is generated, and a straight distance (2.71m) to a point on the 3D bounding box (20) can be displayed on the target image as identification information (201). That is, the animals in FIG. 6(a) and FIG. 6(b) can be identified as different animals by identification information (201). For example, the user interface may display different colors, dashes, or shapes of the three-dimensional bounding box (20) so that different animals can be easily visually distinguished.

[0061] For example, when the height or width of an object or cluster based on identification information (201) changes, the user interface can display the color or shape of the 3D bounding box (20) in real time so that it can be easily visually identified within the video.

[0062] That is, based on identification information (201), the program can detect changes in the height or horizontal width of the animal individual or animal group and provide through the user interface that the behavior / state / posture of the individual or group has changed. For example, when an individual is sitting and then stands up, the user interface can display the color or shape of the 3D bounding box (20) changing in real time so that it can be easily visually confirmed within the video.

[0063] In another embodiment, as shown in FIG. 7(a) and FIG. 7(b), individual animals and animal groups can be detected as individual target objects by the detection model (210). Referring to FIG. 7(a), a group of animals spaced apart (ID: 11) and individual animals (ID: 8 and ID: 9) can be identified as each target object. A three-dimensional bounding box (20) for each target object is generated, and ID and straight-line distance can be displayed on the target image as identification information (201) of the target object. Also, referring to FIG. 7(b), a group of animals densely packed together (ID: 35, ID: 34) and individual animals (ID: 32, ID: 33) can be identified as each target object. A three-dimensional bounding box (20) for each target object is generated, and ID and straight-line distance can be displayed on the target image as identification information (201) of the target object. At this time, the animal community may include a clustered form of 1+n or more individuals.

[0064] Accordingly, the detection model (210) of the present invention can identify single animals and animal groups that are spaced apart or densely packed.

[0065] The behavior pattern analysis model (220) can detect behavior pattern information by receiving images of objects and clusters and identification information about objects and clusters.

[0066] Additionally, the behavior pattern analysis model (220) is a model trained using training data that is labeled with identification information of an object or cluster and behavior pattern information of each type of object or cluster, or unlabeled training data, and can output behavior pattern information of an object and cluster based on an input image.

[0067] For example, a behavior pattern analysis model (220) can learn by labeling behavior pattern information, including movement or behavior change of the animal individual or animal group, to the three-dimensional bounding box (20) of the animal individual or animal group and identification information for the individual or group.

[0068] Accordingly, the program can detect behavioral pattern information of an individual animal or an animal group by means of a behavioral pattern analysis model (220).

[0069] For example, FIGS. 8(a) and FIGS. 8(b) are target images of a situation in which an animal colony eats around a feeding station, illustrating that a colony of five animals is detected as a single target object. In FIGS. 8(a), all individuals of the animal colony maintain a distance from the fence (see arrow), and the behavioral pattern information can be labeled as a normal animal colony. FIGS. 8(b) shows an individual located at the edge of the animal colony being pushed and leaning toward the fence (see arrow), and the behavioral pattern information can be labeled as a dangerous animal colony.

[0070] As another example, the program can output behavioral pattern information based on identification information (201) on the target image. For example, if the behavioral pattern information for a group of five animals changes from a normal animal group to a dangerous animal group, the user interface can output from the 3D bounding box (ID: 11, straight distance 2.90m, orange bounding box) shown in FIG. 8(a) to the 3D bounding box (ID: 26, straight distance 2.62m, light blue bounding box) shown in FIG. 8(b) based on the identification information (201) of the animal group. Thus, the present invention can monitor dangerous situations that may occur when animals push and compete for food due to their nature and gluttony / appetite in a situation with a limited pig pen and a relatively small feeding station width. In addition, in the case of humans, it can also monitor dangerous situations that may occur when a large crowd gathers in a narrow alley and simultaneously moves forward and proceeds against each other.

[0071] The determination unit (230) determines whether an individual and a cluster are normal based on the behavioral pattern information detected by the behavioral pattern analysis model (220), and can provide the behavioral pattern information and normal status together with the input video captured in real time.

[0072] The determination unit (230) can determine that an object and cluster are in an abnormal state when behavior pattern information that has not been learned is detected by the behavior pattern analysis model (220).

[0073] For example, a behavioral pattern analysis model (220) can determine whether an animal has a health abnormality, welfare abnormality, or infection by learning a dataset labeled with identification information for various target objects according to an experiment or field environment and animal behavior patterns according to health abnormalities, welfare abnormalities, or infectious diseases (e.g., TAD).

[0074] Therefore, the present invention can detect suspected individuals infected with an infectious disease very quickly and in real time within a space where each cluster of individuals is housed. In particular, it can provide an efficient monitoring system with relatively low system construction costs.

[0075] In one embodiment, the behavior pattern analysis model (220) can be constructed according to a supervised learning method using training data in which behavior pattern information of objects and clusters is labeled for 3D bounding box images of each object. At this time, the training network may include various architectures such as R(Region)-CNN, YOLO(You Look Only Once), and SSD(Single Shot Detector).

[0076] In another embodiment, the behavior pattern analysis model (220) may be an unsupervised learning model that clusters the patterns of objects based on the image and identification information of each object. For example, the behavior pattern analysis model (220) may be implemented as a Principal Component Analysis (PCA), a K-means Clustering model, a DBSCAN Clustering model, an Affinity Propagation Clustering model, a Hierarchical Clustering model, a Spectral Clustering model, etc.

[0077] For example, the behavioral pattern analysis model (220) includes a clustering algorithm based on the aforementioned unsupervised learning model that identifies detected identification information for each individual and cluster and detects behavioral pattern information based on each identification information. In this case, the behavioral pattern information may be classified based on the similarity between each cluster and the target image and identification information corresponding to each target object. For example, in the case of an animal cluster, the behavioral pattern information labeled on the target image and identification information may include resting states such as sleeping, sitting, and lying down, and states such as foraging, drinking water / eating.

[0078] The behavior pattern analysis device (100) inputs the image and identification information of each object into a behavior pattern analysis model to detect the behavior pattern information of each individual and can determine whether the animal group is normal based on the behavior pattern information. At this time, the behavior pattern analysis device (100) can provide the behavior pattern information and normal status of each individual together with the 3D input image of each individual captured in real time. In addition, if unlearned behavior pattern information is detected by the behavior pattern analysis model (220), it can be determined that the animal group is in an abnormal state.

[0079] In one embodiment, the behavior pattern analysis device (100) inputs the image and identification information of each object into the behavior pattern analysis model (220) to obtain behavior pattern information of each object, and can determine whether each object is in a normal state or an abnormal state based on the distribution of behavior pattern information accumulated over a certain period of time. The behavior pattern information may refer to not only labeled pattern classification information for each object but also unlabeled pattern classification information.

[0080] For example, behavioral pattern information may include 3D bounding box images detected by monitoring an individual in a normal state and labeled training data. For example, in the case of an animal colony, the arrangement of animal individuals lying down to sleep around a water fountain in a barn can be learned as behavioral pattern information.

[0081] Hereinafter, behavioral pattern information of an animal group will be described with reference to FIGS. 9 to 12. FIGS. 9 and 10 are drawings for explaining behavioral pattern information regarding the normal state of a group individual according to an embodiment of the present invention, and FIGS. 11 and 12 are drawings for explaining behavioral pattern information regarding the abnormal state of a group individual according to an embodiment of the present invention.

[0082] Referring to FIGS. 9 and 10, for a normal animal herd, the behavioral pattern information includes a fan-type huddling herd pattern, an irregular huddling herd pattern, a quadrangle-type huddling herd pattern, and an inverted triangle-type huddling herd pattern. In this case, the huddling herd pattern indicates a state in which the individuals are crouching or huddled together.

[0083] Referring to FIGS. 11 and 12, in the case of an abnormal animal herd, the behavioral pattern information includes an inverted triangle type huddling herd pattern, an irregular huddling herd pattern, a quadrangle type huddling herd pattern, a pistol type huddling herd pattern, a trapezoid huddling herd pattern, a detached fan type huddling herd pattern, a V shape huddling herd pattern, a loose order type huddling herd pattern, and a lateral recumbency herd pattern.

[0084] In an additional embodiment, the present invention defines the total number of image frames of an input image of one day as 1, sets weights for each time period (e.g., morning, lunch or / and evening), and applies weights to each behavioral pattern information to more accurately determine whether animal groups are normal.

[0085] In another embodiment, the present invention allows time-series behavioral pattern information regarding the sleeping posture patterns of an animal population according to a specific disease to be trained using training data labeled as specific disease information, such as African zoonotic fever. Subsequently, if the time-series behavioral pattern information of the animal population monitored through the input image (20) corresponds to the previously trained specific disease information, the corresponding specific disease information can be provided. Accordingly, the present invention allows for the verification of changes in the health and welfare of animal populations through the time-series changes in accumulated behavioral pattern information. Furthermore, it provides the effect of managing various diseases of animal populations, or all behaviors ranging from a normal state to an abnormal state, and early detection of infectious diseases.

[0086] In addition, the program can infer and calculate the volume of each object based on the 3D bounding box and identification information of the target object. As a result, the behavior and posture of each individual or / or animal group can be monitored more precisely and accurately. For example, the program can derive a predicted weight value of each individual(s) or / or animal group(s) based on training data labeled with the volume of the 3D bounding box of the target object and the weight information of individual individuals. For example, in the case of livestock, animals are shipped out after the fattening period of a certain rearing management period ends; the present invention provides the effect of being able to continuously monitor the weight or weight change (weight gain rate), which can guarantee the economic value of the pigs to be shipped, at any time until shipment.

[0087] On the other hand, when inferring animal weight by considering only two-dimensional aspects from existing two-dimensional boxes or edge lines, there is a problem in that the inference can only be made using images where only the animal's back is monitored. In particular, the degree of abdominal thickening, which affects body weight, cannot be determined at all. Furthermore, it is almost impossible to determine the thickness of the animal's back, and the predicted weight value is highly inaccurate. Meanwhile, in animals or livestock, the efficiency of weight gain, especially in animals, is determined by feed, exercise levels, or other influencing factors, and is closely related to the improvement of livestock productivity, which generates significant economic benefits.

[0088] Therefore, the appropriate time for shipment can be predicted and calculated through volume or weight data of each individual inferred based on the coordinate information of the three-dimensional bounding box of the present invention. Furthermore, through these data, not only can they serve as a basis for economically adjusting various conditions during the animal shipment process, but changes in weight can also primarily suspect diseases, poisoning, or certain serious infectious diseases. Thus, for example, it can help manage the mortality rate, which has an impact of approximately 70-80% or more on livestock productivity management.

[0089] In addition, the present invention provides a non-face-to-face, non-contact remote monitoring effect for weight prediction of three-dimensional individual(s) or / and group individual(s). Existing two-dimensional-based weight prediction methods for group individuals had limitations in that they had to predict for each individual because they were inferred through edge lines for each individual. However, the present invention can predict the weight of not only individual individuals but also animal groups based on identification information of target objects, thereby solving the problems of the conventional two-dimensional-based technology. As an example, when predicting the weight of an animal group, the average weight of an individual within the group can be inferred by dividing by the number of individuals in the group.

[0090] For example, the present invention forms accurate and precise actual edge lines on the three-dimensional surface (i.e., Shell) of an object (e.g., an animal) composed of a polyhedron of three or more sides using data of three-dimensional distance points. Then, highly reliable, accurate, and precise weight inference values ​​can be obtained from the information on the edge lines of the precise three-dimensional surface of the target individual or animal group obtained from the edge lines. Of course, the present invention can provide information including accurate parts, size, shape, and posture for each object (e.g., an animal), as well as visual information inferred and simulated over time for each piece of information, in various forms in a virtual space through a user interface.

[0091] In addition, 3D recognition is possible not only for the target object (e.g., if it is an animal) but also for objects surrounding the animal (e.g., surrounding animals, animals of different species, feed troughs, water troughs, fences, bedding, or tools such as branches or toys (such as balls)). For example, it is possible to represent and monitor the behavior of the animal itself, as well as the target animal and non-animals, in 3D recognition. Therefore, the present invention can monitor changes in behavior, posture patterns, etc., over time in real time by considering the surrounding environment of individual(s) or / and group(s). In other words, it helps to identify the correlations between various combinations of target objects or / and target animals that are monitored in 3D.

[0092] Referring to FIG. 13, the program may provide a user interface that classifies objects and clusters into normal patterns or abnormal patterns depending on whether they are normal or abnormal, and outputs the frequency of each behavioral pattern information classified as normal or abnormal patterns. Additionally, the user interface may provide a screen of a predetermined area divided according to the ratio of the area occupied by the corresponding behavioral pattern information based on the frequency of each behavioral pattern information. Specifically, the user interface may provide the behavioral pattern information of each cluster object in the form of a map divided by each behavioral pattern information within a screen of a predetermined area, and each behavioral pattern information may be displayed with an area proportional to the frequency within the corresponding normal or abnormal pattern. At this time, the rectangular treemap shape illustrated in black and white in FIG. 13 is merely one embodiment, and may include various shapes in which the area is divided according to the ratio of each behavioral pattern information within a predetermined area, and may be displayed in color as well as in black and white.

[0093] In the following, the description of identical configurations among the configurations illustrated in FIGS. 1 to 13 described above will be omitted.

[0094] FIG. 14 is a flowchart illustrating a three-dimensional based behavioral pattern analysis method according to another embodiment of the present invention.

[0095] Referring to FIG. 14, a method for analyzing behavioral patterns using a 3D-based behavioral pattern analysis device according to another embodiment of the present invention includes the steps of: collecting an input image through at least one camera positioned above a space in which clustered objects are accommodated (S110); inputting the input image into a detection model to generate a 3D bounding box for an object and detecting identification information for the object (S120); and inputting the image and identification information of the object into a behavioral pattern analysis model to detect a behavioral pattern (S130).

[0096] A 3D bounding box is created in a form that encloses all or part of an object, and identification information is generated based on the feature information of the 3D bounding box.

[0097] The identification information may include at least one reference point located on one surface of the three-dimensional bounding box, the planar area of ​​the three-dimensional bounding box, the volume of the three-dimensional bounding box, the straight-line distance from the camera to the reference point of the three-dimensional bounding box, the vertical-line distance from the camera to the bottom surface, or the angle between the straight line and the vertical line.

[0098] Step S120 may include the step of using a detection model to generate a 3D bounding box for a cluster containing more than a preset number of objects from an input image, and detecting identification information for the cluster.

[0099] Step S120 can display the color or shape of the 3D bounding box by changing it when the height or width of an object or cluster based on identification information changes.

[0100] Step S130 includes the step of detecting behavioral pattern information by inputting images and identification information of objects and clusters into a behavioral pattern analysis model.

[0101] A behavioral pattern analysis model is a model trained using labeled or unlabeled training data containing images and identification information of objects or clusters, as well as behavioral pattern information by type of object or cluster, and can output behavioral pattern information of objects and clusters based on input images.

[0102] The behavioral pattern analysis method using the behavioral pattern analysis device of the present invention may further include a step (S140) of determining whether an individual and a cluster are normal based on behavioral pattern information.

[0103] Step S140 provides behavioral pattern information and normality status along with input video captured in real time, and if behavioral pattern information that has not been learned is detected by the behavioral pattern analysis model, it can be determined as an entity or cluster in an abnormal state.

[0104] Step S140 may further include a step of classifying objects and clusters into normal patterns or abnormal patterns based on whether they are normal, and providing a user interface that outputs the frequency of behavioral pattern information classified into normal and abnormal patterns.

[0105] At this time, the user interface may provide a screen of a predetermined area divided according to the ratio of the area occupied by the corresponding behavioral pattern information, based on the frequency of each behavioral pattern information.

[0106] One embodiment of the present invention may also be implemented in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0107] Although the method and system of the present invention have been described in relation to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0108] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical concept or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0109] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0110] 100: Behavior pattern analysis device 110: Communication module 120: Memory 130: Processor 140: Database 210: Detection Model 220: Behavioral Pattern Analysis Model

Claims

Claim 1 A three-dimensional based behavioral pattern analysis device comprises: a data transmission and reception module; a memory in which a behavioral pattern analysis program is stored; and a processor that executes the program stored in the memory, wherein the program collects an input image through at least one camera positioned above a space in which a cluster of objects is accommodated, inputs the input image into a detection model to generate a three-dimensional bounding box for the object, detects identification information for the object, and inputs the image and identification information of the object into a behavioral pattern analysis model to detect a behavioral pattern, wherein the three-dimensional bounding box is generated in a form that encloses all or part of the object, and the identification information includes vertical distance information from the bottom surface of the three-dimensional bounding box to a reference point and volume information of the three-dimensional bounding box, and the behavioral pattern analysis model analyzes the posture change or density form of the object or cluster based on the change in the vertical distance information and volume information to detect the behavioral pattern. Claim 2 A behavior pattern analysis device according to claim 1, wherein the three-dimensional bounding box is overlaid on a target image containing at least one object, and the identification information includes at least one reference point located on one surface of the three-dimensional bounding box, the planar area of ​​the three-dimensional bounding box, the straight-line distance from the camera to the reference point of the three-dimensional bounding box, or the angle between the straight line and the vertical line. Claim 3 A behavior pattern analysis device according to claim 1, wherein the program uses the detection model to generate a 3D bounding box for a cluster containing more than a preset number of objects from the input image, and detects identification information for the cluster. Claim 4 A behavioral pattern analysis device according to paragraph 3, wherein the program changes the color or shape of the three-dimensional bounding box and displays it when the height or width of an object or cluster based on the identification information changes. Claim 5 A behavior pattern analysis device according to paragraph 3, wherein the program inputs identification information of the cluster into the behavior pattern analysis model to detect behavior pattern information, and the behavior pattern analysis model is a model trained using training data labeled with identification information of the individual or cluster and behavior pattern information by type of individual or cluster, or unlabeled training data, and outputs behavior pattern information of the individual and cluster based on the input image. Claim 6 A behavior pattern analysis device according to claim 5, wherein the program determines whether an individual and a cluster are normal based on the behavior pattern information, and provides the behavior pattern information and normal status together with the input video captured in real time, and determines the individual and cluster as being in an abnormal state when behavior pattern information that has not been learned is detected by the behavior pattern analysis model. Claim 7 A behavioral pattern analysis device according to claim 6, wherein the program classifies the object and cluster into normal patterns or abnormal patterns depending on whether they are normal, and provides a user interface that outputs the frequency of each behavioral pattern information classified as normal patterns and abnormal patterns, wherein the user interface provides a screen of a predetermined area divided according to the ratio of the area occupied by the corresponding behavioral pattern information based on the frequency of each behavioral pattern information. Claim 8 A method for analyzing behavioral patterns using a three-dimensional based behavioral pattern analysis device, comprising: (a) collecting input images through at least one camera positioned above a space in which clustered entities are accommodated; (b) inputting the input images into a detection model to generate a three-dimensional bounding box for the entities and detecting identification information for the entities; and (c) inputting the images and identification information of the entities into a behavioral pattern analysis model to detect behavioral patterns, wherein the three-dimensional bounding box is generated in a form that encloses all or part of the entities, and the identification information includes vertical distance information from the bottom surface of the three-dimensional bounding box to a reference point and volume information of the three-dimensional bounding box, and wherein step (c) involves the behavioral pattern analysis model analyzing the posture change or density form of the entities or clusters based on changes in the vertical distance information and volume information to detect the behavioral pattern. Claim 9 A method for analyzing behavioral patterns according to claim 8, wherein the three-dimensional bounding box is overlaid on a target image containing at least one object, and the identification information includes at least one reference point located on one surface of the three-dimensional bounding box, the planar area of ​​the three-dimensional bounding box, the straight-line distance from the camera to the reference point of the three-dimensional bounding box, or the angle between the straight line and the vertical line. Claim 10 A method for analyzing behavioral patterns according to claim 8, wherein step (b) includes the step of generating a three-dimensional bounding box for a cluster containing more than a preset number of objects from the input image using the detection model, and detecting identification information for the cluster. Claim 11 A method for analyzing behavioral patterns, wherein step (b) above changes the color or shape of the three-dimensional bounding box when the height or width of the object or cluster based on the identification information changes. Claim 12 In claim 10, the above step (c) includes the step of inputting the image and identification information of the cluster into the behavior pattern analysis model to detect behavior pattern information, wherein the behavior pattern analysis model is a model trained using labeled training data or unlabeled training data, and outputs behavior pattern information of the individual and cluster based on the input image. Claim 13 A method for analyzing behavioral patterns according to claim 12, further comprising: (d) a step of determining whether an individual and a cluster are normal based on the behavioral pattern information, wherein step (d) provides the behavioral pattern information and normal status together with the input image captured in real time, and if behavioral pattern information that has not been learned is detected by the behavioral pattern analysis model, the individual and cluster are determined to be in an abnormal state. Claim 14 A method for analyzing behavioral patterns according to claim 13, wherein step (d) further includes the step of classifying the object and cluster into normal patterns or abnormal patterns depending on whether they are normal, and providing a user interface that outputs the frequency of each behavioral pattern information classified into normal patterns and abnormal patterns, and wherein the user interface provides a screen of a predetermined area divided by the ratio of the area occupied by the corresponding behavioral pattern information based on the frequency of each behavioral pattern information.

Images