Method, apparatus, and recording medium for generating three-dimensional posture data of animal

Abstract

The present disclosure relates to a method, an apparatus, and a recording medium for generating three-dimensional posture data of an animal. The method for generating three-dimensional posture data for training a three-dimensional posture estimation model for an animal, according to an embodiment of the present disclosure, comprises the steps of: generating a two-dimensional posture prediction model; acquiring, by using the two-dimensional posture prediction model, two-dimensional posture information of a target animal from a two-dimensional image of the target animal captured through a plurality of cameras; and acquiring three-dimensional posture information of the target animal on the basis of the two-dimensional posture information of the target animal, wherein the step of generating the two-dimensional posture prediction model comprises the steps of: collecting an image of the target animal for training; extracting a plurality of frames from the image of the target animal for training; generating two-dimensional posture data by labeling each of the plurality of frames with the two-dimensional posture information of the target animal for training; and training by using the two-dimensional posture data.

Description

Method, device and recording medium for generating three-dimensional posture data of animals

[0001] The present disclosure relates to a method, device, and recording medium for generating three-dimensional posture data of an animal, and more particularly, to a method, device, and recording medium for generating three-dimensional posture data for learning a three-dimensional posture estimation model of an animal by obtaining three-dimensional posture information of the animal from a two-dimensional image.

[0002] Recently, research on animal posture and movement has attracted considerable attention in various fields, including animal behavior research, animal health monitoring, and wildlife conservation. In particular, active research is underway on techniques for estimating 3D animal posture from images using artificial intelligence-based posture estimation models, such as deep learning and machine learning.

[0003] Building such AI-based pose estimation models requires a sufficient dataset for training. However, for animals, especially wild animals, such datasets are scarce, and most of the available data are two-dimensional pose data, making it difficult to use them to train pose estimation models capable of estimating three-dimensional poses.

[0004] Meanwhile, various methods are known for collecting three-dimensional animal posture information by measuring their posture and movements. For example, methods are known for measuring animal posture by forcibly fixing parts of the animal's body or forcing the animal into a confined space. Another method involves attaching various sensors to the animal's joints and using separate equipment to measure the animal's posture. However, these methods suffer from the difficulty of controlling the animal, making it difficult to induce natural postures and movements. Furthermore, the friction and weight of attachments such as sensors restrict the movement of certain muscles or joints, making it difficult to accurately measure posture and movements. Furthermore, the high cost of devices for measuring movements is a limitation.

[0005] Accordingly, there is a need for a method for collecting three-dimensional posture data on various postures and movements of animals.

[0006] The present disclosure is intended to solve the problems of the above-described prior art, and its purpose is to collect three-dimensional posture data of animals for learning a posture estimation model.

[0007] In addition, the purpose is to build an artificial intelligence-based 3D posture estimation model that can accurately estimate the 3D posture of an animal using the 3D posture data of the animal.

[0008] A method for generating 3D posture data for learning a 3D posture estimation model of an animal according to one embodiment of the present disclosure includes the steps of generating a 2D posture prediction model, obtaining 2D posture information of a target animal from 2D images of the target animal captured by a plurality of cameras using the 2D posture prediction model, and obtaining 3D posture information of the target animal based on the 2D posture information of the target animal, wherein the step of generating the 2D posture prediction model includes the steps of collecting images of a target animal for learning, extracting a plurality of frames from the images of the target animal for learning, generating 2D posture data by labeling 2D posture information of the target animal for each of the plurality of frames, and learning using the 2D posture data.

[0009] According to one embodiment of the present disclosure, the two-dimensional posture information of the target animal may include two-dimensional coordinate information of joint elements constituting the target animal.

[0010] According to one embodiment of the present disclosure, in the step of acquiring three-dimensional posture information of a target animal, three-dimensional coordinate information of joint elements constituting the target animal can be derived based on triangulation.

[0011] According to one embodiment of the present disclosure, a two-dimensional pose prediction model can be optimized through hyperparameter tuning. Here, hyperparameter tuning can be performed based on Bayesian optimization.

[0012] According to one embodiment of the present disclosure, the step of acquiring three-dimensional posture information of a target animal may include a step of determining internal parameters and external parameters through calibration of a plurality of cameras.

[0013] According to one embodiment of the present disclosure, a device for generating 3D posture data for learning a 3D posture estimation model of an animal includes a learning model management unit for generating a 2D posture prediction model, a 2D posture information acquisition unit for acquiring 2D posture information of a target animal from 2D images of the target animal captured by a plurality of cameras using the 2D posture prediction model, and a 3D posture information acquisition unit for acquiring 3D posture information of the target animal based on the 2D posture information of the target animal, wherein the learning model management unit includes an image collection unit for collecting images of a target animal for learning, a frame extraction unit for extracting a plurality of frames from images of the target animal for learning, a labeling unit for labeling 2D posture information of the target animal for each of the plurality of frames to generate 2D posture data, and a learning unit for learning using the 2D posture data.

[0014] According to one embodiment of the present disclosure, by obtaining two-dimensional posture information of an animal from a two-dimensional image and obtaining three-dimensional posture information from the two-dimensional posture information of the animal based on triangulation, three-dimensional posture data regarding various postures and movements of the animal can be collected.

[0015] In addition, a 3D pose estimation model with improved accuracy and reliability can be built using 3D pose data.

[0016] FIG. 1 is a schematic diagram of an entire system for estimating a three-dimensional pose of an animal from a two-dimensional image according to one embodiment of the present disclosure.

[0017] FIG. 2 is a functional block diagram schematically illustrating the functional configuration of a device for generating three-dimensional posture data according to one embodiment of the present disclosure.

[0018] FIG. 3 is a block diagram schematically illustrating a detailed configuration of a learning model management unit of a 3D posture data generation device according to one embodiment of the present disclosure.

[0019] FIG. 4 is a drawing exemplarily showing how labeling is performed on an image of a learning target animal according to one embodiment of the present disclosure.

[0020] FIG. 5 is a drawing exemplarily showing a process of calibrating a camera according to one embodiment of the present disclosure.

[0021] FIG. 6 is a diagram exemplarily showing a process of generating three-dimensional posture data according to one embodiment of the present disclosure.

[0022] FIG. 7 is a flowchart illustrating a method for generating three-dimensional posture data according to one embodiment of the present disclosure.

[0023] [Explanation of symbols]

[0024] 100: System for estimating the three-dimensional posture of an animal

[0025] 200: 3D posture data generation device

[0026] 210: Learning Model Management Department

[0027] 220: 2D posture information acquisition unit

[0028] 230: 3D posture information acquisition unit

[0029] 240: Communications Department

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. Hereinafter, specific descriptions of previously known functions and configurations will be omitted if deemed likely to unnecessarily obscure the gist of the present disclosure. Furthermore, it should be noted that the following description relates only to one embodiment of the present disclosure and that the present disclosure is not limited thereto.

[0031] The terminology used in this disclosure is merely used to describe specific embodiments and is not intended to be limiting of the present disclosure. For example, a component expressed in the singular should be understood to include plural components unless the context clearly indicates only a singular meaning. It should be understood that the term "and / or" used in this disclosure encompasses any and all possible combinations of one or more of the listed items. The terms "comprises" or "has" used in this disclosure are intended to specify that a feature, number, step, operation, component, part, or combination thereof described in this disclosure is present, but the use of such terms does not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0032] In the embodiments of the present disclosure, a "module" or "part" refers to a functional component that performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one software module and implemented by at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0033] Additionally, unless otherwise defined, all terms used in this disclosure, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with the contextual meaning of the relevant technology, and should not be interpreted in an unduly limiting or expansive manner unless explicitly defined otherwise in this disclosure.

[0034] Hereinafter, with reference to the attached drawings, embodiments of the present disclosure will be described in detail.

[0035] FIG. 1 is a schematic diagram of an entire system for estimating a three-dimensional pose of an animal from a two-dimensional image according to one embodiment of the present disclosure.

[0036] As illustrated in FIG. 1, according to one embodiment of the present disclosure, a three-dimensional posture of an animal can be estimated from a two-dimensional image through a system (100) for estimating a three-dimensional posture of an animal. Specifically, when a two-dimensional image of an animal is input into the system (100) for estimating a three-dimensional posture of an animal, the three-dimensional posture of the animal can be estimated through a deep learning-based three-dimensional posture estimation model.

[0037] Meanwhile, in order to train such a deep learning-based 3D posture estimation model, 3D posture data containing 3D posture information is essential. However, as previously explained, it is difficult to obtain 3D posture data of an animal by directly measuring the animal's posture by attaching a sensor, etc. Therefore, in the present disclosure, 2D posture information is detected from a 2D image, and 3D posture information is obtained based on the 2D posture information, thereby generating 3D posture data for training a posture estimation model.

[0038] According to one embodiment of the present disclosure, a system (100) for estimating a three-dimensional pose of an animal may include a device for generating three-dimensional pose data for learning a three-dimensional pose estimation model of the animal.

[0039] FIG. 2 is a functional block diagram schematically illustrating the functional configuration of a device for generating three-dimensional posture data according to one embodiment of the present disclosure.

[0040] Referring to FIG. 2, a device (200) for generating three-dimensional posture data according to an embodiment of the present disclosure may include a learning model management unit (210), a two-dimensional posture information acquisition unit (220), a three-dimensional posture information acquisition unit (230), and a communication unit (240). The components illustrated in FIG. 2 do not reflect all functions of the three-dimensional posture data generation device (200), nor are they essential, and thus the three-dimensional posture data generation device (200) may include more or fewer components than the illustrated components.

[0041] According to one embodiment of the present disclosure, at least some of the learning model management unit (210), the 2D posture information acquisition unit (220), the 3D posture information acquisition unit (230), and the communication unit (240) may be program modules that communicate with an external system. These program modules may be included in the 3D posture data generation device (200) in the form of an operating system, an application program module, or other program modules, and may be physically stored in various known memory devices. In addition, these program modules may be stored in a remote memory device that can communicate with the 3D posture data generation device (200). Meanwhile, these program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform specific tasks or execute specific abstract data types, which will be described later according to the present disclosure.

[0042] Meanwhile, although the 3D posture data generation device (200) has been described as above, this description is exemplary and it is obvious to those skilled in the art that at least some of the components or functions of the 3D posture data generation device (200) may be realized or included in an external system (not shown) as needed.

[0043] According to one embodiment of the present disclosure, the learning model management unit (210) performs a function of generating a two-dimensional posture prediction model.

[0044] FIG. 3 is a block diagram schematically illustrating a detailed configuration of a learning model management unit of a 3D posture data generation device according to one embodiment of the present disclosure.

[0045] Referring to FIG. 3, a learning model management unit (210) according to one embodiment of the present disclosure may include an image collection unit (211), a frame extraction unit (212), a labeling unit (213), a learning unit (214), and an optimization unit (215). The components illustrated in FIG. 3 do not reflect all functions of the learning model management unit (210), nor are they essential, and thus the learning model management unit (210) may include more or fewer components than the illustrated components.

[0046] According to one embodiment of the present disclosure, the image collection unit (211) may perform a function of collecting two-dimensional images including various movements and postures of a learning target animal. In one embodiment, the image collection unit (211) may collect images by photographing the learning target animal through a camera for a predetermined period of time (e.g., 10 to 15 seconds). Here, the camera may be a general two-dimensional camera such as a webcam, and the frame rate and resolution of the camera may be set in consideration of data size, data processing speed, etc. For example, the camera may be set to capture images at a resolution of 640 * 480 at a speed of 15 frames per second (fps). In one embodiment, the image collection unit (211) may collect images of the learning target animal through multiple cameras. For example, images may be collected through three cameras. Accordingly, images of the same learning target animal from different angles may be collected. In one embodiment, the image collection unit (211) may collect images of two or more learning target animals of the same species.

[0047] According to one embodiment of the present disclosure, the frame extraction unit (212) may perform a function of dividing the images collected by the image collection unit (211) into frame units and extracting individual frames. In one embodiment, the frame extraction unit (212) may extract more than 10,000 frames from image data of two or more learning target animals captured by multiple cameras. The frame extraction may be performed through image editing software (e.g., Python libraries OpenCV and MoviePy), and each frame may be saved as an image file (e.g., JPEG or PNG).

[0048] According to one embodiment of the present disclosure, the labeling unit (213) may perform a function of generating two-dimensional posture data by labeling two-dimensional posture information of a learning target animal for each frame extracted by the frame extraction unit (212). Specifically, the labeling unit (213) may generate two-dimensional posture data by labeling key points corresponding to the positions of joint elements constituting the learning target animal in each extracted frame. In one embodiment, the labeling task may be performed using manual labeling information generated manually by a worker or may be automatically performed through an algorithm.

[0049] FIG. 4 is a diagram illustrating an example of how labeling is performed on an image of a learning target animal according to one embodiment of the present disclosure. FIG. 4 illustrates a case where the learning target animal is a cat.

[0050] Referring to Fig. 4, labeling can be performed to indicate and designate keypoints corresponding to the locations of joint elements constituting a cat, which is a learning target animal. Here, in the case of a cat, a labeling task can be performed to indicate keypoints corresponding to the locations of a total of 18 joint elements. By performing labeling on frames extracted from multiple images, a large amount of labeling data can be generated. For example, by performing a labeling task to indicate 18 keypoints for 10,000 frames, a total of 180,000 labels can be designated.

[0051] According to one embodiment of the present disclosure, the learning unit (214) may perform a function of constructing a two-dimensional posture prediction model by performing learning using two-dimensional posture data generated by the labeling unit (213). Specifically, the learning unit (214) may construct a two-dimensional posture prediction model by performing training on a pre-trained model using a supervised learning method using two-dimensional posture data. In one embodiment, the two-dimensional posture prediction model may be applied with a deep neural network model, a convolutional neural network model, etc. For example, a neural network model such as ResNet-101 may be applied.

[0052] According to one embodiment of the present disclosure, the optimization unit (214) may perform an optimization task to improve the performance of a two-dimensional pose prediction model. In one embodiment, the optimization unit (214) may improve the performance of the two-dimensional pose prediction model by adjusting parameters such as the number of neurons, activation function, optimizer, learning rate, batch size, data augmentation, and learning period (epoch) through hyperparameter tuning. In one embodiment, the optimization unit (214) may perform hyperparameter tuning based on Bayesian optimization. Bayesian optimization is a method that provides a probabilistic approach for finding an optimal set of hyperparameters, and may include the following steps.

[0053] (1) Select hyperparameter values ​​and train the model using these values.

[0054] (2) Evaluate the model's performance and calculate the score.

[0055] (3) Update hyperparameter values ​​based on potential improvements that can improve model performance.

[0056] (4) This process is repeated until a stopping condition, such as minimizing the loss metric or maximizing the classification accuracy, is met.

[0057]

[0058] *Referring again to FIG. 2, according to one embodiment of the present disclosure, the 2D posture information acquisition unit (220) can perform a function of acquiring 2D posture information of a target animal from a 2D image of the target animal using a 2D posture prediction model generated by the learning model management unit (210).

[0059] According to one embodiment of the present disclosure, a two-dimensional posture information acquisition unit (220) can acquire two-dimensional images showing various postures and movements of a target animal. In one embodiment, the two-dimensional images of the target animal can be acquired through multiple cameras (e.g., three cameras).

[0060] According to one embodiment of the present disclosure, a two-dimensional posture information acquisition unit (220) can detect a target animal from a two-dimensional image of the target animal acquired using a two-dimensional posture prediction model and acquire two-dimensional posture information of the target animal. In one embodiment, the two-dimensional posture information acquisition unit (220) can acquire two-dimensional coordinates (X, Y) of each joint element constituting the target animal.

[0061] According to one embodiment of the present disclosure, the 3D posture information acquisition unit (230) performs a function of acquiring 3D posture information based on 2D posture information of the target animal. In one embodiment, the 3D posture information acquisition unit (230) can acquire 3D posture information of the target animal by calculating 3D coordinate information (X, Y, Z) of joint elements constituting the target animal based on triangulation. Here, triangulation is a technology for estimating the position of a camera or the position of an object by using feature points on a 2D image in a 3D space, and in order to perform triangulation, a process of calibrating the camera must be performed first.

[0062] FIG. 5 is a drawing exemplarily showing a process of calibrating a camera according to one embodiment of the present disclosure.

[0063] Referring to Fig. 5, first, an image of a specific pattern (e.g., a Checkerboard Charuco pattern) to be used for camera calibration is acquired through each camera. The initial parameters of each camera are estimated using the acquired pattern images. Specifically, the intrinsic parameters (focal length, center point, distortion coefficient, etc.) and extrinsic parameters (position and orientation of the camera relative to an object, etc.) of the camera are estimated. Next, image preprocessing is performed to remove distortion and adjust brightness and contrast, and then common keypoints are extracted from each image. These keypoints can be matched with other images, thereby identifying the relationship between the cameras that captured the images.

[0064] Next, the relative positions and orientations between multiple cameras are precisely established, and the matching images are aligned. Then, triangulation is performed to derive the 3D spatial coordinates of the matched keypoints on the image plane. Bundle Adjustment is used to compensate for errors arising during the matching and triangulation stages, accurately adjusting the camera parameters. Through this process, the internal and external parameters of each of the multiple cameras can be determined.

[0065] According to one embodiment of the present disclosure, the communication unit (240) can perform a function to enable data transmission and reception from / to the learning model management unit (210), the two-dimensional posture information acquisition unit (220), and the three-dimensional posture information acquisition unit (230).

[0066] According to one embodiment of the present disclosure, the three-dimensional posture data generating device (200) may further include a learning data generating unit (not shown) that generates a learning data set. In one embodiment, the learning data generating unit may generate a learning data set composed of input data and corresponding output data. For example, the learning data generating unit may generate a learning data set using a two-dimensional image of a target animal as input data and three-dimensional posture data of the target animal as output data. In another example, the learning data generating unit may generate a learning data set using two-dimensional posture data of the target animal as input data and three-dimensional posture data of the target animal as output data.

[0067] FIG. 6 is a diagram exemplarily showing a process of generating three-dimensional posture data according to one embodiment of the present disclosure.

[0068] As illustrated in Fig. 6, by inputting a two-dimensional image of a target animal acquired through multiple cameras, two-dimensional posture information of the target animal can be acquired through a two-dimensional posture prediction model. Here, the two-dimensional posture information can include two-dimensional coordinate information of joint elements constituting the target animal in each frame of the image.

[0069] In one embodiment, 3D pose information can be acquired based on 2D pose information of a target animal. Specifically, 3D coordinate information of the joint elements constituting the target animal can be derived based on triangulation. To achieve this, multiple cameras must be calibrated first, and by performing camera calibration, the internal and external parameters of the cameras can be determined.

[0070] FIG. 7 is a flowchart illustrating a method for generating three-dimensional posture data according to one embodiment of the present disclosure.

[0071] First, in step (S701), the 3D posture data generation device (200) generates a 2D posture prediction model based on deep learning. To this end, 2D posture data for training the 2D posture prediction model is generated. Specifically, a plurality of frames may be extracted from images of a learning target animal collected through a plurality of cameras, and 2D posture information of the learning target animal may be labeled for each frame to generate 2D posture data. In one embodiment, the 2D posture prediction model may be a model trained using 2D posture data, and may be a model optimized through hyperparameter tuning. Here, the hyperparameter tuning may be performed based on Bayesian optimization.

[0072] In step (S703), the 3D posture data generation device (200) can obtain 2D posture information of the target animal from 2D images of the target animal captured by multiple cameras using the 2D posture prediction model generated in step (S701). Here, the 2D posture information can include 2D coordinate information of joint elements constituting the target animal.

[0073] In step (S705), the 3D posture data generation device (200) can acquire 3D posture information of the target animal based on the 2D posture information acquired in step (S705). In one embodiment, the 3D posture information of the target animal can be acquired by calculating 3D coordinate information of joint elements constituting the target animal based on triangulation. To this end, camera calibration may be performed in advance to determine internal and external parameters of the camera. In one embodiment, the camera calibration may be performed using a Checkerboard Charuco pattern.

[0074] The embodiments of the present disclosure described above may be implemented in the form of program instructions that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either singly or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present disclosure or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present disclosure, and vice versa.

[0075] Although the present disclosure has been described with specific details such as specific components and limited examples, the above examples are provided only to help a more general understanding of the present disclosure, and the present disclosure is not limited thereto, and a person having ordinary knowledge in the technical field to which the present disclosure pertains may make various modifications and variations based on this description.

[0076] For example, in one embodiment of the present disclosure, abnormal animal behavior can be detected through dynamic data, such as animal movement information, acquired through the information acquisition unit and the movement analysis unit, and further, the presence of animal disease can be diagnosed at an early stage. Specifically, by acquiring accurate animal movement information through the information acquisition unit and the movement analysis unit, and comparing this with normal animal movement, the presence of abnormal behavior or the presence of disease can be detected.

[0077] Therefore, the spirit of the present disclosure should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below, as well as the claims described below, are considered to fall within the scope of the spirit of the present disclosure.

Claims

1. A method for generating 3D posture data for learning a 3D posture estimation model of an animal, Steps to create a 2D posture prediction model, A step of obtaining 2D posture information of a target animal from a 2D image of the target animal captured through multiple cameras using the 2D posture prediction model, and A step of obtaining three-dimensional posture information of the target animal based on two-dimensional posture information of the target animal is included. The step of generating the above two-dimensional posture prediction model is: Step of collecting images of the target animal for learning; A step of extracting multiple frames from an image of the above learning target animal, A step of generating two-dimensional posture data by labeling two-dimensional posture information of the learning target animal for each of the plurality of frames, and A step of learning using the above two-dimensional posture data Including Method for generating 3D pose data.

2. In paragraph 1, A method for generating three-dimensional posture data, wherein the two-dimensional posture information of the target animal includes two-dimensional coordinate information of joint elements constituting the target animal.

3. In paragraph 1, A method for generating 3D posture data, wherein in the step of acquiring 3D posture information of the target animal, 3D coordinate information of joint elements constituting the target animal is calculated based on triangulation.

4. In paragraph 1, The above two-dimensional posture prediction model is a method for generating three-dimensional posture data optimized through hyperparameter tuning.

5. In paragraph 4, A method for generating 3D pose data, wherein the above hyperparameter tuning is performed based on Bayesian optimization.

6. In paragraph 1, A method for generating 3D posture data, wherein the step of acquiring 3D posture information of the target animal includes a step of determining internal parameters and external parameters through correction of the plurality of cameras.

7. A recording medium recording a computer program for executing the method according to paragraph 1.

8. A device for generating 3D posture data for learning a 3D posture estimation model of an animal, A learning model management unit that generates a 2D posture prediction model, A 2D posture information acquisition unit that acquires 2D posture information of a target animal from 2D images of the target animal captured through multiple cameras using the 2D posture prediction model, and It includes a 3D posture information acquisition unit that acquires 3D posture information of the target animal based on 2D posture information of the target animal, The above learning model management department, Image collection unit that collects images of the target animal for learning; A frame extraction unit that extracts multiple frames from an image of the above learning target animal, A labeling unit that generates two-dimensional posture data by labeling two-dimensional posture information of the learning target animal for each of the plurality of frames; and A learning unit that learns using the above two-dimensional posture data Including 3D posture data generation device.

Images