Apparatus and method for archiving motion data of dance movement

Abstract

An apparatus for archiving motion data for dance movements according to one embodiment disclosed in this document includes a communication circuit, a memory including a database, and a processor electrically connected to the communication circuit and the memory. The processor acquires a video of a dance movement captured by a camera using the communication circuit, and acquires pose estimation data corresponding to the dance movement for each frame constituting the video by analyzing the video using a pose estimation model, and stores motion data by combining the pose estimation data in chronological order in a database, and provides a user interface for searching the motion data stored in the database, and when motion data is selected through the user interface, the motion data can be visualized as a three-dimensional skeleton.

Description

Technology Field

[0001] The embodiments disclosed in this document relate to an apparatus and method for archiving motion data for various dance movements and utilizing the same. Background Technology

[0002] With the advancement of media art, where art and technology converge, the demand for dance creations integrated with new technologies is steadily increasing. For example, dance works utilizing technologies such as artificial intelligence, augmented reality, and virtual reality are becoming more widespread, leading to a growing need for technical support and data management required to create such works.

[0003] Motion capture technology can be utilized in the field of dance to record movements as data. Motion capture refers to a technology that records human movement in digital form through methods such as attaching sensors to the body or using infrared cameras. Generating movement data using motion capture may require expensive specialized equipment and professional expertise. The problem to be solved

[0004] In the case of dance, systematic archiving of movement data can be difficult because movements, as the products of creation, are ephemeral and highly volatile. Due to these archiving challenges, there are limitations to the development of collaboration tools that allow for the sharing of movements and the exchange of feedback in a non-face-to-face environment. Furthermore, since existing archiving relies on video, it is difficult to access or analyze structured data, and processing or modifying the data can also be challenging. Utilizing motion capture technology to acquire structured data may require excessive costs and manpower.

[0005] The embodiments of the present invention are intended to provide a platform that archives artistic movements into structured data without dedicated equipment or systems, and enables creators to utilize the collected data in various ways. means of solving the problem

[0006] An apparatus for archiving motion data for dance movements according to one embodiment disclosed in this document includes a communication circuit, a memory including a database, and a processor electrically connected to the communication circuit and the memory. The processor acquires a video of a dance movement captured by a camera using the communication circuit, and acquires pose estimation data corresponding to the dance movement for each frame constituting the video by analyzing the video using a pose estimation model, and stores motion data by combining the pose estimation data in chronological order in a database, and provides a user interface for searching the motion data stored in the database, and when motion data is selected through the user interface, the motion data can be visualized as a three-dimensional skeleton.

[0007] According to one embodiment, the processor can acquire motion search data by analyzing a motion captured for data search within a database using a pose estimation model, and output a search result based on the similarity between the motion data and the motion search data.

[0008] According to one embodiment, the processor stores motion data along with text describing the motion data in a database and performs text mining within the database, thereby allowing the motion data to be clustered according to the similarity of the text.

[0009] According to one embodiment, the processor receives an input to move a joint included in a three-dimensional skeleton visualizing motion data in a specific frame, modifies the coordinate information of the joint for the specific frame included in the motion data according to the input, and can perform motion blending for the specific frame and a frame adjacent to the specific frame.

[0010] A method for archiving motion data for a dance movement according to one embodiment disclosed in this document may include the steps of: acquiring a video of a dance movement captured by a camera; acquiring pose estimation data corresponding to the dance movement for each frame constituting the video by analyzing the video using a pose estimation model; storing motion data by combining the pose estimation data in chronological order in a database; providing a user interface for searching motion data stored in the database; and, when motion data is selected through the user interface, visualizing the motion data as a three-dimensional skeleton. Effects of the invention

[0011] According to the embodiments disclosed in this document, by constructing an archive of dance movements using motion data obtained using a posture estimation model, a platform can be provided that enables a creator to efficiently utilize data on dance movements for creative activities.

[0012] In addition, by providing features such as motion search and motion clustering, users can search for motion data conveniently and intuitively.

[0013] In addition, by modifying motion data based on input for visualized joints and performing motion blending, users can intuitively edit motion data, and edited frames can be naturally connected with neighboring frames.

[0014] In addition, new motions can be created through the editing of motion data, and the created motions can be utilized in various fields such as art work, computer graphics, and animation.

[0015] In addition, various effects identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0016] FIG. 1 illustrates the operating environment of a device for archiving motion data for dance movements according to one embodiment. FIG. 2 is a block diagram illustrating the configuration of a device for archiving motion data for dance movements according to one embodiment. FIG. 3 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment. FIG. 4 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment. FIG. 5 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment. FIG. 6 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment. FIG. 7 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment. FIG. 8 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment. FIG. 9 is a flowchart illustrating a method for archiving motion data for dance movements according to one embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0017] Hereinafter, some embodiments of the present invention will be described in detail with reference to the exemplary drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that the invention includes various modifications, equivalents, or substitutions of the embodiments. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the embodiments of the present invention, if it is determined that a detailed description of related known components or functions would hinder understanding of the embodiments of the present invention, such detailed description is omitted.

[0019] FIG. 1 illustrates the operating environment of a device for archiving motion data for dance movements according to one embodiment.

[0020] Referring to FIG. 1, a motion data archiving device (100) according to one embodiment can generate and store motion data for a dance movement demonstrated by a user (11), and can share, edit, and reproduce the motion data among multiple users (11). The motion data archiving device (100) can be implemented in the form of a server (e.g., a web server).

[0021] The motion data archiving device (100) can receive video from a user terminal (12) that has captured the dance movements of a user (11). The motion data archiving device (100) can obtain motion data in which the dance movements are expressed as joint coordinates by analyzing the video using an artificial intelligence model. The motion data archiving device (100) can archive motion data obtained from multiple users (11) and can provide a platform in which multiple users (11) can search, edit, and reproduce the stored motion data.

[0023] FIG. 2 is a block diagram illustrating the configuration of a device for archiving motion data for dance movements according to one embodiment.

[0024] Referring to FIG. 2, a motion data archiving device (200) according to one embodiment may be implemented as one of various types of computing devices and may be physically located on-premise or in the cloud. As another example, the motion data archiving device (200) may be implemented as a computing device in an environment distributed to two or more. A motion data archiving device (200) according to one embodiment may include a communication circuit (210), a memory (220), and a processor (230).

[0025] The communication circuit (210) may be an interface that communicates wirelessly or via a wire with a user terminal or other external device. The communication circuit (210) may transmit and receive data with the user terminal or other external device.

[0026] The memory (220) may include volatile memory and / or non-volatile memory. The memory (220) may store various data handled by the motion data archiving device (200), and may store video, motion data, pose estimation models, language models, etc. The memory (220) may include a database for storing motion data.

[0027] The processor (230) may be electrically connected to the communication circuit (210) and the memory (220). The processor (230) may control the communication circuit (210) and the memory (220) and perform various data processing and operations. Although the processor (230) is shown as a single configuration in FIG. 2, it may be implemented by being separated into multiple configurations. The processor (230) may perform the following operations by executing software or instructions stored in the memory (220).

[0028] According to one embodiment, the processor (230) can acquire a video of a dance movement captured by a camera using a communication circuit (210). The processor (230) can receive a video captured by a user using a user terminal. The video may be a video of a dance movement captured by the user that the user wishes to archive.

[0029] According to one embodiment, the processor (230) can obtain pose estimation data corresponding to a dance movement for each frame constituting the video by analyzing the video using a pose estimation model. The processor (230) can perform analysis on each frame included in the video. The pose estimation model may be, for example, a BlazePose and can extract 33 joints from an image of a person performing a dance movement. The pose estimation data may include information regarding three-dimensional coordinates (e.g., x-coordinate, y-coordinate, and z-coordinate), probabilities (accuracy of estimation), and names of joints (e.g., left elbow, right knee, etc.). The three-dimensional coordinates may have normalized values. The pose estimation data may be in JSON format.

[0030] According to one embodiment, the processor (230) can store motion data in a database by combining pose estimation data in chronological order. Since pose estimation data is data obtained from one frame, if pose estimation data is obtained for the entire video, pose estimation data corresponding to each frame included in the video can be generated. Motion data composed of a single file can be generated by arranging and combining the code included in the pose estimation data in chronological order. The motion data can also be in JSON format. The processor (230) can store the motion data in a database.

[0031] According to one embodiment, the processor (230) may provide a user interface for searching motion data stored in a database. The user interface may include a list of motion data uploaded by multiple users. The list may include the names, titles, tags, descriptions, and / or preview images of the motion data uploaded by the uploading users.

[0032] According to one embodiment, when motion data is selected through a user interface, the processor (230) can visualize the motion data as a three-dimensional skeleton. When specific motion data is selected from a list of motion data, the processor (230) can load the information included in the selected motion data and then display the position values ​​of each joint in three dimensions on the screen. Since the coordinates classified by joint and arranged in chronological order included in the JSON file are sequentially visualized as a three-dimensional skeleton, the motion of the three-dimensional skeleton corresponding to the dance movements included in the video can be visually played.

[0033] According to one embodiment, the processor (230) may provide a user interface that can intuitively modify the position of a joint included in the motion data. For example, the processor (230) may receive an input that moves a joint included in a three-dimensional skeleton visualized in the motion data at a specific frame. The input may be a drag input that selects and moves a specific joint. The processor (230) may modify the coordinate information of the joint for a specific frame included in the motion data according to the input. The coordinate information may be modified to the endpoint of the drag input. The processor (230) may perform motion blending for a specific frame and a frame adjacent to that specific frame. If the coordinate information is modified only for a specific frame, unnatural movement may occur when the motion data is played back. Therefore, to enable smooth movement, motion blending may be performed by interpolating the joint coordinates using a graphics engine for the modified frame and adjacent frames.

[0034] According to one embodiment, the processor (230) can perform a search for motion data by receiving a video of a user's motion instead of a search term. Since motion data is data about human movement, it may be difficult to search for the desired motion using text or keywords. Therefore, to search for motion data containing the desired motion, the user can perform the search by taking the desired motion in front of the camera instead of entering a text search term. For example, the processor (230) can acquire motion search data by analyzing the motion captured for data search within the database using a pose estimation model. The method of acquiring motion search data may be similar to the method of acquiring motion data. The processor (230) can output search results based on the similarity between the motion data and the motion search data. The processor (230) can calculate the similarity between the motion data and the motion search data based on the distance between the joints of the motion data and the joints of the motion search data, the velocity and acceleration of the two joints, etc., and can output search results in order of similarity.

[0035] According to one embodiment, the processor (230) can cluster motion data by considering the context of text associated with the motion data. By clustering motion data by considering the context, the user can discover the motion data they want by gathering and verifying motion data with similar keywords. For example, the processor (230) can store text describing the motion data along with the motion data in a database. The text may include the creator, title, descriptive phrase, tags, related questions, and related answers. The processor (230) can cluster motion data according to the similarity of the text by performing text mining within the database. The processor (230) can determine the similarity between texts for each motion data using a language model and cluster motion data with a similarity higher than a specified standard and display it in a user interface.

[0037] FIG. 3 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment.

[0038] Referring to FIG. 3, a motion data archiving device according to one embodiment may provide a user interface to a user terminal for analyzing a video using a pose estimation model. The user may upload a video, and the pose estimation model may extract keypoints (or joints) from the shape of the human body captured in the video. The motion data archiving device may play the uploaded video and display keypoints and connecting lines between keypoints on the video playback screen. The motion data archiving device may generate and store pose estimation data including coordinate information of keypoints for each frame of the video. Motion data may be generated by combining codes included in the pose estimation data so that they are arranged in chronological order.

[0040] FIG. 4 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment.

[0041] Referring to FIG. 4, a motion data archiving device according to one embodiment may provide a user interface for uploading motion data to a user terminal. The user interface may include a preview screen that visualizes motion data and a text input window. The preview screen may visualize a specific frame of the motion data, or it may visualize at least some frames of the motion data to be played back. Text such as the title of the motion data, a description of the motion data, tags, a question about the motion data, and an answer to the question may be entered in the input window. The name of the user uploading the motion data may be automatically registered. When the upload button is activated, the motion data may be archived.

[0043] FIG. 5 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment.

[0044] Referring to FIG. 5, a motion data archiving device according to one embodiment may provide a user interface to a user terminal that displays a list of various uploaded motion data. The list may include information such as a preview image for each motion data, the ID of the uploading user, a title, a description, and tags. The preview image may be a visualization of a specific frame of the motion data, or a visualization of at least some frames of the motion data to be played back. Although FIG. 5 illustrates the list arranged in a grid pattern, it is not limited thereto, and the list may be arranged in various ways, such as vertically or horizontally.

[0046] FIG. 6 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment.

[0047] Referring to FIG. 6, a motion data archiving device according to one embodiment may provide a user interface to a user terminal for playing and editing motion data selected from a list. The user interface may include a playback screen, a navigation bar, a play / stop button, a capture button, a number of people selection button, and a time difference setting bar, and may include a drop-down menu for sequentially selecting and connecting multiple motion data.

[0048] The playback screen is a screen that visually plays motion data, and may display a 3D skeleton represented by the motion data, user ID, title, frame number, etc. The navigation bar allows the user to check the current playback position and select another playback position. The number of people selection button functions to set the number of 3D skeletons displayed on the playback screen. The time difference setting bar functions to set the time difference between multiple skeletons when multiple skeletons are displayed by the number of people selection button.

[0049] The motion data archiving device can provide a function to sequentially combine multiple motion data. Motion data to be combined can be sequentially selected using drop-down menus corresponding to motion 1, motion 2, motion 3, and motion 4 shown in FIG. 7, and the selected motion data can be combined in order and then sequentially displayed on a playback screen. In this case, the movement of the skeleton may appear unnatural when transitioning from the first motion data to the second motion data because the movement is not continuous. The motion data archiving device can implement smooth movement when motion data is switched by performing motion blending, which interpolates joint coordinates using a graphics engine for the last frame of the first motion data and the first frame of the second motion data.

[0051] FIG. 7 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment.

[0052] Referring to FIG. 7, a motion data archiving device according to one embodiment may provide a user interface for modifying motion data. For example, when frame 201 (710) is displayed on a playback screen, the position of a joint included in the three-dimensional skeleton of frame 201 (710) may be modified. For example, a drag input may be applied to move the left knee joint (711), and the left knee joint (711) may be moved from the starting point of the drag input to the end point. The position of the joint may be easily modified by user input on the playback screen.

[0053] If the position of a joint for a specific frame is changed, discontinuous and unnatural movement may occur with the preceding and succeeding frames. For example, if the position of the left knee joint (711) of frame 201 (710) is modified, the positions of the left knee joint (721) of frame 200 (720) and the left knee joint (731) of frame 202 (730) also need to be modified. The motion data archiving device can modify the coordinates of the left knee joint (721) of frame 200 (720) by linearly interpolating the joint coordinates of frame 199, frame 200 (720), and frame 201 (710), and can modify the coordinates of the left knee joint (731) of frame 202 (730) by linearly interpolating the joint coordinates of frame 201 (710), frame 202 (730), and frame 203. In this document, motion blending is described as being performed on the coordinates of one adjacent frame, but is not limited thereto, and motion blending can be performed on the coordinates of two or more adjacent frames.

[0055] FIG. 8 is a drawing illustrating an exemplary operation of a device for archiving motion data for dance movements according to one embodiment.

[0056] Referring to FIG. 8, a motion data archiving device according to one embodiment can detect a user's motion and search for motion data. The user can film a dance movement to be searched using a user terminal. The user terminal can transmit the filmed video to the motion data archiving device. The motion data archiving device can obtain data for motion search by analyzing the video using a pose estimation model.

[0057] A motion data archiving device can compare motion search data with multiple motion data recorded in the archive. For example, similarity can be calculated based on the distance between the joints of the motion search data and the joints of the motion data, the velocity and acceleration of the two joints, etc. When the similarity between the third motion data and the motion search data is highest, the third motion data can be provided as a search result.

[0059] FIG. 9 is a flowchart illustrating a method for archiving motion data for dance movements according to one embodiment.

[0060] In the following, it is assumed that the motion data archiving device of FIG. 2 performs the process of FIG. 9. Also, in the description of FIG. 9, the operation described as being performed by the motion data archiving device can be understood as being controlled by the processor (230).

[0061] Referring to FIG. 9, in step 910, the motion data archiving device can acquire a video of a dance movement captured by a camera.

[0062] In step 920, the motion data archiving device can obtain pose estimation data corresponding to a dance movement for each frame constituting the video by analyzing the video using a pose estimation model.

[0063] In step 930, the motion data archiving device can store motion data, which combines pose estimation data in chronological order, in a database.

[0064] In step 940, the motion data archiving device may provide a user interface for browsing motion data stored in a database.

[0065] In step 950, when motion data is selected through the user interface, the motion data archiving device can visualize the motion data as a three-dimensional skeleton.

[0067] The embodiments of this document and the terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of said embodiments. In relation to the description of the drawings, similar reference numerals may be used for similar components. A singular expression may include a plural expression unless the context clearly indicates otherwise. In this document, expressions such as "A or B," "at least one of A and / or B," "A, B or C," or "at least one of A, B and / or C" may include all possible combinations of items listed together. Expressions such as "first," "second," "first," or "second" may modify said components regardless of order or importance and are used only to distinguish one component from another and do not limit said components. When it is mentioned that a component is "(functionally or telecommunicationally) connected" or "connected" to another component, said component may be directly connected to said other component or connected through said other component.

[0068] In this document, "adapted to or configured to" may be used interchangeably with, depending on the context, for example, hardware- or software-wise, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to." In some cases, the expression "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "processor configured to perform A, B, and C" may mean a dedicated processor for performing those operations (e.g., an embedded processor) or a general-purpose processor (e.g., a CPU) capable of performing those operations by executing one or more programs stored in a memory device.

[0069] As used in this document, the term “module” includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A “module” may be a component formed as a whole or a minimum unit or part thereof that performs one or more functions. A “module” may be implemented mechanically or electronically and may include, for example, an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs), or programmable logic device, known or under development, that performs certain operations.

[0070] At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to one embodiment may be implemented as instructions stored in a computer-readable storage medium in the form of program modules. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions.

[0071] Each component (e.g., module or program module) according to one embodiment may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted or additional sub-components may be included. Generally or additionally, some components (e.g., module or program module) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program module, or other components according to one embodiment may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

Claims

Claim 1 A device for archiving motion data for dance movements, comprising: a communication circuit; a memory including a database; and a processor electrically connected to the communication circuit and the memory, wherein the processor acquires a video of a dance movement captured by a camera using the communication circuit, acquires pose estimation data corresponding to the dance movement for each frame constituting the video by analyzing the video using a pose estimation model, stores motion data formed by combining the pose estimation data in chronological order in the database, provides a user interface for searching the motion data stored in the database, and when the motion data is selected through the user interface, visualizes the motion data as a three-dimensional skeleton, wherein the processor acquires motion search data by analyzing a motion captured for searching data archived in the database using the pose estimation model, and outputs a search result based on the similarity between the motion data and the motion search data. Claim 2 delete Claim 3 An apparatus according to claim 1, wherein the processor stores text describing the motion data together with the motion data in the database, and clusters the motion data according to the similarity of the text by performing text mining within the database. Claim 4 The device according to claim 1, wherein the processor receives an input to move a joint included in a three-dimensional skeleton visualized with the motion data in a specific frame, modifies the coordinate information of the joint for the specific frame included in the motion data according to the input, and performs motion blending for the specific frame and a frame adjacent to the specific frame. Claim 5 A method for archiving motion data for dance movements, comprising: a step of acquiring a video of a dance movement captured by a camera; a step of acquiring pose estimation data corresponding to the dance movement for each frame constituting the video by analyzing the video using a pose estimation model; a step of storing motion data formed by combining the pose estimation data in chronological order in a database; a step of providing a user interface for searching the motion data stored in the database; and a step of visualizing the motion data as a three-dimensional skeleton when the motion data is selected through the user interface; a step of acquiring motion search data by analyzing a captured movement using the pose estimation model for searching the data archived in the database; and a step of outputting a search result based on the similarity between the motion data and the motion search data.

Images