Data generate apparatus and the operation method

Abstract

The present invention provides a data generation device comprising a camera module that outputs first and second video frames capturing a human gesture, and a control module that recognizes a joint position according to the human gesture based on the first and second video frames, calculates a joint depth corresponding to the joint position, and generates pose data that forms a robot gesture set based on the joint position and the joint depth.

Description

Technology Field

[0001] The present invention relates to a data generation device and a method of operation thereof, and more specifically, to a data generation device and a method of operation thereof that generates and transmits 3D type pose data so that a robot configured to correspond to a human gesture takes a gesture. Background Technology

[0002] Computer vision is the implementation of human visual perception capabilities using computers, and its core function is extracting information by analyzing images or videos captured by cameras. In computer vision, the technology for recognizing a person's posture is called Human Pose Estimation.

[0003] Human body estimation is a technique that estimates how a person's body joints are structured in photographs or videos. However, not all joints are visible in a photograph, and even within the same posture, the joints may appear different depending on the shooting angle, be obscured by other objects, wear various clothing, or be affected by lighting, making it difficult to estimate all joints.

[0004] Therefore, although holistic reasoning is required, human pose estimation technology remains a difficult field, despite having been addressed in computer vision for a long time.

[0005] Convolutional Neural Networks (CNNs) demonstrate excellent performance in computer vision applications, such as image and video recognition. Among these, heatmap-based human pose estimation refers to a method that estimates regions where joints are likely to be located, and then estimates the position coordinates of the joints by using weights to specify the precise locations of the joints within those estimated regions.

[0006] Recently, research has been conducted on estimating joint position coordinates by converting the joint position coordinates based on human gestures into data for application to actual robots. Prior art literature

[65535] Republic of Korea Published Patent Application No. 10-2024-0081551 (June 10, 2024) The problem to be solved

[0007] The objective of the present invention is to provide a data generation device and a method of operation thereof for generating and transmitting 3D type pose data so that a robot configured to correspond to a human gesture takes a gesture.

[0008] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0009] A data generation device according to the present invention includes a camera module that outputs first and second video frames capturing a human gesture, and a control module that recognizes a joint position corresponding to the human gesture based on the first and second video frames, calculates a joint depth corresponding to the joint position, and generates pose data that forms a robot gesture based on the joint position and the joint depth. The control module includes a position recognition unit that recognizes the joint position by applying the first video frame to a set joint recognition algorithm, and a depth calculation unit that calculates the joint depth according to a pixel value corresponding to the joint position in the second video frame. The depth calculation unit can calculate the joint depth according to the pixel value by comparing the pixel value with a previous pixel value calculated in a previous video frame input at a time earlier than the second video frame, if the error pixel value between the pixel value and the previous pixel value is within an allowable range.

[0010] The camera module may include an image sensor that outputs the first image frame and a depth sensor that outputs the second image frame corresponding to the first image frame at the same time as the image sensor.

[0011] The control module may further include a control unit that generates pose data to enable the robot to perform a gesture that matches the human gesture based on the joint position and the joint depth.

[0012] The above position recognition unit can recognize the joint position if the visibility of the person's joint in the first image frame is higher than the reference visibility.

[0013] delete

[0014] The control unit above can generate pose data including spatial coordinates that determine the robot joint position to perform a gesture of the robot by synthesizing the joint position and the joint depth.

[0015] The method of operation of a data generation device according to the present invention comprises the steps of: receiving first and second video frames that capture a person’s gesture output from a camera module; recognizing a joint position corresponding to the person’s gesture according to a pixel value corresponding to the joint position in the first video frame; calculating a joint depth corresponding to the joint position based on the second video frame; and generating pose data that forms a robot’s gesture based on the joint position and the joint depth. The step of calculating the joint depth can calculate the joint depth according to the pixel value by comparing the pixel value with a previous pixel value calculated from a previous video frame input at a time earlier than the second video frame, and if the error pixel value between the pixel value and the previous pixel value is within an allowable range.

[0016] The first and second video frames are images taken at the same time, and the step of recognizing the joint position is to apply the first video frame to a set joint recognition algorithm so that if the visibility of the person's joint is higher than the reference visibility, the joint position can be recognized.

[0017] delete

[0018] The step of generating the pose data may include generating the pose data including spatial coordinates that determine the robot joint position to perform a gesture of the robot by synthesizing the joint position and the joint depth, and transmitting the pose data to a set external device. Effects of the invention

[0019] The data generation device and the method of operation according to the present invention have the advantage of facilitating the robot's gesture by recognizing the joint position corresponding to the current human gesture and calculating the joint depth for the joint position based on different first and second video frames capturing the human gesture, and by generating pose data that forms the robot's gesture and transmitting it to an external device.

[0020] Further scopes of the applicability of the present invention will become apparent from the following detailed description. However, since various changes and modifications within the spirit and scope of the present invention are clearly understood by those skilled in the art, specific embodiments, such as the detailed description and preferred embodiments of the present invention, should be understood as being given merely as examples. Brief explanation of the drawing

[0021] FIG. 1 is a control block diagram showing the control configuration of a data generation device according to the present invention. Figure 2 is an example diagram for explaining the operation of the data generation device shown in Figure 1. Figure 3 is an example diagram for explaining the pose data shown in Figure 1. FIG. 4 is a flowchart illustrating the operation method of a data generation device according to the present invention. Specific details for implementing the invention

[0022] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0023] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0024] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0025] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0026] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0027] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0028] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0029] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0030] FIG. 1 is a control block diagram showing the control configuration of a data generation device according to the present invention.

[0031] Referring to FIG. 1, the data generation device (1) may include a camera module (10) and a control module (20).

[0032] The camera module (10) may include an image sensor (110) and a depth sensor (120).

[0033] First, the image sensor (110) can output a first image frame (F1) that captures a color image or an image.

[0034] The depth sensor (120) operates at the same time as the image sensor (110) and can output a second image frame (F2) captured at the time when the first image frame (F1) was captured.

[0035] In an embodiment, the image sensor (110) and the depth sensor (120) can output first and second image frames (F1, F2) that capture a person's gesture, that is, a pose taken by a person.

[0036] The first and second video frames (F1, F2) are one frame of a person's gesture at the current time and may correspond to one of the multiple video frames forming the video.

[0037] The control module (20) may include a filter unit (210), a position recognition unit (220), a depth calculation unit (230), and a control unit (240).

[0038] First, the filter unit (210) can process surrounding objects other than people as noise in each of the first and second video frames (F1, F2), thereby filtering only the parts corresponding to the shape of a person.

[0039] The position recognition unit (220) can recognize the position of a person's joint (JL) by applying the first image frame (F1) filtered by the filter unit (210) to a set joint recognition algorithm.

[0040] In the embodiment, the joint position (JL) of a person is described as including all movable joints of a person, such as the face, eyes, nose, mouth, neck, shoulders, and legs.

[0041] Here, the joint recognition algorithm may be an AI pose recognition algorithm such as Mediapipe, Deep3Dpose, etc., but is not limited thereto.

[0042] For example, the position recognition unit (220) can recognize the joint position (JL) of a person by applying a joint model that briefly represents the joint position of a person included in the joint recognition algorithm to the first image frame (F1).

[0043] At this time, the position recognition unit (220) can recognize the position of a person's joint (JL) when the visibility of the person's joint in the first image frame (F1) is higher than the reference visibility.

[0044] That is, the position recognition unit (220) can recognize the position of a person's joints in the next image frame input after the first image frame (F1) if the visibility is lower than the reference visibility.

[0045] The depth calculation unit (230) can calculate the human joint depth (JD) by applying the area or coordinates corresponding to the human joint position (JL) recognized by the position recognition unit (220) to the second image frame (F2).

[0046] That is, the depth calculation unit (230) can calculate the joint depth (JD) according to the pixel value of the area or coordinate corresponding to the joint position (JL) of a person in the second image frame (F2).

[0047] In the embodiment, the joint position (JL) of a person represents x-axis and y-axis coordinates in a 2D plane, and the joint depth (JD) of a person may be a z-axis coordinate in 3D space representing the joint angle, but is not limited thereto.

[0048] At this time, the depth calculation unit (230) compares the pixel value with the previous pixel value calculated from the previous image frame input at the previous time point, and if the error pixel value between the pixel value and the previous pixel value is within an allowable range, it can calculate the human joint depth (JD) according to the pixel value.

[0049] Additionally, the depth calculation unit (230) can calculate the human joint depth (JD) in the next image frame input after the second image frame (F2) if the error pixel value between the pixel value and the previous pixel value is outside the allowable range.

[0050] The control unit (240) can generate pose data (PD) so that a robot configured to match a person's gesture based on the person's joint position (JL) and joint depth (JD) can form a gesture and take a pose.

[0051] As described above, the position of a person's joint (JL) is the x-axis and y-axis coordinates, and the depth of a person's joint (JD) may be the z-axis coordinate.

[0052] The control unit (240) can synthesize the human joint position (JL) and joint depth (JD) to generate pose data (PD) containing spatial coordinates that determine the robot joint position to take a pose for the actual robot's gesture.

[0053] Afterwards, the control unit (240) can transmit the pose data (PD) to a set external device, such as a robot, a device that controls the robot, etc.

[0054] Additionally, pose data (PD) may be data for forming a joint movement path that includes the position, velocity, and acceleration of the robot's joints.

[0055] The data generation device of the present invention has the advantage of enabling the robot's gesture to be formed smoothly like a human gesture by generating appropriate pose data (PD) based on the number of joints and joint parts of a robot modeled after at least a part of a human, based on the joint position (JL) and joint depth (JD) that move in response to changes in a human gesture according to a plurality of video frames.

[0056] FIG. 2 is an example diagram for explaining the operation of the data generation device shown in FIG. 1, and FIG. 3 is an example diagram for explaining the pose data shown in FIG. 1.

[0057] Referring to FIGS. 2 and FIGS. 3, the camera module (10) may include an image sensor (110) and a depth sensor (120).

[0058] At this time, FIG. 2(a) shows a first image frame (F1) captured by an image sensor (110), and FIG. 2(b) shows a second image frame (F2) captured by a depth sensor (120).

[0059] The first and second video frames (F1, F2) can capture a person's gesture at the same time and output it to the control module (20).

[0060] Here, FIG. 2(a) can recognize joint positions (JL) by applying a joint model (JM) representing the simplified joint positions of a person according to a joint recognition algorithm to the shape of a person included in the first image frame (F1) in the position recognition unit (220).

[0061] FIG. 3(a) shows a packet containing pose data (PD), and FIG. 3(b) may show human pose data (PD). Additionally, the packet containing pose data (PD) may include data for a single joint or all data for all joints, and is not limited thereto.

[0062] FIG. 4 is a flowchart illustrating the operation method of a data generation device according to the present invention.

[0063] Referring to FIG. 4, the camera module (10) of the data generation device (1) can output or transmit first and second video frames (F1, F2) that capture a person's gesture to the control module (20) (S110).

[0064] That is, the camera module (10) may include an image sensor (110) and a depth sensor (120).

[0065] First, the image sensor (110) can output a first image frame (F1) that captures a color image or an image.

[0066] The depth sensor (120) operates at the same time as the image sensor (110) and can output a second image frame (F2) captured at the time when the first image frame (F1) was captured.

[0067] In an embodiment, the image sensor (110) and the depth sensor (120) can output first and second image frames (F1, F2) that capture a person's gesture, that is, a pose taken by a person.

[0068] The first and second video frames (F1, F2) are one frame of a person's gesture at the current time and may correspond to one of the multiple video frames forming the video.

[0069] The control module (20) of the data generation device (1) can recognize the joint position (JL) according to the person's gesture based on the first image frame (F1) (S120), and calculate the joint depth (JD) corresponding to the joint position (JL) based on the second image frame (F1) (S130).

[0070] That is, the control module (20) can recognize the position of a person's joint (JL) by applying the first image frame (F1) to a set joint recognition algorithm.

[0071] Here, the joint recognition algorithm may be an AI pose recognition algorithm such as Mediapipe, Deep3Dpose, etc., but is not limited thereto.

[0072] For example, the control module (20) can recognize the joint position (JL) of a person by applying a joint model that briefly represents the joint position of a person included in the joint recognition algorithm to the first image frame (F1).

[0073] The control module (20) can recognize the position of a person's joint (JL) when the visibility of the person's joint in the first image frame (F1) is higher than the reference visibility.

[0074] That is, the control module (20) can recognize the position of a person's joints in the next image frame input after the first image frame (F1) if the visibility is lower than the reference visibility.

[0075] The control module (20) can calculate the human joint depth (JD) by applying the area or coordinates corresponding to the human joint position (JL) to the second image frame (F2).

[0076] That is, the control module (20) can calculate the joint depth (JD) based on the pixel value of the area or coordinate corresponding to the human joint position (JL) in the second image frame (F2).

[0077] Here, the joint position (JL) of a person represents the x-axis and y-axis coordinates in a 2D plane, and the joint depth (JD) of a person may be the z-axis coordinate in 3D space representing the joint angle, but is not limited thereto.

[0078] At this time, the control module (20) compares the pixel value with the previous pixel value calculated from the previous image frame input at the previous time point, and if the error pixel value between the pixel value and the previous pixel value is within an allowable range, it can calculate the human joint depth (JD) according to the pixel value.

[0079] Additionally, the control module (20) can calculate the human joint depth (JD) in the next image frame input after the second image frame (F2) if the error pixel value between the pixel value and the previous pixel value is outside the allowable range.

[0080] The control module (20) can generate pose data (PD) that forms a gesture of the robot based on the joint position (JL) and joint depth (JD) (S140).

[0081] That is, the control module (20) can synthesize the human joint position (JL) and joint depth (JD) to generate pose data (PD) containing spatial coordinates that determine the robot joint position to take a pose for the actual robot's gesture.

[0082] Afterwards, the control module (20) can transmit the pose data (PD) to a set external device, such as a robot, a device that controls the robot, etc.

[0083] The embodiments according to the present disclosure described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, flash memory, etc.

[0084] Meanwhile, the above computer program may be one specifically designed and configured for the present disclosure, or one known and available to a person skilled in the art of computer software. Examples of computer programs may include machine code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0085] In the specification of this disclosure (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in this disclosure, it is to include an invention to which individual values ​​belonging to said range are applied (unless otherwise stated), as is equivalent to describing each individual value constituting said range in the detailed description of the invention.

[0086] Unless explicitly stated otherwise, the steps constituting the method according to the present disclosure may be performed in a suitable order. The present disclosure is not necessarily limited by the order in which the steps are described. The use of any examples or exemplary terms (e.g., etc.) in the present disclosure is merely for the purpose of describing the present disclosure in detail and, unless limited by the claims, the scope of the present disclosure is not limited by such examples or exemplary terms. Furthermore, a person skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0087] As described above, the disclosed embodiments have been explained with reference to the attached drawings. Those skilled in the art will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be interpreted restrictively.

Claims

Claim 1 A data generation device comprising: a camera module that outputs first and second video frames capturing a human gesture; and a control module that recognizes a joint position corresponding to the human gesture based on the first and second video frames, calculates a joint depth corresponding to the joint position, and generates pose data that forms a robot gesture based on the joint position and the joint depth, wherein the control module includes: a position recognition unit that recognizes the joint position by applying the first video frame to a set joint recognition algorithm; and a depth calculation unit that calculates the joint depth according to a pixel value corresponding to the joint position in the second video frame, wherein the depth calculation unit compares the pixel value with a previous pixel value calculated in a previous video frame input at a time earlier than the second video frame, and if the error pixel value between the pixel value and the previous pixel value is within an allowable range, calculates the joint depth according to the pixel value. Claim 2 A data generation device according to claim 1, wherein the camera module comprises: an image sensor that outputs the first image frame; and a depth sensor that outputs the second image frame corresponding to the first image frame at the same time as the image sensor. Claim 3 A data generation device according to claim 1, wherein the control module further comprises a control unit that generates pose data to enable the robot to perform a gesture that matches the human gesture based on the joint position and the joint depth. Claim 4 In claim 1, the position recognition unit is a data generation device that recognizes the joint position when the visibility of the person's joint in the first image frame is higher than the reference visibility. Claim 5 delete Claim 6 In claim 3, the control unit generates pose data including spatial coordinates that determine the robot joint position to take a gesture of the robot by synthesizing the joint position and the joint depth. Claim 7 A method of operation of a data generation device comprising: receiving first and second video frames that capture a person’s gesture output from a camera module; recognizing a joint position according to the person’s gesture based on the first video frame; calculating a joint depth corresponding to the joint position according to a pixel value corresponding to the joint position in the second video frame; and generating pose data that forms a robot’s gesture based on the joint position and the joint depth, wherein the step of calculating the joint depth compares the pixel value with a previous pixel value calculated in a previous video frame input at a time earlier than the second video frame, and if the error pixel value between the pixel value and the previous pixel value is within an allowable range, calculates the joint depth according to the pixel value. Claim 8 In claim 7, the first and second image frames are images captured at the same time, and the step of recognizing the joint position is to apply the first image frame to a set joint recognition algorithm and recognize the joint position if the visibility of the person's joint is higher than the reference visibility. A method of operation of a data generation device. Claim 9 delete Claim 10 In claim 7, the step of generating the pose data comprises generating the pose data including spatial coordinates that determine the robot joint position to perform a gesture of the robot by synthesizing the joint position and the joint depth, and transmitting the pose data to a set external device.

Images