Video processing program, video processing method, and video processing apparatus

Abstract

To provide a video processing program, a video processing method and a video processing device capable of displaying a key point of a photographing object and a reference line on a video / image of the photographed photographing object.SOLUTION: The image processing terminal 1 includes a data-analyzing unit 13 that estimates a skeleton structure of the target to be captured P captured by the In Camera 5 or the out-camera 7, a reference-line-calculating unit 14 that calculates a reference line based on sensing data of the acceleration sensor 17 and the gyro sensor 18, a data-synchronizing unit 15 that synchronizes the target to be captured, the skeleton structure, and the reference line, and a display unit 2 that displays the target to be captured, the skeleton structure, and the reference line synchronized by the data-synchronizing unit 15.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a video processing program, a video processing method, and a video processing device that display reference lines in video / images that display straight line bones connecting key points (facial features and joint points) of a photographed object detected by a skeleton estimation function and adjacent key points. [Background technology]

[0002] Conventionally, a camera calibration device is known that captures an image of a person or other subject and displays key points identified by a skeleton estimation function in the captured image, as well as straight bones connecting adjacent key points (see, for example, Patent Document 1).

[0003] The camera calibration device described in Patent Document 1 uses skeletal estimation technology using machine learning for camera calibration, making it possible to display a person standing upright, as well as the person's key points and straight bones. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7420146 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since the captured image does not display horizontal, vertical, or other reference lines, it is not possible to know how tilted the object being imaged is.

[0006] Therefore, the present invention aims to solve the above problems and provide an image processing program, an image processing method, and an image processing device that can display key points and reference lines of a photographed subject in a photographed video / image. [Means for solving the problem]

[0007] The video processing program of the present invention is characterized in that it causes an information processing terminal to function as a data analysis unit that estimates the skeletal structure of a subject photographed by a photographing means, a reference line calculation unit that calculates a reference line based on detection data from an inertial sensor, a data synchronization unit that synchronizes the subject, the skeletal structure, and the reference line, and a display unit that displays the subject, the skeletal structure, and the reference line synchronized by the data synchronization unit.

[0008] The image processing method of the present invention is characterized by comprising the steps of photographing an object to be photographed using a photographing means, estimating the skeletal structure of the photographed object, calculating a reference line based on detection data from an inertial sensor, synchronizing the object to be photographed, the skeletal structure, and the reference line, and displaying the synchronized object to be photographed, the skeletal structure, and the reference line on a display unit.

[0009] The video processing device of the present invention comprises an imaging means, a display unit, an inertial sensor, a data analysis unit having a skeletal estimation function, a reference line calculation unit that calculates a reference line based on detection data detected by the inertial sensor, and a data synchronization unit that synchronizes an image or video of the subject captured by the imaging means, the skeletal structure of the subject estimated by the data analysis unit, and the reference line, and is characterized in that the subject, the skeletal structure, and the reference line synchronized by the data synchronization unit are displayed on the display unit. [Effects of the Invention]

[0010] According to the video processing program of the present invention, it is possible to display key points and reference lines of a photographed subject in sync with the photographed video / image of the subject.

[0011] According to the imaging processing method of the present invention, it is possible to display the key points and reference lines of the subject in sync with the video / image of the subject that has been photographed.

[0012] According to the video processing device of the present invention, it is possible to display on the display unit key points and reference lines of a captured video / image of a subject in sync with the captured video / image of the subject. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a front view of an information processing terminal according to an embodiment of the present invention; [Figure 2] FIG. 2 is a rear view of the information processing terminal according to the embodiment of the present invention. [Figure 3] 1 is a block diagram showing an electrical configuration of an information processing terminal according to an embodiment of the present invention; [Figure 4] FIG. 1 is a front view of an information processing terminal that displays a subject to be photographed and a skeletal structure in synchronization with each other according to an embodiment of the present invention. [Figure 5] FIG. 1 is a front view of an information processing terminal according to an embodiment of the present invention, in which a subject to be photographed (front), a skeletal structure, and a reference line are displayed in a synchronized manner. [Figure 6] FIG. 10 is a front view of an information processing terminal according to an embodiment of the present invention, in which a subject to be photographed (right side), a skeletal structure, and a reference line are displayed in a synchronized manner. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Note that not all of the configurations described below are essential requirements for the present invention.

[0015] 1 and 2 show an information processing terminal 1, which is an imaging device according to the present invention. The information processing terminal 1 has a display unit 2 capable of displaying various images, videos, etc., a touch panel type operation unit 3, an in-camera 5 serving as imaging means provided on a front surface 4 of the information processing terminal 1, and an out-camera 7 serving as imaging means provided on a rear surface 6 of the information processing terminal 1. Therefore, the information processing terminal 1 can be realized as, for example, a tablet terminal or a smartphone equipped with imaging means.

[0016] 3 shows the electrical configuration of the information processing terminal 1. In addition to the display unit 2, operation unit 3, in-camera 5, and out-camera 7 described above, the information processing terminal 1 also includes a control means 8, an inertial measurement unit 9, a storage unit 10, and a transmission / reception unit 11.

[0017] The display unit 2 is composed of a liquid crystal module and a liquid crystal panel that are exposed on the front surface 4 of the information processing terminal 1, and as is well known, these liquid crystal modules and liquid crystal panels display images using a dot matrix in which a large number of sub-pixels are arranged in a grid pattern.

[0018] The operation unit 3 receives operations from a user and sends an electrical operation signal to the control means 8. In the information processing terminal 1 of this embodiment, the display unit 2 is a touch panel, and the surface of the display unit 2 functions as the operation unit 3.

[0019] The in-camera 5 and the out-camera 7 can be activated by a user operating the operation unit 3, and can capture still images (hereinafter referred to as "images") and moving images (hereinafter referred to as "video") of a person or other subject P. In this embodiment, a case will be described in which video of the address and swing of a golf shot is captured by the in-camera 5. Specifically, the information processing terminal 1 is fixed to a tripod or the like (not shown), and the front part 4 is directed toward the subject P who will be taking the shot to capture the image. Note that a user other than the subject P may hold the information processing terminal 1 and point the rear part 6 toward the subject P to capture the image with the out-camera 7, or the information processing terminal 1 may be fixed to a tripod or the like, and the rear part 6 may be directed toward the subject P to capture the image with the out-camera 7. When the subject P or the user operates the operation unit 3 to finish capturing the image, the captured video is stored in the storage unit 10.

[0020] The control means 8 includes a CPU (Central Processing Unit) and controls the entire information processing terminal 1 based on a program 12 stored in a storage unit 10. The CPU executes calculations in accordance with the program 12, thereby realizing each function of the information processing terminal 1. The control means 8 also includes a data analysis unit 13, a reference line calculation unit 14, and a data synchronization unit 15, which will be described later.

[0021] The inertial measurement unit 9 incorporates an acceleration sensor 16 and a gyro sensor 17, both of which serve as inertial sensors, as detection means for detecting the movement of the information processing terminal 1. The acceleration sensor 16 can measure acceleration in three orthogonal axis directions, and the gyro sensor 17 can measure angular velocity around each of the three orthogonal axes. The inertial measurement unit 9 measures the acceleration and direction changes of the information processing terminal 1. The acceleration information and angular velocity information measured by the inertial measurement unit 9 are sent to the reference line calculation unit 14 of the control means 8.

[0022] The memory unit 10 is configured using various memory devices such as a magnetic hard disk drive and a semiconductor memory device, and is capable of writing and reading image and video data captured by the in-camera 5 and the out-camera 7, skeleton estimation data detected by the data analysis unit 13, baseline data calculated by the baseline calculation unit 14, and synchronization data synchronized by the data synchronization unit 15.

[0023] The transmitting / receiving unit 11 enables two-way wireless communication with the cloud server 18 via a communication means such as the Internet. Therefore, the information processing terminal 1 can transmit and receive data including various types of information to and from the cloud server 18.

[0024] The data analysis unit 13 analyzes the video data captured from the in-camera 5 and the out-camera 7, and extracts key points for detecting (estimating) the skeletal structure of the subject P based on facial features, joint points, etc., using a skeleton estimation function using machine learning. In this embodiment, 17 key points are detected: a left eye key point K1, a right eye key point K2, a nose key point K3, a left ear key point K4, a right ear key point K5, a left shoulder joint key point K6, a right shoulder joint key point K7, a left elbow joint key point K8, a right elbow joint key point K9, a left wrist key point K10, a right wrist key point K11, a left hip joint key point K12, a right hip joint key point K13, a left knee joint key point K14, a right knee joint key point K15, a left ankle key point K16, and a right ankle key point K17.

[0025] Furthermore, the data analysis unit 13 selects key points K1 to K17 to be connected by straight bones in order to detect (estimate) the skeletal structure from the extracted key points K1 to K17. The skeletal structure is detected (estimated) by connecting the selected key points K1 to K17 by straight bones. In this embodiment, a straight line bone B1 connecting the left eye key point K1 and the right eye key point K2, a straight line bone B2 connecting the left eye key point K1 and the nose key point K3, a straight line bone B3 connecting the right eye key point K2 and the nose key point K3, a straight line bone B4 connecting the left eye key point K1 and the left ear key point K4, a straight line bone B5 connecting the right eye key point K2 and the right ear key point K5, a straight line bone B6 connecting the left ear key point K4 and the left shoulder joint key point K6, a straight line bone B7 connecting the right ear key point K5 and the right shoulder joint key point K7, a straight line bone B8 connecting the left shoulder joint key point K6 and the right shoulder joint key point K7, a straight line bone B9 connecting the left shoulder joint key point K6 and the left elbow joint key point K8, a straight line bone B10 connecting the right shoulder joint key point K7 and the right elbow joint key point K9, a straight line bone B11 connecting the left elbow joint key point K The skeletal structure is detected (estimated) using 19 straight line bones B1 to B19: a straight line bone B11 connecting key point K8 to key point K10 of the left wrist, a straight line bone B12 connecting key point K9 of the right elbow joint to key point K11 of the right wrist, a straight line bone B13 connecting key point K6 of the left shoulder joint to key point K12 of the left hip joint, a straight line bone B14 connecting key point K7 of the right shoulder joint to key point K13 of the right hip joint, a straight line bone B15 connecting key point K12 of the left hip joint to key point K13, a straight line bone B16 connecting key point K12 of the left hip joint to key point K14 of the left knee joint, a straight line bone B17 connecting key point K13 of the right hip joint to key point K15 of the right knee joint, a straight line bone B18 connecting key point K14 of the left knee joint to key point K16 of the left ankle, and a straight line bone B19 connecting key point K15 of the right knee joint to key point K17 of the right ankle. The number of key points and the number of straight bones can be determined appropriately depending on the subject to be photographed and the skeletal structure to be detected (estimated).

[0026] The reference line calculation unit 14 generates reference lines to be displayed on the display unit 2 based on the direction of gravitational acceleration (vertical direction) detected by the acceleration sensor 16. In this embodiment, the reference lines are a vertical reference line L1 parallel to the direction of gravitational acceleration, a horizontal reference line L2 tilted 90 degrees from the direction of gravitational acceleration, and an upper body reference line L3 tilted 40 degrees rightward from the direction of gravitational acceleration. The reference line calculation unit 14 generates the reference lines L1, L2, and L3 to have preset lengths. The reference lines L1, L2, and L3 can also be generated to be displayed in a predetermined color. Furthermore, the reference lines L1, L2, and L3 can also be generated to be displayed in a predetermined line type (thickness, solid line, dashed line, chain line, etc.). In this embodiment, the vertical reference line L1 is generated as a dashed-dotted line, the horizontal reference line L2 is generated as a dotted line, and the upper body reference line L3 is generated as a solid line.

[0027] The data synchronization unit 15 has a function of synchronizing each piece of data with each other so that the skeletal structure (key points K1 to K17 and straight bones B1 to B19) detected by the data analysis unit 13 and the reference lines L1, L2, and L3 calculated by the reference line calculation unit 14 are simultaneously displayed on the display unit 2 in the images captured by the in-camera 5 and the out-camera 7. FIG. 4 shows the subject P and the skeletal structure (key points K1 to K17 and straight bones B1 to B19). FIG. 5 shows the subject P, the skeletal structure (key points K1 to K17 and straight bones B1 to B19), and the intersecting vertical and horizontal reference lines L1 and L2. FIG. 6 shows the subject P, the skeletal structure (key points K1 to K17 and parts of the straight bones B1 to B19), and the upper body reference line L3.

[0028] In this embodiment, the vertical reference line L1 is set to pass through the nose key point K3. The horizontal reference line L2 is set to pass through the left shoulder joint key point K6. The upper body reference line L3 is set to pass through the left shoulder joint key point K6. The upper body reference line L3 is set to the ideal upper body angle of the subject P at address, calculated from past swing data of the subject P. When the nose key point K3 moves due to the movement of the subject P, the vertical reference line L1 moves to follow the nose key point K3. Similarly, when the left shoulder joint key point K6 moves, the horizontal reference line L2 and the upper body reference line L3 also move to follow the left shoulder joint key point K6.

[0029] As shown in Fig. 5, by displaying the vertical reference line L1 and the horizontal reference line L2 together with the subject P and the skeletal structure (key points K1 to K17 and straight bones B1 to B19) on the display unit 2, it is possible to visually confirm that the center of the vertical axis of the body of the subject P does not align with the vertical reference line L1, and to what extent the straight bone B8 connecting the key point K6 of the left shoulder joint and the key point K7 of the right shoulder joint is tilted with respect to the horizontal reference line L2. Also, as shown in Fig. 6, by displaying the upper body reference line L3 together with the subject P and the skeletal structure (key points K1 to K17 and a part of the straight bones B1 to B19) on the display unit 2, it is possible to visually confirm whether the upper body of the subject P at address is aligned with the upper body reference line L3, and if so, how much it is misaligned.

[0030] The vertical reference line L1, horizontal reference line L2, and upper body reference line L3 may be set to pass through other key points, or other reference lines (e.g., left arm reference line, right leg reference line, etc.) may be set and displayed. In this case, only one of the reference lines may be displayed, or multiple reference lines may be displayed simultaneously. Furthermore, the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 may be fixed in a predetermined position and displayed without moving in accordance with the key points. Furthermore, polygonal frame-shaped reference lines may be displayed, such as a triangular frame-shaped reference line formed by three reference lines connecting three key points, or a rectangular frame-shaped reference line formed by four reference lines connecting four key points. Frame-shaped reference lines surrounding the set specific key points K1 to K17 may also be displayed.

[0031] In this embodiment, the image capturing and image processing are performed by the information processing terminal 1, but the image data captured by the information processing terminal 1 may be transmitted to the cloud server 18 via the transceiver unit 11, and the image processing may be performed by the cloud server 18.

[0032] As described above, the video processing program of the present embodiment causes information processing terminal 1 to function as: data analyzer 13 that estimates the skeletal structure of subject P captured by in-camera 5 or out-camera 7; reference line calculator 14 that calculates vertical reference line L1, horizontal reference line L2, and upper body reference line L3 based on detection data from acceleration sensor 17 and gyro sensor 18; data synchronizer 15 that synchronizes subject P, the skeletal structure, and vertical reference line L1, horizontal reference line L2, and upper body reference line L3; and display unit 2 that displays subject P, the skeletal structure, and vertical reference line L1, horizontal reference line L2, and upper body reference line L3 synchronized by data synchronizer 15. Therefore, by capturing an image of subject P with in-camera 5 or out-camera 7, it is possible to simultaneously display subject P, the skeletal structure (key points K1 to K17 and straight bones B1 to B19), the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 on display unit 2 in the captured video. As a result, it is possible to visually confirm whether or not the subject P and the straight bones B1 to B19 are tilted relative to the vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3, and the degree of tilt.

[0033] Furthermore, the video processing program of this embodiment synchronizes the skeletal structure with the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 so that the skeletal structure has key points K1 to K17 that indicate specific positions of the subject P, and the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 pass through at least one of the key points K1 to K17. As a result, the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 are displayed so that at least a portion of them overlaps with the subject P, and the subject P and straight bones B1 to B19 are displayed close to the vertical reference line L1, horizontal reference line L2, and upper body reference line L3, making it easy to see the degree of tilt.

[0034] As described above, the image processing method of this embodiment includes the steps of capturing an image of subject P using in-camera 5 or out-camera 7, estimating the skeletal structure of the captured image of subject P, calculating vertical reference line L1, horizontal reference line L2, and upper body reference line L3 based on detection data from acceleration sensor 17 and gyro sensor 18, synchronizing subject P, the skeletal structure, and vertical reference line L1, horizontal reference line L2, and upper body reference line L3, and displaying the synchronized subject P, skeletal structure, and vertical reference line L1, horizontal reference line L2, and upper body reference line L3 on display unit 2. Therefore, by capturing an image of subject P using in-camera 5 or out-camera 7, subject P, the skeletal structure (key points K1 to K17 and straight bones B1 to B19), vertical reference line L1, horizontal reference line L2, and upper body reference line L3 can be simultaneously displayed on display unit 2 in the captured image. As a result, it is possible to visually confirm whether or not the subject P and the straight bones B1 to B19 are tilted relative to the vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3, and the degree of tilt.

[0035] Furthermore, in the image processing method of this embodiment, in the step of estimating the skeletal structure of the photographed subject P, key points K1-K17 indicating specific positions of the subject P are detected, and in the step of synchronizing the subject P, the skeletal structure, and the vertical reference line L1, horizontal reference line L2, and upper body reference line L3, the skeletal structure and the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 are synchronized so that the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 pass through at least one of the key points K1-K17. As a result, at least a portion of the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 is displayed overlapping the subject P, and the subject P and straight bones B1-B19 are displayed close to the vertical reference line L1, horizontal reference line L2, and upper body reference line L3, making it easy to see the degree of tilt.

[0036] As described above, the image processing device 1 of this embodiment comprises the in-camera 5 or the out-camera 7, the display unit 2, the acceleration sensor 17 and the gyro sensor 18, the data analysis unit 13 having a skeleton estimation function, the reference line calculation unit 14 that calculates the vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3 based on the detection data detected by the acceleration sensor 17 and the gyro sensor 18, and the data synchronization unit 15 that synchronizes the video or image of the subject P captured by the in-camera 5 or the out-camera 7, the skeletal structure of the subject P estimated by the data analysis unit 13, and the vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3, and displays the subject P, skeletal structure, vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3 synchronized by the data synchronization unit 15 on the display unit 2. Therefore, by capturing an image of the subject P with the in-camera 5 or the out-camera 7, the subject P, the skeletal structure (key points K1 to K17 and straight bones B1 to B19), the vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3 can be simultaneously displayed in the captured image on the display unit 2. As a result, it is possible to visually confirm whether or not the subject P and the straight bones B1 to B19 are tilted relative to the vertical reference line L1, the horizontal reference line L2, and the upper body reference line L3, and the degree of tilt.

[0037] Furthermore, in the video processing device 1 of this embodiment, the data analysis unit detects key points K1 to K17 that indicate specific positions of the subject P, and the data synchronization unit synchronizes the skeletal structure with the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 so that the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 pass through at least one of the key points K1 to K17. As a result, the vertical reference line L1, horizontal reference line L2, and upper body reference line L3 are displayed so that at least a portion of them overlaps with the subject P, and the subject P and straight bones B1 to B19 are displayed close to the vertical reference line L1, horizontal reference line L2, and upper body reference line L3, making it easy to see the degree of tilt.

[0038] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention. The information processing terminal 1 and the imaging means may be separate entities. For example, a digital camera or video camera may be used as the imaging means, and the video and image data captured by the imaging means may be transmitted to a personal computer equipped with a display unit 2 and control means 8 as the information processing terminal 1, and the video processing may be performed on the personal computer. Furthermore, the present invention can be used not only for observing golf address and swing movements, but also for observing other sports movements, observing skeletal distortions, and monitoring the progress of people undergoing rehabilitation to recover from injuries or improve physical disabilities. In the above-described embodiment, video processing is performed on a single subject P. However, if multiple people are captured at the same time, video processing of the multiple people can also be performed simultaneously. [Explanation of symbols]

[0039] 1. Information processing terminal 2 Display section 5. In-camera (photography method) 7. Outer camera (photography means) 13 Data Analysis Department 14 Baseline calculation section 15 Data Synchronization Section 17 Acceleration sensor (inertial sensor) 18 Gyro sensor (inertial sensor) K1~K17 Key Points L1 Vertical reference line (reference line) L2 Horizontal reference line (reference line) L3 Upper body reference line (reference line) P Shooting subject

Claims

1. Information processing terminal, a data analysis unit that estimates the skeletal structure of the subject photographed by the photographing means; a reference line calculation unit that calculates a reference line based on detection data of the inertial sensor; a data synchronization unit that synchronizes the subject, the skeletal structure, and the reference line; a display unit that displays the subject to be photographed, the skeletal structure, and the reference line synchronized by the data synchronization unit;

2. the skeletal structure has key points that indicate specific positions of the subject; 2. The video processing program according to claim 1, wherein the skeletal structure and the reference line are synchronized so that the reference line passes through at least one of the key points.

3. a step of photographing an object to be photographed by a photographing means; a step of estimating a skeletal structure of the photographed subject; calculating a reference line based on detection data of the inertial sensor; synchronizing the subject, the skeletal structure, and the reference line; and displaying the synchronized subject, skeletal structure, and reference line on a display unit.

4. In the step of estimating the skeletal structure of the photographed subject, a key point indicating a specific position of the photographed subject is detected; The image processing method according to claim 3, characterized in that in the step of synchronizing the object to be photographed, the skeletal structure, and the reference line, the skeletal structure and the reference line are synchronized so that the reference line passes through at least one of the key points.

5. Photography means, A display unit; an inertial sensor; a data analysis unit having a skeleton estimation function; a reference line calculation unit that calculates a reference line based on detection data detected by the inertial sensor; a data synchronization unit that synchronizes the image or video of the subject captured by the imaging means, the skeletal structure of the subject estimated by the data analysis unit, and the reference line; a display unit that displays the subject to be photographed, the skeletal structure, and the reference line synchronized by the data synchronization unit;

6. the data analysis unit detects key points that indicate specific positions of the subject; The image processing device of claim 5 , wherein the data synchronization unit synchronizes the skeletal structure and the reference line so that the reference line passes through at least one of the key points.

Images