Display device and operation method therefor

Abstract

Disclosed are a display device and an operation method therefor. The display device according to at least one of various embodiments disclosed herein comprises: a display for providing an application execution screen; and a processor for detecting first landmarks for a first image and second landmarks for a second image, and determining the motion similarity between the first image and the second image on the basis of the variation in the second landmarks on the basis of the detected first landmarks.

Description

Display device and method of operation thereof

[0001] The present disclosure relates to a display device and a method of operating the same.

[0002] The functions of recent terminals are becoming more diverse, including data and voice communication, taking pictures and videos through cameras, voice recording, playing music files through speaker systems, and outputting images or videos to displays.

[0003] Some terminals have added electronic game play capabilities or perform multimedia player functions.

[0004] As terminals become more diverse in their functions, they are being implemented in the form of multimedia players with complex functions such as taking photos or videos, playing music or video files, playing games, and receiving broadcasts.

[0005] The present disclosure aims to provide a display device and an operating method thereof capable of calculating mutual similarity even when there is a mismatch in synchronization between a user image and a reference image.

[0006] A display device according to at least one of various embodiments of the present disclosure may include a display providing an application execution screen; and a processor detecting first landmarks for a first image and second landmarks for a second image, and determining motion similarity between the first image and the second image based on an amount of change in the second landmarks based on the detected first landmarks.

[0007] At this time, the processor can calculate a first angle value of a preset major joint part for an object included in the first image, and can calculate a second angle value of a preset major joint part for an object included in the second image.

[0008] And the processor can calculate the difference between the first angle value and the second angle value, and further use the calculated difference between the first angle value and the second angle value to determine the motion similarity between the first image and the second image.

[0009] Additionally, the processor can calculate motion similarity between the first image and the second image using a time series data similarity analysis algorithm.

[0010] And the processor may set the region where each landmark is detected as a region of interest for each frame for the first image and the second image, normalize the coordinate values ​​of each landmark for the set region of interest, calculate the difference value for each landmark from the output of the previous frame, and generate a vector using the calculated difference value for each landmark and the angle value of the main joint.

[0011] Additionally, the processor can store the generated vectors in a sequence vector list by adding a relative time stamp from the start frame for each of the first image and the second image.

[0012] And the processor can compare the size of the second sequence vector list stored for the second image with a preset threshold value on a frame-by-frame basis, and check the timestamp value of the first sequence vector list stored for the first image based on the comparison result to read the corresponding sequence vector list.

[0013] Additionally, the processor can determine the motion similarity by using a vector sequence matching technique with the sequence vector list read for the first image and the second sequence vector list.

[0014] A method of operating a display device according to at least one of various embodiments of the present disclosure may include: detecting first landmarks for a first image and second landmarks for a second image; calculating a change amount of the second landmarks based on the detected first landmarks; calculating a first angle value of a preset major joint part for an object included in the first image and calculating a second angle value of a preset major joint part for an object included in the second image; calculating a difference between the calculated first angle value and the second angle value; and determining a motion similarity between the first image and the second image based on the calculated change amount and the difference between the first angle value and the second angle value.

[0015] At this time, the step of determining the motion similarity between the first and second images may use a time series data similarity analysis algorithm.

[0016] And the step of determining the motion similarity between the first image and the second image may further include the step of setting the region in which each landmark is detected as a region of interest on a frame basis for each of the first image and the second image; and the step of normalizing the coordinate values ​​of each landmark with respect to the set region of interest.

[0017] In addition, the step of determining the motion similarity between the first image and the second image may further include the step of calculating a difference value for each landmark from the output of the previous frame; and the step of generating a vector using the calculated difference value for each landmark and the angle value of the main joint.

[0018] And the step of determining the motion similarity between the first image and the second image may further include the step of adding a relative time stamp from the start frame for each of the first image and the second image and storing the generated vector in a sequence vector list.

[0019] In addition, the step of determining the motion similarity between the first image and the second image may further include the step of comparing the size of the second sequence vector list stored for the second image with a preset threshold value on a frame-by-frame basis, and checking the timestamp value of the first sequence vector list stored for the first image based on the comparison result to read the corresponding sequence vector list.

[0020] And the step of determining the motion similarity between the first image and the second image may use a vector sequence matching technique to match the sequence vector list read for the first image with the second sequence vector list.

[0021] According to at least one of the various embodiments of the present disclosure, there is an effect that enables more accurate calculation of motion similarity between multiple images even if they are not necessarily out of sync, such as when there is a delay between motions, a difference in viewpoint, or a difference in motion speed.

[0022] In addition, there is an effect of improving the user satisfaction of the display device user through the calculation of the above motion similarity.

[0023] FIG. 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present disclosure.

[0024] FIG. 2 is a block diagram of a remote control device according to an embodiment of the present disclosure.

[0025] Fig. 3 shows an example of an actual configuration of a remote control device according to one embodiment of the present disclosure.

[0026] FIG. 4 shows an example of utilizing a remote control device according to an embodiment of the present disclosure.

[0027] FIG. 5 is a drawing for explaining the horizontal mode and vertical mode of a stand-type display device according to an embodiment of the present disclosure.

[0028] FIG. 6 is a schematic diagram of a motion similarity judgment system for multiple images according to one embodiment of the present disclosure.

[0029] FIG. 7 is a flowchart illustrating an analysis process for a reference image according to an embodiment of the present disclosure.

[0030] Figures 8 to 9 are flowcharts illustrating an analysis process for a user image according to an embodiment of the present disclosure.

[0031] FIGS. 10 to 13 are diagrams illustrating a motion similarity determination algorithm according to an embodiment of the present disclosure.

[0032] FIG. 14 is a drawing showing an example of an execution screen of an application for determining motion similarity according to an embodiment of the present disclosure.

[0033] Hereinafter, embodiments related to the present disclosure will be described in more detail with reference to the drawings. The suffixes "module" and "part" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles.

[0034] FIG. 1 is a block diagram illustrating the configuration of a display device (100) according to one embodiment of the present disclosure.

[0035] Referring to FIG. 1, the display device (100) may include a broadcast receiving unit (130), an external device interface unit (135), a storage unit (140), a user input interface unit (150), a control unit (170), a wireless communication unit (173), a voice acquisition unit (175), a display unit (180), an audio output unit (185), and a power supply unit (190).

[0036] The broadcast receiving unit (130) may include a tuner (131), a demodulation unit (132), and a network interface unit (133).

[0037] The tuner (131) can select a specific broadcast channel according to a channel selection command. The tuner (131) can receive a broadcast signal for the selected specific broadcast channel.

[0038] The demodulator (132) can separate the received broadcast signal into a video signal, an audio signal, and a data signal related to the broadcast program, and can restore the separated video signal, audio signal, and data signal into a form that can be output.

[0039] The network interface unit (133) may provide an interface for connecting the display device (100) to a wired / wireless network including the Internet. The network interface unit (133) may transmit or receive data to or from other users or other electronic devices via the connected network or another network linked to the connected network.

[0040] The network interface unit (133) can access a predetermined web page through a connected network or another network linked to the connected network. In other words, it can access a predetermined web page through a network and transmit or receive data with the corresponding server.

[0041] In addition, the network interface unit (133) can receive content or data provided by a content provider or network operator. That is, the network interface unit (133) can receive content such as movies, advertisements, games, VOD (Video on Demand), broadcast signals, etc., and information related thereto provided from a content provider or network provider via a network.

[0042] Additionally, the network interface unit (133) can receive firmware update information and update files provided by the network operator, and transmit data to the Internet or content provider or network operator.

[0043] The network interface unit (133) can select and receive a desired application from among applications open to the public via a network.

[0044] The external device interface unit (135) can receive an application or a list of applications in an adjacent external device and transmit it to the control unit (170) or storage unit (140).

[0045] The external device interface unit (135) can provide a connection path between the display device (100) and the external device. The external device interface unit (135) can receive one or more of images and audio output from an external device connected wirelessly or wiredly to the display device (100) and transmit them to the control unit (170). The external device interface unit (135) can include a plurality of external input terminals. The plurality of external input terminals can include an RGB terminal, one or more HDMI (High Definition Multimedia Interface) terminals, and a component terminal.

[0046] The voice signal of an external device input through the external device interface unit (135) can be output through the display unit (180). The voice signal of an external device input through the external device interface unit (135) can be output through the audio output unit (185).

[0047] An external device that can be connected to the external device interface unit (135) may be any one of a set-top box, a Blu-ray player, a DVD player, a game console, a sound bar, a smartphone, a PC, a USB memory, and a home theater, but this is only an example.

[0048] Additionally, some content data stored in the display device (100) can be transmitted to a selected user or electronic device among other users or other electronic devices pre-registered in the display device (100).

[0049] The storage unit (140) stores programs for each signal processing and control within the control unit (170), and can store signal-processed images, voices, or data signals.

[0050] In addition, the storage unit (140) may perform a function for temporary storage of video, audio, or data signals input from an external device interface unit (135) or a network interface unit (133), and may also store information about a specific image through a channel memory function.

[0051] The storage unit (140) can store an application or a list of applications input from the external device interface unit (135) or the network interface unit (133).

[0052] The display device (100) can play content files (video files, still image files, music files, document files, application files, etc.) stored in the storage unit (140) and provide them to the user.

[0053] The user input interface unit (150) can transmit a signal input by a user to the control unit (170), or transmit a signal from the control unit (170) to the user. For example, the user input interface unit (150) can receive and process control signals such as power on / off, channel selection, and screen setting from the remote control device (200) according to various communication methods such as Bluetooth, Ultra Wideband (UWB), ZigBee, Radio Frequency (RF) communication, or infrared (IR) communication, or process the control signals from the control unit (170) to be transmitted to the remote control device (200).

[0054] In addition, the user input interface unit (150) can transmit control signals input from local keys (not shown) such as a power key, channel key, volume key, and setting value to the control unit (170).

[0055] An image signal processed by the control unit (170) can be input to the display unit (180) and displayed as an image corresponding to the image signal. In addition, an image signal processed by the control unit (170) can be input to an external output device through the external device interface unit (135).

[0056] The voice signal processed in the control unit (170) can be output as audio to the audio output unit (185). In addition, the voice signal processed in the control unit (170) can be input to an external output device through the external device interface unit (135).

[0057] In addition, the control unit (170) can control the overall operation within the display device (100).

[0058] In addition, the control unit (170) can control the display device (100) by a user command or internal program input through the user input interface unit (150), and can connect to a network to enable the user to download a desired application or application list into the display device (100).

[0059] The control unit (170) enables the channel information selected by the user to be output through the display unit (180) or audio output unit (185) together with the processed video or audio signal.

[0060] In addition, the control unit (170) allows a video signal or audio signal from an external device, for example, a camera or camcorder, input through the external device interface unit (135) to be output through the display unit (180) or audio output unit (185) in accordance with an external device video playback command received through the user input interface unit (150).

[0061] Meanwhile, the control unit (170) can control the display unit (180) to display an image, and for example, can control the display unit (180) to display a broadcast image input through the tuner (131), an external input image input through the external device interface unit (135), an image input through the network interface unit, or an image stored in the storage unit (140). In this case, the image displayed on the display unit (180) can be a still image or a moving image, and can be a 2D image or a 3D image.

[0062] In addition, the control unit (170) can control the playback of content stored in the display device (100), received broadcast content, or external input content input from outside, and the content can be in various forms such as broadcast video, external input video, audio file, still image, connected web screen, and document file.

[0063] The wireless communication unit (173) can communicate with an external device through wired or wireless communication. The wireless communication unit (173) can perform short-range communication with the external device. To this end, the wireless communication unit (173) can support short-range communication using at least one of Bluetooth™, BLE (Bluetooth Low Energy), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB, ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. This wireless communication unit (173) can support wireless communication between the display device (100) and a wireless communication system, between the display device (100) and another display device (100), or between the display device (100) and a network where the display device (100, or an external server) is located via a short-range wireless communication network (Wireless Area Network). The short-range wireless communication network may be a short-range wireless personal area network (Wireless Personal Area Network).

[0064] Here, the other display device (100) may be a wearable device (e.g., a smart watch, smart glasses, a head mounted display (HMD), a mobile terminal such as a smart phone) that can exchange data with the display device (100) according to the present disclosure (or can be linked). The wireless communication unit (173) may detect (or recognize) a wearable device capable of communication around the display device (100). Furthermore, if the detected wearable device is an authenticated device to communicate with the display device (100) according to the present disclosure, the control unit (170) may transmit at least a part of the data processed in the display device (100) to the wearable device through the wireless communication unit (173). Accordingly, a user of the wearable device may use the data processed in the display device (100) through the wearable device.

[0065] The voice acquisition unit (175) can acquire audio. The voice acquisition unit (175) can include at least one microphone (not shown) and can acquire audio around the display device (100) through the microphone (not shown).

[0066] The display unit (180) can generate a driving signal by converting a video signal, data signal, OSD signal processed by the control unit (170) or a video signal, data signal, etc. received from the external device interface unit (135) into R, G, B signals, respectively.

[0067] Meanwhile, since the display device (100) illustrated in FIG. 1 is merely an example of the present disclosure, some of the illustrated components may be integrated, added, or omitted depending on the specifications of the display device (100) actually implemented.

[0068] That is, two or more components may be combined into a single component, or a single component may be subdivided into two or more components, as needed. Furthermore, the functions performed by each block are intended to illustrate embodiments of the present disclosure, and their specific operations or devices do not limit the scope of the present disclosure.

[0069] According to another embodiment of the present disclosure, the display device (100) may receive and play back an image through a network interface unit (133) or an external device interface unit (135) without having a tuner (131) and a demodulator (132), unlike as shown in FIG. 1.

[0070] For example, the display device (100) may be implemented separately as an image processing device, such as a set-top box, for receiving broadcast signals or contents according to various network services, and a content playback device for playing contents input from the image processing device.

[0071] In this case, the method of operating the display device according to the embodiment of the present disclosure to be described below may be performed by any one of the display device (100) described with reference to FIG. 1, as well as an image processing device such as a separate set-top box, or a content playback device having a display unit (180) and an audio output unit (185).

[0072] The audio output unit (185) receives a signal processed by the control unit (170) and outputs it as voice.

[0073] The power supply unit (190) supplies power to the entire display device (100). In particular, it can supply power to a control unit (170) that can be implemented in the form of a system on chip (SOC), a display unit (180) for displaying images, and an audio output unit (185) for outputting audio.

[0074] Specifically, the power supply unit (190) may be equipped with a converter that converts AC power into DC power and a dc / dc converter that converts the level of the DC power.

[0075] Next, a remote control device according to an embodiment of the present disclosure will be described with reference to FIGS. 2 and 3.

[0076] FIG. 2 is a block diagram of a remote control device according to an embodiment of the present disclosure, and FIG. 3 shows an example of an actual configuration of a remote control device according to an embodiment of the present disclosure.

[0077] First, referring to FIG. 2, the remote control device (200) may include a fingerprint recognition unit (210), a wireless communication unit (220), a user input unit (230), a sensor unit (240), an output unit (250), a power supply unit (260), a storage unit (270), a control unit (280), and a voice acquisition unit (290).

[0078] Referring to FIG. 2, the wireless communication unit (220) transmits and receives signals with any one of the display devices according to the embodiments of the present disclosure described above.

[0079] The remote control device (200) may be equipped with an RF module (221) capable of transmitting and receiving signals with the display device (100) according to RF communication standards, and may be equipped with an IR module (223) capable of transmitting and receiving signals with the display device (100) according to IR communication standards. In addition, the remote control device (200) may be equipped with a Bluetooth module (225) capable of transmitting and receiving signals with the display device (100) according to Bluetooth communication standards. In addition, the remote control device (200) may be equipped with an NFC module (227) capable of transmitting and receiving signals with the display device (100) according to NFC (Near Field Communication) communication standards, and may be equipped with a WLAN module (229) capable of transmitting and receiving signals with the display device (100) according to WLAN (Wireless LAN) communication standards.

[0080] Additionally, the remote control device (200) transmits a signal containing information about the movement of the remote control device (200) to the display device (100) through the wireless communication unit (220).

[0081] Meanwhile, the remote control device (200) can receive a signal transmitted by the display device (100) through the RF module (221), and, if necessary, can transmit commands for turning the power on / off, changing the channel, changing the volume, etc. to the display device (100) through the IR module (223).

[0082] The user input unit (230) may be configured as a keypad, buttons, a touch pad, or a touch screen. The user can input commands related to the display device (100) to the remote control device (200) by operating the user input unit (230). If the user input unit (230) is equipped with a hard key button, the user can input commands related to the display device (100) to the remote control device (200) by pushing the hard key button. This will be described with reference to FIG. 3.

[0083] Referring to FIG. 3, the remote control device (200) may include a plurality of buttons. The plurality of buttons may include a fingerprint recognition button (212), a power button (231), a home button (232), a live button (233), an external input button (234), a volume control button (235), a voice recognition button (236), a channel change button (237), a confirmation button (238), and a back button (239).

[0084] The fingerprint recognition button (212) may be a button for recognizing a user's fingerprint. In one embodiment, the fingerprint recognition button (212) may be capable of a push operation, and thus may receive a push operation and a fingerprint recognition operation. The power button (231) may be a button for turning the display device (100) on / off. The home button (232) may be a button for moving to the home screen of the display device (100). The live button (233) may be a button for displaying a real-time broadcast program. The external input button (234) may be a button for receiving an external input connected to the display device (100). The volume control button (235) may be a button for adjusting the volume output by the display device (100). The voice recognition button (236) may be a button for receiving a user's voice and recognizing the received voice. The channel change button (237) may be a button for receiving a broadcast signal of a specific broadcast channel. The confirmation button (238) may be a button for selecting a specific function, and the back button (239) may be a button for returning to the previous screen.

[0085] Let's explain Figure 2 again.

[0086] When the user input unit (230) is equipped with a touch screen, the user can input commands related to the display device (100) using the remote control device (200) by touching the soft keys of the touch screen. In addition, the user input unit (230) may be equipped with various types of input devices that can be operated by the user, such as a scroll key or a jog key, and the present embodiment does not limit the scope of the present disclosure.

[0087] The sensor unit (240) may be equipped with a gyro sensor (241) or an acceleration sensor (243), and the gyro sensor (241) may sense information about the movement of the remote control device (200).

[0088] For example, the gyro sensor (241) can sense information about the operation of the remote control device (200) based on the x, y, and z axes, and the acceleration sensor (243) can sense information about the movement speed of the remote control device (200). Meanwhile, the remote control device (200) can further include a distance measuring sensor, so as to sense the distance to the display unit (180) of the display device (100).

[0089] The output unit (250) can output a video or audio signal corresponding to the operation of the user input unit (230) or to a signal transmitted from the display device (100). Through the output unit (250), the user can recognize whether the user input unit (230) is being operated or whether the display device (100) is being controlled.

[0090] For example, the output unit (250) may be equipped with an LED module (251) that lights up when the user input unit (230) is operated or a signal is transmitted and received with the display device (100) through the wireless communication unit (220), a vibration module (253) that generates vibration, a sound output module (255) that outputs sound, or a display module (257) that outputs an image.

[0091] In addition, the power supply unit (260) supplies power to the remote control device (200), and can reduce power waste by stopping the power supply when the remote control device (200) is not moved for a predetermined period of time. The power supply unit (260) can resume the power supply when a predetermined key provided in the remote control device (200) is operated.

[0092] The storage unit (270) can store various types of programs, application data, etc. required for the control or operation of the remote control device (200). If the remote control device (200) wirelessly transmits and receives signals through the display device (100) and the RF module (221), the remote control device (200) and the display device (100) transmit and receive signals through a predetermined frequency band.

[0093] The control unit (280) of the remote control device (200) can store and reference information regarding the frequency band, etc., that can wirelessly transmit and receive signals with the display device (100) paired with the remote control device (200), in the storage unit (270).

[0094] The control unit (280) controls all matters related to the control of the remote control device (200). The control unit (280) can transmit a signal corresponding to a predetermined key operation of the user input unit (230) or a signal corresponding to the movement of the remote control device (200) sensed by the sensor unit (240) to the display device (100) via the wireless communication unit (220).

[0095] Additionally, the voice acquisition unit (290) of the remote control device (200) can acquire voice.

[0096] The voice acquisition unit (290) may include at least one microphone (291) and may acquire voice through the microphone (291).

[0097] Next, Figure 4 is described.

[0098] FIG. 4 shows an example of utilizing a remote control device according to an embodiment of the present disclosure.

[0099] Figure 4 (a) illustrates that a pointer (205) corresponding to a remote control device (200) is displayed on a display unit (180).

[0100] The user can move or rotate the remote control device (200) up and down, left and right. The pointer (205) displayed on the display unit (180) of the display device (100) corresponds to the movement of the remote control device (200). As shown in the drawing, the pointer (205) moves and is displayed according to the movement in 3D space, so the remote control device (200) can be called a space remote control.

[0101] Figure 4 (b) illustrates that when a user moves the remote control device (200) to the left, the pointer (205) displayed on the display unit (180) of the display device (100) also moves to the left in response.

[0102] Information about the movement of the remote control device (200) detected by the sensor of the remote control device (200) is transmitted to the display device (100). The display device (100) can calculate the coordinates of the pointer (205) from the information about the movement of the remote control device (200). The display device (100) can display the pointer (205) to correspond to the calculated coordinates.

[0103] Figure 4 (c) illustrates a case where a user moves the remote control device (200) away from the display unit (180) while pressing a specific button within the remote control device (200). As a result, a selection area within the display unit (180) corresponding to the pointer (205) can be zoomed in and displayed in an enlarged manner.

[0104] Conversely, when the user moves the remote control device (200) closer to the display unit (180), the selection area within the display unit (180) corresponding to the pointer (205) may be zoomed out and displayed in a reduced size.

[0105] Meanwhile, when the remote control device (200) moves away from the display unit (180), the selection area may be zoomed out, and when the remote control device (200) moves closer to the display unit (180), the selection area may be zoomed in.

[0106] In addition, when a specific button within the remote control device (200) is pressed, recognition of up, down, left, and right movements may be excluded. That is, when the remote control device (200) moves away from or toward the display unit (180), up, down, left, and right movements may not be recognized, and only forward and backward movements may be recognized. When a specific button within the remote control device (200) is not pressed, only the pointer (205) moves in accordance with the up, down, left, and right movements of the remote control device (200).

[0107] Meanwhile, the movement speed or movement direction of the pointer (205) can correspond to the movement speed or movement direction of the remote control device (200).

[0108] Meanwhile, the pointer in this specification refers to an object displayed on the display unit (180) in response to the operation of the remote control device (200). Therefore, objects of various shapes other than the arrow shape illustrated in the drawing can be used as the pointer (205). For example, the pointer may be a concept including a point, a cursor, a prompt, a thick outline, etc. In addition, the pointer (205) may be displayed corresponding to one point on the horizontal or vertical axis on the display unit (180), or may be displayed corresponding to multiple points such as lines or surfaces.

[0109] FIG. 5(a) and FIG. 5(b) are drawings for explaining the horizontal mode and vertical mode of a stand-type display device according to an embodiment of the present disclosure.

[0110] Referring to FIG. 5(a) and FIG. 5(b), a stand-type display device (100) is illustrated.

[0111] A shaft (103) and a stand base (105) can be connected to the display device (100).

[0112] The shaft (103) can connect the display device (100) and the stand base (105). The shaft (103) can extend vertically.

[0113] The lower end of the shaft (103) can be connected to the edge of the stand base (105).

[0114] The lower end of the shaft (103) can be rotatably connected to the periphery of the stand base (105).

[0115] The display device (100) and shaft (103) can rotate around a vertical axis with respect to the stand base (105).

[0116] The upper part of the shaft (103) can be connected to the rear of the display device (100).

[0117] The stand base (105) can serve to support the display device (100).

[0118] The display device (100) may be configured to include a shaft (103) and a stand base (105).

[0119] The display device (100) can rotate around the point where the upper part of the shaft (103) and the rear part of the display (180) meet.

[0120] FIG. 5(a) shows that the display (180) operates in a landscape mode in which the horizontal length is greater than the vertical length, and FIG. 5(b) shows that the display (180) operates in a landscape mode in which the vertical length is greater than the horizontal length.

[0121] The user can move the stand-type display device (100). That is, unlike a fixed device, the stand-type display device (100) has improved mobility, so the user is not restricted by the placement location.

[0122] The display device (100) acquires an image of the user through an image acquisition device such as a camera connected thereto, and compares the acquired image of the user with a reference image to determine whether the user is following the movement well and provides feedback.

[0123] However, in the past, during the above comparison process, if the same movement was not performed at the same timing, it was judged as an incorrect imitation, making it difficult to properly judge the user's movement similarity. In reality, since the user follows along while watching the reference video, the timing is bound to be a bit slow. In the process, it is practically impossible to imitate the same movement or posture at the same speed, so performing the same movement at the same timing is bound to be very difficult, and the judgment of movement similarity based on this had the problem of low reliability.

[0124] In this way, in the past, evaluation was performed only when the motion was followed with a timing almost similar to that of the reference image, and if not, a low evaluation was received. To solve this problem, the present disclosure discloses a method for determining motion similarity between a reference image and a user image by inputting body landmarks in units of frames and using the amount of change and the angle values ​​of the major joints set in advance.

[0125] According to this disclosure, not only can similarity be calculated even if the camera shooting positions or user positions of the two images are different, but also the similarity of motions can be determined even if there is a time difference between the motions, that is, they are not at the same time or there is a difference in motion speed.

[0126] Accordingly, this specification discloses a display device (100) and method thereof capable of determining motion similarity of an object in an image according to various embodiments of the present disclosure.

[0127] The motion similarity judgment function according to the present disclosure is provided through a display device (100) as an example, but is not limited thereto.

[0128] The display device (100) related to the present disclosure may be replaced by, for example, a separate terminal (registered on a server or the display device (100), etc.) or various terminals that can be linked with the display device (100) connected through a network.

[0129] For convenience, the motion similarity determination function in this specification may be provided when a user selects and executes an application or a predetermined function installed on the display device (100). Meanwhile, the application may be any one of a native application built into the display device (100), an application provided by the manufacturer of the display device (100) or a third party, etc. The latter application may be downloaded and installed by the user on a display device (100) connected to the Internet.

[0130] FIG. 6 is a schematic diagram of a motion similarity judgment system for multiple images according to one embodiment of the present disclosure.

[0131] A motion similarity judgment system according to one embodiment of the present disclosure may be configured to include a display device (100) and an image acquisition device (300).

[0132] According to an embodiment, the motion similarity determination system may further include a smart pad (400) specifically for motion similarity determination.

[0133] According to another embodiment, the motion similarity determination system may further include a server (500).

[0134] According to another embodiment, the motion similarity determination system may be configured to include both a smart pad (400) and a server (500).

[0135] Hereinafter, for convenience of explanation, a system for providing a body diagnosis function is described using as an example a system including a server (400) illustrated in FIG. 6.

[0136] The display device (100) can receive a request from a user to execute a motion similarity determination function. At this time, the user can request execution of the motion similarity determination function using a terminal (not shown) or a remote control device (200) illustrated in FIG. 2.

[0137] At this time, the terminal may be a mobile terminal owned or registered by the user. Such a terminal may have applications installed for data communication with the display device (100). The mobile terminal may include a smartphone, tablet PC, laptop, or wearable device such as a smartwatch.

[0138] Meanwhile, if the display (180) of the display device (100) is configured as a touch panel, the request for executing the motion similarity determination function may be received by a user's touch input.

[0139] The image acquisition device (300) can be attached to one side of the display device (100) or installed outside the display device (100) to acquire image data for the user.

[0140] At this time, if the image acquisition device (300) is not attached to the display device (100) but is installed externally, it is preferable to link or connect with the display device (100) via a network. It is preferable that the network is the same network, but it is not limited thereto.

[0141] Meanwhile, the image acquisition device (300) may be provided in a built-in form as a component of the display device (100).

[0142] In the above, the image refers to a moving image, i.e., video data, and may also mean a still image, i.e., image data, still image data, etc.

[0143] In FIG. 6, only one image acquisition device (300) is illustrated and described for convenience, but the present disclosure is not limited thereto.

[0144] Depending on the embodiment, there may be multiple image acquisition devices. In this case, the configuration and performance of each image acquisition device may or may not be identical. For example, if there are two image acquisition devices, the images acquired from each image acquisition device may be processed as left image data from one and right image data from the other, thereby forming a three-dimensional (3-dimensional) image. However, for the convenience of explanation, the present disclosure will be described as an example using a single image acquisition device (300) and processing data in a 2D format.

[0145] The smart pad (400) may not necessarily be an essential component. However, the smart pad (400) may be connected or linked to at least one of the display device (100), the image acquisition device (300), and the server (400) to be used for executing the motion similarity determination function. For example, when a user is positioned on the smart pad (400), the smart pad (400) can collect more accurate data for determining the motion similarity of the user's image, and process the data in the display device (100), the server (500), etc.

[0146] Such a smart pad (400) may replace or perform some of the various functions or components of, for example, a display device (100), an image acquisition device (300), and a server (500).

[0147] The server (500) may not necessarily be an essential component. However, the server (500) may replace all or part of the operations or processing related to the operation similarity determination function of the display device (100).

[0148] The server (500) can acquire a user's image from the image acquisition device (300). The server (500) can transmit the user's image data thus acquired to the display device (100).

[0149] At this time, during the above transmission process, the user's image data may be transmitted as raw data, or may be transmitted to the display device (100) after processing. During the above processing process, the server (500) may process the acquired user's image data and transmit the analyzed result information to the display device (100). The analyzed result information may also include a control command for controlling the display device (100).

[0150] The server (500) can obtain information about the user from the smart pad (400). The server (500) can transmit the user's image data thus obtained to the display device (100). During the transmission process, the user's image data may be transmitted as raw data, or may be transmitted to the display device (100) after being processed. During the processing process, the server (500) can process the obtained user's image data and transmit the analyzed result information to the display device (100). The analyzed result information may also include a control command for controlling the display device (100).

[0151] Hereinafter, a process of executing a motion similarity judgment function according to a user's request or selection in a display device (100) will be described in more detail.

[0152] FIG. 7 is a flowchart illustrating an analysis process for a reference image according to an embodiment of the present disclosure.

[0153] Figures 8 to 9 are flowcharts illustrating an analysis process for a user image according to an embodiment of the present disclosure.

[0154] A method of operating a display device according to at least one of various embodiments of the present disclosure may include a process of detecting first landmarks for a first image and second landmarks for a second image, a process of calculating a change amount of the second landmarks based on the detected first landmarks, a process of calculating a first angle value of a preset major joint part for an object included in the first image and a process of calculating a second angle value of a preset major joint part for an object included in the second image, a process of calculating a difference between the calculated first angle value and the second angle value, and a process of determining a motion similarity of the first image and the second image based on the difference between the calculated change amount and the first angle value and the second angle value.

[0155] At this time, the process of determining the motion similarity between the first and second images can use a time series data similarity analysis algorithm.

[0156] And the process of determining the motion similarity between the first image and the second image may further include a process of setting the region where each landmark is detected as a region of interest on a frame-by-frame basis for each of the first image and the second image, and a process of normalizing the coordinate values ​​of each landmark with respect to the set region of interest.

[0157] In addition, the process of determining the motion similarity between the first image and the second image may further include a process of calculating a difference value for each landmark from the output of the previous frame, and a process of generating a vector using the calculated difference value for each landmark and the angle value of the main joint.

[0158] And the process of determining the motion similarity of the first image and the second image may further include a process of adding a relative time stamp from the start frame for each of the first image and the second image and storing the generated vector in a sequence vector list.

[0159] In addition, the process of determining the motion similarity between the first image and the second image may further include a process of comparing the size of the second sequence vector list stored for the second image with a preset threshold value on a frame-by-frame basis, and checking the timestamp value of the first sequence vector list stored for the first image based on the comparison result to read the corresponding sequence vector list.

[0160] And the process of determining the motion similarity between the first image and the second image can use a vector sequence matching technique with the sequence vector list read for the first image and the second sequence vector list.

[0161] First, the reference image analysis process will be specifically described with reference to Fig. 7 as follows.

[0162] In the S101 operation, the processor can sequentially extract body landmarks frame by frame from the reference image and input them.

[0163] In operation S103, the processor can find an area where a body landmark appears and set a region of interest (ROI). The region of interest may be, for example, a rectangular shape, but is not limited thereto.

[0164] In operation S105, the processor can perform normalization on body landmark coordinates for a region of interest (ROI).

[0165] At this time, the above standardization may be for the purpose of forming a common coordinate to compensate for the fact that each user may have a different body type and the camera installation environment may be different, for example.

[0166] In operation S107, the processor can calculate a first difference value per landmark between the normalized output in the previous frame and the normalized output in the current frame.

[0167] In the S109 operation, the processor can calculate the first angle value of the preset main joint.

[0168] In operation S111, the processor can generate a first vector using the first difference value and the first angle value.

[0169] In operation S113, the processor can store the generated first vector in a first sequence vector list (SEQ_A) by adding a relative time timestamp from the start of the image.

[0170] In operation S115, the processor can sequentially and repeatedly perform the aforementioned operations S101 to S113 for frames.

[0171] In operation S117, the processor can store the first sequence vector list (SEQ_A) finally generated as a result of the above repetition.

[0172] Through the above-described operation, preliminary work on a reference image for determining motion similarity can be completed.

[0173] In Fig. 7, for convenience, the above-described operation is described as being performed sequentially and repeatedly for all frames, but it may not be limited thereto.

[0174] For example, it can be defined in various ways, such as not every frame, but a certain number of frames, a frame unit in which a scene change is performed, a frame unit at a preset interval, a frame unit corresponding to a certain time interval, etc.

[0175] Next, the user image analysis process will be specifically described with reference to FIGS. 8 and 9 as follows.

[0176] If Fig. 7 is a reference image (or an instructor image), Figs. 8 to 9 can represent the image analysis process of a user who is the subject of motion similarity judgment.

[0177] First, the basic image analysis process, i.e., part of the user image analysis process (corresponding to FIG. 8), may be identical or similar to the image analysis process of FIG. 7 described above.

[0178] For example, in the S201 operation, the processor can sequentially extract body landmarks frame by frame from the user image and input them as input.

[0179] In operation S203, the processor can find an area where a body landmark appears and set a region of interest (ROI). The region of interest may be, for example, a rectangular shape, but is not limited thereto.

[0180] In operation S205, the processor can perform normalization on body landmark coordinates for a region of interest (ROI).

[0181] In operation S207, the processor can calculate a second difference value per landmark between the normalized output in the previous frame and the normalized output in the current frame.

[0182] In operation S209, the processor can calculate the second angle value of the preset main joint.

[0183] In operation S211, the processor can generate a second vector using the second difference value and the second angle value.

[0184] In operation S213, the processor can store the generated second vector in a second sequence vector list (SEQ_B) by adding a relative time timestamp from the start of the image.

[0185] However, the following is one of the steps in the motion similarity determination process of the two images, and FIG. 9 may be performed after operation S213 of FIG. 8. However, the present disclosure is not limited thereto. In addition, operation S301 described below may not be performed immediately after operation S213 of FIG. 8.

[0186] In operation S301, the processor can determine whether the size of the second sequence vector list (SEQ_B) stored in operation S213 exceeds or is greater than a preset threshold.

[0187] In operation S303, if the processor determines that the size of the stored second sequence vector list (SEQ_B) exceeds or is greater than a preset threshold as a result of the above operation S301, it can load the first sequence vector list (SEQ_A) stored in FIG. 7 and obtain data up to the current timestamp.

[0188] In operation S305, the processor can calculate the operation similarity between the first sequence vector list (SEQ_A) and the second sequence vector list (SEQ_B).

[0189] The result of calculating the motion similarity of the two images thus produced may be provided through the screen of the display device (100) or transmitted to a separately linked terminal (not shown) and output.

[0190] The above motion similarity calculation results can be provided in various forms, such as score form, Pass / Fail form, and graph form.

[0191] Meanwhile, if the processor determines that the size of the stored second sequence vector list (SEQ_B) is equal to or less than a preset threshold as a result of the above S301 operation, it may return to the S201 operation of FIG. 8 and repeat the process of FIG. 8 for the next frame.

[0192] As described above, the operations of FIGS. 8 and 9 can be performed on frames of the acquired user image. This allows for comparison of the user image with a reference image to determine motion similarity.

[0193] In Fig. 8, for convenience, the above-described operation is described as being performed sequentially and repeatedly for all frames, but it may not be limited to this.

[0194] For example, it can be defined in various ways, such as not every frame, but a certain number of frames, a frame unit in which a scene change is performed, a frame unit at a preset interval, a frame unit corresponding to a certain time interval, etc.

[0195] Meanwhile, in step S301 of FIG. 9, the preset threshold may be, for example, 40 to 60. Preferably, the preset threshold may be about 50. However, the present invention is not limited thereto.

[0196] Meanwhile, the preset threshold can be arbitrarily changed.

[0197] In relation to operation S301 of Fig. 9, if the size of the second sequence vector list is greater than the threshold, the current timestamp value can be checked to read values ​​smaller than the corresponding timestamp in the first sequence vector list. In other words, only values ​​for frames that are compared to the user image can be read.

[0198] In operation S305 of FIG. 9, the operation similarity calculation can be performed by calculating the similarity between, for example, the list read from the first sequence vector list (SEQ_A) and the list of the second sequence vector list (SEQ_B) using a vector sequence matching technique.

[0199] The distance measurement used when comparing vectors basically uses the cosine distance, but it can be modified to use other distance measurements such as the Euclidean distance.

[0200] The vector sequence matching method for determining motion similarity between the two images described above is only an example, and the present disclosure is not limited thereto.

[0201] FIGS. 10 to 13 are diagrams illustrating a motion similarity determination algorithm according to an embodiment of the present disclosure.

[0202] FIG. 10 may be an image similarity graph per frame using a motion similarity judgment algorithm according to an embodiment of the present disclosure described above.

[0203] Referring to Fig. 10(a), the vector values ​​can be stored as a list after calculating the position change amount of the body landmark and the angle values ​​of the main joints.

[0204] Through this, Fig. 10(b) enables a comparison of the first sequence vector list (SEQ_A) and the second sequence vector list (SEQ_B) based on time difference. The similarity is approximately 1.0887988, which can be converted to a score of approximately 89.1117.

[0205] FIG. 11 may be a graph illustrating, for example, a method of comparing changes in scores (or similarity) by section with the same image.

[0206] Referring to Figure 11, it may be a method of recording scores by section, generating a motion score (or similarity) graph for the entire image, and then comparing them.

[0207] In addition, in FIG. 11, the first waveform (1110) is a score waveform according to the motion similarity judgment method according to the present disclosure, and the second waveform (1120) may have adopted a similar algorithm because, although the score is different from the first waveform (1110), the trend is similar. On the other hand, it can be seen that the third waveform (1130) shows a completely different waveform from the first waveform (1110) for the same image.

[0208] Figure 12(a) illustrates extracting body landmarks.

[0209] Figure 12(b) explains setting and detecting a region of interest (ROI) using extracted body landmarks.

[0210] Meanwhile, in Fig. 12(b), resizing and scaling can be performed if necessary.

[0211] Additionally, if necessary, a transform matrix can be calculated and applied.

[0212] Figure 12(c) illustrates the normalization operation.

[0213] Referring to the graph of Fig. 13, the first waveform (1310) is an image that follows a reference image (e.g., a national gymnastics image), and the average similarity may be approximately 3.704.

[0214] The second waveform (1320) is, for example, an image downloaded from an application related to a reference image rather than an actual user image, and it can be seen that the average similarity is approximately 12.834.

[0215] The third waveform (1330) is a video of a completely different movement or exercise, not one that follows a user video or a reference video (e.g., a national gymnastics video), and the average similarity is about 29.330.

[0216] A lower similarity may indicate more similar behavior.

[0217] FIG. 14 is a drawing showing an example of an execution screen of an application for determining motion similarity according to an embodiment of the present disclosure.

[0218] FIG. 14 is a scene of use in which a user follows an exercise played in an exercise video VIEW, for example, and a user preview in which the user's appearance is output through an image acquisition device (300) connected to a display device (100), and a body landmark and a responsive avatar that moves in the same way as the user can be provided.

[0219] A reference image may be displayed in the first area (1410) of the application execution screen, a responsive avatar may be displayed in the second area (1420), and a user preview image may be provided in the third area (1430).

[0220] A section of the application execution screen may display the results of motion similarity calculations (e.g., scores). These scores may be provided in real time.

[0221] The exercise score section is expressed in two ways: Average is the average score from the moment you first started exercising to the present, and Current can indicate the score for the movement over the past few seconds.

[0222] In some embodiments, a user preview image may be provided in the second area (1420).

[0223] Depending on the embodiment, the user preview image of the third area (1430) may not be provided.

[0224] Depending on the embodiment, the positions of the second region (1420) and the third region (1430) may be changed. Furthermore, the positions of the two regions may be arbitrarily changed. Similarly, the position of the reference image may also be arbitrarily changed.

[0225] Even if not specifically mentioned, the order of at least some of the operations disclosed in the present disclosure may be performed simultaneously, in a different order than the order described above, or some may be omitted / added.

[0226] According to one embodiment of the present disclosure, the above-described method can be implemented as processor-readable code on a medium in which a program is recorded. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

[0227] The display device described above is not limited to the configuration and method of the embodiments described above, and the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

[0228] The present disclosure relates to a display device and an operating method thereof capable of calculating accurate motion similarity even when synchronization is not necessarily achieved between multiple images, and has industrial applicability as it can be utilized in various electronic devices.

Claims

1. A display that provides an application execution screen; and A processor that detects first landmarks for a first image and second landmarks for a second image, and determines motion similarity between the first image and the second image based on the amount of change in the second landmarks with respect to the detected first landmarks. Display device.

2. In claim 1, The above processor, Calculating a first angle value of a preset major joint part for an object included in the first image, and calculating a second angle value of a preset major joint part for an object included in the second image. Display device.

3. In claim 2, The above processor, The difference between the first angle value and the second angle value is calculated, and the difference between the first angle value and the second angle value is further used to determine the motion similarity between the first image and the second image. Display device.

4. In claim 3, The above processor, Calculating the motion similarity between the first and second images using a time series data similarity analysis algorithm. Display device.

5. In claim 4, The above processor, For the first and second images, the area where each landmark is detected is set as the area of ​​interest in each frame, the coordinate values ​​of each landmark are normalized for the set area of ​​interest, the difference value for each landmark is calculated from the output of the previous frame, and a vector is generated using the calculated difference value for each landmark and the angle value of the main joint. Display device.

6. In claim 5, The above processor, For each of the first and second images, the generated vectors are stored in a sequence vector list by adding a relative time stamp from the start frame. Display device.

7. In claim 6, The above processor, Comparing the size of the second sequence vector list stored for the second image in units of frames with a preset threshold value, and checking the timestamp value of the first sequence vector list stored for the first image based on the comparison result to read the corresponding sequence vector list. Display device.

8. In claim 7, The above processor, The sequence vector list read for the first image is used to determine the motion similarity with the second sequence vector list using a vector sequence matching technique. Display device.

9. A step of detecting first landmarks for the first image and second landmarks for the second image; A step of calculating the amount of change in the second landmarks based on the detected first landmarks; A step of calculating a first angle value of a preset major joint part for an object included in the first image, and calculating a second angle value of a preset major joint part for an object included in the second image; A step of calculating the difference between the first angle value and the second angle value calculated above; and A method comprising: determining the motion similarity between the first image and the second image based on the difference between the calculated change amount and the first angle value and the second angle value; How the display device operates.

10. In claim 9, The step of determining the motion similarity between the first and second images is as follows: Using a time series data similarity analysis algorithm, How the display device operates.

11. In claim 10, The step of determining the motion similarity between the first and second images is as follows: For the first and second images above, respectively, A step of setting the area where each landmark is detected as a region of interest on a frame-by-frame basis; and Further comprising a step of normalizing the coordinate values ​​of each landmark targeting the above-set area of ​​interest. How the display device operates.

12. In claim 11, The step of determining the motion similarity between the first and second images is as follows: A step of calculating the difference value for each landmark and the output of the previous frame; and Further comprising a step of generating a vector using the difference value for each landmark calculated above and the angle value of the main joint part. How the display device operates.

13. In claim 12, The step of determining the motion similarity between the first and second images is as follows: Further comprising a step of storing the generated vector in a sequence vector list by adding a relative time stamp from the start frame for each of the first and second images. How the display device operates.

14. In claim 13, The step of determining the motion similarity between the first and second images is as follows: A method further comprising: comparing the size of the second sequence vector list stored for the second image with a preset threshold value in units of frames, and checking the timestamp value of the first sequence vector list stored for the first image based on the comparison result, and reading the corresponding sequence vector list. How the display device operates.

15. In claim 14, The step of determining the motion similarity between the first and second images is as follows: The sequence vector list read for the first image is used with the second sequence vector list using a vector sequence matching technique. How the display device operates.

Images