Electronic device and method for providing haptic data for guiding motion

Abstract

An electronic device according to an embodiment comprises at least one processor and a memory for storing instructions. The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: acquire a video and haptic data; generate a group of external electronic devices; allocate objects included in the video to the external devices; transmit the haptic data to the external electronic devices; and on the basis of receiving a first user input for playing the video and outputting the haptic data, transmit, to the external electronic devices, a signal for initiating the output of the haptic data.

Description

Electronic device and method for providing haptic data to guide motion

[0001] The present disclosure relates to an electronic device and method for providing haptic data for guiding motion.

[0002] Electronic devices can display objects performing various motions (e.g., exercise, dance) through a display device. For example, the objects may be instructors, images, and / or characters performing specified motions. The electronic devices can provide programs that can help users learn motions by following the motions of the objects.

[0003] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0004] An electronic device is provided. The electronic device may include a display. The electronic device may include a camera. The electronic device may include wireless communication circuitry for wirelessly connecting with external electronic devices. The electronic device may include at least one processor. The electronic device may include a memory storing instructions. The instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: acquire a video through the camera; acquire haptic data related to the motion of an object included in the video; create a group of the external electronic devices connected through the wireless communication circuitry; assign one of the objects to each of the external electronic devices; transmit haptic data corresponding to the assigned one object to each of the external electronic devices; and, based on receipt of a first user input for playing the video and outputting the haptic data, transmit a signal to the external electronic devices to initiate output of the haptic data corresponding to the one object.

[0005] A method of an electronic device is provided. The method may include an operation of obtaining video and haptic data. The method may include an operation of creating a group of one or more external electronic devices for guiding the motion of one or more objects included in the video. The method may include an operation of assigning one of the one or more objects to each of the one or more external electronic devices. The method may include an operation of transmitting, to each of the one or more external electronic devices, the haptic data corresponding to the one object assigned to each of the one or more external electronic devices. The method may include an operation of transmitting, to each of the one or more external electronic devices, a signal for initiating output of the haptic data corresponding to the one object based on reception of a first user input for playing the video and outputting the haptic data.

[0006] An electronic device is provided. The electronic device may include at least one processor. The electronic device may include a memory storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to: obtain video and haptic data; generate a group of external electronic devices for guiding motion of one or more objects included in the video; assign one of the one or more objects to the one or more external electronic devices; transmit, to the external electronic device, the haptic data corresponding to the one object assigned to the external electronic device; and, based on receipt of a first user input for playing the video and outputting the haptic data, transmit, to the external electronic device, a signal for initiating output of the haptic data corresponding to the one object.

[0007] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when individually or collectively executed by at least one processor of an electronic device, can cause the electronic device to acquire video and haptic data. The instructions, when individually or collectively executed by at least one processor of the electronic device, can cause the electronic device to create a group of one or more external electronic devices for guiding the motion of one or more objects included in the video. The instructions, when individually or collectively executed by at least one processor of the electronic device, can cause the electronic device to assign one of the one or more objects to the one or more external electronic devices. The instructions, when individually or collectively executed by at least one processor of the electronic device, may cause the electronic device to transmit, to one or more external electronic devices, the haptic data corresponding to the one object assigned to the one or more external electronic devices. The instructions, when individually or collectively executed by at least one processor of the electronic device, may cause the electronic device to transmit, to the one or more external electronic devices, a signal to initiate output of the haptic data corresponding to the one object based on reception of a first user input for playing the video and outputting the haptic data.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0009] FIG. 2 is a block diagram of a system for guiding users' motions using an electronic device according to one embodiment.

[0010] Figure 3a schematically illustrates the process of generating a haptic notification based on a video.

[0011] Figure 3b is a block diagram of the server.

[0012] Figure 4 is a diagram schematically illustrating haptic notifications according to the type of haptic data.

[0013] Figure 5 illustrates an example of a haptic notification based on haptic data generated based on a video.

[0014] FIG. 6 is a flow chart illustrating the operation of an electronic device and the operation of one or more external electronic devices to guide motion.

[0015] Figure 7a shows a UI displayed on an external electronic device to join a group.

[0016] Figure 7b shows a UI displayed on an external electronic device for selecting an object.

[0017] Figure 8 illustrates a UI displayed on an external electronic device.

[0018] FIG. 9a illustrates a state in which an electronic device according to one embodiment mirrors a video to a display device.

[0019] Figure 9b is a block diagram of an exemplary external electronic device.

[0020] Figure 9c illustrates an example of data acquired through a sensor of an external electronic device.

[0021] Figure 9d illustrates a process in which an external electronic device acquires data related to the user's motion.

[0022] FIG. 10 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to calculate a ranking based on similarity data and display the calculated ranking on a video.

[0023] FIG. 11A is a flowchart illustrating an operation of an electronic device providing feedback according to one embodiment.

[0024] Figure 11b schematically shows the similarity for the timeline.

[0025] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0026] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) to an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] In one embodiment, an event performed by one or more objects may require coordination of motions of the one or more objects.

[0049] According to one embodiment, the electronic device (101) can provide learning about motions performed by one or more objects included in a video. The one or more objects can be referred to as objects capable of expressing motions, such as people (e.g., instructors), virtual objects (e.g., images, characters, avatars), and / or animals included in the video. A user can receive learning about motions by viewing and imitating the motions taken by one or more objects included in the video. The one or more objects can be one or multiple.

[0050] For example, a group dance performed by a plurality of objects can be harmonious by performing not only the unified motion of the plurality of objects but also the unique motion assigned to each object at precise timing. The electronic device (101) according to one embodiment can provide haptic data for guiding the motion of each of one or more objects. The electronic device (101) according to one embodiment can provide feedback on the motion of each of one or more users from one or more external electronic devices (e.g., one or more external electronic devices (220) of FIG. 2). The group dance described above is an example of learning about motions that can be provided through video, but is not limited thereto. The electronic device (101) according to one embodiment can be used to provide various learning, such as playing a musical instrument, in addition to learning about motions such as group dance, yoga, health, and / or gymnastics.

[0051] FIG. 2 is a block diagram of a system for guiding users' motions using an electronic device according to one embodiment.

[0052] Referring to FIG. 2, an electronic device (101) according to one embodiment may include at least one processor (e.g., processor (120) of FIG. 1) and memory (e.g., memory (130) of FIG. 1).

[0053] Any function or operation described in this specification may be processed by the processor (120). The processor (120) is a circuit that performs processing and includes an application processor (AP, eg, a central processing unit (CPU)), a communication processor (CP, eg, a modem), a graphics processing unit (eg, a GPU), a neural processing unit (NPU) (eg, an artificial intelligence (AI) chip), a wireless-fidelity (Wi-Fi) chip, a Bluetooth® chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, connectivity chips, a sensor controller, a touch controller, a finger-print sensor controller, a display drive integrated circuit (DDI), an audio CODEC chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on chip (SoC), an integrated circuit (IC), or a similar circuit.

[0054] For example, the memory (130) can store instructions. The instructions stored in the memory (130), when individually or collectively executed by the processor (120), can cause operations of the electronic device (101). For example, the instructions, when individually or collectively executed by the processor (120), can cause the electronic device (101) to perform the operations described in FIGS. 6, 10, and 11A.

[0055] An electronic device (101) according to one embodiment may further include a wireless communication circuit (e.g., a wireless communication module (192) of FIG. 1), a display (e.g., a display module (160) of FIG. 1), a speaker (e.g., an audio module (170) of FIG. 1), a camera (e.g., a camera module (180) of FIG. 1), and / or an actuator (e.g., a haptic module (179) of FIG. 1).

[0056] For example, the wireless communication circuit (192) may be used to communicate with one or more external electronic devices (220) and other external electronic devices (210) (e.g., a server, a storage device (e.g., USB), a smart phone, a tablet, a computer, a laptop). The electronic device (101) may download video and / or haptic data from the external electronic device (210) using the wireless communication circuit (192). The electronic device (101) may transmit haptic data to one or more external electronic devices (220) using the wireless communication circuit (192) and broadcast a signal to initiate output of the haptic data. The electronic device (101) may communicate with one or more external electronic devices (220) using short-range wireless communication (e.g., Bluetooth communication).

[0057] For example, the display (160) and / or the speaker (170) may be controlled by the processor (120). The display (160) may be configured to display a video based on a control signal of the processor (120). The speaker (170) may be configured to output audio included in the video based on a control signal of the processor (120). The actuator (179) may provide a haptic notification by generating a vibration.

[0058] For example, the external electronic device (210) may provide a video. The video may be referred to as a video related to an event performed by one or more objects, as content including one or more objects. For example, the video may include a video related to a group dance in which one or more objects dance to music. For example, the video may be a stage video in which one or more objects dance to music, or a practice video in which one or more objects practice a choreography. Hereinafter, a group dance video is described as an example of content in the present disclosure, but is not limited thereto. The group dance described below may be replaced with an event performed by one or more objects. For example, the content may be replaced with a play, a musical, a group exercise, health, yoga, or a performance (e.g., playing an instrument).

[0059] An electronic device (101) according to one embodiment may include a camera (180). The camera (180) may be configured to generate images and / or videos by photographing a subject. For example, the electronic device (101) may use the camera (180) to photograph one or more objects, thereby generating a video related to an event performed by the one or more objects. For example, a user may use the electronic device (101) to photograph a performance in which multiple members dance at a concert, thereby generating a video including multiple objects corresponding to the multiple members. For example, the electronic device (101) may use an artificial intelligence model to generate haptic data and / or reference data from a video generated by the camera (180).

[0060] According to one embodiment, the external electronic device (210) may provide haptic data based on a video. The haptic data may be referred to as haptic data generated based on the motion of each of one or more objects included in the video. For example, the external electronic device (210) may extract the motion of each of one or more objects using an artificial intelligence model and generate haptic data based on the extracted motion. If there are multiple objects, the haptic data may be multiple. For example, if the video is a video of four objects dancing to music, the artificial intelligence model may extract the motions of each of the four objects over time. The artificial intelligence model may generate haptic data based on each of the extracted motions. For example, the artificial intelligence model may extract a first motion of a first object and generate first haptic data based on the first motion. The artificial intelligence model may extract a second motion of a second object and generate second haptic data based on the second motion. If the motion of the first object and the motion of the second object are not identical throughout the entire time, the first haptic data and the second haptic data may be different. The haptic data will be described later with reference to FIG. 4. In terms of the artificial intelligence model generating haptic data from a video, the artificial intelligence model may be referred to as, but is not limited to, generative artificial intelligence. The external electronic device (210) may provide reference data based on the haptic data. The reference data will be described later with reference to FIG. 3B.

[0061] In the above-described embodiment, the external electronic device (210) has been described as generating haptic data using an artificial intelligence model, but is not limited thereto. According to one embodiment, the electronic device (101) may generate haptic data from a video. For example, the processor (120) may extract motions of each of one or more objects included in the video and generate haptic data based on each motion. The generated haptic data may be stored in the memory (130).

[0062] According to one embodiment, the electronic device (101) can download video and haptic data from an external electronic device (210) via the wireless communication circuit (192). For example, the electronic device (101) can download video and haptic data stored in the external electronic device (210) for group dance practice of one or more users. As described above, the electronic device (101) can also generate haptic data from the video. For example, the electronic device (101) can generate haptic data by extracting motions of one or more objects included in the video using an artificial intelligence model.

[0063] According to one embodiment, the electronic device (101) may be operatively coupled with a display device (230). For example, the electronic device (101) may mirror content (e.g., a video) displayed on the display (160) of the electronic device (101) onto the display device (230). The electronic device (101) may mirror the video onto the display device (230) so that one or more users may view the video and practice group dance. The display device (230) may be a device that provides a larger display screen than the electronic device (101), and one or more users may practice motions (e.g., a group dance) while viewing the mirrored video.

[0064] According to one embodiment, the electronic device (101) can communicate with one or more external electronic devices (220). The electronic device (101) can transmit haptic data downloaded from the external electronic device (210) or haptic data generated by the electronic device (101) using an artificial intelligence model to the one or more external electronic devices (220). Each of the one or more external electronic devices (220) can receive haptic data from the electronic device (101) and provide a haptic notification based on the received haptic data. For example, the one or more external electronic devices (220) can include a vibration motor (e.g., vibration motor 904 of FIG. 9B) for providing a haptic notification.

[0065] Haptic data generated based on a video can guide the motions of one or more users. The haptic data can be configured to provide various types of haptic notifications, rather than simply providing haptic notifications, so that users can perform specified motions at specified timings. One or more external electronic devices (220) can guide the motions of one or more users possessing one or more external electronic devices (220) by providing haptic notifications.

[0066] According to one embodiment, the one or more external electronic devices (220) may include wearable devices worn by one or more users who practice group dancing. For example, the one or more external electronic devices (220) may include, but are not limited to, a smart watch worn on a wrist, smart earrings worn on an ear, an augmented reality (AR) device or a virtual reality (VR) device worn on a user's body, and / or a smart ring worn on a finger. For example, the one or more external electronic devices (220) may include a mixed reality (MR) device including a video see-through (VST) and / or an optical see-through (OST) device and / or an extended reality (XR) device.

[0067] According to one embodiment, one or more external electronic devices (220) carried by each of one or more users for practicing group dance may be of different types. For example, the first external electronic device (221) and the second external electronic device (222) may be smart watches, the third external electronic device (223) may be a smart ring, and the fourth external electronic device (224) may be AR glasses. Since the wearable device is worn on the user's body, the wearable device may guide the user's motions through haptic notifications.

[0068] Below, haptic notifications generated based on video are described.

[0069] Figure 3a schematically illustrates the process of generating a haptic notification based on an image. Figure 3b is a block diagram of an external electronic device.

[0070] 301 of FIG. 3A represents a video. Referring to 301 of FIG. 3A, according to one embodiment, the video may include images of a plurality of people dancing. In this case, the electronic device (101) may identify each of the plurality of people as objects (310). For example, the video may identify objects (310) corresponding to each of the four people, and may classify the objects into a first object (313), a second object (311), a third object (313), and a fourth object (314). The video may include audio. According to one embodiment, an artificial intelligence model may be used to generate haptic data based on the video.

[0071] 303 of FIG. 3A illustrates a process of identifying the movement (motion) of objects corresponding to each of four people in a video and generating haptic data based on each motion. Referring to 303 of FIG. 3A, an artificial intelligence model may, according to one embodiment, distinguish each of four objects from a video and extract motions based on the movement from the four objects. For example, the artificial intelligence model may detect four objects in a video (e.g., object detection), recognize each motion from the four objects (e.g., action recognition), and track each motion from the four objects (e.g., motion tracking) to extract motions. Based on the extracted motions, the artificial intelligence model may generate motion data over time and generate haptic data (307) based on the motion data.

[0072] In the above description, it has been described that four people are included in the video and the motions of each of the four people are extracted, but the present invention is not limited thereto. For example, in the case of a video in which real people are filmed, objects (310) are identified from the person (or people) included in the video, but the present invention is not limited thereto. For example, the video may include various objects, including virtual objects that are not real (e.g., characters, avatars, 3D models) or objects that are distinct from people (e.g., animals), and the electronic device (101) may also obtain haptic data corresponding to the motions of various objects.

[0073] 305 of FIG. 3A illustrates a process of generating haptic data based on amplitude data regarding audio extracted from a video. Referring to 305 of FIG. 3A, an artificial intelligence model can extract audio from a video. For example, the artificial intelligence model can divide a video into consecutive frames, separate audio channels from the frames, and then extract audio data from each audio channel. However, the present invention is not limited thereto. The extracted audio can be represented as amplitude data with respect to time. The amplitude data can represent melody, meter, beat, and rhythm. When a strong sound is provided at a specific timing, the amplitude data can represent a large amplitude value at the specific timing. In a timing that provides a fast-beat sound, the amplitude data can represent a high frequency, and in a timing that provides a slow-beat sound, the amplitude data can represent a low frequency. The artificial intelligence model can generate haptic data (309) based on the amplitude data. The haptic data (309) illustrated in FIG. 3A is a diagram visualizing haptic data that provides vibration. For example, the type of image included in the haptic data (309) may indicate the type of vibration, the size of the image may indicate the intensity of the vibration, and the length of the image may indicate the duration of the vibration. For example, the larger the size of the image (e.g., the diameter of a circle), the stronger the vibration may be provided, and the longer the length of the image (e.g., the length along the x-axis), the longer the continuous vibration may be provided for a longer period of time. Haptic data (309) corresponding to the timeline of the video may be generated, and while the video is being played, vibration may be provided at a timing corresponding to the time stamp of the video.

[0074] According to one embodiment, the artificial intelligence model can generate haptic data (e.g., haptic data (212) of FIG. 3B) for guiding motion based on haptic data (307) based on motion data and haptic data (309) based on amplitude data. For example, the haptic data can be generated by synthesizing the haptic data (307) and the haptic data (309). The haptic data (212) for guiding motion can be based on the motion and audio of each of four objects according to the playing time of the video.

[0075] According to one embodiment, the artificial intelligence model can generate haptic data corresponding to each target object. For example, the artificial intelligence model can generate haptic data corresponding to at least one object. For example, even if the amplitude data extracted from the audio is the same, the motions of each of the four objects may be different, as illustrated in FIG. 3A. The artificial intelligence model can generate first haptic data corresponding to the first object (313) (e.g., first haptic data (321) of FIG. 3B), second haptic data corresponding to the second object (311) (e.g., second haptic data (322) of FIG. 3B), third haptic data corresponding to the third object (313) (e.g., third haptic data (323) of FIG. 3B), and fourth haptic data corresponding to the fourth object (314) (e.g., fourth haptic data (324) of FIG. 3B). However, the present invention is not limited thereto. As described above, the one or more objects (310) may be objects corresponding to actual people, or may be one or more objects (310) corresponding to virtual objects (e.g., characters, avatars, 3D models) or objects distinct from people (e.g., animals).

[0076] Referring to FIG. 3B, an external electronic device (210) may store video (211), haptic data (212), and / or reference data (213). According to one embodiment, the video (211), haptic data (212), and reference data (213) may be stored in the external electronic device (210), such as, but not limited to, a server, another electronic device (e.g., a smart phone, a computer, and / or a storage device). For example, the video (211), haptic data (212), and / or reference data (213) may be generated and stored by the electronic device (101), or video (211), haptic data (212), and / or reference data (213) generated by another electronic device may be provided to the electronic device (101). For example, the video (211) may be a video captured by a camera of the electronic device (101) (e.g., the camera (180) of FIG. 2), and the electronic device (101) may generate haptic data (212) and / or reference data (213) from the video.

[0077] According to one embodiment, the haptic data (212) may include haptic data corresponding to each of one or more objects (e.g., one or more objects (310) of FIG. 3A) included in the video (211). According to one embodiment, if four objects are included in the video, the haptic data (212) may include first haptic data (321), second haptic data (322), third haptic data (323), and fourth haptic data (324). However, the four objects are merely exemplary and are not limited thereto. For example, the one or more objects (310) may include only one object, in which case the haptic data (212) may include haptic data corresponding to the one object. For example, one or more objects (310) may include five or more objects, in which case the haptic data (212) may include haptic data corresponding to five or more objects. In addition to these, various embodiments may be possible.

[0078] According to one embodiment, the artificial intelligence model generates reference data (213), and the generated reference data (213) can be stored in the external electronic device (210). The reference data (213) can be referenced as data for providing a similarity between motions guided by the haptic data (212) and motions of the user actually measured. For example, if the haptic data (212) is configured to perform a specific motion at a specific timing, the reference data (213) can be configured as reference data for confirming whether the user accurately performed the specific motion at the specific timing. For example, if the user accurately performed the motion at the specific timing, a motion of the user substantially corresponding to the reference data (213) can be measured. The similarity can be calculated through the similarity between the reference data (213) and data measured by the user's actual motion. The reference data (213) can be used to provide feedback to one or more users. For example, the electronic device (101) can download reference data (213) along with video (211) and haptic data (212) from an external electronic device (210).

[0079] Below, the types of haptic data (212) generated by the artificial intelligence model are described.

[0080] Figure 4 is a diagram schematically illustrating haptic notifications according to the type of haptic data.

[0081] The x-axis illustrated in Fig. 4 represents time, and the y-axis represents the intensity of the haptic notification. The types of haptic data illustrated in Fig. 4 are merely exemplary and are not limited thereto.

[0082] According to one embodiment, haptic data generated by an artificial intelligence model may be categorized into haptic data for guiding beats and haptic data for guiding motions. Referring to FIG. 4, the haptic data may include first data, second data, third data, and fourth data.

[0083] According to one embodiment, the first data and the second data may be referred to as haptic data for guiding a beat. 401 of FIG. 4 is a diagrammatic image of a first haptic notification corresponding to the first data. The first data may be configured to provide a haptic notification at a constant intensity and at a constant cycle to guide a rhythm. The first data may be referred to as a rhythm beat type. For example, when there is not much movement, the first data may be used to guide the main beat of the music. For example, the first haptic notification may be provided while waiting in the rear row of one or more objects.

[0084] 403 of FIG. 4 is a diagrammatic image of a second haptic notification corresponding to the second data. The second data may be configured to provide a haptic notification with an intensity that gradually increases in a certain cycle to guide accent or stress. The second data may be referred to as an accent beat type. For example, the second data may be used to guide entry at an accurate timing before entering a personal part. Within the present disclosure, motions may be distinguished into a group part and an individual part. For example, a group part may be referred to as a part in which one or more objects have substantially the same motion or a part in which they have a unified motion. For example, an individual part may be referred to as a part in which at least one object among one or more objects has a different motion from the rest of the objects. For example, the second haptic notification may be provided immediately before entering a personal part among one or more objects.

[0085] According to one embodiment, the third data and the fourth data may be referenced as haptic data for guiding motion.

[0086] 405 of FIG. 4 is a schematic image of a third haptic notification corresponding to the third data. The third data may be configured to continuously provide haptic notifications of smoothly varying intensity to guide continuous movements. The third data may be referred to as a continuous movement type. The third haptic notification may be used to guide smooth and continuous motions (e.g., legato). For example, the third haptic notification may be provided to guide continuous steps or smoothly rotating motions.

[0087] 407 of FIG. 4 is a schematic image of a fourth haptic notification corresponding to the fourth data. The fourth data may be configured to provide irregular, strong haptic notifications to guide precise movements. The third data may be referred to as a precise movement type. The fourth haptic notification may be used to guide a discontinuous motion (e.g., staccato). For example, the fourth haptic notification may be provided to guide motions including precise steps, jumps, and large movements.

[0088] For example, haptic data can be stored in JSON format. Referring to [Table 1] below, the types of haptic notifications can be distinguished by curly brackets, and each haptic notification can be stored according to a timestamp. One or more external electronic devices (220) can be configured to provide haptic notifications in a continuous stream based on the haptic data in JSON format.

[0089] [{ / Output timestamp (ms) timestamp: 3150, / haptic_continuity: single, / haptic_type_id: 5, / haptic_intensity: 10},{timestamp: 3870, / haptic_continuity: continuous,haptic_type_id: 17, / haptic_duration: 3000,}]

[0090] Figure 5 illustrates an example of a haptic notification based on haptic data generated based on a video.

[0091] In one embodiment, the AI ​​model can generate haptic data corresponding to each of one or more objects included in the video. As described above, since the motions of the one or more objects are not always constant over time, the haptic data corresponding to each of the one or more objects may vary. Since the haptic data corresponding to each of the one or more objects differs, the haptic notifications corresponding to each of the one or more objects may be the same or different over time.

[0092] Referring to FIG. 5, when one or more objects include three objects (e.g., a first object, a second object, and a third object), haptic notifications corresponding to the haptic data may be different depending on the haptic data corresponding to each of the three objects. For example, the haptic notifications may be distinguished into three sections over time. For example, the three sections may include a first section (501), a second section (502), and a third section (503). The haptic notification (510) of FIG. 5 is an image that diagrams the haptic notification corresponding to the haptic data corresponding to the first object over time. The haptic notification (520) of FIG. 5 is an image that diagrams the haptic notification corresponding to the haptic data corresponding to the second object over time. The haptic notification (530) of FIG. 5 is an image that diagrams the haptic notification corresponding to the haptic data corresponding to the third object over time.

[0093] For example, the first section (501) may be a section in which the first object, the second object, and the third object all take the motion of a group part. In the first section (501), the first object, the second object, and the third object may all take the same motion or a unified motion. In order to guide the first object, the second object, and the third object to the same motion or a unified motion, in the first section (501), the haptic notification (510), the haptic notification (520), and the haptic notification (530) may be substantially the same. For example, the haptic notification (510), the haptic notification (520), and the haptic notification (530) may be a first haptic notification corresponding to first data for guiding a rhythm. For example, haptic data corresponding to a first object, haptic data corresponding to a second object, and haptic data corresponding to a third object may include first data in the first section (501).

[0094] For example, when the first object enters the personal part in the second section (502) following the first section (501), the haptic notification (510) may be partially different from the haptic notification (520) and the haptic notification (530). For example, the haptic data corresponding to the first object may include second data in which the intensity of the haptic notification increases to guide entry into the personal part before the end of the first section (501). The haptic notification (510) may provide a second haptic notification corresponding to the second data before the end of the first section (501).

[0095] For example, the second section (502) may be a section in which the first object takes the motion of an individual part, and the second object and the third object take the motion of a group part. In the second section (502), the first object may take a different motion from the second object and the third object. The haptic notification (510) may be different from the haptic notification (520) and the haptic notification (530). For example, in the second section (502), when the motion of the individual part of the first object is cut off, the haptic data corresponding to the first object may include the fourth data in the second section (502). In order to guide the same motion in the second section (502), the haptic data corresponding to the second object and the haptic data corresponding to the third object may include the first data in the second section (502). In the second section (502), the haptic notification (520) and the haptic notification (530) may be substantially the same.

[0096] For example, when a third object enters the personal part in the third section (503) following the second section (502), the haptic notification (530) may be partially different from the haptic notification (520). For example, the haptic data corresponding to the third object may include second data in which the intensity of the haptic notification increases to guide entry into the personal part before the end of the second section (502). The haptic notification (530) may provide a second haptic notification corresponding to the second data before the end of the second section (502).

[0097] For example, the third section (503) may be a section in which the third object takes the motion of an individual part, and the first object and the second object take the motion of a group part. In the third section (503), the third object may take a different motion from the first object and the second object. The haptic notification (530) may be different from the haptic notification (510) and the haptic notification (520). For example, if the individual part of the third object in the third section (503) has a smooth and continuous motion, the haptic data corresponding to the third object may include the third data in the third section (503). In order to guide the same motion in the third section (503), the haptic data corresponding to the first object and the haptic data corresponding to the second object may include the first data in the second section (502). In the third section (503), the haptic notification (510) and the haptic notification (520) may be substantially the same.

[0098] As described above, haptic data corresponding to each of one or more objects may be determined based on the motion of each of the one or more objects. According to one embodiment, the electronic device (101) may assign each of one or more objects to each of one or more external electronic devices (220). An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 2) can transmit haptic data corresponding to an assigned object to one or more external electronic devices (e.g., the one or more external electronic devices (220) of FIG. 2). The one or more external electronic devices (220) can receive haptic data corresponding to the assigned object. When a video is played, the video can be mirrored through a display device (e.g., the display device (230) of FIG. 2), and one or more users can make motions while each receiving a unique haptic notification. For example, one or more external electronic devices (220) carried by each of one or more users practicing a group dance can guide the motions of each of the one or more users through a haptic notification corresponding to the assigned object.

[0099] Below, the operation of an electronic device (101) and the operations of one or more external electronic devices (220) according to one embodiment are described.

[0100] Figure 6 is a flowchart illustrating the operation of an electronic device and one or more external electronic devices for guiding motion. Figure 7a illustrates a UI displayed on an external electronic device for joining a group. Figure 7b illustrates a UI displayed on an external electronic device for selecting an object.

[0101] According to one embodiment, instructions stored in a memory (e.g., memory (130) of FIG. 2) may cause the electronic device (101) to perform the operations described in FIG. 6 when individually or collectively executed by a processor (e.g., processor (120) of FIG. 2). In FIG. 6, one or more external electronic devices (220) are illustrated as including a first external electronic device (221) and a second external electronic device (222), but this is for convenience of explanation only and is not limited thereto.

[0102] In operation 601, the electronic device (101) may obtain video and haptic data from an external electronic device (210), according to one embodiment.

[0103] For example, when executing instructions stored in the memory (130), the processor (120) may download video (e.g., video (211) of FIG. 3B) and haptic data (e.g., haptic data (212) of FIG. 3B) from an external electronic device (210) through a communication circuit (e.g., wireless communication circuit (192) of FIG. 2). For example, the video may be a group dance video including one or more objects. The haptic data may be haptic data generated based on the extracted motions by extracting motions of each of one or more objects in the video using an artificial intelligence model. The haptic data may include haptic data corresponding to each of one or more objects.

[0104] According to one embodiment, the external electronic device (210) may include various electronic devices such as a server, a storage device (e.g., USB), a smart phone, a tablet, a computer, a laptop, and the like, and is not limited to a specific electronic device. The electronic device (101) may obtain video and haptic data from the external electronic device (210) via various wireless communication methods (e.g., Wi-Fi, 3G, LTE, 5G) or wired communication methods, and is not limited to the described embodiment.

[0105] In operation 602, the electronic device (101) may create a group of one or more external electronic devices (220) to guide the motions of each of one or more objects.

[0106] For example, when executing instructions stored in the memory (130), the processor (120) may create a group to guide the motions of each of one or more objects included in the video. When creating a group, the number of one or more external electronic devices (220) that can participate in the group may be set. For example, if there are four objects, the number of one or more external electronic devices (220) that can participate in the group may be set to four. However, the present invention is not limited thereto.

[0107] For example, when creating a group, the processor (120) may set the number of one or more external electronic devices (220) that can participate in the group. For example, when creating a group, the user of the electronic device (101) may set the number of participants by considering the number of one or more objects included in the video. For example, when practicing a group dance performed by five objects, if eight users participate, the number of objects that can be assigned may be less than the number of participants. In this case, since it is difficult to properly practice the group dance, it may be necessary to set the number of participants that can participate in the group. The number of participants may be set to the number of one or more external electronic devices (220) that can participate in the group.

[0108] In operation 603, the electronic device (101) may transmit information to be used for invitation request and object selection to one or more external electronic devices (220).

[0109] For example, when executing instructions stored in the memory (130), the processor (120) may transmit information used for invitation requests and object selection to one or more external electronic devices (220) via short-range wireless communication based on the creation of a group.

[0110] In one embodiment, the invitation request may be referred to as transmitting information for displaying a user interface (UI) (e.g., UI (710) of FIG. 7A) for joining a group to one or more external electronic devices (220). For example, the one or more external electronic devices (220) may display the UI (710) on a display of the one or more external electronic devices (220) (e.g., display (701) of FIG. 7A) based on receiving the information for displaying the UI (710).

[0111] Within the present disclosure, selection of an object may be referred to as a user selecting a specific object among one or more objects to provide guidance regarding the motion of said specific object. In one embodiment, the information used for selecting an object may be information for selecting one object among one or more objects included in a video. For example, if one or more objects include four objects, the information may include the name, position, and / or selectability of the object.

[0112] In operation 604, the first external electronic device (221) and the second external electronic device (222) may accept the invitation based on the invitation request received from the electronic device (101), and may select one of one or more objects based on information to be used for selection received from the electronic device (101). For example, the first external electronic device (221) may select the first object, and the second external electronic device (222) may select the second object. The first external electronic device (221) may transmit information regarding the invitation acceptance and information indicating the selection of the first object to the electronic device (101). The second external electronic device (222) may transmit information regarding the invitation acceptance and information indicating the selection of the second object to the electronic device (101).

[0113] Referring to FIG. 7A, an external electronic device (e.g., a first external electronic device (221)) may display a UI (710) based on receiving information for displaying a UI (710) for joining a group from an electronic device (101). For example, if the first external electronic device (221) is a smart watch, the UI (710) may be displayed on a display (701) of the smart watch. As illustrated in FIG. 7A, the UI (710) may include text (711), such as “Would you like to accept the group dance invitation?” The UI (710) may include a first icon (712) and / or a first text (713) that may receive a user input for accepting the invitation and joining the group. The UI (710) may include a second icon (714) and / or a second text (715) that can receive a user input to decline an invitation and not join a group. The first external electronic device (221) may transmit a signal indicating whether to participate, inputted through the first icon (712) and / or the first text (713) of the UI (710), to the electronic device (101). The electronic device (101) may add the external electronic device (e.g., the first external electronic device (221)) that transmitted the signal to the group based on receiving the signal indicating whether to participate, inputted through the UI (710), from one of the one or more external electronic devices (220).

[0114] Referring to FIG. 7B, an external electronic device (e.g., a first external electronic device (221)) may display a UI for selecting one of the objects on a display (701) based on receiving information used for selecting one of the objects from the electronic device (101). As illustrated in FIG. 7B, the UI (720) may include text (721) such as “Select an object.” For example, when the number of one or more external electronic devices (220) that can participate in the group created in operation 602 is set to 4, the UI (720) may include a first UI (722) for selecting a first object, a second UI (723) for selecting a second object, a third UI (724) for selecting a third object, and a fourth UI (725) for selecting a fourth object. If the third object is selected by another external electronic device (e.g., the second external electronic device (222)), the third UI (724) may be deactivated, and the first UI (722), the second UI (723), and the fourth UI (725) may be activated. For example, each of the first UI (722), the second UI (723), and the fourth UI (725) may be assigned a name to distinguish one or more objects. For example, if the video is a group dance video of dance singers, the first UI (722), the second UI (723), and the fourth UI (725) may be assigned the actual names of the dance singers.

[0115] Referring again to FIG. 6, at operation 605, the electronic device (101) may assign one or more objects to one or more external electronic devices (220).

[0116] For example, when executing instructions stored in the memory (130), the processor (120) may assign one or more objects to the first external electronic device (221) and the second external electronic device (222) based on information received from the first external electronic device (221) and the second external electronic device (222). For example, the processor (120) may assign a first object to the first external electronic device (221) and a second object to the second external electronic device (222).

[0117] In operation 606, the electronic device (101) may transmit haptic data to one or more external electronic devices (220).

[0118] For example, when executing instructions stored in the memory (130), the processor (120) may transmit haptic data corresponding to an object assigned to each of the one or more external electronic devices (220) to one or more external electronic devices (220). For example, the processor (120) may transmit haptic data corresponding to a first object to a first external electronic device (221) and may transmit haptic data corresponding to a second object to a second external electronic device (222).

[0119] In operation 607, the first external electronic device (221) may transmit a first user input to the electronic device (101).

[0120] For example, the first external electronic device (221) may be an external electronic device capable of controlling the play of a video among one or more external electronic devices (220). For example, the first external electronic device (221) may be, but is not limited to, a wearable device owned by a user of the electronic device (101). The first external electronic device (221) capable of controlling the play of a video may transmit a first user input to the electronic device (101) to initiate the play of the video. For example, when the first external electronic device (221) receives the first user input from the user, the first user input may be transmitted to the electronic device (101).

[0121] In operation 608, the electronic device (101) may transmit a signal to all of one or more external electronic devices (220) to initiate output of haptic data based on reception of a first user input for playing a video and outputting haptic data.

[0122] For example, when executing instructions stored in the memory (130), the processor (120) may broadcast the signal transmitted to one or more external electronic devices (220) based on reception of a first user input. The one or more external electronic devices (220) may output the received haptic data based on the signal broadcast from the electronic device (101). For example, while outputting the haptic data, the one or more external electronic devices (220) may control a vibration motor (e.g., a vibration motor (904) of FIG. 9B) based on the haptic data.

[0123] For example, the first external electronic device (221) can guide the motion of the first user wearing the first external electronic device (221) by providing a haptic notification based on first haptic data corresponding to the first object (e.g., the first haptic data (321) of FIG. 3B). Since the first haptic data is based on the first motion of the first object, the first external electronic device (221) can guide the motion of the first user to correspond to the first motion through the first haptic data. The second external electronic device (222) can guide the motion of the second user wearing the second external electronic device (222) by providing a haptic notification based on second haptic data corresponding to the second object (e.g., the second haptic data (322) of FIG. 3B). Since the second haptic data is based on the second motion of the second object, the second external electronic device (222) can guide the motion of the second user to correspond to the second motion through the second haptic data.

[0124] According to one embodiment, the first haptic data and the second haptic data may be synchronized with each other because they are output based on a signal broadcast by the electronic device (101). For example, the timestamp of the first haptic data and the timestamp of the second haptic data may be synchronized, thereby allowing one or more users to practice group dancing.

[0125] According to one embodiment, the electronic device (101) can be operatively connected to a display device (230). While the electronic device (101) plays a video, the electronic device (101) can mirror the video on the display device (e.g., the display device (230) of FIG. 2). The display device (230) can display the video mirrored by the electronic device (101). One or more users can watch the video displayed on the display device (230) and practice a group dance using haptic notifications provided from one or more external electronic devices (220). Since the one or more external electronic devices (220) do not provide the same haptic notification, but provide haptic notifications corresponding to selected objects, the electronic device (101) according to one embodiment can provide a group dance practice for harmonizing the motions of one or more users.

[0126] Figure 8 illustrates a UI displayed on an external electronic device.

[0127] According to one embodiment, among one or more external electronic devices (e.g., one or more external electronic devices (220) of FIG. 2), one external electronic device (e.g., a first external electronic device (221)) can control the play of a video. For example, the first external electronic device (221) can be an external electronic device capable of providing a first user input. The first user input capable of controlling the play of the video can be transmitted to an electronic device (e.g., an electronic device (101) of FIG. 2) via the first external electronic device (221).

[0128] Referring to FIG. 8, the first external electronic device (221) may control the play of a video. For example, the play control of the video may include at least one of pausing the video, resuming a paused video, stopping the video, moving the time stamp of the video, adjusting the play speed of the video, or repeating a specified section of the video. For example, moving the time stamp of the video may include jumping to skip a specific part of the video, and / or skipping to skip an intro or introduction video.

[0129] According to one embodiment, the first external electronic device (221) may display a UI (800) for controlling a video on the display (701). The UI (800) may include a first UI (810) for pausing a video, a second UI (820) for resuming a video, a third UI (830) for stopping a video, a fourth UI (840) for moving a time stamp of a video, a fifth UI (850) for adjusting the speed of a video, and / or a sixth UI (860) for repeating a specified section of a video. However, the present invention is not limited thereto. A second user input for the first UI (810), the second UI (820), the third UI (830), the fourth UI (840), the fifth UI (850), and / or the sixth UI (860) may be transmitted to the electronic device (101). The electronic device (101) may control video play based on receipt of a second user input. For example, the electronic device (101) may pause a video being played based on receipt of a second user input for the first UI (810). For example, the electronic device (101) may play a specified section of the video repeatedly based on receipt of a second user input for the sixth UI (860). According to one embodiment, when the first external electronic device (221) is a smart watch, the first external electronic device (221) may include a bezel (801) forming an edge. The bezel (801) may be rotatable with respect to the display (701). For example, either a time stamp movement of the video or a play speed adjustment of the video may be controlled through the rotation of the bezel (801). For example, if the bezel (801) is rotated clockwise, the time stamp of the video may move backward or the playback speed of the video may increase.For example, when the bezel (801) is rotated counterclockwise, the time stamp of the video may advance or the playback speed of the video may decrease. The second user input for controlling the playback of the video may include rotating the bezel (801). The second user input for the bezel (801) may be transmitted to the electronic device (101). The electronic device (101) may control the playback of the video based on receiving the second user input for the bezel (801).

[0130] According to one embodiment, the first external electronic device (221) capable of controlling the playback of a video may be referred to as a leader device or a host device from the perspective of leading the exercise by controlling the playback of the video. The remaining external electronic devices, excluding the first external electronic device (221), may be referred to as follower devices or guest devices.

[0131] Figure 9a illustrates a state in which an electronic device, according to one embodiment, mirrors a video onto a display device. Figure 9b is a block diagram of an exemplary external electronic device. Figure 9c illustrates an example of data acquired through a sensor of the external electronic device. Figure 9d illustrates a process in which the external electronic device acquires data related to a user's motion.

[0132] Referring to FIG. 9A, an electronic device (101) according to one embodiment can mirror a video onto a display device (230). The display device (230) can provide a large display screen, making it easy for one or more users to view the video and follow the motions.

[0133] For example, the electronic device (101) may initiate play of a video based on receipt of a first user input. When play of the video is initiated, the electronic device (101) may broadcast a signal to initiate output of haptic data, such that output of haptic data transmitted to each of one or more external electronic devices (220) is initiated. The one or more external electronic devices (220) may output haptic notifications synchronized by the signal. The timestamps of the synchronized haptic notifications may correspond to the timestamps of the video in which the haptic notifications transmitted to each of the one or more external electronic devices (220) are synchronized. For example, the timestamps of the first haptic data and the second haptic data may be synchronized with the timestamps of the video, such that one or more users may practice a group dance.

[0134] An electronic device (101) according to one embodiment may receive similarity data between data related to users' motions and haptic data from one or more external electronic devices (220).

[0135] For example, when transmitting haptic data to one or more external electronic devices (220), the electronic device (101) may transmit reference data (e.g., reference data (213) of FIG. 3B) together with the haptic data (e.g., haptic data (212) of FIG. 3B). The one or more external electronic devices (220) may receive the reference data from the electronic device (101). As described above, the reference data may be referenced as data for providing a similarity between motions guided by the haptic data and data related to actual measured motions of the user.

[0136] Each of the one or more external electronic devices (220) can measure the motions of users and generate data related to the motions of users. The one or more external electronic devices (220) can calculate the similarity between the reference data and the data related to the measured motions. Referring to FIG. 9B, the one or more external electronic devices (220) (e.g., the first external electronic device (221)) can include a sensor (e.g., sensor (905) of FIG. 9B) for measuring the motions of users. For example, the sensor can include, but is not limited to, at least one of an acceleration sensor and a gyro sensor.

[0137] Referring to FIG. 9b, the first external electronic device (221) may include a processor (901), a memory (902), a communication circuit (903), a vibration motor (904), a sensor (905), and / or a display (701).

[0138] The processor (901) can control the overall operation of the first external electronic device (221). For example, the processor (901) can transmit data to the electronic device (101) or receive data from the electronic device (101) by controlling the communication circuit (903). Although not shown, the first external electronic device (221) can include an antenna for transmitting and / or receiving data. For example, the processor (901) can store first haptic data and / or first reference data received from the electronic device (101) in the memory (902). The memory (902) can store instructions that, when individually or collectively executed by the processor (901), cause operations of the first external electronic device (221). For example, the processor (901) can control the operation of the vibration motor (904) based on the first haptic data. The vibration motor (904) may be configured to provide a first haptic notification based on the first haptic data. The vibration motor (904) may also provide the haptic notification to indicate the occurrence of an event associated with the first external electronic device (221).

[0139] For example, the sensor (905) may be used to measure the motion of a first user wearing the first external electronic device (221). For example, the sensor (905) may include, but is not limited to, an acceleration sensor (906) and / or a gyro sensor (907). For example, if the first external electronic device (221) is a smartwatch worn on the wrist of the first user, the position and / or direction of the wrist may change as the wrist moves according to the motion of the first user. Based on the change in the position and / or direction of the wrist, the sensor (905) may generate data representing the motion of the first user. The acceleration sensor (906) may generate data representing the acceleration of the first external electronic device (221). The gyro sensor (907) may generate data representing the angular velocity of the first external electronic device (221). The processor (901) can generate data related to the motion of the first user based on the above data.

[0140] FIG. 9C illustrates a graph (900a) representing data regarding velocity acquired by a sensor (905) when a user makes a specific motion, and a graph (900b) representing data regarding acceleration. For example, if the first external electronic device (221) is a smartwatch, when the user makes a specific motion while wearing the smartwatch, the position and posture of the smartwatch may change. Within a section (900c) where the position and posture of the smartwatch change, the graph (900a) representing data regarding velocity, and within the section (900d) representing data regarding acceleration, may represent waveforms based on the user's motion. Data according to the waveforms are generated by the sensor (905), and the sensor (905) may provide the data to the processor (901). The processor (901) may generate data related to the motion of the first user based on the data. The graphs (900a, 900b) shown in FIG. 9c are merely exemplary and are not limited thereto.

[0141] 951 of Fig. 9d schematically illustrates a first haptic notification. 952 of Fig. 9d schematically illustrates a first reference motion based on the first haptic data. 953 of Fig. 9d schematically illustrates an actual motion of the first user. 954 of Fig. 9d is a graph of a velocity indicated by the actual motion of the first user. 955 of Fig. 9d is a graph of an acceleration indicated by the actual motion of the first user. 956 of Fig. 9d indicates whether the first reference motion and the actual motion of the first user correspond. 956 of Fig. 9d may be represented as O if the first reference motion and the actual motion of the first user correspond, and may be represented as X if the first reference motion and the actual motion of the first user do not correspond.

[0142] Referring to 951 of FIG. 9D, the first haptic notification provided from the vibration motor based on the first haptic data may include a first type for guiding a moving pose and a second type for guiding a static pose. For example, within a section (957) in which the first haptic data of the first type for guiding a moving pose is provided, the user's moving pose may be guided. Within the section (957), a motion requiring the user's movement may be guided. For example, within a section (958) in which the first haptic data of the second type for guiding a static pose is provided, the user's static pose may be guided. Within the section (958), a motion that does not require the user's movement may be guided.

[0143] Referring to 952 of FIG. 9D , within a section (957) in which the first type of first haptic data is provided, the reference motion of the first user may be a moving motion, and within a section (958) in which the second type of first haptic data is provided, the reference motion of the first user may be a stationary motion. Referring to 953, 954, and 955 of FIG. 9D , the speed and acceleration measured by the motion of the first user may be based on the actual motion of the first user. When the first user makes a moving motion, the speed may be higher than the reference speed, and the acceleration may be higher than the reference acceleration. When the first user makes a moving motion, data related to the motion of the first user acquired through a sensor (e.g., sensor (905) of FIG. 9B ) may indicate data higher than the reference speed and reference acceleration. When the first user makes a stop motion, the speed may be lower than the reference speed, and the acceleration may be lower than the reference acceleration. When the first user makes a stop motion, data related to the first user's motion acquired through the sensor may show data lower than the reference speed and reference acceleration.

[0144] At a timing when the first reference motion is a moving motion, if the data related to the motion of the first user indicates data higher than the reference speed and reference acceleration, the processor (120) (e.g., the processor (901) of FIG. 9B) may determine that the first reference motion and the actual motion of the first user correspond. At a timing when the first reference motion is a stationary motion, if the data related to the motion of the first user indicates data lower than the reference speed and reference acceleration, the processor (901) may determine that the first reference motion and the actual motion of the first user correspond. The processor (901) may calculate a similarity between the actual motion of the first user and the first reference motion. In the example illustrated in FIG. 9D, the similarity may be calculated as 10 / 12 × 100 = 83.3%.

[0145] Although the above descriptions have been described with reference to the first external electronic device (221) as an example, the descriptions of FIG. 9b may be applied to all one or more external electronic devices (220).

[0146] Referring back to FIG. 9A, the first external electronic device (221) may receive first reference data (e.g., first reference data (331) of FIG. 3B) based on first haptic data from the electronic device (101). The first external electronic device (221) may measure a motion of a first user wearing the first external electronic device (221). The first external electronic device (221) may generate data related to the motion of the first user using a sensor (e.g., sensor (905) of FIG. 9B). The first external electronic device (221) may compare the data related to the motion of the first user with reference data for the first haptic data, and calculate a first similarity between the data related to the motion of the first user and the reference data for the first haptic data. The first similarity may be expressed as a percentage between 0% and 100%. For example, if the data related to the motion of the first user generated by measuring the motion of the first user fully corresponds to the first reference data, the first similarity may be 100%. If the data related to the motion of the first user does not fully correspond to the first reference data, the first similarity may be 0%. The above descriptions may be equally applied to the second external electronic device (222) and the third external electronic device (223). The second external electronic device (222) may calculate the second similarity between the data related to the motion of the second user and the second reference data (e.g., the second reference data (332) of FIG. 3B). The third external electronic device (223) may calculate the third similarity between the data related to the motion of the third user and the third reference data (e.g., the third reference data (333) of FIG. 3B).

[0147] According to one embodiment, each of one or more external electronic devices (220) may transmit similarity data representing a calculated similarity to the electronic device (101). A first external electronic device (221) may transmit first similarity data representing a first similarity to the electronic device (101). A second external electronic device (222) may transmit second similarity data representing a second similarity to the electronic device (101). A third external electronic device (223) may transmit third similarity data representing a third similarity to the electronic device (101).

[0148] According to one embodiment, the electronic device (101) may calculate a ranking for each of the motions of one or more users based on similarity data received from one or more external electronic devices (220). For example, the ranking may be calculated in order of high similarity. For example, if the first similarity data is about 90%, the second similarity data is about 80%, and the third similarity data is about 70%, the electronic device (101) may calculate the ranking in the order of the first object, the second object, and the third object.

[0149] According to one embodiment, the electronic device (101) may display a calculated ranking on the video while the video is playing. As illustrated in FIG. 9A, the ranking may be displayed on the video mirrored on the display device (230). For example, the ranking may include, but is not limited to, text (910) and / or an image (920) indicating a similarity value. The similarity value may be converted into a score and displayed. For example, if the similarity value is 90%, the similarity value may be converted into 90 points and displayed. While the video is playing, the electronic device (101) may receive similarity data in real time. The electronic device (101) may calculate a real-time ranking based on the similarity data received in real time and provide the calculated ranking through the display device (230). One or more users practicing group dance may check the real-time ranking through the ranking displayed on the video mirrored on the display device (230).

[0150] An electronic device (101) according to one embodiment can provide real-time evaluations of the motions of one or more users participating in a group dance practice while playing a video.

[0151] FIG. 10 is a flowchart illustrating an operation of an electronic device according to one embodiment of the present invention to calculate a ranking based on similarity data and display the calculated ranking on a video.

[0152] According to one embodiment, instructions stored in a memory (e.g., memory (130) of FIG. 2) may, when individually or collectively executed by a processor (e.g., processor (120) of FIG. 2), cause an electronic device (e.g., electronic device (101) of FIG. 2) to perform the operations described in FIG. 10.

[0153] Referring to FIG. 10, in operation 1001, the electronic device (101) may receive one or more similarity data from one or more external electronic devices (e.g., one or more external electronic devices (220) of FIG. 2).

[0154] For example, a first external electronic device (e.g., the first external electronic device (221) of FIG. 2) can generate data related to the motion of the first user by measuring the actual motion of the first user. The first external electronic device (221) can calculate a similarity between the data related to the motion of the first user and first reference data (e.g., the first reference data (331) of FIG. 3B) and transmit first similarity data representing the calculated similarity to the electronic device (101). The electronic device (101) can receive the first similarity data from the first external electronic device (221), the second similarity data from a second external electronic device (e.g., the second external electronic device (222) of FIG. 2), and the third similarity data from a third external electronic device (e.g., the third external electronic device (223) of FIG. 2).

[0155] In operation 1003, the electronic device (101) may calculate a ranking based on the received similarity data.

[0156] For example, the electronic device (101) can calculate the ranking in the order of high similarity. For example, if the first similarity data is about 90%, the second similarity data is about 80%, and the third similarity data is about 70%, the electronic device (101) can calculate the ranking in the order of the first object, the second object, and the third object.

[0157] In operation 1005, the electronic device (101) can display the ranking on the video.

[0158] For example, the electronic device (101) may display similarity data and rankings on the video while playing the video. The rankings may be displayed on the video mirrored on the display device (230). As described above with reference to FIG. 9A, one or more users may check the rankings displayed in real time while the video is playing.

[0159] In operation 1007, the electronic device (101) may provide feedback to each of one or more external electronic devices (220).

[0160] For example, the electronic device (101) can calculate an overall score for one or more users based on similarity data received from one or more external electronic devices (220). The overall score may be referred to as an average of the similarity data, but is not limited thereto. For example, if the first similarity is 90%, the second similarity is 80%, and the third similarity is 70%, the overall score may be calculated as 80 points. By providing the overall score, the electronic device (101) can provide feedback on the overall completion of the group dance.

[0161] Figure 11a is a flowchart illustrating an operation of an electronic device providing feedback according to one embodiment. Figure 11b schematically illustrates a similarity for a timeline.

[0162] According to one embodiment, instructions stored in a memory (e.g., memory (130) of FIG. 2) may, when individually or collectively executed by a processor (e.g., processor (120) of FIG. 2), cause an electronic device (e.g., electronic device (101) of FIG. 2) to perform the operations described in FIG. 11A.

[0163] An electronic device (101) according to one embodiment can provide feedback to one or more users regarding areas requiring practice. For example, the electronic device (101) can identify a section of a video (e.g., section 1124 of FIG. 11B ) with low overall similarity by analyzing similarity data received from one or more external electronic devices (220). The electronic device (101) can provide focused practice for the section by providing information for outputting haptic data corresponding to the section to one or more external electronic devices (220).

[0164] Referring to FIG. 11A, in operation 1101, the electronic device (101) can identify a low similarity section within the haptic data based on similarity data received from one or more external electronic devices (220).

[0165] For example, a section with low similarity may be referred to as a section having a similarity lower than or equal to a reference value within the haptic data. Referring to FIG. 11B, 1111 of FIG. 11B represents a first similarity based on first similarity data received from a first external electronic device (e.g., the first external electronic device (221) of FIG. 2) with respect to a timeline of a video, and 1112 of FIG. 11B represents a second similarity based on second similarity data received from a second external electronic device (e.g., the second external electronic device (222) of FIG. 2) with respect to a timeline of a video. The first similarity and the second similarity may include a section (1121) having 100% similarity, a section (1122) having a similarity lower than 100% but exceeding a reference value, and a section (1123) having a similarity lower than or equal to the reference value. The electronic device (101) can identify a section (1123) having a similarity lower than a reference value within the first similarity data and the second similarity data.

[0166] According to one embodiment, the electronic device (101) can identify, within the haptic data, a low-similarity section in which one or more users have a low overall similarity. 1113 of FIG. 11B represents the similarity for all one or more users. The electronic device (101) can identify a section (1124) having a low overall similarity for a timeline in which sections (1123) having a similarity below a threshold value overlap.

[0167] Referring again to FIG. 11A, at operation 1103, the electronic device (101) may provide information to one or more external electronic devices (220) to output haptic data corresponding to a low similarity section.

[0168] According to one embodiment, the electronic device (101) may transmit information for outputting haptic data corresponding to a low-similarity section (e.g., section (1124) of FIG. 11B) identified in operation 1101 within the haptic data to one or more external electronic devices (220). The one or more external electronic devices (220) may receive the information for outputting haptic data corresponding to the low-similarity section and provide haptic data of the section based on the information.

[0169] For example, one or more external electronic devices (220) can select a timestamp corresponding to the section within the haptic data. Users of one or more external electronic devices (220) can receive haptic data corresponding to the section by selecting the timestamp corresponding to the section. One or more users can practice motions of a section with low similarity through the haptic data corresponding to the section. For example, a first external electronic device (221) capable of controlling the play of a video can enable one or more users to repeatedly practice the section by selecting a timestamp corresponding to a section with low similarity.

[0170] In one embodiment, when the electronic device (101) identifies a low-similarity section (1124), the electronic device (101) may identify the section so as to guide natural motion. For example, the electronic device (101) may identify the start or end point of the section as a section where motion naturally begins or ends.

[0171] In one embodiment, the electronic device (101) can provide haptic data to guide motion for a hearing-impaired person. Because hearing-impaired people cannot hear sounds, they may find it difficult to learn or practice motion. For example, a hearing-impaired person learning dance can practice by feeling vibrations from a speaker with their hands. However, because their hands cannot be used freely, they may have difficulty learning group dance.

[0172] In one embodiment, motions can be guided by haptic notifications provided by one or more external electronic devices (220) worn by one or more users, allowing even hearing-impaired individuals to learn and practice group dance. Synchronized haptic notifications for group dance enable hearing-impaired individuals to learn and practice group dance.

[0173] An electronic device (101) is provided. The electronic device (101) may include a display (160). The electronic device (101) may include a camera (180). The electronic device (101) may include a wireless communication circuit (192) for wirelessly connecting with external electronic devices (220). The electronic device (101) may include a processor (120). The electronic device (101) may include a memory (130) for storing instructions. The instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to acquire a video through the camera (180), acquire haptic data related to the motion of an object included in the video, create a group of the external electronic devices (220) connected through the wireless communication circuit (192), assign one of the objects to each of the external electronic devices (220), transmit haptic data corresponding to the assigned one object to each of the external electronic devices (220), and, based on receipt of a first user input for playing the video and outputting the haptic data, transmit a signal to the external electronic devices (220) to initiate output of the haptic data corresponding to the one object.

[0174] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to broadcast the signal transmitted to each of the external electronic devices (220) based on receipt of the first user input, such that each of the external electronic devices (220) outputs a synchronized haptic notification.

[0175] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to set a number of the external electronic devices (220) participating in the group when creating the group, and, based on the creation of the group, transmit, via short-range wireless communication, information for displaying a user interface (UI) for joining the group to the external electronic devices (220) and information used for selecting one object from among the objects.

[0176] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to add each of the external electronic devices (220) that transmitted the signal to the group based on receiving a signal indicating participation input through the UI from each of the external electronic devices (220).

[0177] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to receive, from the external electronic devices (220), a signal indicating selection of one object from among the objects, and, based on reception of the signal, assign the one object to the external electronic device that transmitted the signal.

[0178] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to transmit, to a first external electronic device (221) to which a first object is assigned, first haptic data corresponding to a first motion of the first object, among the objects, and to transmit, to a second external electronic device (222) to which a second object is assigned, second haptic data corresponding to a second motion of the second object, among the objects. The first haptic data may be different from the second haptic data.

[0179] According to an exemplary embodiment, the first user input may be a user input provided from a first external electronic device (221) capable of controlling play of the video among the external electronic devices (220).

[0180] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to play the video based on receiving the first user input, and to control the playing of the video based on receiving, from the first external electronic device (221), a second user input for controlling playing of the video while the video is playing.

[0181] According to an exemplary embodiment, controlling the play of the video may include at least one of pausing the video, resuming the paused video, stopping play of the video, moving a timestamp of the video, adjusting a play speed of the video, or repeating a specified section.

[0182] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to receive, from the external electronic devices (220), similarity data indicating a similarity between data related to motions of users wearing the external electronic devices (220) and reference data set based on the haptic data while the video is being played, and to calculate a ranking for the motions of the users based on the received similarity data.

[0183] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to obtain the reference data from another external electronic device (210) and transmit the reference data when transmitting the haptic data to the external electronic devices (220).

[0184] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to identify, based on the similarity data received from the external electronic devices (220), a section within the haptic data having a similarity less than or equal to a reference value, and to provide information to the external electronic devices (220) for outputting haptic data corresponding to the section within the video.

[0185] According to an exemplary embodiment, the instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to mirror the video to a display device (230) while the video is being played, and to display the ranking on the video.

[0186] According to an exemplary embodiment, the haptic data may be haptic data generated by extracting the motion of the object in the video using an artificial intelligence model and based on the extracted motion.

[0187] A method of an electronic device (101) is provided. The method may include an operation of obtaining video and haptic data. The method may include an operation of creating a group of one or more external electronic devices (220) for guiding motion of one or more objects included in the video. The method may include an operation of assigning one of the one or more objects to each of the one or more external electronic devices (220). The method may include an operation of transmitting, to each of the one or more external electronic devices (220), the haptic data corresponding to the one object assigned to each of the one or more external electronic devices (220). The method may include an operation of transmitting, to each of the one or more external electronic devices (220), a signal for initiating output of the haptic data corresponding to the one object based on reception of a first user input for playing the video and outputting the haptic data.

[0188] According to an exemplary embodiment, the method may further include broadcasting the signal transmitted to the one or more external electronic devices (220) based on receipt of the first user input, such that each of the one or more external electronic devices (220) outputs a synchronized haptic notification.

[0189] According to an exemplary embodiment, the method may further include an operation of setting the number of the one or more external electronic devices (220) participating in the group when creating the group. The method may further include an operation of transmitting, based on the creation of the group, information for displaying a UI (user interface) for joining the group to the one or more external electronic devices (220) and information used for selecting the one object from among the one or more objects via short-range wireless communication.

[0190] According to an exemplary embodiment, the method may further include an operation of adding the one or more external electronic devices (220) that transmitted the signal to the group based on receiving a signal indicating participation input through the UI from the one or more external electronic devices (220).

[0191] According to an exemplary embodiment, the method may further include mirroring the video to a display device (230) while the video is being played.

[0192] An electronic device (101) is provided. The electronic device (101) may include a processor (120). The electronic device (101) may include a memory (130) that stores instructions. The instructions, when individually or collectively executed by the processor (120), may cause the electronic device (101) to: obtain video and haptic data; create a group of external electronic devices (220) for guiding motion of one or more objects included in the video; assign one of the one or more objects to the one or more external electronic devices (220); transmit, to the external electronic device (220), the haptic data corresponding to the one object assigned to the external electronic device (220); and, based on receipt of a first user input for playing the video and outputting the haptic data, transmit, to the external electronic device (220), a signal for initiating output of the haptic data corresponding to the one object.

[0193] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when individually or collectively executed by a processor (120) of an electronic device (101), can cause the electronic device (101) to obtain video and haptic data from an external electronic device (210). The instructions, when individually or collectively executed by the processor (120) of the electronic device (101), can cause the electronic device (101) to create a group of a plurality of external electronic devices (220) to guide the motion of each of a plurality of objects included in the video. The instructions, when individually or collectively executed by the processor (120) of the electronic device (101), can cause the electronic device (101) to assign each of the plurality of objects to each of the plurality of external electronic devices (220). The instructions, when individually or collectively executed by the processor (120) of the electronic device (101), may cause the electronic device (101) to transmit, to the plurality of external electronic devices (220), the haptic data corresponding to an object assigned to each of the plurality of external electronic devices (220). The instructions, when individually or collectively executed by the processor (120) of the electronic device (101), may cause the electronic device (101) to transmit, to the plurality of external electronic devices (220), a signal for initiating output of the haptic data based on reception of a first user input for playing the video and outputting the haptic data.

[0194] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0195] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0196] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0197] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (120) (e.g., the processor (120)) of a machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0198] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory (130) of a manufacturer's server, an application store's server, or a relay server.

[0199] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; camera; A wireless communication circuit configured to communicate with external electronic devices; At least one processor comprising a processing circuit; and A memory comprising one or more storage media storing instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Acquire video through the above camera, Obtaining haptic data related to the motions of objects included in the above video, Creating a group of external electronic devices connected via the above wireless communication circuit, Assign one of the objects to each of the above external electronic devices, Transmitting haptic data corresponding to the assigned one object to each of the above external electronic devices, Based on the reception of a first user input for playing the video and outputting the haptic data, causing the external electronic devices to transmit a signal to initiate outputting the haptic data corresponding to the one object. Electronic devices.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Causing each of said external electronic devices to broadcast said signal transmitted to each of said external electronic devices based on receipt of said first user input, so as to cause each of said external electronic devices to output a synchronized haptic notification. Electronic devices.

3. In paragraph 1 or 2, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: When creating the above group, set the number of external electronic devices participating in the group, Based on the creation of the group, causing the external electronic devices to transmit information for displaying a UI (user interface) for joining the group and information used for selecting one object among the objects via short-range wireless communication. Electronic devices.

4. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on receiving a signal indicating whether to participate or not input through the UI from each of the external electronic devices, causing each of the external electronic devices that transmitted the signal to be added to the group. Electronic devices.

5. In paragraph 3 or 4, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Receive a signal from said external electronic devices indicating selection of said one object among said objects, Based on the reception of said signal, causing said one object to be assigned to an external electronic device that transmitted said signal. Electronic devices.

6. In paragraph 5, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Among the above objects, a first object is assigned to a first external electronic device, and first haptic data corresponding to a first motion of the first object is transmitted, Among the above objects, a second object causes a second external electronic device to transmit second haptic data corresponding to a second motion of the second object, The above first haptic data is, Different from the above second haptic data, Electronic devices.

7. In any one of paragraphs 1 to 6, The above first user input is, Among the above external electronic devices, a user input provided from a first external electronic device capable of controlling play of the video, Electronic devices.

8. In paragraph 7, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on receiving the first user input, playing the video, While the video is playing, causing the playing of the video to be controlled based on the reception of a second user input for controlling playing of the video from the first external electronic device. Electronic devices.

9. In paragraph 8, Controls the playback of the above video are: At least one of pausing the video, resuming the paused video, stopping playback of the video, moving the timestamp of the video, adjusting the playback speed of the video, or repeating a specified section. Electronic devices.

10. In any one of paragraphs 1 to 9, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: While the above video is playing, similarity data is received from the external electronic devices, indicating the similarity between data related to motions of users wearing the external electronic devices and reference data set based on the haptic data, Based on the received similarity data, causing a ranking of the motions of the users to be calculated. Electronic devices.

11. In paragraph 10, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining the above reference data from another external electronic device, When transmitting the above haptic data to the external electronic devices, causing the reference data to be transmitted, Electronic devices.

12. In clause 10 or 11, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the similarity data received from the external electronic devices, a section having a similarity lower than a reference value is identified within the haptic data, In the above video, information for outputting haptic data corresponding to the above section is provided to the external electronic devices, causing the same. Electronic devices.

13. In any one of paragraphs 10 to 12, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: While the above video is playing, mirror the above video to the display device, Causing the ranking to be displayed on the above video, Electronic devices.

14. In any one of paragraphs 1 to 13, The above haptic data is, Using an artificial intelligence model, the motion of the object in the video is extracted, and haptic data is generated based on the extracted motion. Electronic devices.

15. In the method of an electronic device, Actions to acquire video and haptic data; An act of generating a group of one or more external electronic devices for guiding the motion of one or more objects included within said video; An action of assigning one of said one or more objects to each of said one or more external electronic devices; An operation of transmitting, to said one or more external electronic devices, said haptic data corresponding to said one object assigned to each of said one or more external electronic devices; and An operation of transmitting a signal to each of the one or more external electronic devices to initiate output of the haptic data corresponding to the one object, based on reception of a first user input for playing the video and outputting the haptic data. method.

Images