A system and a method for collaborative content authoring using object control and interaction in virtual environments based on hand tracking

Abstract

The present invention relates to a system for authoring content in a virtual environment including virtual objects, and is a system and method for implementing an authoring tool for controlling interactions and virtual objects in a virtual environment and performing real-time collaborative authoring activities using hand tracking based on camera images in real time in a PC environment.

Description

Technology Field

[0001] The present invention relates to a system for authoring content in a virtual environment including virtual objects, and is a system and method for implementing an authoring tool for controlling interactions and virtual objects in a virtual environment and performing real-time collaborative authoring activities using hand tracking based on camera images in real time in a PC environment. Background Technology

[0003] A content authoring system refers to a software environment that supports users in creating, editing, and sharing digital content (e.g., images, videos, audio, etc.). Generally, a content authoring system provides a user interface (UI) and an intuitive and easily accessible interface to enable users to intuitively create and modify content.

[0004] The prior art disclosed in the prior art includes a real-space-based content authoring device, a content authoring system, and a method. The prior art includes a real-space-based content authoring system comprising: an augmented reality server for providing augmented reality information including indoor and outdoor location information and simulation data of an augmented reality application user terminal necessary for the simulation of content being authored and the operation of a distributed augmented reality application; a virtual reality server for providing real-space-based virtual reality information including geometric information and image information representing the real-space where the content is to be applied; and a content authoring device for providing a real-space-based virtual reality authoring environment when creating content based on the augmented reality information and virtual reality information, and for providing an augmented reality application and a virtual reality application based on the generated content.

[0005] While it is true that content authoring environments are improving in a direction that enhances productivity, there is nevertheless a need for collaborative authoring systems to move beyond passive methods and provide a more active and innovative authoring environment. Some existing virtual content authoring systems allow users to manipulate virtual objects using physical input devices such as mice and keyboards, but they lack natural interaction methods. Furthermore, to enable multiple users to perform collaborative work in real-time across different workspaces, a function is required to maintain synchronized results by sharing work screens or states.

[0006] Accordingly, the inventor of the present invention conceived a plan to implement a system that allows multiple users to collaborate on creating virtual content in real time through a natural interface and maintain it in a synchronized state by combining hand tracking technology and WebRTC-based real-time communication. Prior art literature

[0008] Prior Art: Republic of Korea Registered Patent 10-1627169 (Registered on May 30, 2016) The problem to be solved

[0009] The objective of the present invention is to provide an authoring tool system that enhances work efficiency by utilizing hand tracking through a camera in a virtual environment, and by using interaction and object control and a simplified interface module.

[0010] Another objective of the present invention is to provide an authoring tool and an interface method for rapidly sharing visual synchronization of high-volume, high-complexity data using WebRTC-based work screen streaming, and for increasing productivity through rapid decision-making via voice chat.

[0011] The system of the present invention aims to enable intuitive interaction by connecting the physical movements of users with virtual objects in real time, particularly in environments supporting collaboration among multiple users, and to minimize network load and provide a stable collaboration environment by utilizing P2P technology. means of solving the problem

[0013] The cooperative authoring system of the present invention for solving the technical problems described above is,

[0014] It is a system that includes a virtual object control module capable of detecting collisions between a virtual hand and a virtual object and performing movement, rotation, and deformation of the object, thereby detecting and processing physical collisions that may occur during user interactions and enabling intuitive control of objects using only hand gestures.

[0015] In addition, it is implemented to enable multiple users to simultaneously share and synchronize work results in real time through WebRTC-based P2P communication. This communication structure contributes to enhancing the performance of the real-time collaboration environment by reducing the load on relay servers and efficiently managing each user's network conditions.

[0016] The present invention can more accurately determine the gesture state by using a physical collision detection function of the fingertips or palms, thereby recognizing various complex gestures and enriching interaction with virtual objects.

[0017] The present invention provides a function capable of maintaining task concurrency and processing data exchange in real time by introducing a network event transmission and reception module using a Node.js server and Socket.io. This ensures a smooth collaborative work process among multiple users and supports effective cooperative work by minimizing conflicts that may occur during operation. Effects of the invention

[0019] The present invention utilizes existing technologies to enable creative activities through the control and interaction of virtual objects using the application's authoring tool via hand tracking with multiple participants, and enables increased productivity through parallel processing and simplified interface modules in a cloud environment.

[0020] The present invention can drastically reduce the latency caused by real-time communication data synchronization by streaming and sharing images that visualize trajectory data over time, such as the creation and placement of multiple objects between peers and the position information of 41 joints of both hands.

[0021] The hand-tracking-based collaborative authoring tool system according to the present invention has a significant advantage in that it can smoothly perform interaction with virtual objects by tracking the user's hand in real time and recognizing gestures. In particular, natural interaction with virtual objects is possible through a hand-tracking module capable of precisely tracking the user's hand movements, which provides an intuitive user experience. Through this, it is expected that the productivity of the worker will be increased and the immersion in the work will be improved.

[0023] Additionally, it features the ability to visualize and share real-time tasks using a P2P-based WebRTC module, enabling multiple users to collaborate more efficiently, and to synchronize them via a virtual reality server. This allows for fast and stable task execution while minimizing network latency and synchronization issues in collaborative environments. Furthermore, by utilizing a method that synchronizes the work of multiple users through augmented reality and virtual reality servers, stable data transmission and simultaneous tasks are ensured even in large-scale collaboration environments, facilitating seamless real-time collaborative creation.

[0024] By utilizing collision detection between a virtual hand and a virtual object, the virtual hand detects the overlay state on an object such as a mouseover, and by using the hand gesture state to implement object control and button function execution functions, authoring tools necessary for a creative environment can be implemented.

[0025] The virtual object control module has the advantage of enabling realistic interactions within the virtual environment by accurately detecting manipulations such as movement, rotation, and resizing of objects based on hand gestures. To compensate for the issue where users cannot experience physical collisions with virtual objects, it can provide more realistic feedback through collision detection, offering a sense of immersion that feels as if the user is experiencing physical sensations in the virtual space.

[0027] Furthermore, the present invention ensures the concurrency of operations by transmitting and receiving network events based on a Node.js server and Socket.io, thereby increasing the speed and stability of data exchange. This allows for the efficient management of workflows between users and prevents delays or omissions during the task sharing process, which is expected to have significant effects in that it greatly improves the completeness and responsiveness of collaboration.

[0029] Consequently, the collaborative authoring tool system of the present invention combines a hand tracking module, a virtual object control module, and a WebRTC-based streaming module to provide technical superiority in offering high intuitiveness, stability, and efficiency in an environment where multiple users perform collaborative tasks in real time, which can significantly enhance practical applicability in various application fields. Brief explanation of the drawing

[0031] Figure 1 illustrates a system configuration diagram according to the present invention. Figure 2 illustrates a conceptual diagram of a system according to the present invention. Figure 3 illustrates an operation flowchart according to the system of the present invention. FIG. 4 illustrates an example of collaborative creation using the P2P cloud-based authoring tool of the present invention. Specific details for implementing the invention

[0032] The present invention is a content authoring tool system that manipulates virtual objects using hand tracking technology based on camera images capturing hand movements in real time in a client environment, and enables multiple users to collaborate to perform real-time creation and work in a virtual environment.

[0033] The present invention relates to a collaborative authoring system that uses hand tracking technology to track a user's hand movements in real time, thereby enabling interaction with virtual objects, and allowing multiple users to simultaneously share and synchronize work screens in a P2P manner. More specifically, the present invention combines hand tracking technology with WebRTC-based streaming technology to enable users to interact with virtual objects in real time through hand gestures, share work progress status among multiple users, and enable simultaneous collaboration. The present invention has the following configuration.

[0034] As shown in FIG. 1, the system is formed by data communication between a user's client (100) and a server (100). The server (200) includes a member management server, a project server, a WebRTC server, a socket communication server, and a database. The client (100) includes a virtual object augmentation module (110) that displays a virtual object in a virtual environment formed by interaction, a hand tracking module (120) to allow intuitive manipulation of the virtual object, a virtual object control module (130) for controlling the virtual object with a virtual hand, a simplified interface module (140) to maximize user convenience, a WebRTC-based streaming module (150) that supports real-time streaming, and a network-based event transmission and reception module (160) to maintain the concurrency of operations. The detailed functions and operational relationships of each component are as follows.

[0036] Since the present invention is based on displaying and authoring virtual objects, it includes a virtual object augmentation module (110).

[0037] The hand tracking module (120) of the present invention performs the role of recognizing the user's gestures by tracking the user's hand movements in real time and analyzing the position of the hand and the movement of the fingers. This module accurately tracks the user's hand using a camera or sensor, thereby enabling the user to grasp, rotate, move, and select virtual objects using only their hand without a physical device.

[0039] The hand tracking module of the present invention detects hand movements in real time from camera images, extracts position and joint data of the hand and fingers, and generates data necessary for virtual object control, and includes the following detailed method.

[0040] Image acquisition unit: Collects high-resolution images from a camera at tens of frames per second and enables real-time processing. By continuously collecting and processing data per frame, it enables accurate recognition of the trajectory of the user's hand even when it moves quickly.

[0041] Image Preprocessing Unit: Emphasizes the contours of the hand through background removal and noise filtering, enabling independent extraction of the hand. It applies color filtering that recognizes skin tones and removes the background, which is not the hand, to clearly recognize the hand shape.

[0042] Joint position recognition and coordinate conversion unit: The present invention extracts 21 joint points included in a single hand to finely track the spreading, bending, and rotation movements of the fingers. That is, it recognizes the position of each joint of the hand and fingers and converts them into 3D coordinates. Specifically, the hand tracking involves a step of converting a real-time camera video image into a binary image, transmitting the binary image to MediaPipe Solutions, and tracking the positions of 42 joints of both hands through a learning model in the transmitted image.

[0043] Gesture Recognition Unit: Determines various gesture states by utilizing collision detection capabilities between the user's fingertips or palm and a virtual object. Tracked joint positions are projected into a 3D space using matrix multiplication. Specifically, the hand gesture recognition recognizes six gesture shapes (OpenPalm, ClosePalm, Thumbsup, Pinch, Point, HangLoose) by checking the collision states between the five fingertips and the palm through collision detection provided by the Unity engine. For example, a specific motion mode can be activated when the thumb and index finger are brought together and their fingertips are recognized as being close to each other, or the position of a virtual object can be controlled through actions such as opening the hand or clenching a fist. Natural interface operation becomes possible through such gesture recognition.

[0044] Gesture command conversion unit: Converts hand gestures recognized based on joint position data into object control commands. For example, it is used as a command to select an area through a gesture of moving left and right with the palm open, to activate screen effects through a gesture of clenching a fist, to specify a background to select through a gesture of pointing with a finger, or to convert a gesture of closing the fingers into a movement command.

[0045] Gesture Information Transmission Unit: Transmits recognized coordinate data and gesture information to other modules in real time to enable immediate interaction with virtual objects.

[0047] As described above, the hand tracking module of the present invention is characterized by assisting the user without inconvenience and facilitating interaction with objects through various gesture recognition functions. For example, complex hand tracking tasks, such as a user pinching their fingers to select a specific object or rotating their wrist to turn an object in a desired direction, can be naturally performed in virtual reality or real-world environments.

[0049] The present invention includes a virtual object control module (130) that processes interaction with a virtual object based on hand position information processed by a hand tracking module. This module detects a collision between the user's hand and the virtual object and performs selection, movement, rotation, and transformation of the virtual object according to the user's gesture. For example, if the user opens their palm to grasp the virtual object, the object is selected, and the virtual object can be rotated or moved through movements such as rotating the wrist or moving the hand.

[0051] The virtual object control module of the present invention controls a virtual object and enables interaction with a user based on data received from an image analysis module.

[0052] A collision detection unit that detects when a virtual hand and a virtual object overlap using the Unity physics engine;

[0053] A movement trajectory tracking unit that tracks the movement trajectory of a virtual hand within a virtual object;

[0054] A time measuring unit that executes a popup window based on the time when a virtual hand hovers over a virtual button;

[0055] Object selection and movement control unit: A user can select a virtual object by placing their hand over a specific virtual object and opening their palm. Subsequently, the object moves along the trajectory of the hand, and the user can intuitively manipulate the position of the object.

[0056] Object Resizing Section: You can adjust the size of objects using gestures that spread your hands apart. For example, the object is enlarged when you spread your hands and reduced when you bring them together.

[0057] Object Rotation Section: Can rotate an object by recognizing wrist rotation or specific finger gestures. Actions such as rotating the wrist or moving fingers in a specific direction cause the virtual object to rotate along the corresponding axis.

[0058] State Storage and Synchronization: The current state of a virtual object (position, size, rotation) is synchronized with a cloud server, ensuring that the same state is reflected to multiple users. This maintains a consistent object state even when multiple users interact with it simultaneously.

[0059] Manipulation Feedback Section: Provides visual feedback on the current operation status while the user manipulates an object, thereby providing an intuitive user experience.

[0061] The present invention includes an interface module (140) and provides a simplified UI / UX that allows a user to intuitively manipulate a virtual object for virtual object manipulation, and enables intuitive interaction through hand gestures. It includes the following detailed configurations.

[0062] Gesture-based UI Provider: Maximizes work efficiency by providing an intuitive interface that allows users to perform commands such as selecting, moving, rotating, and resizing objects through hand movements.

[0063] Status display unit: Displays the currently selected object and the user's action status on the screen in real time to provide an intuitive overview of the work progress.

[0064] Work History and Restore Provider: Saves work history and supports users in returning to previous steps or restoring specific work states at any time.

[0065] Result visualization provider: Visual feedback is provided so that object movement paths and state changes are immediately reflected on the screen, allowing for intuitive verification of work results.

[0066] Convenience of Use Section: Enhances user experience by simplifying interface design, enabling all users to perform tasks intuitively.

[0068] The present invention includes a WebRTC-based streaming module (150) and performs the function of sharing and synchronizing the progress of work among multiple users in real time. A cooperative authoring tool system is provided that establishes peer-to-peer (P2P) communication through a WebRTC signaling server so that multiple users can cooperate in real time to share work results and perform collaborative work. This system is designed to immediately reflect changes made by users in their work, and to this end, it implements a P2P structure in which direct connections are made between users instead of communicating through a server. By using a P2P communication method, direct data transmission and screen sharing between users are possible without passing through a server. Specifically, it may have the following detailed configurations.

[0069] P2P Connection Configuration Section: Enables direct data sharing between users through WebRTC-based P2P connections, enhancing the real-time nature of operations and reducing dependency on the server.

[0070] Data Transmission Optimization and Compression Unit: Object control data and joint position data transmitted between users are compressed to minimize latency, and transmission speeds are adjusted according to network conditions to provide optimized responsiveness.

[0071] Joint Position Data Sharing Unit: Each user's hand movement data is shared in real-time via a P2P method, allowing the movement, rotation, and scaling of virtual objects to be accurately reflected.

[0072] Screen Synchronization Unit: The work screen is provided to all users in a synchronized state, so changes to work occurring during collaboration are reflected immediately.

[0073] Error Prevention and Data Correction Unit: Ensures reliable data transmission by including packet retransmission and data correction functions to prevent data loss due to network delays or errors.

[0074] Real-time Voice Chat Section: Supports seamless communication between users through voice chat functionality and enables real-time exchange of opinions during collaborative work. Voice data utilizes network bandwidth efficiently through compression and transmission optimization.

[0076] A WebRTC signal server allows multiple users to establish P2P communication when initiating collaboration. Specifically, in environments where peer-to-peer communication was previously blocked by firewalls, the WebRTC signal server enables video and audio data streaming in a P2P cloud environment by facilitating the exchange of necessary information between connected peers. The server supports initial connection setup and data channel establishment, and once connected, user work information is synchronized in real time. This allows users to share their work with others in real-time and collaborate.

[0077] The present invention includes a cloud network-based event transmission and reception module (160), wherein each client (Peer) transmits and receives event signals through a NodeJs server and Socket.io, and implements a cooperative authoring environment by transmitting and receiving video and audio between clients (Peers) through a P2P cloud streaming environment after going through a handshake step and connecting to a WebRTC signal server implemented in NodeJs. Specifically, the NodeJs server acts as an intermediary to deliver messages transmitted by a Peer connected to the server to other Peers, and plays a role in implementing synchronization between Peers and event generation and synchronization.

[0079] The present invention is a collaborative authoring tool system comprising a network-based event transmission and reception module that maintains work concurrency and exchanges data in real time using a Node.js server and Socket.io. In one embodiment of the present invention, the Node.js server efficiently processes work requests on the server side based on high-performance asynchronous I / O processing and an event-driven architecture, and enables the efficient use of server resources even when there are connection requests from multiple clients. Socket.io supports the WebSocket protocol for real-time bidirectional communication between the client and the server, thereby minimizing data transmission delays and providing a real-time collaborative environment.

[0081] FIG. 2 illustrates a conceptual diagram of a cooperative authoring system according to the present invention,

[0082] According to the present invention, a user executes an application, connects to a member server to log in as a registered member, is invited as a user to perform a registered project, connects to the project, and enables authoring activities through hand tracking and virtual object control. Each user (Peer) is synchronized in real time through a WebRTC server and a socket communication server to perform cooperative authoring.

[0084] Figure 3 illustrates an operation flowchart according to the system of the present invention, allowing the user to access the project through authentication and security checks.

[0086] FIG. 4 illustrates an example of collaborative creation using the P2P cloud-based authoring tool of the present invention, in which mutual screens are displayed simultaneously to enable mutual collaborative creation between Peer1 and Peer2.

[0087] A specific embodiment of the present invention can be implemented as a system in which, during the process of content creation by multiple users in a collaborative work environment using virtual objects, intuitive interaction occurs through hand tracking, and work synchronization and collaboration are smoothly achieved through WebRTC-based streaming based on P2P communication. Users can handle virtual objects as if they were real using only their hands, without physical devices, and collaborate to complete tasks in real time.

[0088] In conclusion, the present invention provides a virtual environment collaborative authoring system composed of a hand tracking module, an interface module, a virtual object control module, and a WebRTC-based streaming module, wherein each module operates in conjunction with the others to enable multiple users to collaborate in real time to perform creative activities. By enabling efficient and intuitive work performance in situations where multiple users collaborate simultaneously through a hand tracking-based collaborative authoring system, it can make a significant contribution to providing a new paradigm for virtual collaboration. In particular, the present system provides an effective method that allows users in remote locations to work simultaneously in the same virtual space without physical constraints, which can contribute to increased efficiency in collaborative work and improved user experience. Explanation of the symbols

[0090] Fig. 1

Claims

Claim 1 A system that recognizes a user's hand movements to control virtual objects within a virtual environment and supports multiple users in collaborating in real time to create content, comprising: a virtual object augmentation module (110) that implements virtual objects within a virtual environment; a hand tracking module (120) that detects the user's hand movements in real time while capturing them and analyzes the joint positions of the hand and fingers to generate gesture data; a virtual object control module (130) that controls virtual objects to select, move, rotate, and resize them according to the gesture data provided from the hand tracking module; an interface module (140) that icons control commands for virtual objects based on the gesture data provided from the hand tracking module and places them on the screen, and executes creation commands based on the detection of a physical collision between a specific user gesture and the corresponding icon; a WebRTC-based streaming module (150) that streams a work screen so that multiple users can share the same work environment in real time and enables collaboration through voice chat; and a cloud-based network event transmission and reception module (160) that transmits object control data between each user in real time using a P2P method to reduce server dependency and minimize data transmission delay.The hand tracking module (120) includes an image acquisition unit that captures high-resolution images from a camera, an image preprocessing unit that emphasizes the outline of the hand through background removal and noise filtering and independently extracts the hand, a joint position recognition and coordinate conversion unit that extracts 21 joint points included in one hand to finely track the spreading, bending, and rotation movements of the fingers, and recognizes the position of each joint of the hand and fingers to convert them into 3D coordinates, a gesture recognition unit that determines various gesture states using a collision detection function between the user's fingertips or palm and a virtual object, a gesture command conversion unit that converts the recognized hand gestures based on joint position data into object control commands, and a gesture information transmission unit that transmits the recognized coordinate data and gesture information to another module in real time so that interaction with the virtual object occurs immediately, and the virtual object control module (130) detects when the virtual hand and the virtual object overlap, so as to enable virtual object control and virtual button execution by detecting the overlay state of the virtual hand on the virtual object using collision detection between the virtual hand and the virtual object. Collision detection unit; movement trajectory tracking unit that tracks the movement trajectory of a virtual hand within a virtual object; time measurement unit that executes a pop-up window based on the time the virtual hand hovers over a virtual button; object selection and movement control unit in which the user selects the object by positioning the hand on a specific object and opening the palm, and the object moves according to the hand's movement trajectory, allowing the user to intuitively manipulate the object's position; object size adjustment unit that adjusts the size of the object through gestures of widening or narrowing the distance between the two hands; object rotation unit capable of rotating the object by recognizing wrist rotation or finger movements; state storage and synchronization unit that reflects the real-time state of a virtual object to multiple users as the same state; and operation feedback unit that provides an intuitive user experience by providing visual feedback on the current operation state while the user is manipulating the object;A content collaborative authoring system using hand-tracking-based virtual environment object control and interaction, comprising: a WebRTC-based streaming module including a P2P connection setting unit configured to enable direct data sharing between users; a data transmission optimization and compression unit that compresses object control data and joint position data transmitted between users; a joint position data sharing unit that shares each user's hand motion data in real-time via a P2P method; a screen synchronization unit that provides work screens to all users in a synchronized state so that work changes occurring during collaboration are reflected in real-time; an error prevention and data correction unit that includes packet retransmission and data correction functions to prevent data loss due to network delay or errors; and a real-time voice chat unit that supports smooth communication between users through a voice chat function and exchanges opinions in real-time during collaborative work. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 In claim 1, the cloud-based network event transmission and reception module is characterized by utilizing a Node.js server and Socket.io to transmit and synchronize the user's work status to other users in real time, in a content collaborative authoring system using hand-tracking-based virtual environment object control and interaction. Claim 6 A content collaborative authoring system using hand-tracking-based virtual environment object control and interaction, characterized in that, in claim 1 or 5, the interface module includes the steps of: recording work history to visually display the movement path and state change of each object; reflecting work results to the user in real time; recovering errors during work or returning to a specific work state; and managing all work history to maximize work convenience. Claim 7 A content collaborative authoring system using hand tracking-based virtual environment object control and interaction, characterized in that, in claim 1 or 5, it comprises: a step of converting a real-time camera video image into a binary image; a step of tracking the positions of 21 joints of each hand through a learning model for the binary image; a step of projecting the tracked joint positions into a three-dimensional space using matrix multiplication; and a step of checking the collision state between each fingertip and the palm and recognizing the shape of the gesture through collision detection between the five fingertips and the palm.

Images