Electronic device for drawing virtual object through image plane and operating method thereof

Abstract

Disclosed are an electronic device for drawing a virtual object through an image plane and an operating method therefor. The disclosed electronic device comprises a processor and a memory storing instructions, wherein the instructions, when executed by the processor, instruct the electronic device to: generate an image plane corresponding to a touch screen of a tablet in a virtual space; project a virtual object located in the virtual space onto the image plane according to a viewpoint of a user; and on the basis of a pen input inputted to the touch screen from the user, draw a first line on the image plane onto which the virtual object is projected and draw a second line corresponding to the first line on the virtual object projected onto the image plane.

Description

Electronic device for drawing virtual objects through an image plane and method for operating the same

[0001] The disclosure below relates to an electronic device for drawing a virtual object through an image plane and a method of operating the same.

[0002] Many products feature moving parts, and these moving parts can transform the product into various poses. Products with these multiple poses exhibit distinct shapes for each pose. Because shape changes with movement, designers must consider both shape and movement when designing a product. However, it's difficult to consider both shape and movement simultaneously during the initial sketching phase of a design. It's difficult to imagine how shapes will change with movement, and sketching each shape individually can be time-consuming. Failure to sufficiently explore ideas during the sketching phase can result in a product that doesn't move as intended or takes on unexpected shapes.

[0003] The background technology described above is possessed or acquired during the process of deriving the present disclosure, and cannot necessarily be said to be a publicly known technology disclosed to the general public prior to the filing of the present disclosure.

[0004] Various embodiments can provide a virtual object located in a virtual space to a user by projecting it onto an image plane corresponding to a touch screen within the virtual space according to the user's viewpoint.

[0005] Various embodiments can draw lines on a virtual object projected onto an image plane based on pen input input to a touchscreen.

[0006] Various embodiments may display virtual objects to a user through a head mounted display (HMD) and enable the user to draw and control virtual objects in a virtual space through a tablet including a touchscreen.

[0007] Other objects and advantages of the present invention can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0008] An electronic device according to one embodiment includes a processor and a memory storing instructions, wherein the instructions, when executed by the processor, cause the electronic device to create an image plane corresponding to a touch screen of a tablet in a virtual space, project a virtual object located in the virtual space onto the image plane according to a viewpoint of a user, draw a first line on the image plane on which the virtual object is projected based on a pen input input from the user to the touch screen, and draw a second line corresponding to the first line on the virtual object projected onto the image plane.

[0009] The instructions, when executed by the processor, may cause the electronic device to project the virtual object onto the image plane in response to the user's hand approaching the touchscreen within a predetermined distance.

[0010] The instructions, when executed by the processor, may cause the electronic device to determine a viewing frustum from the camera to the image plane based on the camera and the image plane in the virtual space corresponding to the user's viewpoint, and to project an area of ​​the virtual object included in the viewing frustum onto the image plane.

[0011] The instructions, when executed by the processor, may cause the electronic device to determine whether a distance between the user's hand and the touchscreen is less than or equal to the predetermined distance, and, if the distance between the user's hand and the touchscreen is determined to be less than or equal to the predetermined distance and the virtual object is projected onto the image plane, fix a camera that serves as a reference when the virtual object is projected onto the image plane, regardless of movement of the user's viewpoint.

[0012] The instructions, when executed by the processor, may cause the electronic device to, in response to touch inputs inputted to the touch screen, determine a plurality of first fingertip points on the touch screen corresponding to the touch inputs, and generate the image plane in the virtual space based on the plurality of first fingertip points.

[0013] The instructions, when executed by the processor, may cause the electronic device to determine a plurality of second fingertip points corresponding to the user's fingertips in the virtual space based on positions of the user's fingertips detected through sensors of an HMD (head mounted display) worn by the user, and to match the plurality of first fingertip points of the user with the plurality of second fingertip points in the virtual space, thereby determining a position and angle of the image plane with respect to the HMD, and to generate the image plane in the virtual space based on the position and angle of the image plane.

[0014] The instructions, when executed by the processor, may cause the electronic device to calibrate the plurality of first fingertip points based on the plurality of second fingertip points, and to generate the image plane in the virtual space based on the plurality of calibrated first fingertip points.

[0015] The instructions, when executed by the processor, may cause the electronic device to, in response to a predetermined action by the user, shift the user's focus from the virtual object to an image projected onto the image plane.

[0016] The above predetermined action may be that the user's hand or pen approaches the touchscreen within a predetermined distance.

[0017] The above instructions, when executed by the processor, may cause the electronic device to switch from a stereoscopic viewing mode that displays the virtual object in three dimensions to a monoscopic viewing mode that displays an image of the virtual object projected onto the image plane in a planar manner.

[0018] The instructions, when executed by the processor, cause the electronic device to control at least one of the virtual object, an image of the virtual object projected onto the image plane, and a position of an avatar generated in the virtual space to correspond to the user according to a control input inputted through the touch screen, wherein the control input may be inputted by one hand, both hands, or one hand and a pen of the user.

[0019] The instructions, when executed by the processor, may cause the electronic device to, in response to receiving a control input from the user of tapping and dragging from one point on the touchscreen to another point, rotate the reference plane or the avatar within the virtual space about a center point of the reference plane set within the virtual space with respect to the virtual object according to the control input.

[0020] The instructions, when executed by the processor, may cause the electronic device to move a position of the virtual object or the avatar within the virtual space so as to increase a distance between the virtual object and the user within the virtual space in response to receiving the control input of pinching in such that a distance between two or more points touched by the user on the touchscreen decreases.

[0021] The instructions, when executed by the processor, may cause the electronic device to, in response to receiving a control input in which the user taps and drags two or more points of the touchscreen in the same direction, move the virtual object along a reference plane set within the virtual space or the avatar within the virtual space according to the control input.

[0022] The above instructions, when executed by the processor, may cause the electronic device to determine a reference plane for the virtual object based on a point specified through the pen input, and to draw the second line based on the reference plane and the pen input.

[0023] The instructions, when executed by the processor, may cause the electronic device to receive the pen input based on at least one of a location on the touchscreen where the pen input was input, an angle between the pen and the touchscreen, and a pressure applied to the touchscreen.

[0024] An image of the above virtual space can be provided to the user through an HMD worn by the user.

[0025] An operating method of an electronic device according to one embodiment may include an operation of creating an image plane corresponding to a touch screen of a tablet in a virtual space, an operation of projecting a virtual object located in the virtual space onto the image plane according to a user's viewpoint, and an operation of drawing a first line on the image plane onto which the virtual object is projected based on a pen input input from the user to the touch screen, and drawing a second line corresponding to the first line on the virtual object projected onto the image plane.

[0026] The operation of projecting the virtual object onto the image plane according to the user's viewpoint may project the virtual object onto the image plane in response to the user's hand approaching the touchscreen within a predetermined distance.

[0027] Various embodiments enable a user to draw a three-dimensional virtual object via a touchscreen of a tablet, thereby enabling the user to draw stably while supporting his or her hand on the tablet, reducing user fatigue, and increasing drawing accuracy.

[0028] Various embodiments can increase the sense of three-dimensionality and immersion in a virtual object by displaying the three-dimensional virtual object to the user in a virtual space.

[0029] Various embodiments can eliminate binocular parallax and display images and virtual objects clearly by shifting the user's focus to an image plane or virtual object in virtual space when the user wishes to perform a drawing or look at a virtual object.

[0030] Various embodiments allow for free drawing, viewing and controlling of virtual objects through touch input, pen input and control input via a touch screen.

[0031] FIG. 1 is a drawing for explaining an operation of an electronic device performing drawing according to one embodiment.

[0032] FIG. 2 is a drawing for explaining operations provided by an electronic device according to one embodiment.

[0033] FIGS. 3 and 4 are drawings for explaining an operation of creating an image plane in a virtual space according to one embodiment.

[0034] FIGS. 5 to 8 are drawings for explaining an operation of projecting a virtual object onto an image plane according to one embodiment.

[0035] FIGS. 9 to 11 are drawings for explaining an operation of switching a user's focus according to one embodiment.

[0036] FIG. 12 and FIG. 13 are drawings for explaining a user's control input according to one embodiment.

[0037] FIGS. 14 to 17 are drawings for explaining the operation of an electronic device according to a control input according to one embodiment.

[0038] FIG. 18 is a drawing for explaining an operation of displaying a pen in a virtual space according to one embodiment.

[0039] FIG. 19 is a drawing for explaining an operation of drawing a second line corresponding to a first line according to one embodiment.

[0040] FIG. 20 is a drawing for explaining a process of designing a virtual object according to one embodiment.

[0041] Figure 21 is a drawing for explaining a reference plane according to one embodiment.

[0042] Figure 22 is a drawing for explaining a sketching operation according to one embodiment.

[0043] FIG. 23 is a diagram for explaining a segmenting operation according to one embodiment.

[0044] FIG. 24 is a drawing for explaining an exploded view operation according to one embodiment.

[0045] FIG. 25 and FIG. 26 are drawings for explaining a rigging operation according to one embodiment.

[0046] FIG. 27 and FIG. 28 are drawings for explaining a posing motion according to one embodiment.

[0047] FIGS. 29 to 31 are drawings for explaining a filming operation according to one embodiment.

[0048] FIGS. 32 to 34 are drawings for explaining an operation of providing various viewpoints within a virtual space according to one embodiment.

[0049] Fig. 35 is a drawing showing an operating method of an electronic device according to one embodiment.

[0050] Fig. 36 is a drawing showing an electronic device according to one embodiment.

[0051] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0052] In this document, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, or C", and "a combination of one or more of A, B, and C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. Although terms such as first or second may be used to describe various components, such terms should be construed only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0053] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0054] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0055] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0056] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0057]

[0058] FIG. 1 is a drawing for explaining an operation of an electronic device performing drawing according to one embodiment.

[0059] Referring to FIG. 1, an electronic device may receive various inputs from a user (111) and create a virtual object (126) within a virtual space based on the received inputs. The inputs received from the user (111) may be, for example, touch input, pen input, and control input, but the embodiment is not limited thereto.

[0060] An electronic device can generate a virtual space corresponding to a real space. For example, the electronic device can generate an avatar (121) of a virtual space corresponding to a user (111), an image plane (123) corresponding to a touch screen (113) of a tablet, a virtual pen (124) corresponding to a pen (114), and a three-dimensional virtual object (126) corresponding to an object (115) drawn by the user. The virtual space can be implemented as, for example, augmented reality (AR), virtual reality (VR), mixed reality (MR), or extended reality (XR), but the embodiments are not limited thereto.

[0061] The HMD (112) can display the generated virtual space to the user. For example, the HMD (112) can display an avatar (121), an image plane (123), a virtual pen (124), an image (125), and a virtual object (126) generated in the virtual space to the user (111). The HMD (112) can be worn on the user's head.

[0062] In addition, the HMD (112) can detect and recognize objects in a real space. For example, the HMD (112) can recognize and track the hands and fingertips of the user (111) in a real space through sensors. According to an embodiment, the HMD (112) can also detect and recognize a pen (114) and a tablet in a real space. The HMD (112) can include smart glasses, a smart watch, and a smart band. The HMD (112) can be a wearable device that the user (111) can wear or is suitable for wearing. An image of a virtual space can be provided to the user (111) through the HMD (112) worn by the user (111). In this specification, the HMD (112) may also be referred to as an image display device, a wearable device, or a VR (virtual reality) device for convenience of description.

[0063] The tablet may include a touchscreen (113) capable of receiving touch input, pen input, and control input from a user (111). Touch input may be input by touching the touchscreen (113) with the user's (111) hand, and pen input may be input by a pen (114) held by the user (111). In addition, control input may be input by one hand, both hands, or one hand and the pen (114) of the user (111). The tablet may include various computing devices such as a mobile phone, a smart phone, an e-book device, a laptop, a personal computer, a desktop, a workstation, or a server, depending on the embodiment. In this specification, for convenience of description, the touchscreen (113) may also be referred to as a display.

[0064] The pen (114) is a device controlled by the user's (111) hand, and can transmit input to an electronic device by touching the touch screen (113) of the tablet. The pen (114) may be, for example, a touch pen used on the touch screen (113).

[0065] According to one embodiment, the electronic device can generate an image plane (123) corresponding to the touch screen (113) of the tablet in a virtual space. For example, the electronic device can generate an image plane (123) corresponding to the touch screen (113) in a virtual space based on touch inputs input from a user (111) to the touch screen (113) of the tablet. The image plane (123) may be a virtual transparent plane. The image plane (123) may be synchronized with the touch screen (113) so that an object (115) drawn on the touch screen (113) is displayed. In the present specification, for convenience of explanation, the image plane (123) may also be referred to as a transparent plane. An operation of generating an image plane will be described in detail with reference to FIGS. 3 and 4 below.

[0066] An electronic device can project a virtual object (126) located in a virtual space onto an image plane (123) according to the viewpoint of a user (111). The user (111) can draw the virtual object (126) by referring to the image (125) projected onto the image plane (123). The user (111) can observe the three-dimensional virtual object (126) planarly through the image plane (123) corresponding to the touch screen (113) and draw on the hard and flat touch screen (113), thereby performing stable drawing, reducing fatigue of the user (111), and increasing drawing accuracy. The operation of projecting the virtual object (126) onto the image plane (123) will be described in detail with reference to FIGS. 5 to 8 below.

[0067] The electronic device may draw a first line on an image plane (123) on which a virtual object (126) is projected based on a pen input input from a user (111) to a touch screen (113), and may draw a second line corresponding to the first line on the virtual object (126) projected on the image plane (123). In one embodiment, the first line may represent a line drawn on the image (125) according to a sketch of the user (111), and the second line may represent a line drawn on the virtual object (126) according to the sketch of the user (111). In one embodiment, the electronic device may create a reference plane in a virtual space, and draw a second line corresponding to the first line on the virtual object (126) based on the reference plane. In the present specification, for convenience of explanation, the reference plane may also be referred to as a sketch plane. An operation of drawing the second line will be described in detail with reference to FIGS. 18 to 22 below.

[0068]

[0069] FIG. 2 is a drawing for explaining operations provided by an electronic device according to one embodiment.

[0070] Referring to FIG. 2, in operation (210), the electronic device may draw a first line on an image plane according to a user's pen input, and may draw a second line corresponding to the first line on a virtual object projected on the image plane. The virtual object may be, for example, a robot, but the embodiment is not limited thereto. The virtual object may include a plurality of parts (e.g., a torso, legs), and the plurality of parts may be drawn with a plurality of second lines. In the present specification, for convenience of explanation, operation (210) may also be referred to as a sketching operation.

[0071] In operation (220), the electronic device can provide a user with a three-dimensional virtual object generated in a virtual space. The electronic device can perform at least one of a segmenting operation, a rigging operation, a posing operation, and a filming operation based on the user's control input. The electronic device can distinguish parts of the virtual object based on the user's input, and can receive input of the connection relationship and physical characteristics of each part. For example, the electronic device can distinguish parts of the virtual object into links and joints, and can receive input of the mass of each part. In the present specification, for convenience of explanation, operations (210) and (220) may also be referred to as design operations.

[0072] In operation (230), the electronic device may train a model based on sketch data for the virtual object, the connection relationship between each part, and physical characteristics so that the virtual object can perform a preset motion (e.g., a walking motion). For example, the electronic device may reinforcement train the model to determine the next action of each link and joint based on the current state of each link and joint of the virtual object. In one embodiment, the electronic device may determine the moment of inertia of each part based on the mass of each part.

[0073] In operation (240), the electronic device can control the movement of the virtual object based on the learned model and the user's input. For example, the electronic device can control the virtual object to walk based on the user's controller input.

[0074]

[0075] FIGS. 3 and 4 are drawings for explaining an operation of creating an image plane in a virtual space according to one embodiment.

[0076] Referring to FIG. 3, the electronic device can generate an image plane (320) corresponding to the touch screen in a virtual space (300) based on touch inputs (325) input from a user (310) to the touch screen of the tablet.

[0077] Touch inputs (325) can be received as multi-touch interactions via a touchscreen. For example, touch inputs (325) can be input from first fingertip points on the touchscreen touched by the hand of the user (310). In the present specification, for the purpose of explanation, the input for creating an image plane (320) in the virtual space (300) is described as touch inputs (325) by the hand of the user (310), but the embodiment is not limited thereto. For example, depending on the embodiment, the touch inputs (325) can be implemented as inputs by a controller of the HMD or inputs by a 3D marker.

[0078] The electronic device can, in response to touch inputs (325), determine a plurality of first fingertip points on the touchscreen corresponding to the touch inputs (325). Based on the plurality of first fingertip points, the electronic device can generate an image plane (320) in the virtual space (300).

[0079] According to one embodiment, the electronic device may determine the positions of the fingertips of the user (310) through the HMD worn by the user (310), determine a plurality of second fingertip points of the user (310) where the fingertips of the user (310) are located within the virtual space (300), and generate an image plane (320) in the virtual space (300) based on the plurality of first fingertip points and the plurality of second fingertip points. For example, the electronic device may determine the position and angle of the image plane (320) based on the HMD by matching the plurality of first fingertip points and the plurality of second fingertip points, and generate the image plane (320) in the virtual space (300) based on the determined position and angle of the image plane (320). The plurality of second fingertip points may represent positions of the fingertips of the user (310) within the real space acquired through the HMD. The electronic device can position the user's (310) fingertips within a virtual space (300) at multiple second fingertip points.

[0080] For example, the electronic device may determine the position of the touchscreen relative to the user (310) based on a plurality of first fingertip points and a plurality of second fingertip points, and position the image plane (320) at the corresponding position. In addition, the electronic device may determine the angle of the touchscreen relative to the user (310) based on a plurality of second fingertip points, and determine the angle of the image plane (320) based on the angle. In one embodiment, the electronic device may recognize a tablet through an HMD, determine the position and angle of the tablet relative to the user (310), and determine the position and angle of the image plane (320).

[0081] The electronic device can calibrate a plurality of first fingertip points based on a plurality of second fingertip points, and generate an image plane (320) in the virtual space (300) based on the calibrated plurality of first fingertip points. For example, the electronic device can optimize the plurality of first fingertip points so that the plurality of second fingertip points and the plurality of first fingertip points match. Alternatively, the electronic device can calibrate a plurality of second fingertip points based on a plurality of first fingertip points, and generate an image plane (320) in the virtual space (300) based on the calibrated plurality of second fingertip points.

[0082] According to one embodiment, the electronic device receives data from a plurality of first fingertip points. and decides on the data of multiple second fingertip points. can be determined. Here, n can be determined as the product of the number of fingertip points and the number of frames. For example, if 10 fingertip points are acquired for f frames, n can be determined as 10×f. In one embodiment, if the rotational transformation of the image plane (320) is R and the translational transformation of the image plane (320) is t, the electronic device can be determined by minimizing the squared error between the plurality of first fingertip points and the plurality of second fingertip points as in Mathematical Expression 1 below.

[0083] [Mathematical Formula 1]

[0084]

[0085] According to one embodiment, the electronic device can match a plurality of first fingertip points and a plurality of second fingertip points using the Kabsch algorithm. The electronic device can match the data of the plurality of first fingertip points as shown in the following mathematical expression 2. and data from multiple second fingertip points Each centroid can be determined.

[0086] [Equation 2]

[0087]

[0088] Here, N can represent the total number of data for each of the first fingertip points and the second fingertip points.

[0089] The electronic device can shift a plurality of first fingertip points and a plurality of second fingertip points based on their respective center points, and perform SVD (singular value decomposition), as shown in the mathematical expression 3 below.

[0090] [Equation 3]

[0091]

[0092] Here, U, S, and V can represent the left singular vectors, singular values, and right singular vectors used in SVD, respectively. In addition, R can represent the optimal rotation matrix.

[0093] The electronic device can determine the optimal transformation vector t based on the optimal rotation matrix as shown in Equation 4 below.

[0094] [Equation 4]

[0095]

[0096] The electronic device can match a plurality of first fingertip points and a plurality of second fingertip points based on the determined optimal transformation vector.

[0097] In FIG. 3, the touch inputs (325) are illustrated as ten inputs by the fingertips of the user (310), but the embodiment is not limited thereto, and the touch inputs (325) may be implemented as two or more inputs. The number of touch inputs (325) for generating the image plane (320) may be determined differently depending on the embodiment. For example, the touch inputs (325) may be inputs input by five fingertips of one hand. As the number of touch inputs (325) for generating the image plane (320) increases, the electronic device can more accurately determine the position and angle of the image plane (320).

[0098] Referring to FIG. 4, an image plane (410) generated in a virtual space (400) from the user's point of view is exemplarily illustrated. The electronic device can generate the image plane (410) at the corresponding position and angle based on the position and angle of the image plane (410) determined based on the user's position.

[0099]

[0100] FIGS. 5 to 8 are diagrams illustrating an operation of projecting a virtual object onto an image plane according to one embodiment. In this specification, for convenience of explanation, the operation of projecting a virtual object onto an image plane may also be referred to as a flattening operation.

[0101] Referring to FIG. 5, the electronic device may determine a view frustum (545) from the camera (520) to the image plane (530) based on a camera (520) and an image plane (530) in a virtual space corresponding to the viewpoint of the user (510), and may project an area of ​​a virtual object included in the view frustum (545) onto the image plane (530). The camera (520) may be positioned in the virtual space so as to face the viewpoint that the user (510) looks at from the eye position of the user (510). The electronic device may determine a view frustum (545) from the camera (520) to the image plane (530) within the entire view frustum (540) of the camera (520). In FIG. 5, the entire view frustum (540) and the view frustum (545) of the camera (520) are depicted as flat surfaces, but the embodiment is not limited thereto and may have a three-dimensional shape corresponding to a portion of a square pyramid (e.g., a truncated square pyramid). In addition, for convenience of explanation in the present specification, the view frustum (545) may also be referred to as a view frustum or a viewing frustum.

[0102] For example, the electronic device may determine the image plane (530) as a near plane of the viewing frustum (545), and determine planes including each corner of the camera (520) and the image plane as a left plane, a right plane, a top plane, and a bottom plane of the viewing frustum (545), respectively. In addition, the electronic device may determine a plane corresponding to a maximum cull distance of the camera (520) as a far plane of the viewing frustum (545).

[0103] According to one embodiment, the electronic device may project a virtual object onto the image plane (530) in response to a user's hand approaching the touchscreen within a predetermined distance. In one embodiment, the electronic device may use the touchscreen to determine whether the distance between the user's hand and the touchscreen is within a predetermined distance. Here, the predetermined distance may be determined differently depending on the embodiment. By projecting a virtual object onto the image plane (530) in response to a user's hand approaching the touchscreen within a predetermined distance, the electronic device may determine a point in time when the user intends to draw on the virtual object, thereby allowing the user to draw more conveniently. According to one embodiment, the electronic device may project a virtual object onto the image plane (530) in response to a pen approaching the touchscreen within a predetermined distance.

[0104] Referring to FIG. 6, the electronic device may project an area of ​​a virtual object included in a viewing frustum onto the image plane (620) based on respective distances (630, 631, 641, 642) of the viewing frustum from the camera (610) to the image plane (620). Here, the image plane (620) may be orthogonal to the viewpoint direction of the camera (610), and each distance (630, 631, 641, 642) may be determined based on a normal from the camera (610) to the image plane (620). For example, the electronic device may project coordinates on a three-dimensional virtual object into coordinates on a two-dimensional image plane (620) based on the viewing frustum.

[0105] According to one embodiment, the electronic device may project an area of ​​a virtual object included in a viewing frustum onto an image plane (620) using a projection matrix for a camera (610) that projects coordinates on a three-dimensional virtual object into coordinates on a two-dimensional image plane (620). The projection matrix may be represented by a combination of an intrinsic camera property matrix P representing an optical property of a camera lens and an extrinsic camera property matrix (R|t) representing a world transform (rotation R and translation t) of the camera (610). Depending on the embodiment, the projection matrix may be interchangeable with the intrinsic camera property matrix. For example, the electronic device may determine a coordinate c on a two-dimensional image plane (620) by multiplying a coordinate p on a three-dimensional virtual object by the projection matrix P, as in Equation 5 below.

[0106] [Equation 5]

[0107]

[0108] Here, the range of c can be from "-1" (top or bottom edge) to "+1" (right or left edge) ([-1, 1]). Points with coordinates outside that range are outside the field of view of the camera (610) and may not be visible to the user.

[0109] The electronic device can obtain the final pixel location to be displayed on the HMD by scaling the normalized coordinate c to the pixel size of the HMD.

[0110] According to one embodiment, the electronic device can project coordinates on a three-dimensional virtual object into coordinates on a two-dimensional image plane (620) using a projection matrix of a camera (610) as shown in Equation 6 below.

[0111] [Equation 6]

[0112]

[0113] Here, “near” may represent the distance (631) between the camera (610) and the image plane (620), “far” may represent the maximum distance (630) that the camera (610) can see, “left” may represent the left distance (641) of the image plane (620) based on the normal to the image plane (620), and “right” may represent the right distance (642) of the image plane (620) based on the normal to the image plane (620).

[0114] Referring to FIG. 7, the electronic device can fix a reference camera when a virtual object is projected onto an image plane, regardless of the movement of the user's viewpoint. By fixing the reference camera when the virtual object is projected onto the image plane, the electronic device can fix an image of the virtual object projected onto the image plane.

[0115] According to one embodiment, the electronic device determines whether the distance between the user's hand and the touchscreen is less than or equal to a predetermined distance, and if the distance between the user's hand and the touchscreen is determined to be less than or equal to the predetermined distance and a virtual object is projected onto an image plane, the electronic device can fix a camera that serves as a reference when the virtual object is projected onto the image plane, regardless of movement of the user's viewpoint. In one embodiment, the electronic device can use the touchscreen to determine whether the distance between the user's hand and the touchscreen is less than or equal to the predetermined distance. Here, the predetermined distance may be determined differently depending on the embodiment.

[0116] In the example of FIG. 7, when the user's viewpoint moves from the camera's position (710) to the position (720) at the time of projecting the virtual object onto the image plane, the electronic device can determine whether the distance between the user's hand and the touchscreen is less than a predetermined distance and display the projected image by fixing it to the image plane based on the position (710).

[0117] The electronic device can enable the user to conveniently perform drawing on a fixed image by fixing an image on which a virtual object is projected while the user is drawing, and can enable the user to check a flat image and a three-dimensional shape simultaneously or alternately.

[0118] Referring to FIG. 8, a screen is exemplarily illustrated in which an image (815) projected on an image plane (810) is fixed regardless of the movement of the user's viewpoint. The user can simultaneously view a two-dimensional image (815) of a virtual object fixed on the image plane (810) and a three-dimensional stereoscopic virtual object (820).

[0119]

[0120] FIGS. 9 to 11 are drawings for explaining an operation of switching a user's focus according to one embodiment.

[0121] Referring to FIG. 9, when focusing on a virtual object (910) within a virtual space (900), the image plane (920) and the user's hand, which are relatively close, may be out of focus, resulting in an overlapping image. In the situation of FIG. 9, the user may have difficulty drawing accurately on the image plane (920).

[0122] Referring to FIG. 10, in contrast to the situation in FIG. 9, when focusing on the image plane (1020) and the user's hand within the virtual space (1000), the virtual object (1010) located relatively far away may be out of focus and thus appear to overlap. In the situation in FIG. 10, the user may have difficulty accurately interacting with the virtual object (1010) and may have difficulty accurately identifying the shape of the virtual object.

[0123] Referring to FIG. 11, the electronic device may, in response to a predetermined action by the user, shift the user's focus from a virtual object (1110) to an image (1125) projected onto an image plane (1120). In one embodiment, the predetermined action may be a user's hand or pen approaching the touchscreen within a predetermined distance.

[0124] According to one embodiment, the electronic device can switch from a stereoscopic viewing mode that displays a virtual object (1110) in three dimensions to a monoscopic viewing mode that displays an image (1125) of the virtual object (1110) projected on an image plane (1120) in a two-dimensional manner. The stereoscopic viewing mode may refer to a mode that provides the virtual object (1110) in three dimensions by outputting different screens viewed from different angles to each of the user's eyes through the HMD, and the monoscopic viewing mode may refer to a mode that provides the image (1125) in a two-dimensional manner by outputting the same screen to each of the user's eyes.

[0125] The electronic device may switch the screen presented to the user from a stereoscopic viewing mode to a monoscopic viewing mode in response to the user's hand or pen approaching the touchscreen within a predetermined distance.

[0126] According to one embodiment, the electronic device may provide a stereoscopic viewing mode for a virtual object (1110) and a monoscopic viewing mode for an image (1125) on an image plane (1120) in response to a hand or pen approaching the touchscreen within a predetermined distance.

[0127]

[0128] FIG. 12 and FIG. 13 are drawings for explaining a user's control input according to one embodiment.

[0129] Referring to FIG. 12, the electronic device can control at least one of a virtual object, an image of the virtual object projected onto an image plane (1210), and a user's position within the virtual space according to a control input entered through a touch screen.

[0130] Control input can be input by the user's one hand, both hands (1220), or one hand and a pen. In the example of FIG. 12, the user can input control input to the touchscreen using both hands (1220). For example, the user can perform bimanual interaction for control input by using touch input using both hands or by using touch input from one hand and pen input from the other hand. For example, the user can input control input using both the dominant hand and the non-dominant hand. In some embodiments, the control input can be input using a separate button. For example, the user can input control input to the electronic device by performing a touch input on the touchscreen with the dominant hand while holding down a button with the non-dominant hand.

[0131] Referring to FIG. 13, control input can be input by a user's one hand (1321) and a pen (1322). In the example of FIG. 13, the user can perform touch input on the touchscreen with the non-dominant hand, while performing pen input with the pen (1322) using the dominant hand. The user can input control input by referring to an image plane (1310) within the virtual space.

[0132]

[0133] FIGS. 14 to 17 are drawings for explaining the operation of an electronic device according to a control input according to one embodiment.

[0134] Referring to FIG. 14, in response to a control input of a user tapping and dragging from one point (1421) of a touchscreen to another point (1422), the electronic device may rotate a reference plane or an avatar within a virtual space based on the center point of a reference plane set within a virtual space for a virtual object according to the control input. The control input of tapping and dragging from one point (1421) of the touchscreen to another point (1422) may be expressed as a control input of tapping and dragging from one point (1421) of an image plane (1410) to another point (1422) within the virtual space. In the present specification, for the convenience of explanation, the motion of rotating in response to the control input of tapping and dragging may also be referred to as a pan and tumble motion.

[0135] The electronic device can rotate a reference plane or an avatar within a virtual space based on the direction, distance, and speed of the user tapping and dragging the touchscreen from one point (1421) to another point (1422). For example, the electronic device can rotate the reference plane within the virtual space by an angle corresponding to the distance the user taps and drags. Additionally, the electronic device can rotate the reference plane at a speed corresponding to the speed of the user taps and drags.

[0136] Depending on the user's control input, the reference plane may rotate in the direction from point (1421) to point (1422), or the avatar in the virtual space may rotate in the opposite direction.

[0137] Referring to FIG. 15, an example is shown in which a user in a virtual space rotates in a direction opposite to the tap-and-drag direction. The electronic device can rotate an avatar (1510) in the virtual space by an angle (1520) corresponding to the distance the user taps and drags.

[0138] Referring to FIG. 16, in response to receiving a pinch-in control input to decrease a distance between two or more points (1621) touched by a user on a touchscreen, the electronic device may move the position of the virtual object or the avatar within the virtual space so that the distance between the virtual object and the avatar within the virtual space increases. The pinch-in control input to decrease the distance between two or more points (1621) on the touchscreen may be expressed as a pinch-in control input to decrease the distance between two or more points (1621) on an image plane (1610) within the virtual space. In the present specification, for convenience of explanation, the movement operation in response to the pinch-in control input may also be referred to as a dolly out operation.

[0139] Electronic devices can move the position of virtual objects or avatars within a virtual space based on the distance and speed of the pinch. For example, the slower the pinch speed, the slower the virtual object moves away.

[0140] Conversely, the electronic device may, in response to receiving a pinch-out control input to increase the distance between two or more points (1621) touched by the user on the touchscreen, move the position of the virtual object or the avatar within the virtual space so that the distance between the virtual object and the avatar within the virtual space becomes closer. In the present specification, for convenience of description, the motion of moving in response to the pinch-out control input may also be referred to as a dolly-in motion.

[0141] Referring to FIG. 17, a situation is exemplarily illustrated in which an avatar (1710) in a virtual space moves away from a virtual object (1720) according to a pinch-in control input.

[0142]

[0143] FIG. 18 is a drawing for explaining an operation of displaying a pen in a virtual space according to one embodiment.

[0144] Referring to FIG. 18, the electronic device can display the transparency of the pen (1810, 1820) in the virtual space differently depending on the distance between the touch screen and the pen.

[0145] According to one embodiment, the electronic device may display the pen (1810) in the virtual space with a first transparency (e.g., translucent) in response to the pen (1810) approaching the touchscreen within a predetermined distance. Furthermore, the electronic device may determine a point (1815) on the touchscreen with a minimum distance from the pen (1810) and determine a reference plane for the virtual object based on the point (1815) in response to the pen (1810) approaching the touchscreen within a predetermined distance. In the present specification, for convenience of description, the point (1815) on the touchscreen with a minimum distance from the pen (1810) may also be referred to as a hover point.

[0146] In one embodiment, the electronic device may, in response to the pen (1820) touching the touchscreen, display the pen (1820) with a second transparency (e.g., opaque). Furthermore, in response to the pen (1820) touching the touchscreen, the electronic device may determine a point (1825) on the touchscreen touched by the pen (1820) and determine a reference plane for a virtual object based on the point (1825). In the present specification, for convenience of description, the point (1825) on the touchscreen touched by the pen (1820) may also be referred to as a touch point.

[0147] The user can conveniently determine whether the pen (1810, 1820) has made contact with the image plane based on the transparency of the pen (1810, 1820) within the virtual space. In addition, the user can more conveniently and accurately draw on a virtual object by referring to hover points, touch points, and reference planes within the virtual space.

[0148] According to one embodiment, the electronic device can receive pen input based on at least one of a position at which pen input is input on a touchscreen, an angle between the pen (1810, 1820) and the touchscreen, and a pressure applied to the touchscreen. The electronic device can display at least one of the position at which the acquired pen input is input, the angle between the pen (1810, 1820) and the touchscreen, and the pressure applied to the touchscreen on an image plane. For example, the electronic device can display a position at which pen input is input as a dot, and display the size of the dot differently depending on the pressure applied to the touchscreen. For example, the electronic device can display a larger dot indicating a position at which pen input is input as the pressure applied to the touchscreen increases. In addition, the electronic device can change the shape of the dot indicating a position at which pen input is input depending on the angle between the pen (1810, 1820) and the touchscreen.

[0149]

[0150] FIG. 19 is a drawing for explaining an operation of drawing a second line corresponding to a first line according to one embodiment.

[0151] Referring to FIG. 19, the electronic device may draw a first line (1925) on an image plane (1920) on which a virtual object is projected based on a pen input input from a user to a touch screen, and may draw a second line (1935) corresponding to the first line (1925) on the virtual object projected on the image plane (1920). According to one embodiment, the electronic device may determine a reference plane (1930) for the virtual object based on a point specified through the pen input, and may draw the second line (1935) based on the reference plane (1930) and the pen input.

[0152] For example, the electronic device may draw a second line (1935) corresponding to a first line (1925) on a reference plane (1930) based on a camera (1910) corresponding to the user's viewpoint. For example, the electronic device may determine the points where the line segments from the camera (1910) to the first line (1925) intersect the reference plane (1930) as the second line (1935).

[0153] According to one embodiment, the electronic device can draw the first line (1925) and the second line (1935) based on the angle between the pen and the touchscreen or the pressure applied to the touchscreen. For example, the electronic device can draw the second line (1935) thicker as the pressure applied to the touchscreen by the pen increases.

[0154]

[0155] FIG. 20 is a drawing for explaining a process of designing a virtual object according to one embodiment.

[0156] Referring to FIG. 20, the process of designing a virtual object in a three-dimensional virtual space can be divided into a sketching operation (2010), a segmenting operation (2020), a rigging operation (2030), a posing operation (2040), and a filming operation (2050).

[0157] In a sketching operation (2010), the electronic device can create a virtual object by drawing multiple sketch lines in a three-dimensional virtual space under user control. In a segmenting operation (2020), the electronic device can divide the virtual object into multiple parts by grouping some of the multiple sketch lines drawn in the three-dimensional virtual space and defining them as a single part. In a rigging operation (2030), the electronic device can create joints connecting parts to define relative positions or superior-subordinate relationships between the parts. In a posing operation (2040), the electronic device can change the relative positions or angles between the parts by controlling the joints, thereby changing the virtual object into a different pose. In this way, the user can move the virtual object sketched in the three-dimensional virtual space and confirm that the appearance of the virtual object changes, and can perform the sketching operation (2010), the segmenting operation (2020), and the rigging operation (2030) again while the virtual object is changed to a different pose. Additionally, in the filming operation (2050), the electronic device can record the process of the virtual object changing from one pose to another.

[0158] Electronic devices can edit sketches even after moving a virtual object. The electronic device can erase existing sketch lines from the moved virtual object and create new ones. If the newly created sketch lines are included in a previously created part, the newly created sketch lines can move along with the part. In some embodiments, if the user is not satisfied with the movement, the joints that create the movement can be modified to create a different movement. Multiple poses of a virtual object that appear through various movements can be saved, and sketch modifications made to one pose can be reflected in the remaining poses.

[0159] Users can use various types of joints to reproduce the movements of virtual objects created by those joints. Furthermore, multiple joints can be used to achieve movements that would be impossible with a single joint. Furthermore, by layering movable parts on top of one another, users can create motions in which the shape of the product changes depending on the movement of the other parts. This allows users to fully express the movement of a virtual object and observe how its shape changes as it moves.

[0160] The present invention does not require a separate space for users to check the status of sketch lines representing virtual objects in a three-dimensional virtual space. Instead, the status can be indicated through the color, thickness, and type (e.g., solid line, dotted line) of sketch lines visible on the screen. This allows users to easily check the status without having to take their eyes off the object they are working on.

[0161] For example, in the present invention, colors may be distinguished as black, gray, chromatic colors, etc. A sketch line expressed in black may indicate an object that can be utilized in its current state. Selecting a black sketch line may cause a function to be activated or a change to the system to occur. A sketch line expressed in gray may indicate an object to which a specific function is not applied when the function is activated. A sketch line expressed in chromatic colors may indicate an object to which a specific function is applied when the function is activated. The colors described here are exemplary, and sketch lines may be distinguished in various other ways.

[0162] The sketching operations (2010), segmenting operations (2020), rigging operations (2030), posing operations (2040), and filming operations (2050) performed during the process of designing a virtual object are organically interconnected, so that there is no restriction on the next operation to be performed after performing one operation, and various operations can be performed consecutively without limitation. Even if the concept is repeatedly modified in the initial product design operations, all operations can be performed multiple times regardless of the order.

[0163]

[0164] Figure 21 is a drawing for explaining a reference plane according to one embodiment.

[0165] Referring to FIG. 21, a reference plane (2100) is a plane placed in a three-dimensional virtual space on which a sketch line is drawn according to a user's input, and can help draw a sketch line in a three-dimensional virtual space according to the user's intention.

[0166] A reference plane (2100) may be set in advance before a sketch operation is performed. For example, the reference plane (2100) may be generated based on one or more points specified by tapping a pen tip on a grid line set in a three-dimensional virtual space or a previously drawn sketch line. If there is only one specified point, a reference plane (2100) may be generated with the tangent direction of the sketch line including the point as the normal direction. If there are two specified points, a reference plane (2100) may be generated with the average of the tangent directions of the sketch lines including each of the two points as the normal direction. If there are three specified points, a reference plane (2100) including all three points may be generated. However, examples of generating the reference plane (2100) are not limited to the examples described above.

[0167] The reference plane (2100) may include a plurality of axes (2110), a center point (2120), a center region (2130), and a peripheral region (2140). The plurality of axes (2110) may have x, y, and z-axis directions centered around the center point (2120) of the reference plane (2100). The center region (2130) is a region of a predetermined shape (e.g., a square) centered around the center point (2120), and may also be referred to as a face. The peripheral region (2140) is a region surrounding the center region (2130), and may also be referred to as a bezel. These elements of the reference plane (2100) can help the user interpret the three-dimensional position and orientation of the reference plane (2100), and can also be utilized when the position and orientation of the reference plane (2100) are controlled in six degrees of freedom through two-handed multi-touch interaction.

[0168] In some embodiments, a shadow (not shown) projected on the floor surface in a vertical direction in three-dimensional space of a reference plane (2100) may be displayed, an area where the reference plane (2100) goes below the floor may be displayed dark, and an intersection point of a sketch line in three-dimensional space and a reference plane may be displayed when the sketch line passes through the reference plane. Such visualization may help a user interpret the position and orientation of the reference plane in three-dimensional space when the reference plane (2100) is stationary or moving.

[0169]

[0170] Figure 22 is a drawing for explaining a sketching operation according to one embodiment.

[0171] Referring to FIG. 22, a control for performing a sketch using a reference plane in a three-dimensional virtual space is exemplarily illustrated. In operation (2210), when a user places a reference plane in a desired position and direction and then draws a two-dimensional line (e.g., a straight line or a curve) on the touch screen of an electronic device, the two-dimensional line may be projected onto the reference plane placed in the three-dimensional virtual space to determine a three-dimensional line. In FIG. 22, for convenience of explanation, an example in which a sketch is performed using a pen input by the dominant hand is illustrated, but the sketch input is not limited to the above-described example. In operation (2220), when a user crosses a sketched line with a pen input held with the dominant hand while inputting a control input corresponding to deletion with the non-dominant hand, the sketched line may be deleted. In operation (2230), when a user crosses a reference plane with a pen input held with the dominant hand while inputting a control input corresponding to deletion with the non-dominant hand, the reference plane may be deleted.

[0172] A three-dimensional virtual object can be expressed within a three-dimensional virtual space using multiple lines drawn through the sketching operation described above.

[0173]

[0174] FIG. 23 is a diagram for explaining a segmenting operation according to one embodiment.

[0175] Referring to FIG. 23, a control for determining a portion of lines representing a virtual object as a part is exemplarily illustrated. In operation (2310), when a user inputs a control input indicating 'add a line to a part' with a non-dominant hand and selects a portion of a plurality of lines representing a virtual object in a three-dimensional virtual space (e.g., crossing the lines with a pen input with a dominant hand), the selected lines may be set as the same part. For example, the selected lines among the plurality of lines may be visually distinguished from the unselected lines, thereby allowing the user to confirm whether the intended input has been accurately input. Furthermore, according to an embodiment, when a user inputs a control input indicating 'add a portion of lines to a part' with a non-dominant hand and selects a portion of lines, only the selected portion of lines may be set as the corresponding part. In operation (2320), when a user inputs a control input indicating 'release a line from a part' with a non-dominant hand and selects some of the lines belonging to a part (e.g., crossing a line with a pen input with a dominant hand), the selected lines may be released from the part. Further, according to an embodiment, when a user inputs a control input indicating 'release a part of a line from a part' with a non-dominant hand and selects some of the lines, only the selected part of the lines may be released from the part. In operation (2330), an example of a completed segmentation operation is illustrated.

[0176] The segmentation action may not change the shape of the line. The segmentation action allows the user to define a single part as part of multiple lines.

[0177] For example, if you draw a new sketch line while editing a specific part, that line may automatically be assigned to the specific part being edited. Conversely, while editing a specific part, you may not be able to edit other lines that are not assigned to that part. This exclusive feature allows users to focus on specific parts. For example, a user can focus solely on a robot's legs without worrying about unintended changes to the robot's torso.

[0178]

[0179] FIG. 24 is a drawing for explaining an exploded view operation according to one embodiment.

[0180] Referring to FIG. 24, an exploded view control for multiple parts of a virtual object is exemplarily illustrated. In operation (2410), a user may touch a part to be fixed among the multiple parts with a non-dominant hand. In operation (2420), the user may touch multiple (e.g., three or more) parts with a dominant hand in an empty space while maintaining the touch input of the non-dominant hand. In operation (2430), when the user drags the multiple touch inputs by the dominant hand, an exploded view may be activated in which other parts move away from each other with the fixed part as the center.

[0181] With the exploded view active, the user can perform various controls, such as selecting the part they wish to modify, deleting the parent-child part relationship described below, or deleting the joints established between parts.

[0182]

[0183] FIG. 25 and FIG. 26 are drawings for explaining a rigging operation according to one embodiment.

[0184] A user can control a virtual object as if demonstrating a desired movement and pose in a three-dimensional virtual space, and the electronic device can inversely determine the type, location, direction, and range of motion (e.g., range of motion) of a joint from such control. A joint can be created by selecting a part that has already been segmented and overlaps with a reference plane (e.g., a user's touch input is simultaneously input to the reference plane and the part while the reference plane penetrates at least a portion of the part) and moving together with the reference plane. The motion for creating a joint must be repetitive, which may be to distinguish it from a simple translation and / or rotation of the part. The process for creating a hinge joint, a linear slider, a curved slider, and a ball joint will be described with reference to the drawings below.

[0185] Referring to FIG. 25, a control for generating a hinge joint is exemplarily illustrated. In operation (2510), a user may touch an axis set on a reference plane with a non-dominant hand. For example, the axis of the reference plane into which the user's touch input is input may penetrate a part on which a joint is to be generated. In operation (2520), when the user maintains a touch input with the non-dominant hand on the axis of the reference plane, and a single tap-and-drag input with the dominant hand is input to a part overlapping the reference plane, the part may rotate around the axis of the reference plane. In operation (2530), when the user repeatedly rotates the part around the axis of the reference plane through tap-and-drag input with the dominant hand, a motion afterimage of the part may appear along a trajectory, thereby generating a hinge joint in which the part rotates around the axis of the reference plane.

[0186] For example, a hinge joint may be created where the axis of a reference plane passes through the part, allowing the part to rotate relative to an adjacent part. Furthermore, the hinge joint may have a range of motion corresponding to the repetitive motion required to create the joint. For example, if the rotation due to the repetitive motion occurs within a certain angle, the range of motion of the hinge joint may be limited to that angle, but is not limited to the above-described example.

[0187] Referring to Fig. 26, a control for setting a parent-child relationship is exemplarily illustrated. After a joint is created, a parent-child relationship can be set. A part that the user touches with the non-dominant hand to fix can be set as the parent part, and a part that the user touches with the dominant hand to move can be set as the child part. The parent-child relationship can be visually represented in the exploded view described in Fig. 24, and the parent-child relationship can be deleted or the range of motion of the joint can be modified depending on user control.

[0188]

[0189] FIG. 27 and FIG. 28 are drawings for explaining a posing motion according to one embodiment.

[0190] Similar to the objects mentioned as examples when explaining two-handed multi-touch interaction earlier, when physically manipulating an object with joints, the user can typically hold the part they want to hold still with their non-dominant hand and move the part they want to move with their dominant hand.

[0191] Here, the position and strength of the dominant hand's grip may vary depending on the type of joint and the desired movement. For example, when rotating a specific part around a hinge joint, the dominant hand can grasp a part of the part to be moved that is sufficiently far from the axis of rotation to generate sufficient torque with little force. Conversely, when moving in parallel along a slider joint, the dominant hand can grasp a part of the part to be moved that minimizes the distance of the line of action of the force to prevent unnecessary torque from being generated. Furthermore, a light grip can be used for rough pose control, and a firm grip can be used for precise control. By reflecting these physical affordances, two-handed multi-touch interactions can be implemented that change the pose of a virtual object by moving it using forward and inverse kinematics.

[0192] Referring to FIG. 27, a first control for a virtual object pose is exemplarily illustrated.

[0193] Forward kinematics can refer to creating a desired pose by rotating each joint of a virtual object with multiple joints. In operation (2710), the user can touch and fix one of multiple parts of the virtual object with the non-dominant hand. The fixed part can be referred to as a reference part because it serves as a reference for the movement of a target part that the user wishes to move. In operation (2720), when the user maintains a touch input with the non-dominant hand on the reference part and a tap-and-drag input with the dominant hand is input to the target part to be moved, the target part can move according to the tap-and-drag input. If there are multiple joints between the reference part and the target part, only the joints directly connected to the reference part among the multiple joints are activated to allow the target part to move. In other words, the parts between the reference part and the target part can move as a single unit. This movement control can enable the user to determine the optimal position for moving a desired joint in a desired manner without having to consider the joint structure. In operation (2730), when a plurality of tap and drag inputs by the dominant hand are input to target parts to be moved while the user maintains a touch input of the non-dominant hand on the reference part, the target parts can be moved according to the corresponding tap and drag inputs.

[0194] Inverse kinematics can mean moving multiple joints simultaneously to make a specific part of a virtual object with multiple joints into a desired position and orientation. In operation (2740), when a user maintains a touch input of a non-dominant hand on a reference part and multiple tap-and-drag inputs by a dominant hand are input to a target part, an inverse kinematics calculation can be performed to designate a target point to move the target part and control joint angles accordingly. At this time, all joints connected between the reference part and the target part can be activated. A point-shaped position constraint for the target part touched by the dominant hand is applied, and the touch of the dominant hand can be projected onto a 3D plane that includes the point-shaped position constraint and faces the normal direction of a joint directly connected to the reference part.

[0195] Referring to FIG. 28, a second control for a virtual object pose is exemplarily illustrated.

[0196] Inverse kinematics can also be utilized by utilizing a reference plane. In operation (2810), the user can touch one of the elements of the reference plane (e.g., an axis, a periphery, a center point) with the non-dominant hand. In operation (2820), while the user maintains a touch input with the non-dominant hand on one element of the reference plane, if a tap-and-drag input with the dominant hand is input to a target part that overlaps the reference plane, the target part can move along the reference plane according to the tap-and-drag input. This control can be useful when performing inverse kinematic movements that satisfy certain conditions, such as an end-effector of a robot moving along a reference plane.

[0197]

[0198] FIGS. 29 to 31 are drawings for explaining a filming operation according to one embodiment.

[0199] Similar to how product introduction videos featuring multiple poses feature fluid camera movements, users designing virtual objects in multiple poses can control the virtual object to appear at the most prominent moments, highlighting the various poses and the movement between them. In this context, movement can encompass more than simply transitioning a virtual object from one pose to another within a given usage scenario. This is because movement itself closely interacts with other design elements, such as the virtual object's shape (overall appearance), segmentation (how the overall shape is divided into parts), structure (how parts connect to each other), and pose (what structural arrangement performs what function). For example, users might consider how a part with a specific shape would naturally move, or what shape would be appropriate for a part with a specific movement. Electronic devices can support keyframe-based animation, allowing users to showcase multiple poses as a series of movements.

[0200] Referring to FIG. 29, in operations (2910) to (2930), a user can pose a virtual object to assume a desired pose, adjust a camera to a desired viewpoint, and then input a control input indicating "Save Key Frame" to save a key frame. The saved key frame can be displayed as a thumbnail in the lower left corner of the display of the electronic device. The user can repeat these operations to create a desired sequence of key frames.

[0201] For example, a user can touch a keyframe to recall a previously saved pose and viewpoint, which will activate the thumbnail. If the user changes the pose or viewpoint, the thumbnail can be deactivated. The user can touch a thumbnail and drag it left or right to rearrange the order of keyframes within the sequence, or drag it up or down to delete it. Keyframes and thumbnails can be updated as the user modifies sketching, segmentation, and rigging. This allows the user to easily see how their changes are reflected across multiple poses and viewpoints.

[0202] Referring to FIG. 30, in operations (3010) to (3030), to create a movement connecting key frames, the user may input a tap-and-drag input for a sketch line with the dominant hand while inputting a control input indicating 'follow trajectory' with the non-dominant hand. The tap-and-drag input may be performed along a trajectory displayed on the tablet.

[0203] Referring to FIG. 31, a user can use a key frame sequence to cause a virtual object to continuously move through multiple poses. While a user inputs a control input indicating "follow trajectory" with a specific key frame activated, when the user touches a part to be moved, a trajectory (3110) that the part will be drawn along as it moves continuously can be visualized. The user can continuously move between key frames by moving the touch along the trajectory (3110) at a desired speed. At this time, a frame box can indicate which key frame the current frame is between. The user can record such movements by additionally inputting a specific control input.

[0204]

[0205] FIGS. 32 to 34 are drawings for explaining an operation of providing various viewpoints within a virtual space according to one embodiment.

[0206] Electronic devices can provide various viewpoints for displaying virtual objects. For example, as shown in FIGS. 4, 12, and 13, the electronic device can display virtual objects within a virtual space from the viewpoint of an avatar corresponding to the user. Furthermore, the electronic device can provide the user with a viewpoint from the image plane, a viewpoint from behind the avatar, or a viewpoint from any fixed location within the virtual space.

[0207] Referring to FIG. 32, a situation (3200) in which an electronic device displays a virtual object from a viewpoint on an image plane is exemplarily illustrated.

[0208] Referring to FIG. 33, a situation (3300) in which an electronic device displays a virtual object from a viewpoint looking from behind an avatar is exemplarily illustrated.

[0209] Referring to FIG. 34, a situation (3400) in which an electronic device displays a virtual object from a point of view viewed from an arbitrary fixed location within a virtual space is exemplarily illustrated.

[0210]

[0211] Fig. 35 is a drawing showing an operating method of an electronic device according to one embodiment.

[0212] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (3510) to (3530) may be performed by at least one component (e.g., a processor, etc.) of the electronic device.

[0213] In operation (3510), the electronic device may generate an image plane corresponding to the touch screen of the tablet in the virtual space. The electronic device may determine, in response to touch inputs, a plurality of first fingertip points on the touch screen corresponding to the touch inputs, and generate an image plane in the virtual space based on the plurality of first fingertip points. The electronic device may determine, based on positions of the user's fingertips detected through sensors of an HMD worn by the user, a plurality of second fingertip points corresponding to the user's fingertips in the virtual space, and may match the plurality of first fingertip points of the user with the plurality of second fingertip points in the virtual space, thereby determining a position and an angle of the image plane with respect to the HMD, and generate an image plane in the virtual space based on the position and the angle of the image plane. The electronic device may correct the plurality of first fingertip points based on the plurality of second fingertip points, and generate an image plane in the virtual space based on the corrected plurality of first fingertip points.

[0214] In operation (3520), the electronic device may project a virtual object located in a virtual space onto an image plane according to the user's viewpoint. The electronic device may project the virtual object onto the image plane in response to the user's hand approaching the touchscreen within a predetermined distance. The electronic device may determine a view frustum from the camera to the image plane based on a camera and an image plane in the virtual space corresponding to the user's viewpoint, and may project an area of ​​the virtual object included in the view frustum onto the image plane. The electronic device may determine whether the distance between the user's hand and the touchscreen is within the predetermined distance, and if the distance between the user's hand and the touchscreen is determined to be within the predetermined distance and the virtual object is projected onto the image plane, the electronic device may fix the camera, which serves as a reference when the virtual object is projected onto the image plane, regardless of the movement of the user's viewpoint. The electronic device may, in response to a predetermined action of the user, switch the user's focus from the virtual object to the image projected onto the image plane. The predetermined action for switching the user's focus may be the user's hand or pen approaching the touchscreen within a predetermined distance. The electronic device can switch from a stereoscopic viewing mode, which displays a virtual object in three dimensions, to a monoscopic viewing mode, which displays an image of the virtual object projected onto an image plane in a flat manner.

[0215] In operation (3530), the electronic device may draw a first line on an image plane on which a virtual object is projected based on a pen input input from a user to a touchscreen, and may draw a second line corresponding to the first line on the virtual object projected on the image plane. The electronic device may determine a reference plane for the virtual object based on a point specified through the pen input, and may draw the second line based on the reference plane and the pen input. The electronic device may receive the pen input based on at least one of a position on the touchscreen where the pen input is input, an angle between the pen and the touchscreen, and a pressure applied to the touchscreen.

[0216] An electronic device may control at least one of a virtual object, an image of the virtual object projected onto an image plane, and a position of an avatar generated in a virtual space to correspond to a user, based on a control input received through a touchscreen. The control input may be input by one hand, both hands, or one hand and a pen of the user. In response to receiving a control input in which the user taps and drags from one point on the touchscreen to another, the electronic device may rotate the reference plane or the avatar in the virtual space around a center point of a reference plane set in the virtual space for the virtual object, based on the control input. In response to receiving a control input in which the user pinches in to decrease a distance between two or more points touched on the touchscreen, the electronic device may move the position of the virtual object or the avatar in the virtual space so that a distance between the virtual object and the user in the virtual space increases. In response to receiving a control input in which the user taps and drags two or more points on the touchscreen in the same direction, the electronic device may move the reference plane set in the virtual space or the avatar in the virtual space for the virtual object, based on the control input. Images of virtual spaces can be provided to users through HMDs worn by the users.

[0217] Since the matters described above through FIGS. 1 to 34 are applied to each operation illustrated in FIG. 35, a more detailed description is omitted.

[0218]

[0219] Fig. 36 is a drawing showing an electronic device according to one embodiment.

[0220] Referring to FIG. 36, the electronic device (3600) may include a processor (3610). The processor (3610) may include at least one processor. In addition, the electronic device (3600) may further include a memory (3620). The processor (3610) and the memory (3620) may communicate with each other via a bus.

[0221] The memory (3620) may store instructions (e.g., programs) executable by the processor (3610). For example, the instructions may include instructions for executing operations of the processor (3610) and / or operations of each component of the processor (3610).

[0222] The electronic device (3600) can communicate with the HMD (3630) and the tablet (3640) via a wired and / or wireless network to transmit and receive data. For example, the electronic device (3600) can receive the position of the user's hand detected by the HMD (3630), and the HMD (3630) can receive data about the virtual space from the electronic device (3600). In addition, the electronic device (3600) can receive touch input, pen input, and control input from the tablet (3640).

[0223] Additionally, depending on the embodiment, the HMD (3630) and the tablet (3640) may communicate directly or via the electronic device (3600). In another embodiment, the operation of the electronic device (3600) described above may be performed by the HMD (3630) and / or the tablet (3640), in which case the electronic device (3600) illustrated in FIG. 36 may be omitted.

[0224] The processor (3610) is a device that executes commands or programs or controls the electronic device (3600), and may include various processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), for example. The processor (3610) may generate an image plane corresponding to the touch screen of the tablet in a virtual space. The processor (3610) may project a virtual object located in the virtual space onto the image plane according to the user's viewpoint. Based on a pen input input from the user to the touch screen, the processor (3610) may draw a first line on the image plane on which the virtual object is projected, and may draw a second line corresponding to the first line on the virtual object projected onto the image plane.

[0225] The processor (3610) may project a virtual object onto an image plane in response to a user's hand approaching the touchscreen within a predetermined distance. The processor (3610) may determine a view frustum from the camera to the image plane based on a camera and an image plane in a virtual space corresponding to the user's viewpoint, and may project an area of ​​the virtual object included in the view frustum onto the image plane. The processor (3610) may determine whether the distance between the user's hand and the touchscreen is within the predetermined distance, and if the distance between the user's hand and the touchscreen is determined to be within the predetermined distance and the virtual object is projected onto the image plane, the processor may fix the camera, which serves as a reference when the virtual object is projected onto the image plane, regardless of movement of the user's viewpoint. The processor (3610) may determine a plurality of first fingertip points on the touchscreen corresponding to the touch inputs inputted onto the touchscreen, and may generate an image plane in the virtual space based on the plurality of first fingertip points. The processor (3610) determines a plurality of second fingertip points corresponding to the user's fingertips in the virtual space based on the positions of the user's fingertips detected by the sensors of the HMD worn by the user, and matches the plurality of first fingertip points of the user with the plurality of second fingertip points in the virtual space, thereby determining the position and angle of an image plane with respect to the HMD, and can generate an image plane in the virtual space based on the position and angle of the image plane. The processor (3610) can correct the plurality of first fingertip points based on the plurality of second fingertip points, and can generate an image plane in the virtual space based on the corrected plurality of first fingertip points. The processor (3610) can, in response to a predetermined action for the user, switch the user's focus from a virtual object to an image projected on the image plane.The processor (3610) can switch from a stereoscopic viewing mode that displays a virtual object in three dimensions to a monoscopic viewing mode that displays an image of the virtual object projected onto an image plane in a two-dimensional manner. The processor (3610) can control at least one of a virtual object, an image of the virtual object projected onto an image plane, and a position of an avatar generated in a virtual space to correspond to the user, according to a control input inputted through the touchscreen. In response to receiving a control input of a user tapping and dragging from one point on the touchscreen to another, the processor (3610) can rotate the reference plane or the avatar within the virtual space based on the center point of a reference plane set within the virtual space for the virtual object, according to the control input. In response to receiving a control input of pinching in to decrease a distance between two or more points touched by the user on the touchscreen, the processor (3610) can move the position of the virtual object or the avatar within the virtual space so that the distance between the virtual object and the user within the virtual space increases. The processor (3610) may, in response to receiving a control input in which a user taps and drags two or more points of a touchscreen in the same direction, move a reference plane set in a virtual space for a virtual object or an avatar in a virtual space according to the control input. The processor (3610) may determine a reference plane for a virtual object based on a point specified through a pen input, and may draw a second line based on the reference plane and the pen input. The processor (3610) may receive a pen input based on at least one of a position on the touchscreen where the pen input is input, an angle between the pen and the touchscreen, and a pressure applied to the touchscreen.

[0226] In addition, the electronic device (3600) can process the above-described operations.

[0227] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0228] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0229] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0230] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0231] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0232] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, processor; and Memory that stores instructions Including, The above instructions, when executed by the processor, cause the electronic device to: Create an image plane corresponding to the tablet's touchscreen in virtual space, Projecting a virtual object located in the above virtual space onto the image plane according to the user's viewpoint, Based on a pen input input from the user to the touch screen, a first line is drawn on the image plane on which the virtual object is projected, and a second line corresponding to the first line is drawn on the virtual object projected on the image plane. Electronic devices.

2. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: In response to the user's hand approaching the touchscreen within a predetermined distance, the virtual object is projected onto the image plane. Electronic devices.

3. In paragraph 2, The above instructions, when executed by the processor, cause the electronic device to: Based on the camera and the image plane in the virtual space corresponding to the viewpoint of the user, a viewing frustum is determined from the camera to the image plane, Projecting the area of ​​the virtual object included in the above viewing frustum onto the image plane, Electronic devices.

4. In paragraph 2, The above instructions, when executed by the processor, cause the electronic device to: Determining whether the distance between the user's hand and the touchscreen is less than the predetermined distance, When the distance between the user's hand and the touchscreen is determined to be less than or equal to the predetermined distance and the virtual object is projected onto the image plane, the camera that serves as a reference is fixed when the virtual object is projected onto the image plane, regardless of the movement of the user's viewpoint. Electronic devices.

5. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: In response to touch inputs inputted to the touch screen, a plurality of first fingertip points on the touch screen corresponding to the touch inputs are determined, Based on the plurality of first fingertip points, generating the image plane in the virtual space, Electronic devices.

6. In paragraph 5, The above instructions, when executed by the processor, cause the electronic device to: Based on the positions of the user's fingertips detected through the sensors of the HMD (head mounted display) worn by the user, a plurality of second fingertip points corresponding to the user's fingertips are determined within the virtual space, By matching the plurality of first fingertip points of the user and the plurality of second fingertip points in the virtual space, the position and angle of the image plane are determined based on the HMD, Based on the position and angle of the image plane, the image plane is created in the virtual space. Electronic devices.

7. In paragraph 6, The above instructions, when executed by the processor, cause the electronic device to: Calibrate the plurality of first fingertip points based on the plurality of second fingertip points, Based on the above-mentioned plurality of first fingertip points, generating the image plane in the virtual space, Electronic devices.

8. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: In response to a predetermined action for said user, to shift the focus of said user from said virtual object to an image projected onto said image plane, Electronic devices.

9. In paragraph 8, The above predetermined actions are The user's hand or pen approaches the touchscreen within a predetermined distance, Electronic devices.

10. In paragraph 8, The above instructions, when executed by the processor, cause the electronic device to: Switching from a stereoscopic viewing mode that displays the virtual object in three dimensions to a monoscopic viewing mode that displays an image projected onto the image plane in a planar manner. Electronic devices.

11. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Control at least one of the virtual object, the image of the virtual object projected onto the image plane, and the position of the avatar generated in the virtual space to correspond to the user according to the control input entered through the touch screen, The above control input is Input by the user's one hand, two hands, or one hand and a pen, Electronic devices.

12. In paragraph 11, The above instructions, when executed by the processor, cause the electronic device to: In response to receiving a control input from the user to tap and drag from one point to another on the touchscreen, to rotate the reference plane or the avatar within the virtual space based on the control input with respect to the center point of the reference plane set within the virtual space for the virtual object. Electronic devices.

13. In paragraph 11, The above instructions, when executed by the processor, cause the electronic device to: In response to receiving the control input of pinching in to decrease the distance between two or more points touched by the user on the touchscreen, move the position of the virtual object or the avatar within the virtual space so that the distance between the virtual object and the user within the virtual space increases. Electronic devices.

14. In paragraph 11, The above instructions, when executed by the processor, cause the electronic device to: In response to receiving the control input of the user tapping and dragging two or more points of the touchscreen in the same direction, moving the virtual object along a reference plane set within the virtual space or the avatar within the virtual space according to the control input. Electronic devices.

15. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Based on the point specified through the above pen input, determine a reference plane for the virtual object, To draw the second line based on the above reference plane and the above pen input, Electronic devices.

16. In paragraph 15, The above instructions, when executed by the processor, cause the electronic device to: Receiving the pen input based on at least one of the position where the pen input is input on the touch screen, the angle between the pen and the touch screen, and the pressure applied to the touch screen. Electronic devices.

17. In paragraph 1, The image of the above virtual space is provided to the user through the HMD worn by the user. Electronic devices.

18. In the method of operating an electronic device, The act of creating an image plane in virtual space corresponding to the touch screen of a tablet; An operation of projecting a virtual object located in the virtual space onto the image plane according to the user's viewpoint; and An operation of drawing a first line on the image plane on which the virtual object is projected based on a pen input input from the user to the touch screen, and drawing a second line corresponding to the first line on the virtual object projected on the image plane. Including How an electronic device operates.

19. In paragraph 18, The action of projecting the above virtual object onto the image plane according to the user's viewpoint is In response to the user's hand approaching the touchscreen within a predetermined distance, the virtual object is projected onto the image plane. How an electronic device operates.

20. A computer-readable recording medium storing a computer program for executing the method of any one of claims 18 and 19.

Images