Virtual reality interaction device and method based on bimanual gestures for artificial protein skeleton design

Abstract

The present disclosure relates to a virtual reality interaction device and method based on bimanual gestures for artificial protein skeleton design, the device and the method detecting bimanual gestures of a user gripping the same virtual axis in a space, and generating a protein secondary structure corresponding to the bimanual gestures along the virtual axis in the space. In the disclosure, the protein secondary structure can include at least one from among an alpha-helix, a beta-strand and a loop. In various embodiments, the bimanual gestures can include at least one from among a grab gesture, a closed pinch gesture and an open pinch gesture.

Description

A two-handed gesture-based virtual reality interaction device and method for designing artificial protein scaffolds

[0001] The present disclosure relates to a two-hand gesture-based virtual reality interaction device and method for designing artificial protein scaffolds.

[0002] Proteins, after water, are the most abundant structural component in the human body, acting as biological micromachines involved in nearly every bodily function, including breathing, digestion, movement, and immune response. Proteins perform these functions by binding to other proteins or chemicals. The occurrence and location of a protein are determined by its 3D structure, and its composition is determined by the sequence of the various amino acids that compose it.

[0003] Each of the 20 types of amino acids found in the human body consists of a structural backbone identical to all types and a unique side chain. When dozens, hundreds, or even thousands of amino acids are sequentially linked, the forces between the side chains and water molecules cause the sequence to fold, causing the backbone molecules to locally form secondary structures that constitute the protein's 3D structure.

[0004] Synthetic biologists envision a future in which they can design and synthesize novel proteins crucial to meeting human needs, such as those that can cure cancer, fight infectious diseases, and decompose plastic waste, and recent developments in generative AI models are driving this future forward.

[0005] When developing new proteins, exploring a wide range of design options is essential to reduce unnecessary trial and error in the early stages of design. However, tools for this have not yet caught up with advances in AI.

[0006] The present disclosure provides a two-hand gesture-based virtual reality interaction device and method for designing artificial protein scaffolds.

[0007] In the present disclosure, a method of operating an electronic device providing a two-hand gesture-based virtual reality interaction for designing an artificial protein scaffold may include the steps of detecting a two-hand gesture of a user gripping the same virtual axis in space, and generating a protein secondary structure corresponding to the two-hand gesture along the virtual axis in space.

[0008] In the present disclosure, an electronic device providing a two-hand gesture-based virtual reality interaction for designing an artificial protein scaffold includes a camera module, a display module, and a processor configured to detect a gesture of at least one hand of a user in space through the camera module, and design an artificial protein scaffold in the space according to the gesture through the display module, wherein the processor may be configured to detect a two-hand gesture of the user gripping the same virtual axis in space, and generate a protein secondary structure corresponding to the two-hand gesture along the virtual axis in the space.

[0009] In the present disclosure, a computer program stored in a non-transitory computer-readable recording medium for executing a two-hand gesture-based virtual reality interaction method for designing an artificial protein scaffold on an electronic device may include the steps of detecting a two-hand gesture of a user gripping the same virtual axis in space, and generating a protein secondary structure corresponding to the two-hand gesture along the virtual axis in the space.

[0010] According to the present disclosure, an electronic device can generate an artificial protein scaffold in virtual reality through interaction with a user. That is, the user can intuitively design an artificial protein scaffold through two-hand gestures in space. Thus, the electronic device can easily fabricate an artificial protein scaffold with complex and irregular shapes. Specifically, the electronic device can easily generate a protein secondary structure based on two-hand gestures. Here, at least one of the type, position, length, or angle of the protein secondary structure can be set. Furthermore, the electronic device can easily control the protein secondary structure based on variations in the two-hand gesture or additional one-hand gestures. At this time, the electronic device can control the protein secondary structure by at least one of translation, rotation, bending, or twisting, and can connect or disconnect it with respect to another protein secondary structure. Here, the curvature of the protein secondary structure can be set, or at least one of the position, length, angle, or curvature of the protein secondary structure can be changed. The protein 3D structure produced in this way can be output in the Protein Structure Standard File (PDB) format, the most widely used protein structure data format, which can be utilized in the subsequent AI-based protein design process.

[0011] FIG. 1 is a block diagram schematically illustrating an electronic device according to various embodiments.

[0012] FIG. 2 is an exemplary diagram illustrating an artificial protein scaffold designed by an electronic device according to various embodiments.

[0013] FIG. 3 is a diagram schematically illustrating an operating method of an electronic device according to various embodiments.

[0014] FIG. 4 is a diagram illustrating a method of operation of an electronic device based on a protein secondary structure according to various embodiments.

[0015] Figures 5a, 5b, and 5c are exemplary diagrams illustrating the steps of generating the protein secondary structure of Figure 4.

[0016] Figures 6a, 6b, and 6c are exemplary diagrams illustrating steps for controlling the protein secondary structure of Figure 4.

[0017] Hereinafter, various embodiments of the present disclosure are described with reference to the attached drawings.

[0018] FIG. 1 is a block diagram schematically illustrating an electronic device (100) according to various embodiments. FIG. 2 is an exemplary diagram illustrating an artificial protein scaffold (200) designed by an electronic device (100) according to various embodiments.

[0019] Referring to FIG. 1, an electronic device (100) is configured to create a virtual reality 3D curved network for designing an artificial protein skeleton (200) based on a two-hand gesture, and may include at least one of a camera module (110), a communication module (120), an input module (130), a display module (140), an audio module (150), a memory (160), or a processor (170). In this case, the electronic device (100) may be implemented as a near-eye display (NED) device that can be worn on a user's face or head. For example, the near-eye display device may include at least one of smart glasses or a head mount display (HMD) device. In some embodiments, at least one of the components of the electronic device (100) may be omitted, and at least one other component may be added to the electronic device (100). In some embodiments, at least two of the components of the electronic device (100) may be implemented as a single integrated circuit.

[0020] The camera module (110) can capture images. In this case, when the electronic device (100) is implemented as a near-eye display device, the camera module (110) can capture images of the front of a user wearing the electronic device (100). For example, the camera module (110) can include at least one lens, at least one image sensor, at least one image signal processor, and at least one flash.

[0021] The communication module (120) can perform communication with an external device (not shown) in the electronic device (100). The communication module (120) establishes a communication channel between the electronic device (100) and the external device, and can perform communication with the external device through the communication channel. The communication module (120) can include at least one of a wired communication module and a wireless communication module. For example, the wireless communication module can perform communication with the external device through at least one of a long-distance communication network and a local area communication network.

[0022] The input module (130) can input a signal to be used in at least one component of the electronic device (100). The input module (130) can be configured to detect a signal directly input by a user or to generate a signal by detecting a change in the surroundings. For example, the input module (130) can include at least one of a microphone, at least one physical button, or a touch pad. The touch pad can be configured to detect contact by a user's hand, i.e., a touch, and generate a touch signal in response thereto. Here, the touch pad can include at least one of a touch circuitry configured to detect a touch or a sensor circuitry configured to measure the intensity of a force generated by a touch.

[0023] The display module (140) can visually output information to the outside of the electronic device (100). Specifically, the display module (140) can be configured to display visual content. In this case, if the electronic device (100) is implemented as a near-eye display device, the display module (140) can be placed in front of the eyes of a user wearing the electronic device (100). The display method of the display module (140) can vary depending on the type of electronic device (100). For example, if the electronic device (100) is an optical see-through type, at least a portion of the display module (140) is made of a transparent or translucent material, and the user can directly view the actual environment through the display module (140). In this case, as the display module (140) displays the visual content, the user can view both the actual environment and the visual content. As another example, if the electronic device (100) is a video see-through type, the user can view an image of the actual environment captured by the camera module (110) through the display module (140). In this case, as the display module (140) displays visual content, the user can view the image of the actual environment and the visual content together.

[0024] The audio module (150) can output information audibly to the outside of the electronic device (100). For example, the audio module (150) can include at least one of a speaker or a receiver.

[0025] The memory (160) can store various data used by at least one component of the electronic device (100). For example, the memory (160) can include at least one of volatile memory and non-volatile memory. The data can include at least one program and input data or output data related thereto. The program can be stored in the memory (160) as software including at least one command and can include at least one of an operating system, middleware, or an application.

[0026] The processor (170) can control at least one component of the electronic device (100) by executing a program in the memory (160). Through this, the processor (170) can perform data processing or calculation. At this time, the processor (170) can execute a command stored in the memory (160). The processor (170) can detect a gesture of at least one hand of the user in space through the camera module (110). The processor (170) can output information corresponding to the gesture in space through the display module (140).

[0027] According to various embodiments, the electronic device (100) can create a virtual reality 3D curved network for designing an artificial protein scaffold (200) based on a user's two-hand gesture. Specifically, while the user wears the electronic device (100), the electronic device (100) can design an artificial protein scaffold (200), as illustrated in FIG. 2, based on the user's two-hand gesture in space. At this time, the artificial protein scaffold (200) can be implemented as at least one protein secondary structure (210, 220, 230). Here, the protein secondary structure (210, 220, 230) includes at least one of an alpha-helix (α-helix) (210), a beta-strand (β-strand) (220), or a loop (230), and the alpha-helix (210), the beta-strand (220), and the loop (230) can have different structural characteristics.

[0028] More specifically, the processor (170) can generate protein secondary structures (210, 220, 230) between the user's two hands in response to a two-hand gesture. In some embodiments, the two-hand gesture can include at least one of a grab gesture, a closed-pinch gesture, or an open-pinch gesture. The grab gesture can indicate a state in which all five fingers are bent with the thumb and index finger not facing each other for each hand, the closed-pinch gesture can indicate a state in which the thumb and index finger are facing each other for each hand and the remaining fingers are bent, and the open-pinch gesture can indicate a state in which the thumb and index finger are facing each other for each hand and the remaining fingers are extended. For example, the processor (170) may generate an alpha helix (210) in response to a grab gesture, a beta strand (220) in response to a closed pinch gesture, and a loop (230) in response to an open pinch gesture.

[0029] Alternatively or additionally, the processor (170) may adjust the protein secondary structure (210, 220, 230) in response to a deformation of the two-handed gesture. For example, the processor (170) may translate, rotate, bend, or twist the protein secondary structure (210, 220, 230). Alternatively or additionally, the processor (170) may adjust the protein secondary structure (210, 220, 230) in response to a one-handed gesture on the protein secondary structure (210, 220, 230). For example, the processor (170) can move, rotate, link, or separate the protein secondary structure (210, 220, 230) from another protein secondary structure (210, 220, 230).

[0030] FIG. 3 is a drawing schematically illustrating an operation method of an electronic device (100) according to various embodiments.

[0031] Referring to FIG. 3, first, in step 310, the electronic device (100) can detect a user's two-hand gesture in space. While the user wears the electronic device (100), the processor (170) can monitor the space through the camera module (110). During this time, the processor (170) can detect the user's two-hand gesture in space. Specifically, the processor (170) can detect the user's two hands in space and then detect the two-hand gesture from the hand shapes of the two hands. The two-hand gesture can include at least one of a grab gesture, a closed pinch gesture, or an open pinch gesture.

[0032] In response, at step 320, the electronic device (100) can generate a new protein secondary structure (210, 220, 230) in space. Specifically, the processor (170) can generate a protein secondary structure (210, 220, 230) corresponding to a two-hand gesture in space through the display module (140). The processor (170) can generate the protein secondary structure (210, 220, 230) between the user's two hands. The protein secondary structure (210, 220, 230) includes at least one of an alpha helix (210), a beta strand (220), or a loop (230), and the alpha helix (210), the beta strand (220), and the loop (230) can have different structural characteristics.

[0033] In this way, the electronic device (100) can generate at least one protein secondary structure (210, 220, 230) in space. Specifically, the processor (170) can generate one protein secondary structure (210, 220, 230), and can also individually generate a plurality of protein secondary structures (210, 220, 230). In the case of a plurality of protein secondary structures (210, 220, 230), the processor (170) can generate at least two of the plurality of protein secondary structures (210, 220, 230) to overlap with each other, or to be separated from each other without overlapping with each other. In this case, after generating each protein secondary structure (210, 220, 230), the two-hand gesture can be maintained or released.

[0034] Next, at step 330, the electronic device (100) can detect a deformation of a two-hand gesture or a one-hand gesture for any protein secondary structure (210, 220, 230) in space. While the user wears the electronic device (100), the processor (170) can monitor the space through the camera module (110). During this time, the processor (170) can detect a deformation of the user's two-hand gesture or a one-hand gesture in space. Specifically, after the protein secondary structure (210, 220, 230) is generated, the processor (170) can detect a deformation of the two-hand gesture without releasing the two-hand gesture or as the two-hand gesture is released and then re-formed. Meanwhile, after the protein secondary structure (210, 220, 230) is generated, the two-hand gesture is released, and then the processor (170) can detect a one-hand gesture.

[0035] In response to this, at step 340, the electronic device (100) can manipulate the corresponding protein secondary structure (210, 220, 230) in space. Specifically, the processor (170) can manipulate the corresponding protein secondary structure (210, 220, 230) in response to the deformation of the two-handed gesture. For example, the processor (170) can move, rotate, bend, or twist the corresponding protein secondary structure (210, 220, 230). Alternatively, the processor (170) can manipulate the corresponding protein secondary structure (210, 220, 230) in response to a one-handed gesture. For example, the processor (170) can move, rotate, connect to another protein secondary structure (210, 220, 230), or separate from another protein secondary structure (210, 220, 230).

[0036] In this manner, the electronic device (100) can individually control at least one protein secondary structure (210, 220, 230) generated in space. Specifically, when one protein secondary structure (210, 220, 230) is generated, the processor (170) can control one protein secondary structure (210, 220, 230) once or multiple times. Alternatively, when multiple protein secondary structures (210, 220, 230) are generated, the processor (170) can control at least one of the multiple protein secondary structures (210, 220, 230) once or multiple times. In this case, after each protein secondary structure (210, 220, 230) is controlled once, the two-hand gesture or the one-hand gesture can be maintained or released.

[0037] Finally, in step 350, the electronic device (100) can determine whether the design of the artificial protein scaffold (200) is complete. Specifically, the processor (170) can determine whether the design of the artificial protein scaffold (200) is complete based on user input. At this time, if it is determined that the design of the artificial protein scaffold (200) is not complete, the processor (170) can repeat at least some of steps 310 to 340. Meanwhile, if it is determined that the design of the artificial protein scaffold (200) is complete, the processor (170) can determine a combination of at least one protein secondary structure (210, 220, 230) in space as the artificial protein scaffold (200). As a result, the artificial protein scaffold (200) can be generated, as illustrated in FIG. 2.

[0038] FIG. 4 is a diagram illustrating an operating method of an electronic device (100) according to various embodiments. Here, FIG. 4 illustrates an operating method of an electronic device (100) based on one protein secondary structure (210, 220, 230). FIGS. 5A, 5B, and 5C are exemplary diagrams for explaining a step (step 420) of generating a protein secondary structure (210, 220, 230) of FIG. 4. FIGS. 6A, 6B, and 6C are exemplary diagrams for explaining a step (step 443 and 453) of controlling a protein secondary structure (210, 220, 230) of FIG.

[0039] Referring to FIG. 4, first, at step 410, the electronic device (100) can detect a user's two-hand gesture in space. While the user wears the electronic device (100), the processor (170) can monitor the space through the camera module (110). During this time, the processor (170) can detect the user's two-hand gesture in space. Specifically, the processor (170) can detect the user's two hands in space and then detect the two-hand gesture from the hand shapes of the two hands.

[0040] At this time, the two-hand gesture may be generated so that the user's two hands grasp the same virtual axis in space. In some embodiments, the two-hand gesture may include at least one of a grab gesture, a closed pinch gesture, or an open pinch gesture. As illustrated in FIG. 5A, the grab gesture may indicate a state in which all five fingers are bent while the thumb and index finger are not facing each other for each hand, as illustrated in FIG. 5B, the closed pinch gesture may indicate a state in which the thumb and index finger are facing each other for each hand while the remaining fingers are bent, and as illustrated in FIG. 5C, the open pinch gesture may indicate a state in which the thumb and index finger are facing each other for each hand while the remaining fingers are spread out.

[0041] In response to this, at step 420, the electronic device (100) can generate protein secondary structures (210, 220, 230) in space. Specifically, the processor (170) can generate protein secondary structures (210, 220, 230) corresponding to a two-hand gesture in space through the display module (140). The processor (170) can generate protein secondary structures (210, 220, 230) along a virtual axis between the user's two hands. More specifically, the processor (170) can generate vertices at the positions of the two hands on the virtual axis, and then generate protein secondary structures (210, 220, 230) corresponding to the two-hand gesture along a straight line connecting the vertices along the virtual axis. In this way, based on the two-hand gesture, the type, position, length, and angle of the protein secondary structure (210, 220, 230) can be set.

[0042] At this time, the protein secondary structure (210, 220, 230) includes at least one of an alpha helix (210), a beta strand (220), or a loop (230), and the alpha helix (210), the beta strand (220), and the loop (230) may have different structural characteristics. For example, the processor (170) may generate an alpha helix (210) in a helical structure wrapping around a virtual axis in response to a grab gesture, as illustrated in FIG. 5A. Alternatively, the processor (170) may generate a beta strand (220) in a band-shaped arrow structure running along the virtual axis in response to a closed pinch gesture, as illustrated in FIG. 5B. Alternatively, the processor (170) may generate a loop (230) in a line structure matching the virtual axis in response to an open pinch gesture, as illustrated in FIG. 5C.

[0043] Optionally or additionally, at step 431, the electronic device (100) can detect movement and / or rotation of a two-hand gesture in space. Specifically, after generating a protein secondary structure (210, 220, 230), the two-hand gesture can be moved and / or rotated directly by the user without being released. Alternatively, after generating a protein secondary structure (210, 220, 230), the two-hand gesture can be released and then re-formed by the user for the protein secondary structure (210, 220, 230). Here, the user can re-form the two-hand gesture by positioning both hands respectively at the vertices of the protein secondary structure (210, 220, 230). Then, the two-hand gesture can be moved and / or rotated by the user. Accordingly, the processor (170) can detect movement and / or rotation of the two-hand gesture in space.

[0044] In response to this, at step 433, the electronic device (100) can move and / or rotate the protein secondary structure (210, 220, 230) in space. Specifically, the processor (170) can move the protein secondary structure (210, 220, 230) according to the movement of the two-hand gesture. Meanwhile, the processor (170) can rotate the protein secondary structure (210, 220, 230) according to the rotation of the two-hand gesture. At this time, when the distance between the two hands changes during the movement and / or rotation of the two-hand gesture, the processor (170) can change the length of the protein secondary structure (210, 220, 230) corresponding to the distance between the two hands while moving and / or rotating the protein secondary structure (210, 220, 230). In this way, based on the movement and / or rotation of the two-handed gesture, at least one of the position, length, or angle of the protein secondary structure (210, 220, 230) can be changed.

[0045] Alternatively or additionally, at step 441, the electronic device (100) can detect a transformation of the user's two hands in space from one two-hand gesture to another two-hand gesture. Specifically, after generating a protein secondary structure (210, 220, 230), the two-hand gesture may not be released, and the two hands may be gradually transformed into another two-hand gesture. Alternatively, after generating a protein secondary structure (210, 220, 230), the two-hand gesture may be released and then re-formed by the user for the protein secondary structure (210, 220, 230). Here, the user may re-form the two-hand gesture by positioning the two hands respectively at the vertices of the protein secondary structure (210, 220, 230). Then, the two hands may be gradually transformed into another two-hand gesture. This allows the processor (170) to detect a transformation of the user's hands from one two-hand gesture to another two-hand gesture. In one embodiment, as illustrated in FIG. 6A, the other two-hand gesture may be generated such that the user's hands each grip different virtual axes. In another embodiment, the other two-hand gesture may be generated such that the user's hands each rotate at different angles around the virtual axis of the two-hand gesture. In yet another embodiment, as illustrated in FIG. 6B, the other two-hand gesture may be generated such that the user's hands each rotate at different angles while gripping different virtual axes.

[0046] In response, at step 443, the electronic device (100) can bend and / or twist the protein secondary structure (210, 220, 230) in space. In one embodiment, as illustrated in FIG. 6A, when different two-handed gestures are generated such that the user's two hands each grip different virtual axes, the processor (170) can bend the protein secondary structure (210, 220, 230) in space. More specifically, the processor (170) can bend the protein secondary structure (210, 220, 230) by pulling the ends, i.e., vertices, of the protein secondary structure (210, 220, 230) along the positions of the two hands. In another embodiment, when another two-hand gesture is generated such that the user's two hands rotate at different angles about the virtual axes of the two-hand gesture, the processor (170) can twist the protein secondary structure (210, 220, 230) in space. More specifically, the processor (170) can twist the protein secondary structure (210, 220, 230) by rotating the ends, i.e., the vertices, of the protein secondary structure (210, 220, 230) along the angles of the two hands. In yet another embodiment, as illustrated in FIG. 6B , when another two-hand gesture is generated such that the user's two hands grip at different virtual axes and rotate at different angles, the processor (170) can twist the protein secondary structure (210, 220, 230) in space while bending it. In this way, based on the deformation of the user's two hands from one two-hand gesture to another, for the protein secondary structure (210, 220, 230), the straight lines between the vertices are deformed into curves, and the curvature of the protein secondary structure (210, 220, 230) can be set. Here, the curvature for each portion of the protein secondary structure (210, 220, 230) can be set. In addition, at least one of the position, length, or angle of the protein secondary structure (210, 220, 230) can be changed.

[0047] Alternatively or additionally, at step 451, the electronic device (100) may detect a one-handed gesture with respect to a vertex of a protein secondary structure (210, 220, 230) in space. Specifically, after generating the protein secondary structure (210, 220, 230), the two-handed gesture may be released. Then, the processor (170) may detect a one-handed gesture in which the user grips and moves one of the vertices of the protein secondary structure (210, 220, 230), as illustrated in FIG. 6C , although the one-handed gesture is illustrated as a one-handed open pinch gesture, the present invention is not limited thereto. Various one-handed gestures are possible, including a one-handed grab gesture and a one-handed closed pinch gesture. In one embodiment, the one-handed gesture may be to place one of the vertices of a protein secondary structure (210, 220, 230) on another protein secondary structure (210, 220, 230). In another embodiment, the one-handed gesture may be to place a vertex of a protein secondary structure (210, 220, 230) that is connected to another protein secondary structure (210, 220, 230) at a location outside of the other protein secondary structure (210, 220, 230).

[0048] In response, at step 453, the electronic device (100) can spatially connect or disconnect the protein secondary structure (210, 220, 230) to another protein secondary structure (210, 220, 230). In one embodiment, if the one-handed gesture places one of the vertices of the protein secondary structure (210, 220, 230) on the other protein secondary structure (210, 220, 230), the processor (170) can connect the protein secondary structure (210, 220, 230) to the other protein secondary structure (210, 220, 230) via that vertex. At this time, the processor (170) can change at least one of the length or curvature of the protein secondary structure (210, 220, 230) while moving the vertex onto another protein secondary structure (210, 220, 230). In another embodiment, if the one-handed gesture places a vertex of a protein secondary structure (210, 220, 230) connected to another protein secondary structure (210, 220, 230) at a location outside of the other protein secondary structure (210, 220, 230), the processor (170) can separate the protein secondary structure (210, 220, 230) from the other protein secondary structure (210, 220, 230) via the vertex. At this time, the processor (170) can change at least one of the length or curvature of the protein secondary structure (210, 220, 230) by moving the vertex to a position outside of another protein secondary structure (210, 220, 230). In this way, based on a one-hand gesture for a vertex of the protein secondary structure (210, 220, 230), at least one of the position, length, angle, or curvature of the protein secondary structure (210, 220, 230) can be changed.

[0049] Alternatively or additionally, at step 461, the electronic device (100) can detect a one-hand gesture for a line between vertices of a protein secondary structure (210, 220, 230) in space. Specifically, after generating the protein secondary structure (210, 220, 230), the two-hand gesture can be released. Then, the processor (170) can detect a one-hand gesture in which the user draws and moves the line between the vertices of the protein secondary structure (210, 220, 230). Here, the one-hand gesture can include various gestures, including a one-hand grab gesture, a one-hand closed pinch gesture, and a one-hand open pinch gesture. In this case, the one-hand gesture can be moving and / or rotating the line.

[0050] In response, at step 463, the electronic device (100) can move and / or rotate the protein secondary structure (210, 220, 230) in space. Specifically, the processor (170) can move the protein secondary structure (210, 220, 230) according to the movement of the one-hand gesture. Meanwhile, the processor (170) can rotate the protein secondary structure (210, 220, 230) according to the rotation of the one-hand gesture. In this way, at least one of the position or angle of the protein secondary structure (210, 220, 230) can be changed based on the one-hand gesture with respect to the line between the vertices of the protein secondary structure (210, 220, 230).

[0051] Finally, in step 470, the electronic device (100) can determine whether the design of the artificial protein scaffold (200) is complete. Specifically, the processor (170) can determine whether the design of the artificial protein scaffold (200) is complete based on user input. At this time, if it is determined that the design of the artificial protein scaffold (200) is not complete, the processor (170) can repeat at least some of steps 410 to 463. Meanwhile, if it is determined that the design of the artificial protein scaffold (200) is complete, the processor (170) can determine a combination of at least one protein secondary structure (210, 220, 230) in space as the artificial protein scaffold (200). As a result, as illustrated in FIG. 2, an artificial protein scaffold (200) having a three-dimensional protein structure can be generated. The protein 3D structure produced in this way can be output in the Protein Structure Standard File (PDB) format, the most widely used protein structure data format, which can then be utilized in the AI-based protein design process.

[0052] According to the present disclosure, an electronic device (100) can generate an artificial protein scaffold (200) in virtual reality through interaction with a user. That is, the user can intuitively design an artificial protein scaffold (200) through two-hand gestures in space. Accordingly, the electronic device (100) can easily produce an artificial protein scaffold (200) of a complex and irregular shape. Specifically, the electronic device (100) can easily generate a protein secondary structure (210, 220, 230) according to a two-hand gesture. Here, at least one of the type, position, length, or angle of the protein secondary structure (210, 220, 230) can be set. In addition, the electronic device (100) can easily adjust the protein secondary structure (210, 220, 230) based on a variation of a two-hand gesture or an additional one-hand gesture. At this time, the electronic device (100) can control the protein secondary structure (210, 220, 230) to at least one of movement, rotation, bending, or twisting, and can connect or separate it with respect to another protein secondary structure (210, 220, 230). Here, the curvature of the protein secondary structure (210, 220, 230) can be set, or at least one of the position, length, angle, or curvature of the protein secondary structure (210, 220, 230) can be changed. The protein 3D structure produced in this way can be output in the protein structure standard file (PDB) format, which is the most widely used among protein structure data formats, and this can be utilized in a subsequent AI-based protein design process.

[0053] The interaction for designing the artificial protein scaffold (200) of the present disclosure can facilitate an AI-based protein design workflow in three ways. First, if a user generates a portion of a desired protein 3D structure, the generative AI model can fill in the remaining portion. Second, if a user generates a full 3D structure, the AI ​​can generate a better-optimized alternative for the specified conditions. Third, if the AI ​​generates a full 3D structure only under the specified conditions, the user can modify the result as desired.

[0054] In addition to creating an artificial protein scaffold (200), there are two types of conditions that users can specify to achieve the desired results from the AI ​​model. The first is the total volume that the designed protein must not exceed or must fill as much as possible. The second is a hotspot, which is one or more amino acids present on the surface of the target protein and possesses specific favorable physicochemical properties suitable for binding to the designed protein.

[0055] The system of the present disclosure uses a hand pose to create a shape and then repeatedly strokes the hand in mid-air, as if stroking the surface of a virtual object. This allows the user to roughly but quickly create the volume conditions of an organic 3D shape using the cross-section of the hand shape as a sweep profile. Furthermore, the user can identify the physicochemical properties of each amino acid on the surface by viewing an overlay that appears when the hand approaches the surface of the target protein, and can designate hotspots by making rapid hand gestures similar to pulling the desired amino acid toward the designed protein.

[0056] In summary, the present disclosure provides a two-hand gesture-based virtual reality interaction device (i.e., electronic device (100)) and method for designing an artificial protein scaffold (200).

[0057] In the present disclosure, the method of operating an electronic device (100) may include a step of detecting a two-hand gesture of a user that draws the same virtual axis in space (steps 310 and 410), and a step of generating a protein secondary structure (210, 220 and 230) corresponding to the two-hand gesture along the virtual axis in space (steps 320 and 420).

[0058] In various embodiments, the protein secondary structure (210, 220, 230) may include at least one of an alpha helix, a beta strand, or a loop.

[0059] In some embodiments, the two-handed gesture may include at least one of a grab gesture, a closed pinch gesture, or an open pinch gesture.

[0060] For example, the step of generating a protein secondary structure (210, 220, 230) may include at least one of the steps of generating an alpha helix as a helical structure wrapping around a virtual axis when a grab gesture is detected, the step of generating a beta strand as a band-shaped arrow structure running along the virtual axis when a closed pinch gesture is detected, or the step of generating a loop as a line structure matching the virtual axis when an open pinch gesture is detected.

[0061] In various embodiments, the step of detecting a two-hand gesture (steps 310 and 410) may include the step of detecting the user's two hands in space, and the step of detecting the two-hand gesture from the hand shapes of the two hands.

[0062] In various embodiments, the steps of generating a protein secondary structure (steps 320, 420) (steps 210, 220, 230) may include the steps of generating vertices at the positions of the two hands on a virtual axis, and the steps of generating a protein secondary structure (210, 220, 230) corresponding to a two-hand gesture along a straight line connecting the vertices along the virtual axis.

[0063] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting movement of a two-hand gesture in space (steps 330 and 431), and a step of moving a protein secondary structure (210, 220, 230) along the movement (steps 340 and 433).

[0064] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting rotation of a two-hand gesture in space (steps 330 and 431), and a step of rotating a protein secondary structure (210, 220, 230) along the rotation (steps 340 and 433).

[0065] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting deformation of the user's two hands from a two-hand gesture in space to another two-hand gesture gripping different virtual axes (steps 330 and 441), and a step of bending the protein secondary structure (210, 220, 230) by pulling both ends of the protein secondary structure (210, 220, 230) along the two hands in space (steps 340 and 443).

[0066] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting a deformation of the two hands from a two-hand gesture in space to another two-hand gesture in which the two hands of the user rotate at different angles about a virtual axis (steps 330 and 441), and a step of twisting the protein secondary structure (210, 220, 230) by rotating both ends of the protein secondary structure (210, 220, 230) along angles in space (steps 340 and 443).

[0067] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting a one-handed gesture of drawing one of the vertices in space and placing it on another protein secondary structure (210, 220, 230) (steps 330, 451), and a step of changing at least one of a length or curvature of the protein secondary structure (210, 220, 230) while moving one of the vertices in space onto the other protein secondary structure (210, 220, 230) to connect the protein secondary structure (210, 220, 230) to the other protein secondary structure (210, 220, 230) (steps 340, 453).

[0068] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting a one-handed gesture of drawing a vertex of a protein secondary structure (210, 220, 230) that is connected to another protein secondary structure (210, 220, 230) in space and placing it at a location outside the other protein secondary structure (210, 220, 230) (steps 330, 451), and a step of changing at least one of a length or curvature of the protein secondary structure (210, 220, 230) while moving the vertex to a location in space, thereby separating the protein secondary structure (210, 220, 230) from the other protein secondary structure (210, 220, 230) (steps 340, 453).

[0069] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting a one-handed gesture of gripping and moving one of the vertices in space (steps 330 and 451), and a step of changing at least one of a length or curvature of a protein secondary structure (210, 220, and 230) while moving one of the vertices in space.

[0070] In various embodiments, the method of operating the electronic device (100) may further include a step of detecting a one-handed gesture of drawing a line between vertices in space and moving or rotating the line (steps 330 and 461), and a step of moving or rotating a protein secondary structure (210, 220, 230) in space (steps 340 and 463).

[0071] In the present disclosure, an electronic device (100) includes a camera module (110), a display module (140), and a processor (170) configured to detect a gesture of at least one hand of a user in space through the camera module (110), and to design an artificial protein scaffold according to the gesture in space through the display module (140), wherein the processor (170) is configured to detect a gesture of both hands of the user gripping the same virtual axis in space, and to generate a protein secondary structure (210, 220, 230) corresponding to the two-hand gesture along the virtual axis in space, and the protein secondary structure (210, 220, 230) may include at least one of an alpha helix, a beta strand, or a loop.

[0072] In the present disclosure, a computer program stored in a non-transitory computer-readable recording medium for executing a two-hand gesture-based virtual reality interaction method for designing an artificial protein scaffold (200) on an electronic device (100) includes the steps of detecting a two-hand gesture of a user gripping the same virtual axis in space, and generating a protein secondary structure (210, 220, 230) corresponding to the two-hand gesture along the virtual axis in space, wherein the protein secondary structure (210, 220, 230) may include at least one of an alpha helix, a beta strand, or a loop.

[0073] The systems described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the systems and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing unit may execute an operating system (OS) and one or more software applications running on the operating system. The processing unit may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit is sometimes described as being used singly; however, those skilled in the art will appreciate that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors, or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0074] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0075] Methods according to various embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0076] The various embodiments of this document and the terminology used herein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0077] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0078] According to various embodiments, each component (e.g., a module or a program) of the described components may include one or more entities. According to various embodiments, one or more components or steps of the aforementioned components may be omitted, or one or more other components or steps may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the steps performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the steps may be executed in a different order, omitted, or one or more other steps may be added.

Claims

1. A method of operating an electronic device providing two-hand gesture-based virtual reality interaction for designing an artificial protein skeleton, A step of detecting a user's two-hand gesture that draws the same virtual axis in space; and A step of generating a protein secondary structure corresponding to the two-hand gesture along the virtual axis in the space. Including, The protein secondary structure comprises at least one of an alpha helix, a beta strand, or a loop. How an electronic device operates.

2. In paragraph 1, The above two-handed gesture includes at least one of a grab gesture, a closed pinch gesture, or an open pinch gesture. How an electronic device operates.

3. In paragraph 1, The step of detecting the above two-hand gesture is: A step of detecting both hands of the user in the space; and A step of detecting the two-hand gesture from the hand shapes of the two hands including, How an electronic device operates.

4. In paragraph 1, The step of generating the above protein secondary structure is: A step of generating vertices at the positions of the two hands on the virtual axis; and A step of generating the protein secondary structure corresponding to the two-hand gesture along a straight line connecting the vertices along the virtual axis. including, How an electronic device operates.

5. In paragraph 1, A step of detecting movement of the two-hand gesture in the space; and A step of moving the protein secondary structure along the above movement. including more, How an electronic device operates.

6. In paragraph 1, A step of detecting rotation of the two-hand gesture in the space; and A step of rotating the protein secondary structure along the rotation. including more, How an electronic device operates.

7. In paragraph 1, A step of detecting a transformation of the user's two hands from the two-hand gesture to another two-hand gesture that each grips different virtual axes in the space; and A step of bending the protein secondary structure by pulling both ends of the protein secondary structure along the two hands in the space above. including more, How an electronic device operates.

8. In paragraph 1, A step of detecting a transformation of the two hands from the two-hand gesture in the space to another two-hand gesture in which the two hands of the user are rotated at different angles around the virtual axis; and A step of twisting the protein secondary structure by rotating both ends of the protein secondary structure along the angles in the space. including more, How an electronic device operates.

9. In paragraph 4, A step of detecting a one-handed gesture of drawing one of the vertices in the space and placing it on another protein secondary structure; and A step of connecting the protein secondary structure to the other protein secondary structure by changing at least one of the length or curvature of the protein secondary structure while moving one of the vertices on the space onto the other protein secondary structure. including more, How an electronic device operates.

10. In paragraph 4, A step of detecting a one-hand gesture of drawing a vertex of the protein secondary structure connected to the other protein secondary structure in the space and placing it at a position outside the other protein secondary structure; and A step of separating the protein secondary structure from the other protein secondary structure by changing at least one of the length or curvature of the protein secondary structure while moving the vertex to the position in the space. including more, How an electronic device operates.

11. In paragraph 4, A step of detecting a one-handed gesture of drawing and moving one of the vertices in the space; and A step of changing at least one of the length or curvature of the protein secondary structure while moving one of the vertices in the space. including more, How an electronic device operates.

12. In paragraph 2, The step of generating the above protein secondary structure is: When the grab gesture is detected, a step of generating the alpha helix as a spiral structure surrounding the virtual axis; When the closed pinch gesture is detected, a step of generating the beta strand as a band-shaped arrow structure proceeding along the virtual axis; or When the above open pinch gesture is detected, a step of generating the loop with a line structure matching the virtual axis Containing at least one of, How an electronic device operates.

13. In paragraph 4, A step of detecting a one-hand gesture of drawing a line between the vertices in the space and moving or rotating the line; and A step of moving or rotating the protein secondary structure in the space above. including more, How an electronic device operates.

14. In an electronic device providing two-hand gesture-based virtual reality interaction for designing an artificial protein skeleton, camera module; display module; and A processor configured to detect a gesture of at least one hand of a user in space through the camera module, and design an artificial protein skeleton in space according to the gesture through the display module. Including, The above processor, Detecting a user's two-hand gesture that draws the same virtual axis in space, configured to generate a protein secondary structure corresponding to the two-hand gesture along the virtual axis in the space; The protein secondary structure comprises at least one of an alpha helix, a beta strand, or a loop. Electronic devices.

15. A computer program stored in a non-transitory computer-readable recording medium for executing a two-hand gesture-based virtual reality interaction method for designing an artificial protein skeleton on an electronic device, The above method, A step of detecting a user's two-hand gesture that draws the same virtual axis in space; and A step of generating a protein secondary structure corresponding to the two-hand gesture along the virtual axis in the space. Including, The protein secondary structure comprises at least one of an alpha helix, a beta strand, or a loop. Computer program.

Images