Electronic device and system for generating objects

Abstract

An electronic device and system are disclosed. The electronic device includes: a housing; a touch screen display disposed in the housing and exposed through at least a portion of the housing; a camera; a wireless communication circuit disposed in the housing; a position sensor; a motion sensor; a processor disposed in the housing and operably connected to the display, the camera, the wireless communication circuit, the position sensor, and the motion sensor; and a memory disposed in the housing and operably connected to the processor, wherein the memory is configured to store instructions that, when executed, cause the processor to: capture an image using the camera, receive the captured image from the camera, receive position data associated with the captured image from the position sensor, receive orientation data associated with the captured image from the motion sensor, receive user input for adding an object to the captured image through the display, and send the object, the captured image, the position data, and the orientation data to an external server through the wireless communication circuit.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and system for intuitively generating and manipulating objects, particularly three-dimensional objects, in response to touch-sensitive user input. Background Art

[0002] In recent years, the role of displays in portable electronic devices has become increasingly important. Specifically, a portable electronic device with a touch-sensitive display such as a touch screen can receive various user inputs (e.g., touch input, stylus input, hover input, etc.) through the display and perform a specific function corresponding to the received user input.

[0003] The portable electronic device may display a user interface (e.g., an augmented reality (AR) interface) through a display, and generate or store an object (e.g., content) based on the user interface. That is, the portable electronic device may recognize an input data value in response to a user input, and then generate or store a specific object based on the recognized data value. Summary of the invention

[0004] Technical issues

[0005] In order to generate such an object based on an AR interface, the electronic device can capture an image through its camera and receive the captured image and position and orientation data associated with the captured image. The electronic device can then implement the AR interface based on the captured image, position data, and orientation data.

[0006] When generating a specific object (e.g., content) based on an AR interface, the electronic device requires information about the object to be generated (e.g., size, length, slope, height, area and / or coordinate data, such as coordinate values ​​relative to the X, Y, and Z axes). Typically, such object information may be a data value input by a user. For example, the electronic device may recognize a numerical value input by a user and generate a specific object corresponding to the recognized numerical value. In particular, in order to generate a three-dimensional object, the user further inputs numerical values ​​of the rotation information and depth information of the object. Unfortunately, this may complicate the operation of generating an object.

[0007] Solution to the problem

[0008] According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes: a housing; a touch screen display, the touch screen display is disposed in the housing and exposed through at least a portion of the housing; a camera; a wireless communication circuit, the wireless communication circuit is disposed in the housing; a position sensor; a motion sensor; a processor, the processor is disposed in the housing and is operably connected to the display, the camera, the wireless communication circuit, the position sensor, and the motion sensor; and a memory, the memory is disposed in the housing and is operably connected to the processor, wherein the memory is configured to store instructions, which when executed cause the processor to: capture an image using the camera, receive the captured image from the camera, receive position data associated with the captured image from the position sensor, receive orientation data associated with the captured image from the motion sensor, receive user input for adding an object to the captured image through the display, and send the object, the captured image, the position data, and the orientation data to an external server through the wireless communication circuit.

[0009] According to another aspect of the present disclosure, a system is provided. The system includes: a network interface; a processor, the processor is operably connected to the network interface; and a memory, the memory is operably connected to the processor, wherein the memory is configured to store instructions, and when the instructions are executed, the processor: receives first data from a first external electronic device including a display and a camera via the network interface, wherein the first data includes an image, position data associated with the image, orientation data associated with the image, and an object to be overlapped with the image generated in response to a user input, processes the first data based on at least a portion of the position data, the orientation data, or the user input received from the first external electronic device, so that the direction of the object is applied to the image, and sends the processed first data to another external electronic device via the network interface.

[0010] Advantageous Effects of the Invention

[0011] One aspect of the present disclosure provides a touch-sensitive environment that enables a user to easily input information about a 3D object into an electronic device when a three-dimensional (3D) object is generated in a user interface based on an AR interface and a two-dimensional plane is displayed. The environment not only enables the user to easily input 3D related information (e.g., rotation information and depth information) into the electronic device through touch input, but also enables the electronic device to generate a 3D object based on the 3D related information input by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a block diagram of an electronic device in a network environment according to an embodiment;

[0014] Figure 2 is a flow chart of a method for generating an object based on an augmented reality interface according to an embodiment;

[0015] 3A and 3B are flowcharts of a method for generating a 3D object according to an embodiment;

[0016] 4A , 4B, 4C and 4D are diagrams of generating a 3D object according to an embodiment;

[0017] 5A , 5B, 5C, 5D and 5E are diagrams of generating a 3D object according to an embodiment;

[0018] Figure 6 is a flowchart of a method for generating a 3D object by utilizing pressure information according to an embodiment;

[0019] 7A , 7B and 7C are diagrams of generating a 3D object by using pressure information according to an embodiment;

[0020] Figure 8 is an illustration of generating a 3D object by utilizing pressure information according to an embodiment;

[0021] Fig. 9 is an illustration of generating a 3D object by utilizing pressure information according to an embodiment;

[0022] 10A , 10B, 10C and 10D are diagrams of rotating a 3D object in response to additional user input according to an embodiment;

[0023] Fig.11 is a flowchart of a method of generating a 3D object by using pen tilt information according to an embodiment; and

[0024] 12A , 12B and 12C are diagrams of generating a 3D object by using pen tilt information according to an embodiment. DETAILED DESCRIPTION

[0025] Various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0026] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0027] Reference Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit.For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented to be embedded in the display device 160 (eg, a display).

[0028] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 , or as part of the main processor 121 .

[0029] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123.

[0030] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0031] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0032] The input device 150 may receive commands or data to be used by other components (eg, the processor 120) of the electronic device 101 from outside the electronic device 101 (eg, a user). The input device 150 may include, for example, a microphone, a mouse, or a keyboard.

[0033] According to an embodiment, the processor 120 may sense input of a stylus on the display device 160 (e.g., a touch screen). The stylus having an independent resonant circuit may interact with an electromagnetic induction panel (e.g., a digitizer) included in the display device 160. The stylus may be implemented using electromagnetic resonance (EMR), active electric stylus (AES), or electrically coupled resonance (ECR) technology.

[0034] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as a part of the speaker.

[0035] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0036] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0037] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0038] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0039] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0041] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0042] According to an embodiment, the processor 120 may capture an image using the camera module 180 and implement an AR interface based on the captured image. In addition, the processor 120 may display the implemented AR interface through the display device 160.

[0043] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0044] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0046] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include one or more antennas, and thus, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna.

[0047] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0048] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0049] Figure 2 is a flowchart of a method for generating an object based on an AR interface according to an embodiment.

[0050] In step 201, the processor 120 of the electronic device 101 may capture an image using the camera module 180. For example, the processor 120 may activate the camera 180 and capture a still image through the activated camera 180.

[0051] In step 203, the processor 120 may receive the captured image and the position and orientation data associated with the captured image. For example, the processor 120 may receive the captured image and the position and orientation data associated with the captured image by utilizing a position sensor (e.g., Figure 1 The sensor module 176 of the embodiment of the present invention receives the position data associated with the captured image, and also receives the position data associated with the captured image by utilizing the motion sensor (e.g., Figure 1 The sensor module 176 of the electronic device 101 receives the position data associated with the captured image. The position data may include data about the coordinates at which the electronic device 101 is located, and the position data may include data about the direction in which the camera 180 of the electronic device 101 is facing. According to an embodiment, the electronic device 101 may implement an AR interface based on the captured image and the position and direction data associated with the captured image.

[0052] In step 205, the processor 120 may receive a user input (e.g., a stylus input) for adding an object to the captured image. For example, the processor 120 may receive the user input through the input device 150 based on the implemented AR interface. The user input may include a touch input for generating an object (e.g., content). Depending on the embodiment, the object may include text, a symbol, a geometric shape, or an image.

[0053] At step 207, the processor 120 may send the object generated in response to the user input, the captured image, the position data, and the orientation data to the server 108. In addition, the processor 120 may store the generated object in the memory 130. In an embodiment, the processor 120 may send the generated object, the image captured at step 201, and the position / orientation data received at step 203 to the server 108, at least in part.

[0054] As described above, the electronic device 101 can capture a specific image through the camera 180 and receive position and orientation data associated with the captured image. Then, the electronic device 101 can implement an AR interface based on the captured image, the received position data, and the received orientation data. In addition, the electronic device 101 can receive user input through the AR interface and then generate an object in response to the received user input. In addition, the electronic device 101 can store the generated object in the memory 130 and send the generated object, the captured image, the position data, and the orientation data to the server.

[0055] 3A and 3B are flowcharts 301 and 302 of a method for generating a 3D object according to an embodiment.

[0056] Referring to FIG. 3A , a flowchart 301 relates to a method for generating a 3D object. The flowchart 301 may correspond to Figure 2 The above steps 205 and 207.

[0057] In step 311, the processor 120 of the electronic device 101 may receive user input associated with an object (or content). For example, the processor 120 may receive a user input associated with an object (or content) via a touch screen display (e.g., Figure 1 The display device 160 of the processor 120 displays a user interface for generating an object, and receives user input through the user interface. The user input may include a touch input of a finger or a touch input of a pen (e.g., a stylus). According to an embodiment, the user interface may be an AR interface. For example, the processor 120 may activate the camera module 180, capture a specific image through the activated camera, display the captured image through the display, and implement an AR interface based on the captured image.

[0058] In step 313, the processor 120 may obtain 3D related information for generating an object from the received user input. For example, the processor 120 may obtain 3D related information (e.g., data on coordinates, angles, slopes, widths, lengths, heights, and / or areas) about an object (e.g., text, geometric shapes, etc.) to be generated according to the received user input.

[0059] In step 315, the processor 120 may send the object generated based on the acquired 3D-related information to the server 108. For example, the processor 120 may generate the object based on the acquired 3D-related information, display the generated object on the display 160, and store the generated object in the memory 130. In addition, the processor 120 may send the generated object to an external electronic device (e.g., the server 108).

[0060] As described above, the electronic device 101 can receive user input through the display 160, and then obtain 3D related information for generating an object from the received user input. In addition, the electronic device 101 can generate an object based on the obtained 3D related information, display the generated object on the display 160, store the generated object in the memory 130, and send the generated object to the server 108.

[0061] Referring to FIG. 3B , flowchart 302 relates to a method for generating a 3D object. Flowchart 302 may correspond to Figure 2 The above steps 205 and 207.

[0062] In step 321, the processor 120 of the electronic device 101 may receive a first user input associated with an object. For example, the processor 120 may receive a user input of writing a text object "HELLO" through a display (eg, a touch screen display).

[0063] In step 323, the processor 120 may identify a geometric shape having 3D-related information. For example, the processor 120 may receive an input to construct a specific geometric shape on the display and identify the geometric shape from the received input. The geometric shape may have 3D-related information such as length, height, and slope, so that the processor 120 may also identify the 3D-related information from the identified geometric shape.

[0064] Although it is described that step 323 is performed after step 321, step 323 may be performed before step 321. The geometric shape with 3D-related information may be associated with the first user input received in step 321. In one example above, the input for constructing the geometric shape may be received after receiving the first user input associated with the object. However, in another example, the processor 120 may identify the geometric shape with 3D-related information and then receive the first user input associated with the object. For example, the processor 120 may identify the first user input in a specific area defined by the geometric shape.

[0065] In step 325, the processor 120 may generate an object in response to the first user input and taking into account the identified geometric shape. For example, taking into account the 3D related information of the identified geometric shape, the processor 120 may generate a specific object corresponding to the first user input. That is, the generated object may be a 3D object to which the 3D related information is applied.

[0066] In step 327, the processor 120 may receive a second user input for the generated object. For example, the second user input may be at least one input of directly touching the generated object. For example, the second user input may include a drag input.

[0067] In step 329, the processor 120 may send the object to the server 108 in response to the received second user input. For example, when the second user input is a drag input for the object, the processor 120 may move the object in response to the drag input. At this time, the processor 120 may display the real-time drag state of the object through the display 160. In addition, the processor 120 may store information related to the object (e.g., position, coordinates, slope, width, length, etc.) in the memory 130, and send the object to an external electronic device (e.g., the server 108).

[0068] 4A , 4B, 4C, and 4D are diagrams of generating a 3D object according to an embodiment.

[0069] 4A, the processor 120 of the electronic device 400 may display a 3D-based user interface (e.g., an AR interface) through a display 410 such as a touch screen display. For example, the 3D-based user interface may include a 3D image, a photo, a picture, or a real-time preview image obtained by the camera module 180. The processor 120 may receive a user input for generating a 3D object on the 3D-based user interface. For example, the processor 120 may receive a first user input of writing a text object 401 (e.g., "HELLO") and display the text object 401 on the display 410. The user input may be a touch input by a user through a finger or a stylus. According to an embodiment, the object generated in response to the user input is not limited to the text object 401.

[0070] 4B , the processor 120 may receive a second user input for constructing a specific geometric shape 403 (e.g., defining a 3D-based plane in a 3D-based user interface) that at least partially surrounds the text object 401. The processor 120 may then identify the geometric shape 403 from the received second user input and display the identified geometric shape 403 on the display 410. The geometric shape 403 may have 3D-related information, such as data about coordinates, angles, slopes, widths, lengths, heights, and / or areas. Therefore, the processor 120 may obtain 3D-related information from the identified geometric shape 403.

[0071] 4C , the processor 120 may receive a third user input for constructing a complete geometric shape 404 that completely surrounds the text object 401. Then, the processor 120 may recognize the complete geometric shape 404 from the received third user input, display the recognized geometric shape 404 on the display 410, and obtain 3D related information from the recognized geometric shape 404. That is, based on the second user input and the third user input, the complete geometric shape 404 required by the user is constructed to surround the text object 401. As a result, the text object 401 such as "HELLO" is contained in the complete geometric shape 404.

[0072] 4D , the processor 120 may modify the shape of the text object 401 based on the complete geometric shape 404. For example, according to the acquired 3D related information of the complete geometric shape 404, the processor 120 may change the corresponding 3D related information of the text object 401.

[0073] 5A , 5B, 5C, 5D, and 5E are diagrams of generating a 3D object according to an embodiment.

[0074] Referring to FIG5A , the processor 120 of the electronic device 400 may display a 3D-based user interface (e.g., an AR interface) through a display 410 such as a touch screen display. In addition, the processor 120 may receive a first user input for constructing a specific geometric shape 501, which defines, for example, a 3D-based plane in a 3D-based user interface. The processor 120 may then identify the geometric shape 501 from the received first user input and display the identified geometric shape 501 on the display 410. The first user input may be a touch input by a user through a finger or a stylus. The geometric shape 501 may have 3D-related information, such as data about coordinates, angles, slopes, widths, lengths, heights, and / or areas. Therefore, the processor 120 may obtain 3D-related information from the identified geometric shape 501.

[0075] 5B , the processor 120 may receive a second user input within a geometric shape 501 displayed in a 3D-based user interface. The second user input is used to generate a 3D object. For example, the processor 120 may receive a second user input of writing a text object 503 “HELLO” and display the text object 503 in the geometric shape 501 on the display 410.

[0076] 5C , the processor 120 may modify the shape of the text object 503 based on the geometric shape 501. For example, according to the acquired 3D related information of the geometric shape 501, the processor 120 may change the corresponding 3D related information of the text object 503.

[0077] 5D , the processor 120 may receive a third user input 510 associated with the text object 503. For example, the third user input may be a drag input for moving the text object 503 in the 3D-based user interface. Therefore, the processor 120 may move the text object 503 in response to the third user input 510. At this time, the geometric shape 501 may disappear from the display 410.

[0078] 5E, after the third user input 510 is released, the processor 120 may display the moved text object 503 on the display 410. As a result, the moved text object 503 may serve as one of the things included in the 3D-based user interface.

[0079] Figure 6 is a flowchart 600 of a method of generating a 3D object by utilizing pressure information according to an embodiment.

[0080] In step 601, the processor 120 of the electronic device 101 may receive a first user input associated with an object. The first user input may be a touch input and include pressure information associated with the touch input. The processor 120 may display a user interface (e.g., an AR interface) for generating an object through a display such as a touch screen display, and receive the first user input through the user interface.

[0081] In step 603, the processor 120 may obtain 3D related information from the received first user input including pressure information. For example, the 3D related information may include data on coordinates, angles, slopes, widths, lengths, heights, and / or areas for generating objects such as text or geometric shapes.

[0082] In step 605 , the processor 120 may generate an object based on the acquired 3D related information. In addition, the processor 120 may display the generated object on the display 160 and also store the generated object in the memory 130 .

[0083] In step 607, the processor 120 may receive a second user input associated with the object. The second user input may be a touch input and include pressure information associated with the touch input. For example, the second user input may be an input for moving or rotating the generated object in the user interface or for modifying the shape of the generated object.

[0084] In step 609, the processor 120 may process the object in response to the received second user input including pressure information. For example, the processor 120 may rotate the object displayed on the display 160 according to the second user input including pressure information. In an embodiment, the touch position of the second user input may determine the rotation axis of the object, and the pressure information may determine the rotation angle of the object. That is, as the pressure input increases, the rotation angle may increase.

[0085] In step 611, the processor 120 may transmit the processing result for the object to the server 108. For example, the processor 120 may store information about the rotation object in the memory 130 and transmit such information to the external electronic device (eg, the server 108).

[0086] 7A , 7B and 7C are diagrams of generating a 3D object by using pressure information according to an embodiment.

[0087] 7A, the processor 120 of the electronic device 400 may receive a user input associated with an object. The user input may be a touch input and include pressure information associated with the touch input. Therefore, upon receiving the user input, the processor 120 may extract the pressure information from the received user input.

[0088] For example, when it is assumed that the user input has pressure information within 120 pressure units, the processor 120 may predefine the pressure level according to the range of the pressure unit, as shown in the following Table 1. The following Table 1 shows an example in which four pressure levels are defined. In this example, the processor 120 may detect a specific pressure unit of the received user input by extracting the pressure information, and find a specific pressure level corresponding to the detected pressure unit by referring to the following Table 1.

[0089] [Table 1]

[0090]

[0091]

[0092] After finding the pressure level associated with the user input, the processor 120 can determine the Z-axis coordinate information (i.e., depth information) of the object to be generated based on the pressure level. For example, the processor 120 can establish a virtual coordinate system consisting of X, Y, and Z axes in a user interface (e.g., an AR interface) displayed on the display 410. When the received user input for generating an object (e.g., a text object "HELLO") is a touch input with a specific pressure level, the processor 120 can determine the X-axis coordinate and the Y-axis coordinate according to the position of the touch input, and also determine the Z-axis coordinate according to the pressure level. For example, when the pressure level is "1", the processor 120 can generate and display an object with a Z-axis coordinate "a" as shown in Figure 7A.

[0093] 7B , the user input for generating the text object “HELLO” may have different pressure levels for the respective letters of “HELLO.” For example, when the letter “H” is input at a pressure level of “1” and then the letter “E” is input at a pressure level of “2,” the processor 120 may generate and display the letter “H” to have a Z-axis coordinate of “a,” and also generate and display the letter “E” to have a Z-axis coordinate of “0.”

[0094] Similarly, referring to FIG. 7C , when the letters “H,” “L,” and “O” are input at a pressure level of “1” and the letters “E” and “L” are input at a pressure level of “2,” the processor 120 may generate and display the letters “H,” “L,” and “O” with Z-axis coordinates of “b,” and also generate and display the letters “E” and “L” with Z-axis coordinates of “0.”

[0095] As described herein, when displayed in a user interface, an object generated in response to a user input having a particular pressure level may have a depth determined according to the pressure level.

[0096] Figure 8is an illustration of generating a 3D object by using pressure information according to an embodiment.

[0097] refer to Figure 8 , the processor 120 of the electronic device 400 may receive a user input associated with the object. The user input may be a touch input and include pressure information associated with the touch input. Therefore, when receiving the user input, the processor 120 may extract the pressure information from the received user input.

[0098] For example, when it is assumed that the received user input is an input for constructing a heart shape, the input for drawing the left portion 810 of the heart shape and the input for drawing the right portion 820 of the heart shape may have different pressure levels. In this case, the processor 120 may calculate the pressure level difference between the two inputs, and determine the rotation angle of the heart shape based on the calculated difference. Table 2 below shows a related example.

[0099] [Table 2]

[0100] Differences in pressure levels Rotation Angle 0 0° 1 30° 2 60° 3 90°

[0101] For example, when the input for the left portion 810 may have a pressure level of "1", and when the input for the right portion 820 may have a pressure level of "2", the difference in pressure level becomes "1". Therefore, the processor 120 may determine the rotation angle to be 30° by referring to Table 2 above. Then, the processor 120 may rotate a portion of the heart shape having a larger pressure level by the determined rotation angle. That is, the right portion 820 of the heart shape having a larger pressure level is rotated by 30°.

[0102] As described herein, when displayed in a user interface, an object generated in response to user input having different pressure levels may be partially rotated according to the difference in pressure levels.

[0103] Fig. 9 is an illustration of generating a 3D object by using pressure information according to an embodiment.

[0104] refer to Fig. 9 , the processor 120 of the electronic device 400 may receive a user input for generating a text object "HELLO", generate the text object "HELLO" in response to the received user input, and display the generated text object on the display 410. In this case, the depth of the text object may vary according to the pressure information of the received user input.

[0105] exist Fig. 9In the three examples, according to pressure level "1", the first exemplary text object 910 may have a lower depth. On the other hand, according to pressure level "2", the second exemplary text object 920 may have a medium depth, and according to pressure level "3", the third exemplary text object 930 may have a higher depth.

[0106] 10A , 10B, 10C, and 10D are diagrams of rotating a 3D object in response to additional user input, according to an embodiment.

[0107] 10A , the processor 120 of the electronic device 400 may establish a virtual coordinate system consisting of X, Y, and Z axes in a user interface (e.g., an AR interface) displayed on the display 410. For example, the X axis may be set to correspond to the length A of the electronic device 400, and the Y axis may be set to correspond to the width B of the electronic device 400. In addition, the Z axis may be set to correspond to the vertical direction from the display 410. According to an embodiment, the processor 120 may display an object on the display 410. The processor 120 may then receive a user input to the displayed object, and rotate the object at least in part in response to the received user input.

[0108] The processor 120 may display a text object 1001 (e.g., "HELLO") on the display 410. Then, the processor 120 may receive a multi-touch input including a first user input 1011 and a second user input 1013 regarding the text object 1001. For example, the first user input 1011 may be a finger touch input, and the second user input 1013 may be a pen touch input.

[0109] In an embodiment, the processor 120 may determine the rotation axis of the object based on the first user input 1011, and may also determine the rotation direction of the object based on the second user input 1013. For example, when the first user input 1011 is detected from the left end of the object "HELLO" (i.e., the letter "H"), and when the second user input 1013 is detected from the right end of the object (i.e., the letter "O"), the processor 120 may determine the rotation axis of the object as the Y axis 1010 passing through the left end of the object based on the first user input 1011, and further determine the rotation direction of the object as the right direction on the rotation axis based on the second user input 1013. Here, the rotation angle may be determined in proportion to the duration of the second user input 1013 or the pressure level of the second user input 1013.

[0110] In an embodiment, when the first user input 1021 is detected from the lower end of the text object 1001 “HELLO”, and when the second user input 1023 is detected from the upper end of the object, the processor 120 may determine the rotation axis of the object as the X-axis 1020 passing through the lower end of the object based on the first user input 1021, and also determine the rotation direction of the object as the downward direction on the rotation axis based on the second user input 1023. Here, the rotation angle may be determined in proportion to the duration of the second user input 1023 or the pressure level of the second user input 1023.

[0111] 10B, the processor 120 may receive a multi-touch input including a first user input 1031 such as a finger touch input and a second user input 1033 such as a pen drag input. The processor 120 may then determine the rotation axis of the object based on the first user input 1031, and may also determine the rotation direction of the object based on the second user input 1033.

[0112] For example, when a first user input 1031 is detected from the left end of the object “HELLO” (i.e., the letter “H”), and when a second user input 1033 is detected from the right end of the object (i.e., the letter “O”), the processor 120 can determine the rotation axis of the object as the Z axis 1030 near the left end of the object based on the first user input 1031, and also determine the rotation direction 1032 of the object based on the second user input 1032 (dragging direction).

[0113] 10C , the processor 120 may receive a multi-touch input including a first user input 1041 (e.g., a finger touch input) and a second user input 1043 (e.g., a pen touch input) relative to a predetermined single area 1040 in the object 1001. In this case, the processor 120 may linearly move the displayed object 1001 in response to the received multi-touch input.

[0114] For example, when the first user input 1041 and the second user input 1043 are detected from the same area 1040 (i.e., the middle letter "L") of the object 1001 "HELLO", the processor 120 may linearly move the object 1001 along the Z axis, i.e., adjust the depth of the object. Here, the moving distance of the object or the adjustment depth may be determined in proportion to the duration of the first user input 1041 and the second user input 1043 or the pressure level of at least one of the first user input 1041 and the second user input 1043.

[0115] 10D, the processor 120 may receive a multi-touch input including a first user input 1051 such as a finger touch input and a second user input 1053 such as a pen touch input. The processor 120 may then determine the rotation axis of the object 1001 based on the first user input 1051, and may also determine the rotation direction of the object based on the second user input 1053.

[0116] For example, when the first user input 1051 is detected from a certain internal point of the object 1001 “HELLO” (e.g., the space between the letters “E” and “L”), and when the second user input 1053 is detected from the right end of the object (i.e., the letter “O”), the processor 120 may determine the rotation axis of the object as the Y axis 1030 passing through the internal point of the object based on the first user input 1051, and also determine the rotation direction of the object as the right direction on the rotation axis based on the second user input 1053. Here, the rotation angle may be determined in proportion to the duration of the second user input 1053 or the pressure level of the second user input 1053.

[0117] Fig.11 is a flowchart 1100 of a method of generating a 3D object by utilizing pen tilt information according to an embodiment.

[0118] In step 1101, the processor 120 of the electronic device 101 may receive a user input associated with an object. The user input may be a touch input of a stylus pen.

[0119] In step 1103, the processor 120 may acquire the tilt information of the pen. For example, the processor 120 may detect the tilt information of the pen based on electromagnetic waves that change according to the tilt of the pen. For example, the pen may have a sensor for detecting its tilt information and acquire the tilt information through the sensor. Therefore, the processor 120 may receive the tilt information of the pen from the pen.

[0120] In step 1105, the processor 120 may generate an object based on both the received user input and the acquired pen tilt information. For example, the processor 120 may determine the direction and slope of the object based on the pen tilt information. The processor 120 may then display the object to which the determined direction and slope are applied on the display.

[0121] 12A , 12B and 12C are diagrams of generating a 3D object by using pen tilt information according to an embodiment.

[0122] 12A , the processor 120 of the electronic device 400 may establish a virtual coordinate system consisting of X, Y, and Z axes in a user interface (e.g., an AR interface) displayed on the display 410. For example, the X axis may be set to correspond to the length A of the electronic device 400, and the Y axis may be set to correspond to the width B of the electronic device 400. In addition, the Z axis may be set to correspond to the vertical direction from the display 410.

[0123] According to an embodiment, the processor 120 may display an object 1201 (e.g., a text object "HELLO") on the display 410. Then, the processor 120 may receive a user input by the pen 1210 for the displayed object 1201, acquire tilt information of the pen 1210, and modify the object based on the acquired tilt information.

[0124] When the tilt information indicates that the pen 1210 is tilted 0 degrees relative to the Z axis (ie, the pen is perpendicular to the display 410), the processor 120 may display the object 1201 without any tilt.

[0125] 12B , when the tilt information indicates that the pen 1210 has a tilt of α° with respect to the X-axis 1221 , the processor 120 may display a text object 1203 “HELLO” having a tilt of α° with respect to the X-axis 1221 .

[0126] 12C , when the tilt information indicates that the pen 1210 has a tilt of β° with respect to the Y-axis 1223 , the processor 120 may display a text object 1205 “HELLO” having a tilt of β° with respect to the Y-axis 1223 .

[0127] According to various embodiments, an electronic device includes a housing; a touch screen display disposed within the housing and exposed through at least a portion of the housing; a camera; a wireless communication circuit disposed within the housing; a position sensor; a motion sensor; a processor disposed within the housing and operably connected to the display, the camera, the wireless communication circuit, the position sensor, and the motion sensor; and a memory disposed within the housing and operably connected to the processor. The memory may store instructions that, when executed, cause the processor to: capture an image using the camera, receive a captured image from the camera, receive position data associated with the captured image from the position sensor, receive direction data associated with the captured image from the motion sensor, receive user input for adding an object to the captured image through the display, and send the object, the captured image, the position data, and the direction data to an external server through the wireless communication circuit.

[0128] The processor may receive user input using the stylus.

[0129] The object may include at least one of text, a symbol, a geometric shape, or an image.

[0130] The processor may receive a first user input constructing a geometric shape on the captured image, and receive a second user input writing text on the captured image that at least partially overlaps the geometric shape.

[0131] The processor may change the direction of the text according to the geometry and send information about the changed direction to the server.

[0132] The processor may receive a third user input to move the text in the captured image and display the moved text on the display without displaying the geometric shape.

[0133] The processor may identify three-dimensional (3D) related information corresponding to the geometric shape, and change the direction of the text based on the identified 3D related information.

[0134] The 3D related information may include data of at least one of coordinates, angles, slopes, widths, lengths, heights, or areas corresponding to the geometric shapes.

[0135] The processor may receive a multi-touch input and rotate the object at least in part based on the received multi-touch input.

[0136] The multi-touch input may include a first touch input for determining a rotation axis of the object, and a second touch input for determining a rotation direction of the object.

[0137] The processor may receive tilt information of the stylus from the stylus and rotate the object at least in part based on the received tilt information.

[0138] The electronic device may further include a pressure sensor, and the processor may receive pressure information associated with the user input from the pressure sensor and generate the object based on the received pressure information.

[0139] The processor may generate an object according to the received user input and store the generated object in the memory.

[0140] According to various embodiments, a system includes a network interface; a processor operably connected to the network interface; and a memory operably connected to the processor. The memory may store instructions that, when executed, cause the processor to: receive first data from a first external electronic device including a display and a camera via the network interface, wherein the first data includes an image, position data associated with the image, orientation data associated with the image, and an object to be overlapped with the image generated in response to user input, process the first data based at least in part on the position data, orientation data, or user input received from the first external electronic device so that the orientation of the object is applied to the image, and send the processed first data to another external electronic device via the network interface.

[0141] The processor can receive an image, location data associated with the image, and orientation data associated with the image from a second external electronic device including a display and a camera via a network interface, and send an object to the second external electronic device via the network interface so that the object is displayed on the display of the second external electronic device, wherein the orientation data is applied to the displayed object.

[0142] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0143] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” when the terms “operably” or “communicatively” are used or when the terms “operably” or “communicatively” are not used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0144] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0145] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0146] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) The computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0147] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0148] While the invention has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.

Claims

1. An electronic device, comprising: case; a touch screen display disposed within the housing and exposed through at least a portion of the housing; camera; A wireless communication circuit, wherein the wireless communication circuit is disposed in the housing; Position sensor; Motion sensor; Pressure sensor; a processor disposed within the housing and operably connected to the touch screen display, the camera, the wireless communication circuitry, the position sensor, the motion sensor, and the pressure sensor; as well as a memory disposed within the housing and operably connected to the processor, The memory is configured to store instructions which, when executed, cause the processor to: capturing an image using the camera, receiving a captured image from the camera, receiving position data associated with the captured image from the position sensor, receiving position data associated with the captured image from the motion sensor, receiving a first user input to construct a geometric shape on the captured image; receiving, via the touch screen display, a second user input writing text on the captured image that at least partially overlaps the geometric shape, wherein the text includes a plurality of letters, and wherein the second user input is a touch input and includes pressure information associated with the touch input, changing the orientation of the text according to the geometric shape, extracting the pressure information from the received second user input and finding a pressure level associated with the second user input, determining depth information of text to be added to the image based on the pressure level, wherein the depth information is determined for each of a plurality of letters of the text based on received pressure information for each of a plurality of letters of the text, generating and displaying each of the plurality of letters of the text based on the depth information determined for each of the plurality of letters, and The text, the captured image, the position data, the orientation data, and the geometric shape are transmitted to an external server through the wireless communication circuit.

2. The electronic device according to claim 1, wherein: When the instructions are executed, the processor further causes the processor to receive the first user input or the second user input using a stylus, The first user input or the second user input further includes at least one of a symbol, a geometric shape and an image.

3. The electronic device according to claim 1, wherein: The instructions, when executed, further cause the processor to: receiving a third user input to move the text in the captured image, and The moved text is displayed on the touch screen display without displaying the geometric shape.

4. The electronic device according to claim 1, wherein: The instructions, when executed, further cause the processor to: identifying 3D related information corresponding to the geometric shape, and Based on the identified 3D related information, changing the direction of the text, The 3D related information includes data on at least one of coordinates, angles, slopes, widths, lengths, heights and areas corresponding to the geometric shapes.

5. The electronic device according to claim 1, wherein: The instructions, when executed, further cause the processor to: Receive multi-touch input, and rotating the text at least partially based on the received multi-touch input, The multi-touch input includes a first touch input for determining a rotation axis of the text, and a second touch input for determining a rotation direction of the text.

6. The electronic device according to claim 2, wherein: The instructions, when executed, further cause the processor to: receiving tilt information from the stylus, and The text is at least partially rotated based on the received tilt information.

Images