Electronic device for providing three-dimensional virtual object related to product, and operation method thereof

Abstract

The present invention relates to a method by which an electronic device displays a three-dimensional virtual object. The method comprises the steps of: generating a three-dimensional virtual object on the basis of a user input; and combining the three-dimensional virtual object with a real image and displaying same in augmented reality, wherein the display size of the three-dimensional virtual object is determined on the basis of the real size of the three-dimensional virtual object and the real size of a real object in the real image.

Description

Electronic device for providing a three-dimensional virtual object related to a product and its operating method

[0001] The present disclosure relates to an electronic device and an operating method thereof for providing a three-dimensional virtual object related to a product.

[0002] Augmented Reality (AR) is a technology that enhances the user experience by adding virtual elements (e.g., 3D virtual objects) to the real world. This technology provides rich user interactions by superimposing or synthesizing visual information onto the real world, and can be applied in a variety of fields. In particular, the retail industry is exploring ways to utilize AR to provide product information and enhance interaction with consumers.

[0003] According to various embodiments of the present disclosure, a technical problem is to provide a three-dimensional virtual object displayed in augmented reality by synthesizing it with an actual image based on an actual object in an actual image, thereby enabling a user to easily recognize the size of the three-dimensional virtual object.

[0004] According to various embodiments of the present disclosure, a technical problem is that a processor determines a display size of a three-dimensional virtual object based on an actual size of the three-dimensional virtual object and an actual size of the actual object in a real image.

[0005] According to various embodiments of the present disclosure, a technical challenge is to generate a three-dimensional virtual object even when an electronic device does not include a lidar sensor.

[0006] According to various embodiments of the present disclosure, a technical problem is to determine a display size of a three-dimensional virtual object based on a magnification determined by a processor according to zoom-in and zoom-out.

[0007] According to various embodiments of the present disclosure, a technical problem is to have a processor display various product information (e.g., delivery type, price discount, etc.) around a three-dimensional virtual object.

[0008] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.

[0009] A method for displaying a three-dimensional virtual object performed by an electronic device according to one embodiment may include the steps of generating a three-dimensional virtual object based on a user's input; and the steps of synthesizing the three-dimensional virtual object with an actual image and displaying it as augmented reality. The display size of the three-dimensional virtual object according to one embodiment may be determined based on the actual size of the three-dimensional virtual object and the actual size of the actual object within the actual image.

[0010] The step of generating a three-dimensional virtual object according to one embodiment may include the step of generating at least one of a set of images acquired from two or more different angles based on a user input, lidar sensor data, or an actual size of the three-dimensional virtual object; and the step of generating the three-dimensional virtual object based on at least one of the set of images, lidar sensor data, or an actual size.

[0011] The step of displaying a three-dimensional virtual object in augmented reality by synthesizing it with an actual image according to one embodiment may include a step of displaying a texture of the three-dimensional virtual object on a surface of the three-dimensional virtual object.

[0012] The step of generating a three-dimensional virtual object according to one embodiment may include the step of determining at least one of an actual size or shape of the three-dimensional virtual object based on a user input.

[0013] The step of generating a three-dimensional virtual object according to one embodiment may include the step of transmitting to a server at least one of a set of images acquired from two or more different angles generated based on a user input, lidar sensor data, or an actual size of the three-dimensional virtual object; and the step of receiving from the server the three-dimensional virtual object generated based on at least one of the set of images, lidar sensor data, or an actual size.

[0014] The method according to one embodiment may further include a step of determining whether the actual image includes a reference object having a predetermined actual size and shape.

[0015] The step of displaying a three-dimensional virtual object in augmented reality by synthesizing it with an actual image according to one embodiment may include a step of determining a display size of the three-dimensional virtual object based on an actual size of the three-dimensional virtual object and an actual size of the reference object, when a reference object is included in the actual image.

[0016] The step of displaying a three-dimensional virtual object in augmented reality by synthesizing it with an actual image according to one embodiment may include a step of determining a position at which to display the three-dimensional virtual object based on a position of a reference object.

[0017] The step of displaying a three-dimensional virtual object in augmented reality by synthesizing it with a real image according to one embodiment may include a step of moving or rotating the position of the three-dimensional virtual object in the real image based on a user input.

[0018] The step of displaying a three-dimensional virtual object in augmented reality by synthesizing it with an actual image according to one embodiment may include the step of adjusting the magnification of a camera included in an electronic device based on a user input; and the step of changing the display size of the three-dimensional virtual object based on the adjusted magnification.

[0019] According to one embodiment, the method may further include a step of displaying, within a certain distance from the three-dimensional virtual object, at least one of delivery type data of a product, price discount data of the product, review data of the product, or a purchase object corresponding to the three-dimensional virtual object.

[0020] Delivery type data according to one embodiment is data related to a delivery form, and the delivery form may include at least one of general delivery, expedited delivery, fresh food delivery, or international delivery.

[0021] According to one embodiment, the method may further include the step of displaying, in a real image, product sales list data of a seller account selling a product corresponding to a three-dimensional virtual object; or the step of displaying, in a real image, recommended product data having the same product and product category.

[0022] According to various embodiments of the present disclosure, by presenting information about products to customers using augmented reality technology, consumers can check the products in a virtual environment as if they were actually experiencing the products and make purchasing decisions.

[0023] According to various embodiments of the present disclosure, an electronic device displays a three-dimensional virtual object corresponding to a product to be purchased in augmented reality, so that a user can easily compare the size of a three-dimensional real object included in an actual image with the size of the product to be purchased.

[0024] According to various embodiments of the present disclosure, even if an electronic device does not have a lidar sensor, a three-dimensional virtual object can be generated based on images captured from multiple angles through a camera.

[0025] According to various embodiments of the present disclosure, even when zooming in or out on an actual image, the display size of a three-dimensional virtual object is adjusted to match the display size of the actual objects according to the magnification, thereby allowing a user to compare the size of a three-dimensional actual object included in an actual image with the size of a product to be purchased.

[0026] The effects according to the technical idea of ​​the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the specification.

[0027] FIG. 1 is a diagram illustrating a system according to various embodiments of the present disclosure.

[0028] FIG. 2 is a block diagram illustrating an electronic device according to various embodiments of the present disclosure.

[0029] FIG. 3 is a drawing for explaining a method of displaying a three-dimensional virtual object in augmented reality by synthesizing it with an actual image according to various embodiments of the present disclosure.

[0030] FIG. 4 is a drawing for explaining a screen for creating a three-dimensional virtual object according to various embodiments of the present disclosure.

[0031] FIG. 5 is a diagram illustrating a method for acquiring lidar sensor data for generating a three-dimensional virtual object according to various embodiments of the present disclosure.

[0032] FIG. 6 is a diagram illustrating a method for obtaining images of multiple angles for generating a three-dimensional virtual object according to various embodiments of the present disclosure.

[0033] FIGS. 7A and 7B are drawings for explaining a three-dimensional virtual object according to the actual size or shape of the three-dimensional virtual object determined based on user input according to various embodiments of the present disclosure.

[0034] FIG. 8 is a flowchart of operations of an electronic device according to various embodiments of the present disclosure.

[0035] The embodiments of this disclosure are provided for the purpose of illustrating the technical concepts of this disclosure. The scope of rights under this disclosure is not limited to the embodiments presented below or the specific descriptions of these embodiments.

[0036] All technical and scientific terms used in this disclosure, unless otherwise defined, have the meanings commonly understood by those of ordinary skill in the art to which this disclosure pertains. All terms used in this disclosure have been selected for the purpose of more clearly explaining this disclosure and are not intended to limit the scope of rights under this disclosure.

[0037] Expressions such as “including,” “comprising,” “having,” and the like used in this disclosure should be understood as open-ended terms that imply the possibility of including other embodiments, unless otherwise stated in the phrase or sentence in which the expression is included.

[0038] The singular forms described in this disclosure may include plural meanings unless otherwise stated, and the same applies to the singular forms described in the claims.

[0039] The expressions “first,” “second,” etc. used in this disclosure are used to distinguish between multiple components, and do not limit the order or importance of the components.

[0040] The term "unit" as used herein refers to software or hardware components such as field-programmable gate arrays (FPGAs) and application specific integrated circuits (ASICs). However, "units" are not limited to hardware and software. "Units" may be configured to reside on an addressable storage medium, and may be configured to execute one or more processors. Thus, by way of example, "units" include components such as software components, object-oriented software components, class components, and task components, as well as processors, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within a component and "unit" may be combined into a smaller number of components and "units" or further separated into additional components and "units."

[0041] The expression "based on" as used in this disclosure is used to describe one or more factors that influence the decision, act of judgment, or action described in the phrase or sentence containing the expression, and this expression does not exclude additional factors that influence the decision, act of judgment, or action.

[0042] In this disclosure, when a component is referred to as being "connected" or "connected" to another component, it should be understood that the component can be directly connected or connected to the other component, or can be connected or connected via a new other component.

[0043] Hereinafter, embodiments of the present disclosure will be described with reference to the attached drawings. In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, redundant descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0044] Figure 1 illustrates an environment in which an electronic device according to one embodiment of the present disclosure can be applied. An electronic device (110) and a user terminal (120, hereinafter referred to as a "terminal") are connected via a network and can communicate with each other. The electronic device (110) may be a server or a different terminal from the terminal (120).

[0045] The terminal (120) may be implemented as a terminal capable of transmitting and receiving various information to and from the electronic device (110) via a network. For example, the terminal (120) may be one of a computer, a laptop, a portable communication terminal (such as a smartphone), a portable multimedia device, a wearable device, or an HMD. However, the type of the terminal (120) is not limited thereto, and the terminal (120) may be any device that includes an input / output interface capable of receiving information from a user or outputting information to a user, and that can communicate with the electronic device (110) or other devices via a network. The terminal (120) may provide information received from the electronic device (110) to the user, and may receive input from the user and transmit it to the electronic device (110). The input obtained from the user may include various forms of input, such as a click using a mouse, a touch using a touchpad or a touch screen, voice recognition, or other electronic input.

[0046] A network may serve to connect an electronic device (110) and a terminal (120) or other external device. For example, the network may provide a connection path so that the terminal (120) can be connected to the electronic device (110) and transmit and receive packet data with the electronic device (110). The network may be implemented as any type of wired or wireless network, such as a Local Area Network (LAN), a Wide Area Network (WAN), a Mobile Radio Communication Network, or Wibro (Wireless Broadband Internet).

[0047] FIG. 2 is a block diagram of an electronic device according to an embodiment of the present disclosure. In one embodiment, the electronic device (200) may be a server or a terminal. In one embodiment, the electronic device (200) may include a communication circuit (210), one or more processors (220), and / or one or more memories (230) as components. In one embodiment, at least one of the components of the electronic device (200) may be omitted, or another component may be added to the electronic device (200). In one embodiment, additionally or alternatively, some of the components may be implemented in an integrated manner, or may be implemented as a single or multiple entities. In the present disclosure, one or more processors (220) may be expressed as a processor (220). The expression “processor (220)” may mean a set of one or more processors, unless the context clearly indicates otherwise. In the present disclosure, one or more memories (230) may be expressed as a memory (230). The expression "memory (230)" may mean a set of one or more memories, unless the context clearly indicates otherwise. In one embodiment, at least some of the components inside / outside the electronic device (200) may be connected to each other via a bus, a General Purpose Input / Output (GPIO), a Serial Peripheral Interface (SPI), or a Mobile Industry Processor Interface (MIPI), and may exchange information (data, signals, etc.).

[0048] The communication circuit (210) can communicate with the user's terminal and an external device. The communication circuit (210) can perform wireless or wired communication between the electronic device (200) and the terminal. For example, the communication circuit (210) can perform wireless communication according to a method such as eMBB (enhanced Mobile Broadband), URLLC (Ultra Reliable Low-Latency Communications), MMTC (Massive Machine Type Communications), LTE (Long-Term Evolution), LTE-A (LTE Advance), NR (New Radio), UMTS (Universal Mobile Telecommunications System), GSM (Global System for Mobile communications), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), WiBro (Wireless Broadband), WiFi (Wireless Fidelity), Bluetooth (Bluetooth), NFC (Near Field Communication), GPS (Global Positioning System), or GNSS (Global Navigation Satellite System). For example, the communication circuit (210) may perform wired communication according to a method such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard-232 (RS-232), or Plain Old Telephone Service (POTS). In one embodiment, the electronic device (200) may be implemented by being integrated with another device. In this case, the communication circuit (210) may function as a connection circuit or interface connecting the electronic device (200) and the other device.

[0049] The processor (220) can control at least one component of the electronic device (200) connected to the processor (220) by running software (e.g., commands, programs, etc.). In addition, the processor (220) can perform various operations related to the present disclosure, such as calculations, processing, data generation, and processing. In addition, the processor (220) can load data from the memory (230) or store data in the memory (230). Furthermore, the processor (220) can transmit and receive various information with the user's terminal and external devices through the communication circuit (210). In one embodiment, the processor (220) can control the communication circuit (210) to transmit various information, such as information on various pages, to the user's terminal.

[0050] The memory (230) can store various information (data). The information stored in the memory (230) is information acquired, processed, or used by at least one component of the electronic device (200), and may include software (e.g., commands, programs, etc.). The memory (230) may include volatile and / or non-volatile memory. In the present disclosure, the commands or programs are software stored in the memory (230), and may include an operating system for controlling the resources of the electronic device (200), an application, and / or middleware for providing various functions to the application so that the application can utilize the resources of the electronic device (200). In one embodiment, the memory (230) may store commands that, when executed by the processor (220), cause the processor (220) to perform operations. The memory (230) may store at least a portion of information received from the terminal via the communication circuit (210) and / or information transmitted to the terminal via the communication circuit (210). The processor (220) can store at least a portion of information received from the terminal through the communication circuit (210) and / or information transmitted to the terminal through the communication circuit (210) in the memory (230).

[0051] Hereinafter, the operations described as being performed by the electronic device in FIGS. 3 to 8 can be understood as being performed by the processor (220) of the electronic device (200) described in FIG. 2.

[0052] Although the process steps, method steps, algorithms, etc., are described in a sequential order in the schematics and flowcharts illustrated in FIGS. 3 and 8, such processes, methods, and algorithms may be configured to operate in any suitable order. In other words, the steps of the processes, methods, and algorithms described in various embodiments of the present disclosure need not be performed in the order described herein. Furthermore, even if some steps are described as being performed asynchronously, in other embodiments, such some steps may be performed concurrently. Furthermore, the illustration of a process by depiction in the drawings does not imply that the illustrated process excludes other variations and modifications thereof, nor does it imply that the illustrated process or any of its steps is essential to one or more of the various embodiments of the present disclosure, nor does it imply that the illustrated process is preferred.

[0053] FIG. 3 is a drawing for explaining a method of displaying a three-dimensional virtual object in augmented reality by synthesizing it with an actual image according to various embodiments of the present disclosure.

[0054] According to one embodiment, the processor (220) may generate a three-dimensional virtual object based on a user's input. When the electronic device (200) according to one embodiment is a terminal, the terminal may generate a three-dimensional virtual object based on the user's input. The processor (220) may generate at least one of an image set acquired from two or more different angles, lidar sensor data, or the actual size (and / or shape) of the three-dimensional virtual object based on the user's input. The processor (220) may generate the three-dimensional virtual object based on at least one of the image set, lidar sensor data, or the actual size (and / or shape). In another embodiment, the processor (220) may transmit data generated based on the user's input to a server. The server may generate a three-dimensional virtual object based on the received data and transmit the same to the terminal. The processor (220) included in the terminal may display the three-dimensional virtual object received from the server on the screen.

[0055] According to one embodiment, a user input may correspond to a request to create a three-dimensional virtual object for a specific product. For example, referring to FIG. 4, when receiving a user input for at least one of a scan object (430), an image upload object (450), or a size input object (470), the processor (220) may generate data corresponding to the user input and create a three-dimensional virtual object based on the generated data.

[0056] According to one embodiment, the processor (220) can generate a three-dimensional virtual object based on a user's input for a scan object (430). The scan object (430) may be an object for generating a three-dimensional virtual object based on lidar sensor data. The lidar sensor data can be obtained from a lidar sensor. A lidar (Light Detection and Ranging) sensor is a technology that measures distance and detects the environment using the wavelength of light. A lidar sensor mainly calculates distance by emitting a laser beam and measuring the time it takes for the beam to reflect off an object and return, thereby generating lidar sensor data (e.g., spatial information of the surrounding environment, the size of an object, the distance to the object, the shape of the object).

[0057] When a scan object (430) is selected in the screen (400) of FIG. 4, the screen (500) of FIG. 5 may be displayed. Referring to FIG. 5, the screen (500) is a screen that displays guide lines on an actual image based on lidar sensor data. The guide lines may be lines for representing spatial information of the surrounding environment. When multiple guide lines intersect, grids may be created at regular intervals, and the grid length may correspond to a specific length (e.g., 1 cm, 2 cm). The user can know the length of a specific object based on the number of grids. The processor (220) may calculate the distance between the electronic device (200) and the actual object (510) based on the lidar sensor data. Through this, the processor (220) may calculate the actual size (e.g., width, length, height, etc.) of the object (510). Similarly, the above-described operation is performed on the real object (530), so that the processor (220) can calculate the real size of the real object (530). The processor (220) can generate a three-dimensional virtual object based on lidar sensor data captured at multiple different angles for the real object (510). For example, a user can capture various sides, such as the front, back, left side, and right side, of the real object (510) while moving the electronic device (200). Through this, the processor (220) can obtain lidar sensor data for various sides of the real object (510).

[0058] The term “actual size” used in this specification refers to the size of an object in an actual environment, and the “display size” may refer to the size of an object displayed on a screen within an actual image displayed on a display unit of an electronic device (200).

[0059] According to one embodiment, the processor (220) may display a texture of a 3D virtual object on the surface of a virtual object generated based on the actual size and shape of the 3D virtual object included in the lidar sensor data. A texture is an image or pattern used in computer graphics, and may be data applied to the surface of a 3D model to determine the appearance of the object. The texture may be mapped to the surface of the object to simulate a specific material, color, uniformity, texture, etc., to provide visual realism or to create a specific design. The texture may be generated based on an image acquired through a camera included in the electronic device (200). The processor (220) may generate the size and shape of the 3D virtual object based on the lidar sensor data. However, a texture may be required to display the surface of the 3D virtual object identically to an actual product. Therefore, the processor (220) may display the surface of the 3D virtual object on the surface using the texture. Through this, the processor (220) may provide a more realistic 3D virtual object to the user. By displaying textures on the surface of objects, users can compare the colors of real objects in the actual image with those of 3D virtual objects to determine whether they harmonize. Therefore, when making purchase decisions, users can consider not only the size of a product but also its color, determining whether it harmonizes with its surroundings.

[0060] According to one embodiment, the processor (220) may generate a three-dimensional virtual object based on a user's input for an image upload object (450). The image upload object (450) may be an object for generating a three-dimensional virtual object based on a set of images acquired from two or more different perspectives. The electronic device (220) may or may not include a lidar sensor. Even when the electronic device (200) does not include a lidar sensor, there may still be a need to generate a three-dimensional virtual object. Therefore, without lidar sensor data, the processor (220) may generate a three-dimensional virtual object based on a set of images acquired from two or more different perspectives. The processor (220) may generate an actual size of the three-dimensional virtual object based on depth data (e.g., the distance between the electronic device and the object) acquired by analyzing images included in the set of images. In addition, the processor (220) may know the shape of the three-dimensional virtual object based on the depth information. The processor (220) may form a virtual object based on the actual size and shape of the 3D virtual object, and then display the surface of the virtual object based on a texture. The texture may be generated based on a set of images. In another embodiment, the processor (220) may transmit the set of images to a server. The server may generate data necessary for generating a 3D virtual object based on the set of images. The processor (220) may generate a 3D virtual object using the data necessary for generating the 3D virtual object received from the server. Alternatively, the server may generate the 3D virtual object based on the set of images.

[0061] According to one embodiment, the processor (220) can generate a three-dimensional virtual object based on a user's input for a size input object (470). The size input object (470) may be an object for generating a three-dimensional virtual object without a lidar sensor data and image set. When the size input object (470) of FIG. 4 is selected, the screen (700) of FIG. 7A may be switched. The processor (220) can receive the shape (710) (e.g., cylinder, rectangular parallelepiped, triangular pyramid, etc.) and actual size (720, 721, 722) of the three-dimensional virtual object from the user through the screen (700). The actual size may include, for example, a width (720), a length (721), and a height (722). The processor (220) can determine the actual size (720, 721, 722) and the shape (710) based on the user's input. The processor (220) can then generate a 3D virtual object based on the determined actual size (720, 721, 722) and shape (710). In reality, the actual size of the 3D virtual object acquired through a lidar sensor or camera may not be accurate. Alternatively, the environment may be difficult to acquire from the lidar sensor or image set. Even in such cases, the user can input the actual size of the product displayed in the product details to generate a 3D virtual object with an accurate size and shape similar to the product.

[0062] According to one embodiment, the processor (220) can display a three-dimensional virtual object generated based on the actual size (720, 721, 722) and shape (710) by synthesizing it with an actual image and displaying it as augmented reality. Referring to FIG. 7B, not only can a real object (750) be displayed on a screen (701) of an actual image captured by a camera of the electronic device (220), but a three-dimensional virtual object (730) can also be synthesized and displayed simultaneously. Through this, the processor (220) can display a three-dimensional virtual object whose size can be compared with an actual object in an actual image without using a lidar sensor or a camera. In addition, a user can compare the size of a product corresponding to the three-dimensional virtual object with another actual object by comparing the three-dimensional virtual object of an accurate size with the actual object. In one embodiment, the user can input texture information of the three-dimensional virtual object together so that the color or texture of the three-dimensional virtual object can be displayed together.

[0063] The screens of FIGS. 4, 5, 7a, and 7b described above may be screens of a terminal connected to a seller account (or administrator account). The terminal of the seller account can create a 3D virtual object for an e-commerce platform and upload it along with product information. The terminal of the seller account can synthesize the 3D virtual object with an actual image and display it in augmented reality. Through this, the seller can check whether the product sold by the seller is accurately represented as a 3D virtual object within the actual image. For example, the screen (300) of FIG. 3 may be displayed on the terminal of the seller account. Through this, the seller can check whether the 3D virtual object (310) corresponding to the product he or she sells is well expressed to represent the product characteristics.

[0064] In one embodiment, a three-dimensional virtual object corresponding to a product needs to be displayed to a consumer in augmented reality by synthesizing it with a real image. To this end, a three-dimensional virtual object generated corresponding to a product can be transmitted to a server of an e-commerce platform. The server can receive a request for a three-dimensional virtual object for a specific product from a terminal of a consumer account. The server can transmit the three-dimensional virtual object to the terminal of the consumer account. The screen of the terminal of the consumer account can display the real object and the three-dimensional virtual object, as shown in screen (300) of FIG. 3. For example, a terminal of a seller account can generate a three-dimensional virtual object of product A. The terminal of the seller account can transmit the three-dimensional virtual object of product A to the server of the e-commerce platform. The terminal of the consumer account can transmit a request for a three-dimensional virtual object for product A to the server. The server can transmit the three-dimensional virtual object for product A to the terminal of the consumer account. The terminal of the consumer account can display the received three-dimensional virtual object for product A on a screen (e.g., an actual image).

[0065] In one embodiment, the processor (220) can determine whether a reference object having a predetermined actual size and shape is included in the actual image. A reference object may refer to an object whose actual size and shape data is stored in memory (or a server memory). For example, the reference object may include items commonly encountered in real life. For example, cans, tins, boxes of snacks, mobile phones, books, electrical outlets, etc. are commonly encountered in real life. After pre-storing data regarding the actual size and shape of such items in memory, the processor (220) can compare the actual size of the reference object with the actual size of a 3D virtual object when the reference object is present in the actual image, thereby more accurately displaying the 3D virtual object on the screen in augmented reality. The processor (220) can retrieve the actual size and shape of the reference object from the memory included in the electronic device or request the actual size and shape of the reference object from a server. For example, referring to FIG. 3, an electrical outlet can be a reference object because it is commonly encountered in real life. Accordingly, the processor (220) can determine the display size of the three-dimensional virtual object (310) based on the actual size of the reference object A (330) corresponding to the outlet and the actual size of the three-dimensional virtual object. The size of the three-dimensional virtual object (310) can be determined based on the display position of the three-dimensional virtual object. For example, when the three-dimensional virtual object (310) is positioned at a position close to the user, the display size of the three-dimensional virtual object (310) can be larger than when the three-dimensional virtual object (310) is positioned at a position far from the user. The display position of the three-dimensional virtual object (310) can be determined and / or changed by the user. As another example, a canned food can also be a reference object.The processor (220) can determine the display size of the three-dimensional virtual object (310) based on the actual size of the reference object B (370) corresponding to the can and the actual size of the three-dimensional virtual object.

[0066] According to one embodiment, the processor (220) may determine the display size of the 3D virtual object by comparing the actual size of the reference object and the actual size of the 3D virtual object when the actual image includes a reference object. When the actual image does not include the reference object, the processor (220) may determine the display size of the 3D virtual object based on the actual size of the real object in the actual image and the actual size of the 3D virtual object. The actual size of the real object may be calculated based on the display size of the real object in the real image and the distance from the electronic device (200) to the real object.

[0067] According to one embodiment, the processor (220) may determine a location at which to display a three-dimensional virtual object based on the location of a reference object. For example, the processor (220) may identify the location of the reference object within an actual image. Based on the location of the identified reference object, the processor (220) may determine a location at which to display a three-dimensional virtual object within the actual image. According to one embodiment, when the location of the reference object within the actual image is located at the bottom of the actual image, the processor (220) may display the three-dimensional virtual object at a location a certain distance upward from the location of the actual reference object. Referring to FIG. 3, the reference object B (370) may be located at the bottom of the actual image. Accordingly, the processor (220) may display the three-dimensional virtual object (310) at a location a certain distance upward from the location of the reference object B (370). According to another embodiment, when the location of the reference object within the actual image is located at the top of the actual image, the processor (220) may display the three-dimensional virtual object at a location a certain distance downward from the location of the actual reference object. Referring to FIG. 3, the reference object A (330) may be positioned at the top of the actual image. Accordingly, the processor (220) may display a three-dimensional virtual object (310) at a position a certain distance downward from the position of the reference object A (330). Through this, the processor (220) may display both the reference object and the three-dimensional virtual object within the actual image.

[0068] According to one embodiment, the processor (220) may move or rotate the position of a three-dimensional virtual object in an actual image based on a user input. According to one embodiment, the processor (220) may display the three-dimensional virtual object (310) in a predetermined position (e.g., the center within the actual image). According to one embodiment, the processor (220) may determine the predetermined position by considering the position at which the actual object is displayed within the actual image. For example, the processor (220) may determine the predetermined position so that the actual object and the three-dimensional virtual object do not overlap. According to one embodiment, the processor (220) may move the position of the three-dimensional virtual object (310) based on a user input. In addition, the processor (220) may rotate the three-dimensional virtual object (310) based on the user input. Through this, the user may simulate the placement of products (e.g., furniture, electronic products, etc.) by placing products corresponding to the three-dimensional virtual object within the actual image.

[0069] In one embodiment, the processor (220) can adjust the magnification of a camera included in the electronic device based on user input. For example, the electronic device can be zoomed in or out based on the user input. Zooming in can mean increasing the magnification, and zooming out can mean decreasing the magnification. The processor (220) can change the display size of a 3D virtual object based on the adjusted magnification. When the magnification is adjusted, the display size of the actual object in the actual image can change. Since the display size of the actual object changes, the display size of the 3D virtual object also needs to change. The processor (220) can know the ratio of the actual size of the actual object to the display size. Therefore, the processor (220) can change the display size of the 3D virtual object based on the ratio of the actual size of the actual object to the display size. Through this, even if the magnification changes, the actual size of the 3D virtual object changes, so that the user can accurately compare the sizes of the actual object and the 3D virtual object.

[0070] According to one embodiment, the processor (220) may display at least one of product delivery type data, product price discount data, product review data, or purchase object corresponding to a 3D virtual object within a certain distance from the 3D virtual object. Referring to FIG. 3, delivery type data (381) may be displayed within a certain distance from the 3D virtual object (310). The delivery type refers to the type of product delivery, and may include, for example, general delivery, expedited delivery, fresh food delivery, and overseas delivery. Expedited delivery may be a delivery type in which the average delivery time is shorter than that of Japanese delivery and delivery is completed within a guaranteed time (e.g., 1 day). Fresh food delivery may be a type in which products are delivered through packaging to maintain the freshness of the food for fresh food delivery. Overseas delivery may be a type in which products traded overseas are imported into Korea and delivered to customers. Since the delivery type is a factor that directly influences a user's decision to purchase a product, by displaying delivery type data (381) along with a three-dimensional virtual object, the user can intuitively check the delivery type and make a purchase decision. In addition, by displaying price discount data (380) and review data, which influence the user's decision to purchase a product, on the screen, the user can comprehensively grasp the information necessary for purchasing a product through a single screen (300). According to one embodiment, the processor (220) may also display a purchase object on the screen (300). When a purchase object is selected, the user can purchase the product corresponding to the three-dimensional virtual object (310). This simplifies the user's product purchase process. Only some of the aforementioned data may be displayed on the screen (300), or other data may be additionally displayed on the screen (300), and the present disclosure is not limited thereto.

[0071] In one embodiment, the processor (220) can display not only the product corresponding to the 3D virtual object but also other products on the actual image (or screen). In one embodiment, the processor (220) can display product sales list data of a seller account selling the product. For example, there may be a seller account A selling a product corresponding to the 3D virtual object (310). Seller account A may also sell other products. The processor (220) can also display other products sold by seller account A on the actual image. For example, referring to FIG. 3, the processor (220) can display other products (391, 392, 393) sold by seller account A in an area where other products (390) are displayed. This allows a user to compare other products sold by a specific seller account with the product corresponding to the 3D virtual object. In another embodiment, the processor (220) can display recommended product data where the product and product category are the same. The recommended product data may be data determined based on user preference data. User preference data can be generated based on user product purchase log data. For example, if the product corresponding to the 3D virtual data is milk, the recommended product data may include other products within the milk category. For example, if the user preference data indicates that user A prefers to purchase less than 1 L of milk, other milks less than 1 L may be included in the recommended product data. Referring to FIG. 3, the processor (220) can display products (391, 392, 393) included in the recommended product data in an area where other products (390) are displayed. This allows the user to view other products within the actual image without having to navigate to another screen, thereby simplifying the online shopping process.

[0072] FIG. 4 is a drawing for explaining a screen for creating a three-dimensional virtual object according to various embodiments of the present disclosure.

[0073] The screen (400) of FIG. 4 may include a three-dimensional virtual object display area (410). The three-dimensional virtual object display area (410) may display a generated three-dimensional virtual object. The three-dimensional virtual object displayed in the three-dimensional virtual object display area (410) may be a three-dimensional virtual object generated based on a user input for at least one of a scan object (430), an image upload object (450), or a size input object (470).

[0074] FIG. 5 is a diagram illustrating a method for acquiring lidar sensor data for generating a three-dimensional virtual object according to various embodiments of the present disclosure. Since a detailed description of FIG. 5 has been described above with reference to FIG. 3, a detailed description thereof is omitted in this drawing.

[0075] FIG. 6 is a diagram illustrating a method for obtaining images of multiple angles for generating a three-dimensional virtual object according to various embodiments of the present disclosure.

[0076] According to one embodiment, the processor (220) may generate a three-dimensional virtual object based on a set of images acquired from two or more different perspectives. For example, in order to generate a three-dimensional virtual object for a product (610), the processor (220) may acquire a set of images (611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622) captured from two or more different perspectives by a camera. The image set illustrated in FIG. 6 is merely an example, and the number of images included in the image set may be less or more than that illustrated in FIG. 6. The processor (220) can generate a three-dimensional virtual object for the product (610) based on a set of images (611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622).

[0077] FIGS. 7A and 7B are diagrams illustrating a three-dimensional virtual object according to its actual size or shape determined based on user input according to various embodiments of the present disclosure. Since a detailed description of FIGS. 7A and 7B has been described above with reference to FIG. 3, a detailed description thereof is omitted in the present drawings.

[0078] FIG. 8 is a flowchart of operations of an electronic device according to various embodiments of the present disclosure.

[0079] An electronic device (200) according to one embodiment can generate (810) a three-dimensional virtual object based on a user input. The user input can be an input for at least one of a scan object (430), an image upload object (450), or a size input object (470). The electronic device (200) can generate the three-dimensional virtual object based on at least one of lidar sensor data, an image set, or an actual size (and / or shape) of the three-dimensional virtual object. Some or all of the operations during the process of generating the three-dimensional virtual object may be performed on a server.

[0080] An electronic device (200) according to one embodiment can display (820) a three-dimensional virtual object by synthesizing it with an actual image in augmented reality. The display size of the three-dimensional virtual object can be determined based on the actual size of the three-dimensional virtual object and the actual size of the actual object within the actual image. Through this, a user can intuitively compare the size of the actual object and the size of the three-dimensional virtual object within the actual image.

[0081] While the method has been described through specific embodiments, the method can also be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Examples of computer-readable recording media include ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc. Furthermore, the computer-readable recording medium can be distributed across network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the above embodiments can be readily inferred by programmers skilled in the art to which the present disclosure pertains.

Claims

1. A method for displaying a three-dimensional virtual object performed by an electronic device, A step of generating a three-dimensional virtual object based on user input; and A step of displaying the above 3D virtual object as augmented reality by synthesizing it with an actual image - the display size of the above 3D virtual object is determined based on the actual size of the above 3D virtual object and the actual size of the actual object in the actual image - A method comprising:

2. In paragraph 1, The step of creating the above 3D virtual object is A step of generating at least one of a set of images acquired from two or more different angles based on the user input, lidar sensor data, or the actual size of a three-dimensional virtual object; and A step of generating a three-dimensional virtual object based on at least one of the image set, the lidar sensor data, or the actual size. A method comprising:

3. In paragraph 1, The step of displaying the above 3D virtual object as augmented reality by synthesizing it with an actual image is A step of displaying the texture of the three-dimensional virtual object on the surface of the three-dimensional virtual object; A method comprising:

4. In paragraph 1, The step of creating the above 3D virtual object is: A step of determining at least one of an actual size or shape of the three-dimensional virtual object based on user input; A method comprising:

5. In paragraph 1, The step of creating the above 3D virtual object is: A step of transmitting to a server at least one of image sets acquired from two or more different angles generated based on the user input, lidar sensor data, or the actual size of a three-dimensional virtual object; and A step of receiving a three-dimensional virtual object generated based on at least one of the image set, lidar sensor data, or the actual size from the server. A method comprising:

6. In paragraph 1, A step of determining whether a reference object having a predetermined actual size and shape is included in the above actual image; A method further comprising:

7. In paragraph 6, The step of displaying the above 3D virtual object as augmented reality by synthesizing it with an actual image is as follows: When the reference object is included in the actual image, a step of determining the display size of the 3D virtual object based on the actual size of the 3D virtual object and the actual size of the reference object A method comprising:

8. In paragraph 6, The step of displaying the above 3D virtual object as augmented reality by synthesizing it with an actual image is as follows: A step of determining a location to display the three-dimensional virtual object based on the location of the reference object. A method comprising:

9. In paragraph 1, The step of displaying the above 3D virtual object as augmented reality by synthesizing it with an actual image is as follows: A method comprising the step of moving or rotating the position of the three-dimensional virtual object in the real image based on a user input.

10. In paragraph 1, The step of displaying the above 3D virtual object as augmented reality by synthesizing it with an actual image is as follows: A step of adjusting the magnification of a camera included in the electronic device based on user input; and A step of changing the display size of the three-dimensional virtual object based on the adjusted magnification; A method comprising:

11. In paragraph 1, A step of displaying at least one of delivery type data of a product corresponding to the three-dimensional virtual object, price discount data of the product, review data of the product, or a purchase object within a certain distance from the three-dimensional virtual object. A method further comprising:

12. In paragraph 11, The above delivery type data is data related to the delivery form, wherein the delivery form is at least one of general delivery, expedited delivery, fresh food delivery, or international delivery.

13. In paragraph 1, A step of displaying product sales list data of a seller account selling a product corresponding to the three-dimensional virtual object in the above real image; or A step of displaying recommended product data having the same product and product category in the above actual image. A method further comprising:

14. In electronic devices, processor; and Memory storing instructions configured to be executed by the processor Including, The above processor, An electronic device configured to execute a method according to any one of claims 1 to 13.

15. In a non-transitory computer-readable recording medium recording a computer program to be executed by a processor, A non-transitory computer-readable recording medium configured to cause the processor to execute a method according to any one of claims 1 to 13.

Images