Method and apparatus for generating an image for rearrangement objects

Abstract

A method and device for generating a virtual layout image in which a plurality of objects are rearranged are disclosed. The device acquires an image of a plurality of objects and determines at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired image; generates data regarding a result in which the plurality of objects are arranged based on at least one of the visual feature, the physical feature, or the utility feature; generates a virtual layout image in which the plurality of objects are arranged based on the data regarding the result in which the plurality of objects are arranged; and can display the virtual layout image.

Description

Technology Field

[0001] The present invention relates to a system and method for interaction between a human and a device (e.g., a robot) in a home environment. More specifically, it relates to a method and device for guiding a user to reposition household items or other personal belongings. Background Technology

[0003] There is an increasing trend in robotics for home applications. Currently, home robots in the mass market may include means for navigating the house, means for receiving video streams, and other sensors such as heart rate monitors.

[0004] Future mass-market robots are predicted to have means for physical interaction, such as robotic arms, vacuum hands, or other means for interacting with objects. One applicable application for such robotic arms is cleaning and repositioning items.

[0005] Therefore, there is a need for research on methods for devices to organize items by virtually rearranging them. The problem to be solved

[0007] According to the present invention, a device and a method for performing a simulation of rearranging a plurality of objects may be provided.

[0008] According to the present invention, a device and a method for generating a virtual arrangement image in which a plurality of objects are relocated may be provided. means of solving the problem

[0010] The present disclosure provides a method for a device to generate a virtual layout image in which a plurality of objects are rearranged. The method for generating a virtual layout image in which a plurality of objects are rearranged may include: acquiring an image of the plurality of objects; determining at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired image; generating data regarding the result of the arrangement of the plurality of objects based on at least one of the visual feature, the physical feature, or the utility feature; generating a virtual layout image in which the plurality of objects are arranged based on the data regarding the result of the arrangement of the plurality of objects; and displaying the virtual layout image.

[0011] According to one embodiment of the present disclosure, the utility feature may include at least one of the frequency of use for each of the plurality of objects, information on whether it is a target to be placed, information on the intended use, or information on stability.

[0012] According to one embodiment of the present disclosure, the step of generating data for a result in which the plurality of objects are arranged based on at least one of the visual feature, the physical feature, or the utility feature; may include the step of obtaining a usage frequency for a first object included in the plurality of objects based on the utility feature; and the step of determining the location of the first object based on the usage frequency.

[0013] According to one embodiment of the present disclosure, the step of determining the position of the first object based on the frequency of use may include the step of determining the position of the first object such that, when the frequency of use for the first object is greater than the frequency of use for the second object, the first object is positioned closer to the user preference area than the second object.

[0014] According to one embodiment of the present disclosure, the user preference area may be determined based on the distance between the device and at least one area where the plurality of objects are placed.

[0015] According to one embodiment of the present disclosure, the step of determining at least one of the visual feature, the physical feature, or the utility feature for each of the plurality of objects based on the acquired image; may include the step of determining the frequency of use for each of the plurality of objects based on the amount of positional movement for each of the plurality of objects.

[0016] According to one embodiment of the present disclosure, the step of determining the frequency of use for each of the plurality of objects may include: tracking a change in the position of a first object included in the plurality of objects from at least one image; determining a positional displacement of the first object based on information regarding the change in the position of the first object; and determining the frequency of use of the first object based on the positional displacement.

[0017] According to one embodiment of the present disclosure, the step of generating data for a result in which the plurality of objects are arranged based on at least one of the visual feature, the physical feature, or the utility feature may include: a step of determining a similarity between a first object included in the plurality of objects and other objects based on at least one of the visual feature, the physical feature, or the utility feature; and a step of determining the location of the first object based on the similarity between the first object and other objects.

[0018] According to one embodiment of the present disclosure, the step of determining the position of the first object based on similarity with the other objects may include, when the similarity between the first object and the second object is greater than the similarity between the first object and the third object, the step of determining the position of the first object such that the first object is located closer to the second object than to the third object.

[0019] According to one embodiment of the present disclosure, the step of generating data for a result of the arrangement of the plurality of objects based on at least one of the visual feature, the physical feature, or the utility feature; may include the step of determining whether the first object is a target for arrangement based on the utility feature of the first object included in the plurality of objects; and the step of determining the location of the first object based on whether the first object is a target for arrangement.

[0020] According to one embodiment of the present disclosure, the step of generating a virtual placement image in which the plurality of objects are placed based on data regarding the result of the placement of the plurality of objects may include, when the first object is not a target for placement, the step of generating a virtual placement image in which the first object is distinguished from the objects that are targets for placement.

[0021] According to one embodiment of the present disclosure, the step of displaying the virtual layout image includes the step of creating a UI for selecting at least one virtual layout image, and the at least one virtual layout image may be an image in which a plurality of objects are virtually arranged based on different layout algorithms.

[0022] According to one embodiment of the present disclosure, the step of displaying the virtual placement image may further include: receiving an input for selecting one virtual placement image among the at least one virtual placement image; and generating a control signal for placing the plurality of objects according to the selected virtual placement image.

[0023] According to one embodiment of the present disclosure, the step of determining at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired image may include: a step of identifying each of the plurality of objects based on the acquired image; and a step of acquiring the physical feature corresponding to each of the identified objects from a database.

[0024] According to one embodiment of the present disclosure, the step of determining the location of each of the plurality of objects in a virtual space based on a placement algorithm may be further included.

[0025] According to one embodiment of the present disclosure, the step of determining the location for each of the plurality of objects in the virtual space based on the placement algorithm may include: the step of assigning the visual feature, the physical feature, or the utility feature corresponding to the first object included in the plurality of objects to the first virtual object; and the step of determining the location of the first virtual object in the virtual space based on the placement algorithm.

[0026] According to one embodiment of the present disclosure, the step of determining the position of the first virtual object in the virtual space based on the placement algorithm may include: determining a force acting on the first virtual object in the virtual space; and determining the position of the first virtual object based on the force acting on the first virtual object.

[0027] According to one embodiment of the present disclosure, the step of determining a force acting on the first virtual object in the virtual space; wherein the attractive force acting between the first virtual object and the second virtual object in the virtual space can be determined based on the similarity between the first virtual object and the second virtual object.

[0028] According to the present disclosure, a device for generating a virtual layout image in which a plurality of objects are relocated may be provided. A device for generating a virtual layout image in which a plurality of objects are relocated may include a memory for storing at least one instruction; and at least one processor for controlling the device by executing the at least one instruction.

[0029] According to one embodiment of the present disclosure, the at least one processor acquires an image of the plurality of objects and determines at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired image; generates data regarding a result of the arrangement of the plurality of objects based on at least one of the visual feature, the physical feature, or the utility feature; generates a virtual arrangement image of the plurality of objects based on the data regarding the result of the arrangement of the plurality of objects; and can display the virtual arrangement image.

[0030] According to one embodiment of the present disclosure, the processor may obtain a usage frequency for a first object included in the plurality of objects based on the utility feature, and determine the location of the first object based on the usage frequency. Effects of the invention

[0032] According to the present invention, there is an effect of performing a simulation of rearranging a plurality of objects.

[0033] According to the present invention, there is an effect of generating a virtual arrangement image in which a plurality of objects are repositioned.

[0034] However, the effects that can be achieved by the method and apparatus for generating an image of arranging objects according to one embodiment are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which this disclosure belongs from the description below. Brief explanation of the drawing

[0036] FIG. 1 is a drawing for schematically illustrating the operation of the present disclosure according to embodiments of the present disclosure. FIG. 2 relates to a method in which a device (100) generates a virtual layout image in which a plurality of objects are rearranged, according to one embodiment of the present disclosure. FIG. 3 is a drawing illustrating images of a plurality of objects according to one embodiment of the present disclosure. FIG. 4 is a drawing for illustrating a text label according to one embodiment of the present disclosure. FIG. 5 is a drawing illustrating virtual arrangement images in which a plurality of objects are arranged, according to one embodiment of the present disclosure. FIG. 6 is a flowchart showing how a device (100) performs a 3D simulation according to one embodiment of the present disclosure. FIG. 7 is a flowchart of a method for performing a simulation by considering the similarity between objects according to one embodiment of the present disclosure. FIG. 8 is a drawing illustrating the similarity between objects determined according to one embodiment of the present disclosure. FIG. 9 is a drawing illustrating the physical force acting between objects according to one embodiment of the present disclosure. FIG. 10 is a graph illustrating the relationship of physical forces considered when modeling the positions of objects according to one embodiment of the present disclosure. FIG. 11 is a drawing illustrating the change in the position of an object over time of modeling according to one embodiment of the present disclosure. FIG. 12 is a diagram illustrating the results of a simulation of placing objects using the utility features of the objects according to one embodiment of the present disclosure. FIG. 13 is a flowchart of a method for determining the location of an object based on the frequency of use of the object, according to one embodiment of the present disclosure. FIG. 14 is a flowchart of a method for determining the frequency of use of an object according to one embodiment of the present disclosure. FIG. 15 is a drawing illustrating a method for determining the frequency of use of an object according to one embodiment of the present disclosure. FIG. 16 is a drawing illustrating a method for determining the frequency of use of an object according to one embodiment of the present disclosure. FIG. 17 is a drawing illustrating a method for determining the frequency of use of an object according to one embodiment of the present disclosure. FIG. 18 is a drawing illustrating a method for determining the positions of objects by considering the characteristics of the objects according to one embodiment of the present disclosure. FIG. 19 is a drawing illustrating a method of arranging objects considering usage frequency according to one embodiment of the present disclosure. FIG. 20 is a diagram of a method for determining a user preference area according to one embodiment of the present disclosure. FIGS. 21 and 22 are drawings illustrating a UI in which a device (100) displays a virtual layout image according to one embodiment of the present disclosure. FIG. 23 is a flowchart of a method for a device to generate a control signal for arranging a plurality of objects based on a virtual placement image, according to one embodiment of the present disclosure. FIG. 24 is a flowchart of a method for creating a virtual object according to one embodiment of the present disclosure. FIG. 25 is a drawing for explaining semantic segmentation according to one embodiment of the present disclosure. FIG. 26 is a drawing for explaining inpainting according to one embodiment of the present disclosure. FIG. 27 is a drawing for explaining a method of creating a plurality of virtual objects corresponding to a plurality of objects according to one embodiment of the present disclosure. Figure 28 is a diagram illustrating a method for creating a virtual object. FIG. 29 is a diagram illustrating a method in which a device performs rearrangement of objects located in a specific area according to one embodiment of the present disclosure. FIG. 30 is a block diagram showing the configuration of a device (100) according to one embodiment of the present disclosure. Specific details for implementing the invention

[0037] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0038] In describing the embodiments, technical details that are well known in the technical field to which this disclosure belongs and are not directly related to this disclosure are omitted. This is intended to convey the essence of this disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0039] For the same reason, some components in the attached drawings have been exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual dimensions. Identical or corresponding components in each drawing have been assigned the same reference numbers.

[0040] The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms. The embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the present disclosure, and the present disclosure is defined only by the scope of the claims. Throughout the specification, like reference numerals refer to like components.

[0041] At this time, it will be understood that each block of the process flow diagrams and combinations of the flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory can also produce a manufactured item containing the means of instruction to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0042] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0043] In this embodiment, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. Thus, as an example, the "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, the components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. In addition, in the embodiments, '~part' may include one or more processors.

[0044] FIG. 1 is a drawing for schematically illustrating the operation of the present disclosure according to embodiments of the present disclosure.

[0045] According to the present disclosure, a method for organizing items in an unorganized state may be disclosed. In this case, the device (100) may generate a virtual arrangement image in which a plurality of items are rearranged. Referring to FIG. 1, items may exist in an unorganized state in a specific space. For example, when personal belongings, etc., are unorganized in a guest room, floor, etc., of a house inhabited by a person, the present disclosure may be used to organize the items.

[0046] The present invention may be used in, but is not limited to, household robots, AR glasses, mobile communication devices, IoT, and other home appliances.

[0047] A device (100) according to one embodiment of the present disclosure may be implemented in various forms. For example, the device (100) may include a smart TV, a set-top box, a mobile phone, a tablet PC, a digital camera, a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a wearable device, AR glasses, a robot, etc. In addition, in an IoT environment, all home appliances may be connected to other home appliances, and for example, the device may be connected to a surveillance camera, a microphone, etc. However, the device (100) is not limited to the examples described above, and any device capable of receiving commands for human-computer interaction may be possible. The device (100) may be any device capable of performing a simulation of rearranging items. In addition, the device (100) may be any device capable of displaying several options for rearranging items generated through the simulation to the user.

[0048] According to one embodiment of the present disclosure, the device (100) may be a robot (120) having an arm capable of organizing items according to the results of a simulation. In this case, the device (100) may organize items according to a generated virtual layout image according to the user's selection.

[0049] According to one embodiment of the present disclosure, when the device (100) is AR glasses (130), the AR glasses worn by the user can capture the surrounding environment. The AR glasses (130) can perform a simulation of rearranging items based on the captured image. Additionally, the AR glasses (130) can display a selection interface generated based on the result of the simulation of rearranging items to the user. At this time, the device (100) may transmit a command to an external device so that the external device can organize items according to a generated virtual arrangement image, depending on the user's selection.

[0050] At this time, the external device may refer to a separate device separated from the device (100). The external device may move items based on the result of the item relocation simulation of the device (100) based on a control signal received from the device (100).

[0051] In the present disclosure, the device (100) can perform a simulation of placing items according to a placement algorithm. That is, according to the simulation result according to the placement algorithm, the position of each item can be determined as a new position different from the existing position. At this time, the device (100) can determine not only the position of the items but also the direction and angle in which the items are placed by performing the simulation. For convenience of explanation, the determination of the new position, direction, and angle of the items as described above in the present disclosure is described as rearranging, placement, arrangement, etc.

[0052] For the sake of convenience of explanation, the case where the device (100) is a robot (120) having a robot hand capable of moving items is described as an example. There may be infinite ways for the robot (120) to rearrange items that are not organized (hereinafter, arrangement algorithm).

[0053] Referring to Fig. 1, the items are in each batch algorithm It can be arranged accordingly. As a non-limited example, the first arrangement algorithm of FIG. 1 is a diagram illustrating the state in which items are arranged according to the class of the items. When following the first arrangement algorithm, it can be seen that markers are placed close together and batteries are placed close together by considering the class of the items.

[0054] In addition, the second arrangement algorithm of FIG. 1 is a diagram illustrating the state in which items are arranged so that the gaps between items are minimized by considering the volume of the arranged items. According to the second arrangement algorithm, it can be seen that the items are arranged so that the gaps between items are minimized.

[0055] In this case, examples of unlimited methods (arrangement algorithms) for rearranging items may include methods for organizing items without gaps by considering spatial elements (compact arrangement), methods for organizing by considering aesthetic elements, and methods for organizing by considering convenience.

[0056] Additionally, when the robot (120) relocates items, it is necessary to consider the usability properties of the items. More specifically, when the robot (120) relocates items, it must consider cases where any item cannot be moved. For example, if an item is a cup filled with water, a cup filled with water cannot be placed horizontally. If the robot (120) performs this operation, the user's item may be damaged. That is, any item cannot be moved for various reasons, such as when the item is attached to a specific space with adhesive, or when moving the item is dangerous (e.g., a glass bottle filled with water that is not closed with a stopper). Also, the user will not move the item if it is suitable for the specific location where the item is currently situated. Therefore, for the robot (120) to place items, it is necessary to consider the usability property of the object, and in this specification, the usability property of the object may refer to the habit properties of the object as described above.

[0057] The present disclosure discloses a plurality of embodiments of a method for arranging unaligned items (hereinafter, objects) and a device (100) for arranging unaligned objects. According to one embodiment of the present disclosure, the device (100) may receive a user command to perform rearrangement of objects located in a specific area. Additionally, the device (100) may capture at least one image (2D or RGBD) of the specific area.

[0058] According to one embodiment of the present disclosure, a device (100) can perform semantic segmentation on objects in at least one image. The device (100) can remove the segmented objects one by one from the image and fill the gaps with inpainting. The device (100) can repeat the above steps until there are no objects remaining as segments.

[0059] According to one embodiment of the present disclosure, the device (100) can perform a 3D reconstruction of immovable 3D objects, such as objects and planes, on a region. The device (100) can calculate at least two possible rearrangements of 3D objects in the 3D reconstructed region. The device (100) can render at least two possible rearrangements as bitmap images of video files.

[0060] Additionally, the present disclosure discloses a plurality of embodiments relating to a user interface for selecting one of a method of arranging a plurality of items generated by a device (100). At least two possible rearrangements may be presented to the user.

[0061] According to one embodiment of the present disclosure, the device (100) may be selected by a user of at least two possible rearrangements. Additionally, the device (100) may send a command to a robot (external device (100)) to perform the selected rearrangement on the possessions (items). Alternatively, the device (100) may perform the selected rearrangement.

[0062] According to one embodiment of the present disclosure, the device (100) can perform the function of recognizing objects in 3D, the function of visualizing objects, and the function of physically modeling.

[0063] According to one embodiment of the present disclosure, a 2D approach for visualization may be used to require inpainting technology for performing reconstruction on closed parts of objects.

[0064] According to one embodiment of the present disclosure, the device (100) may use an end-to-end deep learning approach for 3D reconstruction of object templates and complete scene understanding.

[0065] According to one embodiment of the present disclosure, the device (100) performs a deep semantic understanding of a scene (e.g., an acquired image) to obtain the best rearrangement result applicable to personal belongings.

[0066] More details regarding the present disclosure will be described below.

[0067] FIG. 2 relates to a method in which a device (100) generates a virtual layout image in which a plurality of objects are rearranged, according to one embodiment of the present disclosure.

[0068] According to one embodiment of the present disclosure, the device (100) can provide a method for planning the rearrangement of a plurality of physical objects in a home environment.

[0069] The items described above in FIG. 1 may correspond to the objects in FIG. 2. In this disclosure, an object may refer to an item located without regularity in a specific space where the device (100) is to perform organization. At this time, there are no restrictions on the type, characteristics, etc. of the item corresponding to the object. Multiple objects may be located on shelves, floors, chairs, tables, etc., and are not limited to the examples described above. In addition, in this specification, objects may differ from each other in shape, size, etc.

[0070] In step S210, the device (100) can acquire images of multiple objects.

[0071] According to one embodiment of the present disclosure, the device (100) can acquire images of a plurality of physical objects from an external device. For example, an external device connected to the device (100) via a wired or wireless connection can capture images of a plurality of physical objects and transmit the captured images to the device (100).

[0072] According to another embodiment of the present disclosure, the device (100) can take at least one image of a plurality of physical objects.

[0073] In this specification, an image or picture may represent a still image, a video composed of a plurality of consecutive still images (or frames), or a video. For example, in this disclosure, an image may mean an RGB or RGBD image.

[0074] For example, the device (100) may take only one image of multiple objects.

[0075] As another example, the device (100) can take multiple images of multiple objects. If the device (100) acquires one or more images of multiple objects, the device (100) may additionally recognize the physical visual characteristics of the objects.

[0076] According to one embodiment of the present disclosure, the device (100) may take multiple images of multiple objects at different times. As a non-limiting example, the device (100) may acquire images of multiple objects twice a day for a month. In this case, the device (100) may take multiple images of the objects with the same layout to acquire additional usability characteristics of the objects.

[0077] According to one embodiment of the present disclosure, an image of an object may be taken at a different location to improve the visual features of the object acquired by the device (100). In this case, the visual features of the object in the present disclosure may refer to information about the shape, texture, size, etc. of the object.

[0078] Additionally, an image of the object may be taken at a different location to improve the physical characteristics of the object acquired by the device (100). In this specification, the physical characteristics of the object may refer to information regarding the object's weight, friction coefficient, weight distribution, whether it can be bent, elasticity, etc.

[0079] According to one embodiment of the present disclosure, images of a plurality of unaligned objects may be images taken under various conditions. More specifically, RGB images and RGBD images may be taken under shooting conditions that combine various shooting times, various shooting angles, and various shooting lighting conditions.

[0080] Objects can be seen in their entirety from a specific angle. Additionally, objects can be partially obscured by other objects from other angles. Therefore, the device (100) needs to acquire images of multiple objects captured at various shooting angles.

[0081] FIG. 3 is a drawing illustrating images of a plurality of objects according to one embodiment of the present disclosure.

[0082] According to one embodiment of the present disclosure, the device (100) may include RGB and IR cameras.

[0083] According to one embodiment of the present disclosure, lighting conditions may be changed by applying various lighting techniques. This is a method to improve the recognition of objects by the device (100). Examples of lighting conditions described above, which are not limited to, may include turning on or off a lamp located on the body of the device (100) (e.g., a robot), using structured light, using Time-Of-Flight (Tof) lighting, using lighting in various spectral bands such as multispectral lighting and hyperspectral lighting, which are not limited to white, red, green, and blue, and using modulated light.

[0084] FIG. 3(a) may represent a Time-Of-Flight (Tof) image acquired from an RGBD camera. FIG. 3(b) may represent a structured light image acquired from an RGBD camera.

[0085] In step S220, the device (100) can determine at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired image.

[0086] In step S220, objects can be recognized by the device (100). At this time, the result of the device (100) recognizing the objects can be defined as at least one of the visual characteristics of each object, the physical characteristics of the object, or the utility characteristics of the object.

[0087] In this specification, the visual characteristics of an object may refer to the shape, size, texture, optical properties such as a reflective surface, etc. of the object.

[0088] In this specification, the physical characteristics of an object may refer to the object's weight, weight distribution, bending or stretching capabilities, friction coefficient, etc.

[0089] In this specification, the usefulness features of an object may refer to information regarding whether the object's position is changed and the frequency of the position change, the frequency of use, whether the object is garbage (whether it is subject to removal), the purpose for which the object is used, whether the object is safe when tilted or held, etc.

[0090] According to one embodiment of the present disclosure, the device (100) can identify each of a plurality of objects based on an acquired image. Additionally, it can determine features for each of the identified objects.

[0091] A method for determining the usefulness features for each of a plurality of objects based on an image acquired by the device (100) is described in more detail below in the following drawings (Figs. 14 to 17).

[0092] According to one embodiment of the present disclosure, the device (100) may obtain parameters for physical features corresponding to each of the objects from a database in order to determine physical features for each of the plurality of objects based on an acquired image. The device (100) may generate the parameters for physical features for each of the objects as text labels.

[0093] FIG. 4 is a drawing for illustrating a text label according to one embodiment of the present disclosure.

[0094] As described above, objects can be described by text labels. In particular, text labels can be used to retrieve parameters regarding the average physical characteristics of an object, such as its weight and coefficient of friction, from a database.

[0095] FIG. 4(a) illustrates a plurality of objects (410) and a flipchart marker (420) which is one of the plurality of objects (410).

[0096] Additionally, FIG. 4(b) illustrates a text label summarizing the physical parameters of the flipchart marker (420). Referring to FIG. 4(b), it can be seen that the average weight of the flipchart marker (420) is 15g, the coefficient of friction is 0.2, the center of mass is centered, it is not bendable, it is not stretchable, and it is not dangerous. In other words, it can be seen that the flipchart marker (420) has physical characteristics such as those shown in FIG. 4(b).

[0097] FIG. 4(b) illustrates a text label for a flipchart marker (420), which is one of the objects (410), for convenience of explanation, but is not limited thereto. That is, the text label may store physical parameters for each of the objects (410).

[0098] In another embodiment, an end-to-end deep neural network can be trained for all predictions in an end-to-end manner.

[0099] In addition, methods for determining the physical properties of an object are not limited to the examples described above.

[0100] In step S230, the device (100) can generate data on the result of a plurality of objects being placed based on at least one of a visual feature, a physical feature, or a usability feature.

[0101] According to one embodiment of the present disclosure, the device (100) may obtain a usage frequency for a first object included in a plurality of objects based on a utility feature. Additionally, the device (100) may determine the location of the first object based on the usage frequency. For example, when the device (100) determines the location of the first object based on the usage frequency, the device (100) may determine the location of the first object such that the first object is closer to the user preference area than the second object if the usage frequency for the first object is greater than the usage frequency for the second object. In this case, the first object and the second object may refer to any object included in the plurality of objects. More details are described later in FIGS. 13 to 20.

[0102] According to one embodiment of the present disclosure, the device (100) can determine the similarity between a first object included in a plurality of objects and other objects based on at least one of a visual feature, a physical feature, or a usability feature. Additionally, the device (100) can determine the location of the first object based on the similarity between the other objects.

[0103] For example, when the device (100) determines the position of a first object based on similarity with other objects, if the similarity between the first object and the second object is greater than the similarity between the first object and the third object, the device may determine the position of the first object such that the first object is positioned closer to the second object than to the third object. In this case, the first object, the second object, and the third object may refer to any object included in a plurality of objects. More details are described later in FIGS. 7 to 11.

[0104] According to one embodiment of the present disclosure, the device (100) can determine whether the first object is a target for placement based on a utility feature for the first object included in a plurality of objects. Additionally, the device (100) can determine the location of the first object based on whether the first object is a target for placement. More details are described later in FIG. 12.

[0105] According to one embodiment of the present disclosure, the device (100) can determine the location of each of a plurality of objects in a virtual space based on a placement algorithm. This is described later in FIG. 5.

[0106] Additionally, the device (100) may generate a first virtual object corresponding to a first object included in a plurality of objects. In this specification, virtual objects may be generated for each of the plurality of objects, and the virtual object may correspond to a 3D object used when performing a simulation. A more detailed description is provided later in FIGS. 24 to 28.

[0107] For example, when the device (100) determines the location of each of a plurality of objects in a virtual space based on a placement algorithm, it may assign a visual feature, a physical feature, or a utility feature corresponding to a first object included in the plurality of objects to a first virtual object. Additionally, the device (100) may determine the location of the first virtual object in a virtual space based on a placement algorithm. This is described later in FIG. 5.

[0108] For example, when the device (100) determines the position of the first virtual object in virtual space based on a placement algorithm, it can determine the force acting on the first virtual object in virtual space. Additionally, the device (100) can determine the position of the first virtual object based on the force acting on the first virtual object.

[0109] For example, when the device (100) determines a force acting on a first virtual object in a virtual space, the attractive force acting between the first virtual object and the second virtual object in the virtual space can be determined based on the similarity between the first virtual object and the second virtual object. This is described later in FIGS. 7 to 10.

[0110] That is, the device (100) can use 3D simulation to simulate how virtual objects with extracted features can be placed in a virtual space according to a placement algorithm. More specifically, the device (100) can perform at least one preset of repositioning for a plurality of recognized physical objects.

[0111] In this specification, "preset of repositioning" may mean performing a simulation to generate virtual objects corresponding to each of a plurality of objects, and placing the virtual objects in the virtual space according to a placement algorithm based on determined features and the features of the virtual space. That is, "preset of repositioning" may mean placing 3D objects (virtual objects) at a location different from their initial positions. Furthermore, since the preset of repositioning can be performed according to each placement algorithm, it may correspond to each option that the user can select.

[0112] In this specification, preset relocation may be performed by virtual physical simulation. Preset relocation may be performed using at least one of the physical features of an object, the visual features of an object, or the usability features of an object.

[0113] According to one embodiment of the present disclosure, there may be one or more presets of repositioning. For example, a first preset repositioning may be performed by simple 3D object positioning described below, while other preset object repositionings may be performed by physical 3D simulation.

[0114] In a preferred embodiment, preset relocation can be performed by a physical simulation of the positions of 3D objects (virtual objects).

[0115] In a simple exemplary embodiment, 3D objects may be placed within 3D rendering software. Here, 3D rendering software may refer to software capable of setting simple gravity and integrated friction coefficients for all objects, and in non-limiting embodiments, 3D rendering software may include Blender ⓒ. According to one embodiment of the present disclosure, 3D objects may be placed one by one on a plane. Additionally, a simulation may be performed so that 3D objects fall onto the plane by gravity from a point located higher than the plane. By this procedure, the objects may be placed very close to each other. The simulation method described above may be an embodiment for performing the simulation in the simplest way. In another embodiment, after this procedure, virtual vertical walls may hold multiple objects together. More details are described later in the drawings below.

[0116] In step S240, the device (100) can generate a virtual layout image in which a plurality of objects are arranged according to a layout algorithm based on data regarding the arranged result.

[0117] More specifically, the device (100) can render the result of at least one pre-repositioning setting as at least one visual representation by considering at least one of the physical characteristics of the object, the visual characteristics of the object, or the usability characteristics of the object. In this case, each of the at least one visual representation may correspond to at least one preset of repositioning.

[0118] Objects repositioned by the simulator according to the aforementioned method can be visually rendered based on various software such as Blender.

[0119] FIG. 5 is a drawing illustrating virtual arrangement images in which a plurality of objects are arranged, according to one embodiment of the present disclosure.

[0120] With reference to Fig. 5, steps S230 and S240 can be explained in more detail.

[0121] In step S230, the device (100) may generate virtual objects for a plurality of objects in order to generate data for a result of arranging a plurality of objects based on at least one of a visual feature, a physical feature, or a usability feature. A more detailed explanation thereof may be provided later in FIGS. 24 to 28 below.

[0122] The device (100) can arrange virtual objects created in a virtual space according to a placement algorithm. That is, the device (100) can apply at least one placement algorithm (e.g., a rearrangement algorithm) to virtual objects reconstructed in 3D. At this time, the device (100) can generate data for a result in which multiple objects are arranged according to the placement algorithm based on the result of arranging in the virtual space.

[0123] According to one embodiment of the present disclosure, the batch algorithm may include a semantic similarity-based algorithm, a color similarity-based algorithm, a compactness-based algorithm, etc.

[0124] After that, in step S240, the device (100) can generate a virtual placement image of a plurality of objects based on data regarding the result of a plurality of objects being placed, and FIG. 5 may correspond to the virtual placement image.

[0125] First, referring to FIG. 5(a), the left image of FIG. 5(a) may be an image (initial image) of a plurality of objects obtained by the device (100) in step S210. Then, the device (100) may place virtual objects in a virtual space according to a first placement algorithm. That is, at this step, the device (100) may perform a 3D simulation on the virtual objects according to the first placement algorithm. For example, if the first placement algorithm is a similarity-based algorithm, the device (100) may generate a first virtual placement image such as the right image of FIG. 5(a). That is, when referring to the right image of FIG. (a), it can be seen that objects of similar types are arranged closely together.

[0126] Likewise, with reference to FIG. 5(b), the device (100) can place virtual objects in a virtual space according to a second placement algorithm. For example, if the second placement algorithm is a miniaturization-based algorithm, the device (100) can generate a second virtual placement image such as the right image of FIG. 5(b).

[0127] In step S250, the device (100) can display a virtual placement image.

[0128] When the device (100) displays a virtual layout image, the device (100) may generate a UI that allows at least one virtual layout image to be selected. At this time, the at least one virtual layout image may be an image in which multiple objects are virtually arranged based on different layout algorithms.

[0129] According to one embodiment of the present disclosure, the device (100) may display at least one virtual batch image corresponding to each batch algorithm. Additionally, the device (100) may receive input from a user to select one of the at least one displayed virtual batch images.

[0130] A more detailed explanation is provided in Figures 21 to 25 below.

[0131] FIG. 6 is a flowchart showing how a device (100) performs a 3D simulation according to one embodiment of the present disclosure.

[0132] More specifically, the device (100) can determine the location of each of the multiple objects in virtual space based on a placement algorithm.

[0133] In step S610, the device (100) can create a first virtual object corresponding to a first object included in a plurality of objects.

[0134] In the present disclosure, a virtual object is a 3D object created by a device (100) to perform a 3D simulation in a virtual space, and may be a 3D virtual object corresponding to a real object. A method for creating a virtual object is described in more detail later in FIGS. 24 to 28 below.

[0135] In step S620, the device (100) can assign a visual feature, physical feature, or utility feature corresponding to a first object included in a plurality of objects to a first virtual object.

[0136] In step S630, the device (100) can determine the location of the first virtual object in virtual space based on a placement algorithm.

[0137] In the present disclosure, the term "virtual space" may refer to a virtual space for performing 3D simulations. New positions of objects can be determined by changing the forces in the virtual space acting on objects and the forces acting between objects within the virtual space.

[0138] FIG. 7 is a flowchart of a method for performing a simulation by considering the similarity between objects according to one embodiment of the present disclosure.

[0139] FIG. 7 may be a specific embodiment of step S230 of FIG. 2 described above. More specifically, FIG. 7 may be a case where a plurality of objects are arranged considering the similarity between the objects.

[0140] According to one embodiment of the present disclosure, when the device (100) performs a simulation of relocating the positions of objects, the device (100) can generate data regarding the result of a plurality of objects being placed according to a placement algorithm by utilizing semantic similarity between the objects.

[0141] In this case, similarity between multiple objects may refer to a value, numerical value, etc., indicating the degree of similarity between objects. Additionally, similarity may refer to the aforementioned semantic similarity.

[0142] In step S710, the device (100) can determine the similarity between a first object and other objects included in a plurality of objects based on at least one of a visual feature, a physical feature, or a utility feature.

[0143] In this case, the first object may refer to any object included in a plurality of objects.

[0144] For example, if there are N objects, there may be n * (n-1) / 2 pairwise semantic similarities between each pair of objects. In order to utilize semantic similarities between objects, the characteristics of the objects must be appropriately recognized. In this case, as previously mentioned, the usefulness characteristics of the objects may be recognized, and in particular, the functions of objects represented as 3D objects (virtual objects) must be recognized. Semantic similarity based on the functions of objects means, for example, that a marker and a pen are more similar than a marker and a fingernail.

[0145] In step S720, the device (100) can determine the location of the first object based on the similarity between the first object and other objects.

[0146] According to one embodiment of the present disclosure, when the device (100) determines the position of the first object based on similarity with the other objects, if the similarity between the first object and the second object is greater than the similarity between the first object and the third object, the device may determine the position of the first object such that the first object is positioned closer to the second object than to the third object.

[0147] FIG. 8 is a drawing illustrating the similarity between objects determined according to one embodiment of the present disclosure.

[0148] According to one embodiment of the present disclosure, the device (100) may obtain the names of the objects when determining the similarity between a plurality of objects based on determined features. And, the device (100) may determine the similarity between the objects based on the names of the objects.

[0149] That is, for example, the similarity between objects can be determined by natural language similarity. For example, to determine natural language similarity, cosine similarity between two words can be calculated, and in a non-limited embodiment, GloVe NLP can be used to calculate the cosine similarity between two words.

[0150] For example, referring to Fig. 8, the pairwise distances of the four words “marker,” “pen,” “battery,” and “charger” are shown.

[0151] Physical forces (e.g., attraction) acting between virtual objects in a virtual space may be proportional to semantic similarity. According to one embodiment of the present invention, the physical force attraction between objects may serve as a measure for positioning objects close to or far from other objects when placing objects. For example, if the physical force attraction between two objects is large, objects with large attraction may be positioned close to each other. Additionally, if the physical force attraction between two objects is small, objects with small attraction may be positioned far apart from each other.

[0152] That is, the attractive force acting between the first virtual object and the second virtual object in virtual space can be determined based on the similarity between the first virtual object and the second virtual object. More specifically, the attractive force acting between the first virtual object and the second virtual object in virtual space can be determined in proportion to the similarity between the first virtual object and the second virtual object.

[0153] In some exemplary embodiments, the physical force between two objects can be determined as shown in the following mathematical formula.

[0154] [Mathematical Formula 1]

[0155] F1(t)=a(t)d 12 min(m1,m2) 2 / 3

[0156] At this time, a may represent a scaling factor, and in an embodiment not limited to, a may be set to 30 at the start of the simulation and a may be set to 0 at the end of the simulation.

[0157] d 12 may refer to a semantic similarity coefficient. For example, the semantic similarity coefficient may refer to the GloVe similarity between words or phrases describing objects using linear or empiric kind of monotonic increasing function similarity. In a non-limiting embodiment, d 12 =GloVe(word1, word2) 3 It can be set to.

[0158] In addition, m1 may refer to the mass of the first object, and m2 may refer to the mass of the second object.

[0159] In the present disclosure, 'a' may represent the strength of the bonding target to which the objects are bonded together. This simulation step may be performed in a viscosity environment. In this case, the frictional force may be proportional to the velocity during the dynamic simulation (to prevent unnecessary vibration), and the simulation may be performed in a zero-gravity state where the objects are bound to a surface.

[0160] In addition, powers of 2 / 3 can mean that pressure can be the determining factor when many objects exist. That is, it can mean that it is proportional to the second power of the average size. On the other hand, it can be confirmed that mass is proportional to the cube of the average size.

[0161] In another embodiment, the force between objects may not be determined by their geometric size. In this case, the physical force between objects can be determined according to the following mathematical formula.

[0162] [Mathematical Formula 2]

[0163] F2(t)=a(t)d 12

[0165] In this case, during the simulation, objects begin to adhere to each other, which can create clusters of similar objects.

[0166] There may be numerous ways to derive the characteristic that objects attract each other in proportion to their semantic similarity, and the two examples described above may be disclosed merely as exemplary embodiments.

[0167] Referring to Figure 10 below, in the second stage, gravity takes precedence over the force between objects, and it can be seen that the object is ultimately influenced by gravity rather than the attractive force between objects.

[0168] FIG. 9 is a drawing illustrating the physical force acting between objects according to one embodiment of the present disclosure.

[0169] Referring to FIG. 9, it can be seen that the physical force acting between the objects (910, 920, 930, 940) is depicted. More specifically, it can be seen that the value related to the attractive force acting between the objects of FIG. 8 (e.g., F2 / a(t)) is depicted.

[0170] First, the device (100) can determine the similarity between objects based on features. At this time, the similarity between objects may be proportional to the attractive force acting between the objects.

[0171] Referring to FIG. 9, it can be seen that the force acting between the battery (920) and the charger (910) is 175, and the force acting between the pen (940) and the marker (930) is 43.6. These values ​​are greater than the forces acting between other objects. For example, the force acting between the marker (930) and the battery (920) is 0.12, and the force acting between the marker (930) and the charger (910) is 3.3. Through this, it can be seen that the device (100) has the greatest similarity acting between the battery and the charger. Additionally, it can be seen that the similarity acting between the pen and the marker is the next greatest.

[0172] In this case, similarity can be a relative value. In particular, similarity requires determining relatively how similar the first object is to other objects, based on a single object (e.g., the first object).

[0173] Based on FIG. 9, that is, the object with the greatest similarity to the battery (920) is the charger (910). On the other hand, when judged based on the marker (930), the marker has an attractive force of 43.6 with the pen (940), which is a value smaller than 175, but when viewed based on the marker, the similarity to the pen is the greatest. Therefore, the object with the greatest similarity to the marker can be determined to be the pen.

[0174] The device (100) can generate data regarding the result of arranging multiple objects in a virtual space based on the derived value (similarity). For example, the device (100) can position objects with high similarity close together. Referring to FIG. 9, the device (100) can generate data such that a marker and a pen are positioned close together, and a charger and a battery are positioned close together. Additionally, the device (100) can align multiple objects such that objects with low similarity values ​​are positioned far apart.

[0175] As a result, looking at Fig. 9, placing objects with high similarity close to each other can be one embodiment that enhances user convenience.

[0176] FIG. 10 is a graph illustrating the relationship of physical forces considered when modeling the positions of objects according to one embodiment of the present disclosure.

[0177] Referring to FIG. 10(a), a first force (1020) and a second force (1030) may act on an object (1010). According to one embodiment of the present disclosure, the first force (1020) may be gravity acting on the object. Additionally, the second force (1030) may be an attractive force acting on the object with other objects. In this case, the attractive force acting on other objects may be the force according to FIGS. 8 and 9 described above.

[0178] FIG. 10(b) illustrates an embodiment in which the first force (1020) and the second force (1030) shown in FIG. 10(a) change during the simulation. Referring to FIG. 10(b), there may be two steps for modeling the positions of the objects.

[0179] Referring to FIG. 10(b), the first phase (1040) may be an inter-object attraction phase. In the first phase (1040), it can be seen that the coefficient a(t) is not zero, but the gravitational acceleration g(t) is close to zero. Since there is no friction in this phase (1040), the objects can move freely without friction, forming clusters of objects aligned according to semantic similarity between the objects.

[0180] Referring to FIG. 10(b), the second stage section (1050) may be a finalization phase. In the second stage section (1050), gravity (g(t)) may act preferentially over the force between objects. Therefore, objects may move under the influence of gravity rather than the attractive force between objects.

[0181] FIG. 11 is a drawing illustrating the change in the position of an object over time of modeling according to one embodiment of the present disclosure.

[0182] Referring to Fig. 11, it can be seen that as modeling time progresses, objects that were located far apart from each other move closer together. In other words, it can be seen that objects that were in an unorganized state are arranged to be close to each other through modeling.

[0183] There may be various methods for aligning objects through simulation. Examples, but are not limited to, include using virtual force fields of objects (e.g., a virtual bulldozer), combining various forces, using virtual wind, virtual elastic forces, virtual points of attraction, vibration-simulated annealing, shaking within a contracting sphere, and forcibly repositioning objects without using physical modeling.

[0184] Other exemplary embodiments may include other methods and means for determining the attractive force between objects and determining immovable objects.

[0185] All objects in the aforementioned exemplary embodiments do not possess utility features and are embodiments in which only physical features are used for simulation. The following embodiments describe an example of rearranging objects using the utility features of the objects.

[0186] FIG. 12 is a diagram illustrating the results of a simulation of placing objects using the utility features of the objects according to one embodiment of the present disclosure.

[0187] According to one embodiment of the present disclosure, in the above-described step S230, the device (100) may determine whether the first object is a target to be placed (e.g., an object to be removed during tidying, trash, etc.) based on a utility feature of the first object included in a plurality of objects. Additionally, the device (100) may determine the location of the first object based on whether the first object is a target to be placed.

[0188] FIG. 12(a) may be a drawing illustrating objects before they are sorted. That is, FIG. 12(a) may correspond to an image acquired or captured by the device (100) in step S210. At this time, if an object (1210) that is not a target for placement is included among the objects to be sorted, the device (100) needs to perform sorting excluding the trash (1210).

[0189] According to one embodiment of the present disclosure, the device (100) can determine whether each of a plurality of objects is a target (1210) to be removed (e.g., garbage) based on extracted utility features. That is, in one embodiment, additional utility features of the objects may include a classification of whether the object is non-garbage or garbage.

[0190] In the aforementioned step S250, the device (100) can generate a virtual placement image in which the first object is distinguished from the objects that are the objects to be placed, as in FIG. 12(b), when the first object is not the object to be placed (i.e., when the first object is garbage (1210)).

[0191] For example, the device (100) can generate an image by separating an object determined not to be a target for placement. More specifically, as shown in FIG. 12(b), an object classified as trash (1210) can be displayed with a visually altered representation (e.g., trash may be marked) during the rendering and display processes. That is, trash (1210) can be displayed to be distinguished from other objects.

[0192] FIG. 13 is a flowchart of a method for determining the location of an object based on the frequency of use of the object, according to one embodiment of the present disclosure.

[0193] In step S1310, the device (100) can obtain the frequency of use for a first object included in a plurality of objects based on a utility feature.

[0194] In the present disclosure, the usability feature may include the frequency of use for each of the plurality of objects. In this case, the frequency of use for each of the objects may be a relative concept.

[0195] According to one embodiment of the present disclosure, the device (100) can determine the frequency of use for each of the plurality of objects based on the amount of positional movement for each of the plurality of objects. More details are described later in FIGS. 14 to 17.

[0196] In step S1320, the device (100) can determine the location of the first object based on the frequency of use.

[0197] According to one embodiment of the present disclosure, when the device (100) determines the location of a first object based on usage frequency, if the usage frequency for the first object is greater than the usage frequency for a second object, the location of the first object can be determined such that the first object is closer to the user preference area than the second object.

[0198] According to one embodiment of the present disclosure, a user preference area may be determined based on the distance between at least one area where the plurality of objects are placed and the device (100).

[0199] A more detailed explanation is provided in Figures 18 to 20 below.

[0200] FIG. 14 is a flowchart of a method for determining the frequency of use of an object according to one embodiment of the present disclosure.

[0201] According to one embodiment of the present disclosure, data describing the frequency of use of objects (hereinafter referred to as usage frequency) may exist. Items may be rearranged according to usage frequency. At this time, various other metrics using usage frequency may exist. At this time, for example, the usage frequency may be determined as the front line during a simulation.

[0202] According to one embodiment of the present disclosure, the device (100) can check the location of objects and track whether the location of the objects is changed.

[0203] FIG. 14 may be a flowchart for a method of obtaining the frequency of use for a first object included in a plurality of objects based on the usefulness feature in step S1310.

[0204] In step S1410, the device (100) can track the position change of a first object included in a plurality of objects from at least one image. At this time, the first object may refer to one of the plurality of objects. That is, the first object may refer to an object that is the target when tracking the position and movement of the object.

[0205] In this case, at least one image may refer to an image captured to track changes in the position of an object in order to determine the frequency of use of the object. According to one embodiment of the present disclosure, at least one image may be an image captured at specific time intervals. Additionally, according to another embodiment, at least one image may be an image captured when the object has moved.

[0206] In step S1420, the device (100) can determine the amount of positional movement of the first object based on information about the positional change of the first object.

[0207] In step S1430, the device (100) can determine the frequency of use of the first object based on the amount of positional movement.

[0208] At this time, embodiments for tracking the position change of the device (100) are described in more detail below in FIGS. 15 to 17.

[0209] FIG. 15 is a drawing illustrating a method for determining the frequency of use of an object according to one embodiment of the present disclosure.

[0210] More specifically, FIG. 15 illustrates a method for determining the frequency of use of objects in order to perform a simulation of organizing objects using the usefulness characteristics of the objects.

[0211] In one embodiment, to determine the frequency of use of an object, all observations of the object may be processed to calculate an RMS (root mean square) value such as the following mathematical formula.

[0212] [Mathematical Formula 3]

[0213]

[0215] Here, , and is the average value of all coordinate values ​​for an object acquired during observation. In this embodiment, a higher RMS value may mean that the item is used more frequently by the user.

[0216] Referring to FIG. 15(a), a method for obtaining an RMS value by continuously tracking the position of the pen (1510) can be seen. In this case, according to one example, the first image may be an image taken on the first day, the second image may be an image taken on the second day, the third image may be an image taken on the third day, and the fourth image may be an image taken on the fourth day.

[0217] Looking at FIG. 15(a), it can be seen that the device (100) has obtained the position of the pen (1510) in the first image as (2, 3, 7). At this time, the device (100) can obtain the position of the object as (x, y, z) coordinates, and the x, y, z coordinates of the object can be bound to a global coordinate system.

[0218] Looking at FIG. 15(a), it can be seen that the device (100) has obtained the position of the pen (1510) as (3, 6, 7) in the second image. Additionally, it can be seen that the device (100) has obtained the position of the pen (1510) as (2, 5, 6, 7) in the third image, and the device (100) has obtained the position of the pen (1510) as (4, 7, 7) in the fourth image.

[0219] More specifically, regarding the position of the pen (1510). , , The values ​​are each =(2+3+2.5+4) / 4=2.875; =(3+6+6+7) / 4=5.5; It can be calculated as =7. Therefore, if the RMS value of the pen (1510) in the first to fourth images is calculated according to the above mathematical formula 3, the RMS value of the pen (1510) can be calculated as follows.

[0220] [Mathematical Formula 4]

[0221] RMS=0.25*((2-2.875) 2 +(3-5.5) 2 +0 2 +(2.875-3) 2 +(5.5-6) 2 +0 2 +(2.875-4) 2 +(5.5-6) 2 +0 2 +(2.875-4) 2 +(5.5-7) 2 +0 2 ) 1 / 2 =0.25*(11.1875) 1 / 2 =0.836

[0222] Therefore, the RMS value of the pen (1510) can be 0.836.

[0223] By looking at FIG. 15(b), the RMS values ​​for each of the multiple objects shown in FIG. 15(a) can be checked. It can be seen that the RMS value of the pen (1510) is larger than that of the other objects. Accordingly, it can be seen that the user changed the position of the pen (1510) the most frequently and that the user uses the pen (1510) more frequently than other objects.

[0224] FIG. 16 is a drawing illustrating a method for determining the frequency of use of an object according to one embodiment of the present disclosure.

[0225] In another embodiment, each object may be recognized using two anchor points. In the case of FIG. 16, according to one example, the position of the tip (1620) and the end (1610) of the pen (1600) was determined.

[0226] Referring to FIG. 16, in the first captured image, the positions of the end (1610) and tip (1620) of the pen (1600) are (3.2, 4.5, 3.1) and (1.6, 7.3, 3.1), respectively. In the second image, the positions of the end (1610) and tip (1620) of the pen (1600) are (7.2, 5.2, 3.1) and (4.5, 4.0, 3.1), respectively. In the third image, the positions of the end (1610) and tip (1620) of the pen (1600) are (2.6, 2.5, 3.1) and (1.0, 3.8, 3.1), respectively.

[0227] In this way, the device (100) can track the position of an object using two anchor points for the object. Additionally, the device (100) can obtain an RMS value using two anchor points for the object.

[0229] RMS can be calculated in the same way as described in FIG. 15 using the following mathematical formula.

[0230] [Mathematical Formula 5]

[0231]

[0232] Similarly, objects with a higher RMS value compared to other objects can be determined to have a higher usage frequency.

[0233] FIG. 17 is a drawing illustrating a method for determining the frequency of use of an object according to one embodiment of the present disclosure.

[0234] FIG. 17 is a diagram illustrating an example in which the positions of objects change according to the frequency of use of the objects. In addition to the examples described in FIG. 15 and FIG. 16, various methods for determining the frequency of use of objects may exist.

[0235] For example, in another embodiment, three anchor points per object can be used to track 6DoF position changes.

[0236] If an object disappears from the camera's field of view (FoV) or is not tracked by similar means for tracking the object, the coordinate RMS value or any other frequency of use metric may be calculated using various rules or methods. In one embodiment, when the object disappears from the camera's FoV, a constant positive value may be heuristically added to the RMS value.

[0237] In another embodiment, an end-to-end deep learning method may be applied to calculate the usage frequency of an object. If a dataset synthesized based on the CleVR methodology, which sequentially includes the locations of objects, and a series of images are provided, a utility value (based on the coordinate values ​​of the RMS value) may be used. A deep neural network based on a 3D convolutional network may be used to learn the utility values ​​of objects in an end-to-end manner.

[0238] The aforementioned methods are not limited to the methods described above, and the frequency of use of objects can be obtained in various ways.

[0239] FIG. 18 is a drawing illustrating a method for determining the positions of objects by considering the characteristics of the objects according to one embodiment of the present disclosure.

[0240] First, to determine the positions of the objects, the aforementioned frequency of use as a characteristic of the objects may be considered. Additionally, the attractive force acting between the objects in proportion to the frequency of use may also be considered. Furthermore, referring to FIG. 18, an additional force (1840) may be applied to the objects during the simulation.

[0241] For example, additional force (1840) may include a spring force on an imaginary line passing through a user-preferred area. In this case, the preferred area is an area that the user prefers to access, and more specifically, may mean the most convenient area where the user can select, use, or pick up an object.

[0242] Referring to FIG. 18, values ​​related to the attractive force acting between the charger (1810), battery (1812), marker (1814) and pen (18165) are shown.

[0243] Referring to FIG. 18, the case where the first object to be positioned is a pen (1816) is described as an example. The line of center of preferred area (1820) in FIG. 18 may mean an imaginary line located on the user preferred area.

[0244] At this time, the distance (1830) may mean the distance from the pen (1816) to the center line (1820) of the user's preferred area.

[0245] At this time, the force (1840) acting on the pen (1816) can be calculated as k * distance (1830). Here, k may represent a coefficient that acts to position the object close to the preferred area and may represent an elastic modulus. Additionally, according to one embodiment of the present disclosure, k may be a value set in proportion to the frequency of use of the object.

[0246] FIG. 19 is a drawing illustrating a method of arranging objects considering usage frequency according to one embodiment of the present disclosure.

[0247] FIG. 19(a) may correspond to a virtual placement image in which objects are placed considering the frequency of use of the pen (1900) when the pen (1900) is not used. In this case, the value of k may be 0.

[0248] FIG. 19(b) may correspond to a virtual placement image in which an object is placed when the pen (1900) is used with an intermediate frequency of use. In this case, the value of k may be 5.

[0249] FIG. 19(c) may correspond to a virtual placement image in which objects are placed when the pen (1900) is used with a high frequency of use. In this case, the value of k may be 20. In this case, the frequency of use may be a relative concept between objects.

[0250] According to one embodiment of the present disclosure, the pen (1900) may be positioned differently during the simulation depending on additional force. If the object is used more frequently by the user, the value of the elastic modulus (k) that pulls the object toward a virtual line determining the center of the most accessible area (user preferred area) may increase. As the object gets closer to this line, the force may decrease. During the simulation, the balance of forces may be set at the end of the simulation.

[0251] In the second stage of the simulation (see Fig. 10), where gravity increases and elasticity decreases, the forces affecting usability may decrease. When the simulation ends, the objects may be located in the user preference area.

[0252] According to one embodiment of the present disclosure, an object located in a user preference area can be determined based on usage frequency.

[0253] FIG. 20 is a diagram of a method for determining a user preference area according to one embodiment of the present disclosure.

[0254] The easy or accessible area (user preferred area) can be determined in an unlimited manner.

[0255] According to one embodiment of the present disclosure, the user preferred area may be an area located at the shortest distance to a camera mounted on the device (100) during all movements of the device (100) in a certain space (e.g., an apartment), an area closest to the user on a table, shelf, etc., or an area explicitly designated by the user.

[0256] According to one embodiment of the present disclosure, a user preference area may be determined by comparing a first distance between the location of the device (100) and a first point (2050) and a second distance between the location of the device (100) and a second point (2040).

[0257] In one embodiment, the area designated by the user as a cleanup target may be covered by multiple points. As a non-limited example, the area designated by the user as a cleanup target may be divided into three points. The first point may be designated as an access area preferred by the user (e.g., an area where the user can easily use objects), the second point as an access area of ​​medium difficulty, and the other area as an area that is difficult for the user to access (e.g., a case where it is far away). However, it is not limited to three points as described above.

[0258] Referring to FIG. 20, the explanation assumes that the first point (2050) and the second point (2040) exist in the area designated by the user as a target for organization. The first point (2050) is the closest area to the user among the areas of the table (closest area), and the second point (2040) may mean the middle area among the areas of the table (middle area).

[0259] At this time, the device (100) may include a camera. By considering geometric rules and the first position (2010), second position (2020), and third position (2030) of the device (100) (e.g., a robot), a first distance between the position of the device (100) and a first point (2050) and a second distance between the position of the device (100) and a second point (2040) can be determined.

[0260] At this time, the sum of the distances from the first position (2010), second position (2020), and third position (2030) of the device (100) (2000) to the second point (2040) is greater than the sum of the distances from the first position (2010), second position (2020), and third position (2030) of the device (100) (2000) to the first point (2050). That is, d21 + d22 + d23 > d11 + d12 + d13 holds true. In other words, d21 + d22 + d23, which is the sum of the distances between the positions of the device (100) and the second point (5640), is greater than d11 + d12 + d13, which is the sum of the distances between the positions of the device (100) and the first point (5650). Therefore, it can be seen that the second point is further away than the first point.

[0261] At this time, d21 may mean the second point (2040) and the first position (2010), d22 may mean the second point (2040) and the second position (2020), and d23 may mean the second point (2040) and the third position (2030). Also, d11 may mean the first point (2050) and the first position (2010), d12 may mean the first point (2050) and the second position (2020), and d13 may mean the first point (2050) and the third position (2030).

[0262] Through the method described above, the first point (2050) and the surrounding environment of the first point (2050) can be identified as areas that are more easily accessible to the user. Through this, the device (100) (2000) can determine the area of ​​the first point (2050) as a user-preferred area.

[0263] FIGS. 21 and 22 are drawings illustrating a UI in which a device (100) displays a virtual layout image according to one embodiment of the present disclosure.

[0264] FIGS. 21 and 22 are drawings illustrating an embodiment of step S250 described above in FIG. 2. According to one embodiment of the present disclosure, the rendered result may be generated as a UI. In this case, the device (100) may generate and display several options that a user can select as a UI.

[0265] According to one embodiment of the present disclosure, the option may also include a selection regarding whether to agree to the rearrangement of objects proposed by the device (100). Referring to FIG. 21, a UI may be created to select whether to accept the arrangement of items presented by the device (100). For example, as shown in FIG. 21, an image of the state of the objects before organization (current) and an image of the state of the objects after organization may be displayed. If the user selects to agree to organization (yes), the device (100) may arrange a plurality of objects according to the selected image based on the received input. In addition, as another example, if the user selects to reject organization (no), the device (100) may not perform organization of the objects based on the received input.

[0266] According to one embodiment of the present disclosure, several options may be presented to the user. Referring to FIG. 22, several options may be displayed for the user to select. For example, several options may correspond to virtual batch images generated based on different batch algorithms. Referring to FIG. 22, selection A may be a virtual batch image generated based on a first batch algorithm, and selection B may be a virtual batch image generated based on a second batch algorithm.

[0267] According to one embodiment of the present disclosure, if a user rejects a simulation of the rearrangement of the presented objects, a different rearrangement of the objects may be generated. For example, if the device (100) receives user input for rejection (No) of FIG. 21, the device (100) may generate a virtual rearrangement image of the objects based on a different arrangement algorithm.

[0268] FIG. 23 is a flowchart of a method for a device to generate a control signal for arranging a plurality of objects based on a virtual placement image, according to one embodiment of the present disclosure.

[0269] As described above in FIGS. 21 and 22, when the device (100) displays a virtual layout image, it can generate a UI for selecting at least one virtual layout image. At this time, at least one virtual layout image may be an image in which a plurality of objects are virtually arranged based on different layout algorithms.

[0270] In step S2310, the device (100) can receive an input for selecting one virtual batch image from at least one virtual batch image.

[0271] In step S2320, the device (100) can generate a control signal for placing multiple objects according to a selected virtual placement image.

[0272] According to one embodiment of the present disclosure, the device (100) may generate a control signal for arranging a plurality of objects according to a selected image based on a received input. At this time, the device (100) may arrange a plurality of objects based on the control signal. For example, if the device (100) is a robot capable of performing a function to organize objects based on the control signal, the device (100) may arrange a plurality of objects based on the generated control signal.

[0273] As another example, the device (100) can transmit a control signal to an external device. For example, if the external device is a robot capable of performing a function to organize objects based on the control signal, the device (100) transmits the control signal to the external device, and the external device (e.g., robot) can arrange multiple objects based on the received control signal.

[0274] FIG. 24 is a flowchart of a method for creating a virtual object according to one embodiment of the present disclosure.

[0275] In step S2410, the device (100) can perform segmentation on the first object in the acquired image.

[0276] At this time, according to one embodiment of the present disclosure, the segmentation of objects may utilize semantic segmentation. More specifically, the device (100) may perform semantic segmentation of objects in at least one image.

[0277] Referring to FIG. 25, semantic segmentation may mean that a first object included in a plurality of objects is deleted or separated from an acquired image. More specifically, as shown in FIG. 25(a), an initial image may be acquired. Then, as shown in FIG. 25(b), segmentation may be performed on the first object, a battery (2520), in the initial image. The separated object (2520) may be stored. As shown in FIG. 25(c), the segmented object (2530) may be cut from the image.

[0278] According to one embodiment of the present disclosure, the device (100) can remove divided objects one by one from an image and fill the gaps by inpainting. That is, a mask for inpainting can be performed on the image. In this disclosure, inpainting may mean an algorithm that masks a specific area of ​​an image and reconstructs the gaps.

[0279] Referring to FIG. 26, an initial image (2610) and a mask (2620) can be input into an inpainting algorithm to perform inpainting. Through this process, an image (2630) with the object removed can be obtained.

[0280] Referring to FIG. 27, it can be seen that the steps described above are performed sequentially for each of the multiple objects. That is, segmentation and inpainting can be performed sequentially for each of the multiple objects included in an initial image such as FIG. 27(a). The device (100) can repeat the segmentation and inpainting until there are no remaining objects.

[0281] In step S2420, the device (100) can assign a determined feature to the first object to the first object.

[0282] At this time, the determined feature for the first object may mean at least one of the visual feature, physical feature, or utility feature of the first object as described above.

[0283] In step S2430, the device (100) can shape the first object in 3D.

[0284] More specifically, the device (100) can perform 3D reconstruction of immovable 3D objects such as objects and planes on a region.

[0285] FIG. 28 is a diagram illustrating a method for creating a virtual object. As described above, segmentation and masking can be performed on an initial image (2810). Then, the segmented object can be saved for subsequent 3D shaping. Then, a virtual object (2830) can be created by performing 3D shaping on the saved object.

[0286] FIG. 29 is a diagram illustrating a method in which a device performs rearrangement of objects located in a specific area according to one embodiment of the present disclosure.

[0287] Referring to FIG. 29(a), first, the device (100) may receive a user command to have the device (100) or an external device (100) (e.g., a robot) perform a rearrangement of objects located in a specific area (an area where objects to be placed are located). The device (100) may capture at least one image (2D or RGBD) of the specific area. As described above in FIG. 1, according to an unlimited embodiment, the device (100) may be a robot or a device capable of communicating with an external device (e.g., a robot).

[0288] According to FIGS. 24 to 27 described above, the device (100) can perform semantic segmentation of objects in at least one image. The device (100) can remove the segmented objects one by one from the image and fill the gaps with inpainting. The device (100) can repeat the segmentation and inpainting until there are no objects remaining as segments.

[0289] The device (100) performs a simulation so that the device (100) can perform a 3D reconstruction of immovable 3D objects, such as objects and planes, on a region. As shown in FIG. 29(b), the device (100) can calculate at least two possible rearrangements of 3D objects in the 3D reconstructed region.

[0290] At this point, an example in which the placement algorithm is a method of placing objects based on semantic similarity is described in more detail.

[0291] First, the device (100) can recognize objects. Textual labels may be applied. The device (100) can calculate the pairwise semantic distance between pairs of objects. For example, GloVe-300 embedding may be used.

[0292] The device (100) can estimate the physical characteristics of objects (e.g., weight of objects) based on a database of weights based on the types of objects. (Virtual) objects with corresponding weights can be input into a 3D physical simulator. At this time, virtual 3D objects can be input into the 3D physical simulator along with a database of physical characteristics of objects (e.g., weight of objects, friction force, etc.).

[0293] According to one embodiment of the present disclosure, a simulation may be performed such that a physical force (e.g., pairwise force, attraction) proportional to the semantic similarity of the objects acts between the objects within a 3D physical simulator. During the simulation, a vibration that decreases over time may be applied to a surface to which a decreasing pairwise force is applied.

[0294] By the method described above, objects can be semantically grouped and positioned in optimal locations according to the simulated annealing.

[0295] Accordingly, virtual objects can be virtually placed in a virtual space according to at least one placement algorithm as shown in FIG. 29(b).

[0296] Referring to FIG. 29(c), a UI can be created and displayed to the user. At this time, the rendered view can be saved so that it is shown to the user. At this time, the device (100) can render at least two possible rearrangements as bitmap images of video files. At this time, according to one embodiment of the present disclosure, at least two possible rearrangements can be presented to the user.

[0297] According to one embodiment of the present disclosure, one of at least two possible rearrangements may be selected by a user. Additionally, according to one example, the device (100) may send a command to a robot to perform the selected rearrangement of the items.

[0298] FIG. 30 is a block diagram showing the configuration of a device (100) according to one embodiment of the present disclosure.

[0299] As illustrated in FIG. 30, the device (100) of the present disclosure may include a processor (3030), a communication unit (3010), a memory (3020), a display (3030), and a shooting unit (3050). However, the components of the device (100) are not limited to the examples described above. For example, the device (100) may include more components or fewer components than the components described above. In addition, the processor (3030), the communication unit (3010), and the memory (3020) may be implemented in the form of a single chip.

[0300] The device (100) may perform a method of generating a virtual layout image in which a plurality of objects are relocated according to the method described above, and redundant operations are omitted from description. According to one embodiment, the processor (3030) may control a series of processes in which the device (100) can operate according to the embodiment of the present disclosure described above. For example, the components of the device (100) may be controlled to perform a method of providing a service of generating a virtual layout image in which a plurality of objects are relocated according to the embodiment of the present disclosure. There may be multiple processors (3030), and the processor (3030) may perform an operation to provide a service of generating a virtual layout image in which a plurality of objects are relocated according to the present disclosure described above by executing a program stored in memory (3020).

[0301] The communication unit (3010) can transmit and receive signals with an external device (e.g., an external device that captures images, an external device that can move objects according to an image selected by a user, etc.). The signals transmitted and received with the external device may include control information and data. The communication unit (3010) may be composed of an RF transmitter that up-converts and amplifies the frequency of a transmitted signal, and an RF receiver that low-noise amplifies a received signal and down-converts the frequency. However, this is merely one embodiment of the communication unit (3010), and the components of the communication unit (3010) are not limited to an RF transmitter and an RF receiver. Additionally, the communication unit (3010) can receive a signal through a wireless channel and output it to a processor (3030), and transmit the signal output from the processor (3030) through a wireless channel.

[0302] According to one embodiment, the memory (3020) may store programs and data necessary for the operation of the device (100). Additionally, the memory (3020) may store control information or data included in signals transmitted and received by the device (100). The memory (3020) may be composed of a storage medium or a combination of storage media such as ROM, RAM, hard disk, CD-ROM, and DVD. Additionally, the memory (3020) may be a plurality of units. According to one embodiment, the memory (3020) may store a program for performing an operation to provide a service for generating a virtual layout image in which a plurality of objects, which are embodiments of the present disclosure described above, are rearranged.

[0303] According to one embodiment, the processor (3030) acquires an image of a plurality of objects, determines at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired image, generates data for a result of the arrangement of the plurality of objects based on at least one of the visual feature, a physical feature, or a utility feature, generates a virtual arrangement image of the plurality of objects based on the data for the result of the arrangement of the plurality of objects, and can display the virtual arrangement image.

[0304] According to one embodiment of the present disclosure, the utility feature may include at least one of the frequency of use for each of the plurality of objects, information on whether it is a target to be placed, information on the intended use, or information on stability.

[0305] According to one embodiment, the processor (3030) can obtain a usage frequency for a first object included in a plurality of objects based on a utility feature, and determine the location of the first object based on the usage frequency.

[0306] According to one embodiment, the processor (3030) can determine the position of the first object such that the first object is located closer to the user preference area than the second object when the frequency of use of the first object is greater than the frequency of use of the second object.

[0307] According to one embodiment, the processor (3030) can determine a user preference area based on the distance between the device and at least one area where the plurality of objects are placed.

[0308] According to one embodiment, the processor (3030) can determine the frequency of use for each of the plurality of objects based on the amount of positional movement for each of the plurality of objects.

[0309] According to one embodiment, the processor (3030) tracks the position change of a first object included in a plurality of objects from at least one image, determines the amount of position movement of the first object based on information about the position change of the first object, and determines the frequency of use of the first object based on the amount of position movement.

[0310] According to one embodiment, the processor (3030) can determine the similarity between a first object included in a plurality of objects and other objects based on at least one of a visual feature, a physical feature, or a utility feature, and determine the location of the first object based on the similarity between the first object and other objects.

[0311] According to one embodiment, the processor (3030) can determine the position of the first object such that the first object is located closer to the second object than the third object when the similarity between the first object and the second object is greater than the similarity between the first object and the third object.

[0312] According to one embodiment, the processor (3030) can determine whether the first object is a target for placement based on a utility feature for the first object included in a plurality of objects, and determine the location of the first object based on whether the first object is a target for placement.

[0313] According to one embodiment, the processor (3030) can generate a virtual placement image in which the first object is distinguished from the objects that are the objects to be placed when the first object is not the object to be placed.

[0314] According to one embodiment, the processor (3030) can generate a UI for selecting at least one virtual placement image. At this time, the at least one virtual placement image may be an image in which a plurality of objects are virtually placed based on different placement algorithms.

[0315] According to one embodiment, the processor (3030) receives an input for selecting one virtual placement image among at least one virtual placement image, and can generate a control signal for placing a plurality of objects according to the selected virtual placement image.

[0316] According to one embodiment, the processor (3030) can identify each of a plurality of objects based on an acquired image and obtain the physical features corresponding to each of the identified objects from a database.

[0317] According to one embodiment, the processor (3030) can determine the location of each of a plurality of objects in a virtual space based on a placement algorithm.

[0318] According to one embodiment, the processor (3030) may assign a visual feature, physical feature, or utility feature corresponding to a first object included in a plurality of objects to a first virtual object, and determine the location of the first virtual object in virtual space based on a placement algorithm.

[0319] According to one embodiment, the processor (3030) can determine a force acting on a first virtual object in a virtual space and, based on the force acting on the first virtual object, determine the position of the first virtual object.

[0320] According to one embodiment, the processor (3030) can determine the attractive force acting between the first virtual object and the second virtual object in virtual space based on the similarity between the first virtual object and the second virtual object.

[0321] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0322] When implemented as software, a computer-readable storage medium or computer program product storing one or more programs (software modules) may be provided. One or more programs stored in the computer-readable storage medium or computer program product are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the embodiments described in the claims or specification of this disclosure.

[0323] Such programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, ROM (Read Only Memory), Electrically Erasable Programmable Read Only Memory (EEPROM), magnetic disc storage devices, Compact Disc-ROM (CD-ROM), Digital Versatile Discs (DVDs), or other forms of optical storage devices, magnetic cassettes. Alternatively, they may be stored in memory composed of some or all of these. Additionally, each constituent memory may include multiple units.

[0324] Additionally, the program may be stored on an attachable storage device accessible via a communication network such as the Internet, Intranet, Local Area Network (LAN), Wide LAN (WLAN), or Storage Area Network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0325] In the present disclosure, the terms "computer program product" or "computer readable medium" are used to collectively refer to media such as memory, a hard disk installed in a hard disk drive, and signals. These "computer program product" or "computer readable medium" are means provided to a software computer system comprising instructions for acquiring images of a plurality of objects according to the present disclosure, determining at least one of a visual feature, a physical feature, or a utility feature for each of the plurality of objects based on the acquired images, generating data regarding a result of the arrangement of the plurality of objects based on at least one of the visual feature, a physical feature, or a utility feature, generating a virtual arrangement image of the plurality of objects based on the data regarding the result of the arrangement of the plurality of objects, and displaying the virtual arrangement image.

[0326] In the specific embodiments of the present disclosure described above, the components included in the present disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0327] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

Claim 1 A method for generating a virtual arrangement image in which a plurality of real objects are rearranged by a device comprises: acquiring an image including the plurality of real objects; determining visual features, physical features, and utility features for each of the plurality of real objects based on the acquired image, wherein the physical features include at least one of the weight, weight distribution, elasticity, and friction coefficient of each of the plurality of real objects; generating data for a result in which each of the plurality of virtual objects corresponding to the plurality of real objects is arranged based on the visual features, physical features, and utility features; generating a virtual arrangement image in which the plurality of real objects are arranged based on the data for a result in which each of the plurality of virtual objects is arranged based on the data for a result in which each of the plurality of virtual objects is arranged; and displaying the virtual arrangement image; wherein the step of generating data for a result in which each of the plurality of virtual objects corresponding to the plurality of real objects is arranged includes a step of determining similarity between the plurality of virtual objects corresponding to the plurality of real objects based on the visual features, physical features, and utility features. and a step of generating data for a result in which each of the plurality of virtual objects is placed, based on the similarity between the plurality of virtual objects corresponding to the plurality of actual objects; comprising a method. Claim 2 A method according to claim 1, wherein the utility feature comprises at least one of the frequency of use for each of the plurality of actual objects, information on whether it is a target to be placed, information on the intended use, or information on stability. Claim 3 A method according to claim 1, comprising: a step of generating data for a result in which each of the plurality of virtual objects corresponding to the plurality of actual objects is placed based on the visual feature, the physical feature, and the usability feature; a step of obtaining a usage frequency for a first object included in the plurality of actual objects based on the usability feature; and a step of determining the location of the first object based on the usage frequency. Claim 4 A method according to claim 3, wherein the step of determining the position of the first object based on the usage frequency; and the step of determining the position of the first object such that, when the usage frequency for the first object is greater than the usage frequency for the second object, the first object is positioned closer to the user preference area than the second object. Claim 5 A method according to claim 4, wherein the user preference area is determined based on the distance between the device and at least one area in which the plurality of virtual objects corresponding to the plurality of actual objects are placed. Claim 6 A method according to claim 1, comprising the step of determining the visual features, the physical features, and the utility features for each of the plurality of actual objects based on the acquired image; and the step of determining the frequency of use for each of the plurality of actual objects based on the amount of positional movement for each of the plurality of actual objects. Claim 7 A method according to claim 6, wherein the step of determining the frequency of use for each of the plurality of actual objects comprises: the step of tracking the position change of a first object included in the plurality of actual objects from at least one image; the step of determining the amount of position movement of the first object based on information regarding the position change of the first object; and the step of determining the frequency of use of the first object based on the amount of position movement. Claim 8 A method according to claim 1, wherein the step of determining the similarity between the plurality of virtual objects corresponding to the plurality of actual objects based on the visual feature, the physical feature, and the utility feature; the step of determining the similarity between a first object included in the plurality of actual objects and other objects based on the visual feature, the physical feature, and the utility feature; and the step of generating data for the result of each of the plurality of virtual objects being placed based on the similarity between the plurality of virtual objects corresponding to the plurality of actual objects; the step of determining the position of the first object based on the similarity between the first object and other objects. Claim 9 A method according to claim 8, wherein the step of determining the position of the first object based on similarity with the other objects comprises the step of determining the position of the first object such that the first object is positioned closer to the second object than to the third object when the similarity between the first object and the second object is greater than the similarity between the first object and the third object. Claim 10 A method according to claim 1, comprising: a step of generating data for a result in which each of the plurality of virtual objects corresponding to the plurality of actual objects is placed based on the visual feature, the physical feature, and the utility feature; a step of determining whether the first object is a target for placement based on the utility feature of the first object included in the plurality of actual objects; and a step of determining the location of the first object based on whether the first object is a target for placement. Claim 11 A method according to claim 10, wherein the step of generating a virtual placement image in which the plurality of actual objects are placed based on data regarding the result of each of the plurality of virtual objects being placed; and the step of generating a virtual placement image in which the first object is distinguished from the objects being placed when the first object is not the object being placed. Claim 12 The method according to claim 1, wherein the step of displaying the virtual placement image; includes the step of creating a UI for selecting at least one virtual placement image, and wherein the at least one virtual placement image is an image in which the plurality of virtual objects corresponding to the plurality of actual objects are arranged virtually based on different placement algorithms. Claim 13 The method according to claim 12 further comprises the step of displaying the virtual placement image; the step of receiving an input for selecting one virtual placement image among the at least one virtual placement image; and the step of generating a control signal for placing the plurality of virtual objects corresponding to the plurality of actual objects according to the selected virtual placement image. Claim 14 A method according to claim 1, wherein the step of determining visual features, physical features, and utility features for each of the plurality of actual objects based on the acquired image; the step of identifying each of the plurality of actual objects based on the acquired image; and the step of acquiring the physical features corresponding to each of the identified objects from a database. Claim 15 A method according to claim 1, further comprising the step of determining the position of each of the plurality of virtual objects corresponding to the plurality of actual objects in a virtual space based on a placement algorithm. Claim 16 A method according to claim 15, comprising: a step of determining the position of each of the plurality of virtual objects in the virtual space based on the placement algorithm; a step of assigning the visual feature, the physical feature, and the utility feature corresponding to the first object included in the plurality of actual objects to the first virtual object; and a step of determining the position of the first virtual object in the virtual space based on the placement algorithm. Claim 17 A method according to claim 16, wherein the step of determining the position of the first virtual object in the virtual space based on the placement algorithm comprises: the step of determining a force acting on the first virtual object in the virtual space; and the step of determining the position of the first virtual object based on the force acting on the first virtual object. Claim 18 In claim 17, the step of determining the force acting on the first virtual object in the virtual space; is a method in which the attractive force acting between the first virtual object and the second virtual object in the virtual space is determined based on the similarity between the first virtual object and the second virtual object. Claim 19 A device for generating a virtual layout image in which a plurality of real objects are rearranged comprises: a memory for storing instructions; and at least one processor for controlling the device by executing the instructions, wherein, when the instructions are executed by the at least one processor, the device acquires an image containing the plurality of real objects and determines visual features, physical features, and utility features for each of the plurality of real objects based on the acquired image, wherein the physical features include at least one of the weight, weight distribution, elasticity, and friction coefficient of each of the plurality of real objects; generates data regarding the result of each of the plurality of virtual objects corresponding to the plurality of real objects based on the visual features, physical features, and utility features; generates a virtual layout image in which the plurality of real objects are arranged based on the data regarding the result of each of the plurality of virtual objects; and displays the virtual layout image. Claim 20 In claim 19, the device obtains a frequency of use for a first object included in the plurality of actual objects based on the utility feature, and determines the location of the first object based on the frequency of use.

Images