Device, operating method of device, and non-transitory recording medium

Abstract

An operating method of a device is disclosed. The objective of the present disclosure can be to provide a method by which a user can interactively divide a 3D environment in real time without a separate boundary value by clicking on an object. According to one embodiment of the present disclosure, the instability of contrastive learning in which the same objects are classified into different segment IDs in different views can be resolved.

Description

Device, method of operating the device and non-transitory recording medium

[0001] The present invention relates to a device capable of segmenting and recognizing an object in an image.

[0002] Conventional techniques for segmenting objects in a 3D environment have various limitations.

[0003] Although some technologies can segment objects within a 3D scene, they have limitations in precisely segmenting the detailed structure of the object, making it difficult to accurately segment objects with complex shapes.

[0004] In particular, in many cases, there is an inconvenience in that the user must manually set the segmentation threshold while directly checking the shape or location of each object.

[0005] This method places a high workload on the user and may be difficult to segment accurately unless you are an expert.

[0006] Additionally, some existing models include complex preprocessing or postprocessing steps for object segmentation or adopt a network structure with a large computational volume, making real-time processing difficult.

[0007] This immediately raises the problem that it is difficult to apply to interactive applications or user participation-based applications.

[0008] Furthermore, systems that require complex computational processes or manual adjustments have the disadvantage of being unintuitive to general users and having a high barrier to entry.

[0009] Additionally, it is not very versatile as it requires repeated user feedback for accurate segmentation or a separate pre-training dataset.

[0010] Therefore, there is a growing demand for technologies that can efficiently segment objects in 3D environments in a more precise and automated manner.

[0011] The purpose of the present disclosure may be to solve the problems pointed out above.

[0012] An object of the present disclosure may be to provide a novel 3D object segmentation technique having both real-time performance and user friendliness.

[0013] The purpose of the present disclosure may be to provide a method for easily and quickly performing various segmentations by dividing an object into two stages (an object unit and a smaller unit constituting the object).

[0014] The purpose of the present disclosure may be to provide a method that allows a user to interactively segment a 3D environment in real time with just a click without setting separate boundary values.

[0015] According to an embodiment of the present disclosure, a method of operating a device may include: generating a first mask image of a first level and a second mask image of a second level based on a plurality of two-dimensional (2-dimensional) images corresponding to a plurality of viewpoints, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object; generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images; performing contrastive learning on the first mask image and the first feature field, and on the second mask image and the second feature field, respectively; obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints; clustering the obtained average feature set to obtain a plurality of global clusters; and performing contrastive learning between a global feature candidate representing each of the obtained global clusters and a first feature vector corresponding to the first feature field or a second feature vector corresponding to the second feature field.

[0016] According to one embodiment of the present disclosure, a device generates a first mask image of a first level and a second mask image of a second level based on a plurality of two-dimensional (2-Dimensional) images corresponding to a plurality of viewpoints, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object.

[0017] The method may include generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images, performing contrastive learning on the first mask image and the first feature field, and on the second mask image and the second feature field, respectively, obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints, clustering the obtained average feature set to obtain a plurality of global clusters, and performing contrastive learning between a global feature candidate representing each of the obtained global clusters and a first feature vector corresponding to the first feature field or a second feature vector corresponding to the second feature field.

[0018] In accordance with another embodiment of the present disclosure, a non-transitory recording medium having recorded thereon a computer-readable program for performing an operating method of a device, the operating method comprising: generating a first mask image of a first level and a second mask image of a second level based on a plurality of two-dimensional (2-dimensional) images corresponding to a plurality of viewpoints, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object; generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images; performing contrastive learning on the first mask image and the first feature field, and on the second mask image and the second feature field, respectively; obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints; clustering the obtained average feature set to obtain a plurality of global clusters; And it may include a step of performing contrastive learning between the global feature candidate representing each of the acquired global clusters and the first feature vector corresponding to the first feature field or the second feature vector corresponding to the second feature field.

[0019] According to an embodiment of the present disclosure, since editing is possible in units of segmented objects on a 3D image, properties of each object can be individually adjusted, thereby providing precise customization and an intuitive interface without affecting the entire scene.

[0020] According to an embodiment of the present disclosure, by segmenting detailed objects constituting a unit object on a 3D image, the properties of each detailed object can be individually adjusted, and the user can perform precise customization at the detailed element level of the object.

[0021] According to an embodiment of the present disclosure, since segmented 3D objects and detailed objects of 3D objects on a 3D image are displayed on separate areas, the cognitive burden of manipulating specific objects in the entire scene can be reduced, and the accuracy of user input can be increased.

[0022] According to an embodiment of the present disclosure, by displaying coarse and fine level results separately and enabling selection and attribute editing for each segment, a user can perform intuitive and precise editing work based on a hierarchical structure even within a complex 3D scene.

[0023] According to an embodiment of the present disclosure, the instability of contrastive learning in which the same object is classified with different segment IDs in different views can be resolved.

[0024] FIG. 1 is a block diagram illustrating components of an artificial intelligence device according to one embodiment of the present disclosure.

[0025] FIG. 2 is a diagram for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0026] FIGS. 3 and 4 are drawings illustrating a learning method for generating a 3D image including a 3D object according to one embodiment of the present disclosure.

[0027] FIGS. 5A and 5B are diagrams illustrating a process of generating a first mask image of a first level and a second mask image of a second level from a 2D image through SAM according to one embodiment of the present disclosure.

[0028] FIG. 6A and FIG. 6B are diagrams illustrating a process of generating a first feature field of a first level and a second feature field of a second level based on a plurality of 2D images.

[0029] FIG. 7a and FIG. 7b are diagrams illustrating a process of performing contrastive learning according to an embodiment of the present disclosure.

[0030] Figure 8 is a diagram illustrating that the segment ID to which a pixel belongs may vary depending on the view of the 2D image.

[0031] FIGS. 9A to 9D are diagrams for explaining global feature induction learning according to an embodiment of the present disclosure.

[0032] FIG. 10 is a flowchart illustrating an operation method of a device according to another embodiment of the present disclosure.

[0033] FIG. 11 is a drawing for explaining a 3D segment model according to one embodiment of the present disclosure.

[0034] FIGS. 12A to 12I are drawings for explaining a process of editing a selected 3D object on a 3D image including a plurality of 3D objects according to an embodiment of the present disclosure.

[0035] FIGS. 13A to 13D are drawings for explaining a method for providing information on segmented 3D objects in a 3D image according to an embodiment of the present disclosure.

[0036] FIG. 14 is a drawing illustrating a method for guiding shooting when shooting a 2D image to create a 3D image according to one embodiment of the present disclosure.

[0037] FIG. 15 is a drawing illustrating an example of performing furniture arrangement in a home based on a 3D image obtained through a 3D segment model according to one embodiment of the present disclosure.

[0038] FIG. 16 may be a drawing illustrating a scenario of generating a top view image based on a 3D image and linking the 3D image and the top view image according to one embodiment of the present disclosure.

[0039] Artificial intelligence refers to a field that studies artificial intelligence or the methodologies for creating it, and machine learning refers to a field that defines various problems in the field of artificial intelligence and studies the methodologies for solving them.

[0040] Machine learning is sometimes defined as an algorithm that improves its performance on a task through continuous experience.

[0041] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities that is composed of artificial neurons (nodes) that form a network through the combination of synapses.

[0042] An artificial neural network can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates the output values.

[0043] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0044] Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0045] The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

[0046] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0047] Supervised learning refers to a method of training an artificial neural network given labels for training data. The labels can refer to the correct answer (or result value) that the artificial neural network must infer when training data is input to the artificial neural network.

[0048] Unsupervised learning can refer to a method of training an artificial neural network without being given labels for the training data.

[0049] Reinforcement learning can refer to a learning method that teaches an agent defined in an environment to select an action or action sequence that maximizes the cumulative reward in each state.

[0050] Among artificial neural networks, machine learning implemented with a deep neural network (DNN) that includes multiple hidden layers is also called deep learning, and deep learning is a part of machine learning.

[0051] Hereinafter, machine learning is used to mean deep learning.

[0052] FIG. 1 is a block diagram illustrating components of an artificial intelligence device according to one embodiment of the present disclosure.

[0053] The artificial intelligence device (100) can be implemented as a fixed device or a movable device, such as a TV, a projector, a mobile phone, a smart phone, a laptop, a digital broadcasting terminal, a PDA (personal digital assistant), a PMP (portable multimedia player), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, digital signage, a robot, a vehicle, etc.

[0054] Referring to FIG. 1, an artificial intelligence device (100) may include a communication interface (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180).

[0055] The communication interface (110) can transmit and receive data with external devices such as other artificial intelligence devices or AI servers (200) using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

[0056] Communication technologies used by the communication interface (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0057] The input interface (120) can obtain various types of data.

[0058] The input interface (120) may include a camera (121) for capturing images, a microphone (122) for receiving audio signals, and a user input interface (123) for receiving information from a user.

[0059] By treating the camera (121) or microphone (122) as a sensor, the signal obtained from the camera (121) or microphone (122) can be called sensing data or sensor information.

[0060] The input interface (120) can acquire input data to be used when obtaining output using learning data and a learning model for model learning. The input interface (120) can also acquire raw input data, in which case the processor (180) or learning processor (130) can extract input features as preprocessing for the input data.

[0061] The camera (121) processes image frames, such as still images or moving images, obtained by the image sensor in video call mode or shooting mode. The processed image frames can be displayed on the display (151) or stored in the memory (170).

[0062] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application program) being performed by the artificial intelligence device (100). Meanwhile, various noise removal algorithms can be applied to the microphone (122) to remove noise generated during the process of receiving external acoustic signals.

[0063] The user input interface (123) is for receiving information from a user. When information is input through the user input interface (123), the processor (180) can control the operation of the artificial intelligence device (100) to correspond to the input information.

[0064] The user input interface (123) may include a mechanical input means (or a mechanical key, for example, a button located on the front / rear or side of the artificial intelligence device (100), a dome switch, a jog wheel, a jog switch, etc.) and a touch input means.

[0065] As an example, the touch input means may be composed of virtual keys, soft keys, or visual keys displayed on a touch screen through software processing, or may be composed of touch keys placed on a part other than the touch screen.

[0066] The learning processor (130) can train a model composed of an artificial neural network using learning data. The trained artificial neural network can be referred to as a learning model. The learning model can be used to infer result values ​​for new input data other than the learning data, and the inferred values ​​can be used as a basis for judgment to perform a certain action.

[0067] The running processor (130) can perform AI processing together with the running processor (240) of the AI ​​server (200).

[0068] The running processor (130) may include a memory integrated or implemented in the artificial intelligence device (100). The running processor (130) may also be implemented using a memory (170), an external memory directly coupled to the artificial intelligence device (100), or a memory maintained in an external device.

[0069] The sensor (140) can obtain at least one of internal information of the artificial intelligence device (100), information about the surrounding environment of the artificial intelligence device (100), and user information by using various sensors.

[0070] The sensor (140) may include one or more of a proximity sensor, a light sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar sensor, and a radar sensor.

[0071] The output interface (150) can generate output related to visual, auditory, or tactile sensations.

[0072] The output interface (150) may include a display (151) that outputs images, an audio output interface (152) that outputs audio, a haptic device (153) that outputs tactile information, and a light output interface (154) that outputs light.

[0073] The display (151) displays (outputs) information processed in the artificial intelligence device (100). For example, the display (151) may display execution screen information of an application program running in the artificial intelligence device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0074] The display (151) can be implemented as a touch screen by forming a mutual layer structure with the touch sensor or forming an integral structure. The touch screen can function as a user input interface (123) that provides an input interface between the artificial intelligence device (100) and the user, and at the same time, provide an output interface between the artificial intelligence device (100) and the user.

[0075] The audio output interface (152) can output audio data received from the communication interface (110) or stored in the memory (170) in a call signal reception mode, call mode, recording mode, voice recognition mode, broadcast reception mode, etc.

[0076] The audio output interface (152) may include at least one of a receiver, a speaker, and a buzzer.

[0077] The haptic device (153) generates various tactile effects that can be felt by the user. A representative example of the tactile effect generated by the haptic device (153) may be vibration.

[0078] The light output interface (154) outputs a signal to notify the occurrence of an event using light from a light source of the artificial intelligence device (100). Examples of events occurring in the artificial intelligence device (100) may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, and information reception through an application.

[0079] The memory (170) can store data that supports various functions of the artificial intelligence device (100). For example, the memory (170) can store input data, learning data, learning models, learning history, etc. obtained from the input interface (120).

[0080] The processor (180) can determine at least one executable operation of the artificial intelligence device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0081] The processor (180) can control components of the artificial intelligence device (100) to perform determined operations.

[0082] To this end, the processor (180) can request, retrieve, receive or utilize data from the running processor (130) or memory (170), and control components of the artificial intelligence device (100) to execute at least one of the executable operations, a predicted operation or an operation determined to be desirable.

[0083] When the processor (180) requires connection to an external device to perform a determined operation, it can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

[0084] The processor (180) can obtain intent information for user input and determine the user's requirements based on the obtained intent information.

[0085] The processor (180) can obtain intent information corresponding to the user input by using at least one of a STT (Speech To Text) engine for converting voice input into a string or a natural language processing (NLP) engine for obtaining intent information of natural language.

[0086] At least one of the STT engine or the NLP engine may be configured with an artificial neural network, at least in part, trained according to a machine learning algorithm. Furthermore, at least one of the STT engine or the NLP engine may be trained by the learning processor (130), the learning processor (240) of the AI ​​server (200), or through distributed processing thereof.

[0087] The processor (180) can collect history information including the operation details of the artificial intelligence device (100) or the user's feedback on the operation, and store the information in the memory (170) or the learning processor (130), or transmit the information to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0088] The processor (180) can control at least some of the components of the artificial intelligence device (100) to run an application program stored in the memory (170).

[0089] The processor (180) can operate two or more of the components included in the artificial intelligence device (100) in combination to drive the application program.

[0090] FIG. 2 is a diagram for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0091] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0092] The AI ​​server (200) may be composed of multiple servers and perform distributed processing, and may be defined as a 5G network device. The AI ​​server (200) may be included as part of the artificial intelligence device (100) and may perform at least a portion of the AI ​​processing.

[0093] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0094] The communication interface (210) can transmit and receive data with an external device such as an artificial intelligence device (100).

[0095] The memory (230) may include a model memory (231). The model memory (231) may store a model (or artificial neural network, 231a) being learned or learned through the learning processor (240).

[0096] A learning processor (240) can train an artificial neural network (231a) using learning data. The learning model can be used while mounted on the AI ​​server (200) of the artificial neural network, or can be mounted on an external device such as an artificial intelligence device (100).

[0097] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0098] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0099] Hereinafter, the artificial intelligence device (100) of FIG. 1 may be referred to as a device (100). The device (100) may include all components of the artificial intelligence device (100) illustrated in FIG. 1.

[0100] Hereinafter, one or more processors (180) may be provided.

[0101] FIGS. 3 and 4 are drawings illustrating a learning method for generating a 3D image including a 3D object according to one embodiment of the present disclosure.

[0102] Hereinafter, a unit object may represent an object corresponding to a relatively large and distinct subject, and a detailed object may represent a partial object that constitutes the unit object.

[0103] The first level can be a Coarse level, and the second level can be a Fine level.

[0104] The Coarse level can be a level that identifies (segments) multiple unit objects contained within an image.

[0105] The Fine level can be a level that identifies (segments) detailed objects from each unit object contained in the image.

[0106] Object segmentation can be a process of identifying unit objects or detailed objects of each unit object within an image, and distinguishing each identified unit object from other unit objects or detailed objects.

[0107] A segment can be a spatially divided area or representation of an object.

[0108] A mask can be a binary or multi-class image generated to represent a segment on an image.

[0109] A 3D object can represent either a unit object or a detailed object.

[0110] Steps S301 to S311 of FIG. 3 may be a learning process of a 3D segment model.

[0111] A 3D segment model may be a model that generates a 3D image in which one or more objects are segmented from a plurality of 2D images corresponding to a plurality of viewpoints.

[0112] Referring to FIGS. 3 and 4, the processor (180) of the device (100) can generate a first mask image of a first level and a second mask image of a second level based on a plurality of 2D images (S301).

[0113] Each of the plurality of 2D images (400-1 to 400-n) may be images captured from different views (or viewpoints, view angles). The plurality of 2D images (400-1 to 400-n) may be referred to as training view images.

[0114] The processor (180) can obtain a first mask image of the first level and a second mask image of the second level from each 2D image using a SAM (Segment Anything model).

[0115] SAM may be a model capable of identifying the location of an object within an image based on simple user input (click, box), and automatically segmenting the object based on the identified location. SAM may be stored in memory (170).

[0116] FIGS. 5A and 5B are diagrams illustrating a process of generating a first mask image of a first level and a second mask image of a second level from a 2D image through SAM according to one embodiment of the present disclosure.

[0117] The training image (500) may be a 2D image captured from a specific viewing angle. The training image (500) may be composed of a plurality of pixels.

[0118] The processor (180) can independently execute SAM for each training image (500) to generate a plurality of segments (511, 512, 513). The plurality of segments (511, 512, 513) can include a first segment (511), a second segment (512), and a third segment (513). Each of the plurality of segments (511, 512, 513) can be a partial region included in the entire region formed by the training image (500). Each of the plurality of segments (511, 512, 513) can correspond to an object included in the training image (500).

[0119] The pixels (501) included in the training image (500) may be included in each of the first segment (511), the second segment (512), and the third segment (513).

[0120] The processor (180) can generate a first level mask (521) by assigning pixels (501) included in the training image (500) to a first segment (511) having the largest area among a plurality of segments (511, 512, 513). The first level mask (521) can have a segment identifier that identifies a unit object.

[0121] The processor (180) can generate a first mask image (520) including a plurality of first level masks by assigning each of a plurality of pixels included in the training image (500) to a segment with the largest area in order of priority.

[0122] The processor (180) can generate a second level mask (531) by assigning a pixel (501) included in a training image (500) to a third segment (513) having the smallest area among a plurality of segments (511, 512, 513). The second level mask (531) can have a segment identifier that identifies a detailed object.

[0123] The processor (180) can generate a second mask image (530) including a plurality of second level masks by assigning each of the plurality of pixels included in the training image (500) to a segment with the smallest area in order of priority.

[0124] Referring to FIG. 5b, the results of applying SAM to an actual 2D image (540) to obtain a first mask image (550) at a coarse level and a second mask image (560) at a fine level are shown.

[0125] Again, Figures 3 and 4 are described.

[0126] The processor (180) can generate a first feature field of a first level and a second feature field of a second level based on a plurality of 2D images (S303).

[0127] The processor (180) can generate a first level feature field and a second level feature field based on a plurality of 2D images through a 3D Gaussian Splatting (3DGS) method.

[0128] The 3D Gaussian splatting method can classify a 3D view image generated by a plurality of 2D images (400-1 to 400-n) into a plurality of 3D Gaussians, and project the classified plurality of 3D Gaussians onto a 2D screen to render a feature field.

[0129] The processor (180) can generate a first feature field from a coarse level feature vector and a second feature field from a fine level feature vector using a Rasterizer rendering method.

[0130] FIG. 6A and FIG. 6B are diagrams illustrating a process of generating a first feature field of a first level and a second feature field of a second level based on a plurality of 2D images.

[0131] Referring to FIG. 6a, a 3D view image (600) reconstructed based on a plurality of 2D images (600-1 to 600-n) corresponding to a plurality of view points is illustrated.

[0132] The 3D view image (600) may be a scene rendering result based on a virtual camera that is configured by integrating multiple viewpoint information.

[0133] The processor (180) can extract a plurality of 3D Gaussians from a 3D view image (600). A 3D Gaussian can be a unit element that expresses visual information such as shape, position, and color of an object in a 3D space in the form of a probability distribution.

[0134] Each 3D Gaussian can be expressed by the following [Mathematical Formula 1].

[0135] [Mathematical Formula 1]

[0136]

[0137] gi is the ith 3D Gaussian, pi is the position of the ith 3D Gaussian, si is the scale of the ith 3D Gaussian, qi is the quaternion of the ith 3D Gaussian, oi is the opacity of the ith 3D Gaussian, and ci can be the color of the ith 3D Gaussian.

[0138] The processor (180) can obtain an augmented 3D Gaussian by combining a 3D Gaussian and a corresponding feature vector.

[0139] The augmented 3D Gaussian can be expressed as follows [Mathematical Formula 2].

[0140] [Equation 2]

[0141]

[0142] is the i-th augmented 3D Gaussian, gi is the i-th 3D Gaussian, and may be the i-th feature vector at the fine level. The feature vector may be referred to as a feature. The augmented 3D Gaussian may be a Gaussian in which feature vectors corresponding to the original 3D Gaussian are combined.

[0143] The i-th feature vector of fine level ( ) is the i-th feature vector of the coarse level ( ) may be included.

[0144] The i-th feature vector of fine level ( ) can be obtained by the following [Mathematical Formula 3].

[0145] [Equation 3]

[0146]

[0147] can be the feature vector of the ith coarse level. It can be the i-th transformed feature vector that is processed (projected or upsampled) from the i-th coarse level feature vector to the fine level.

[0148] The i-th feature vector of fine level ( ) is the feature vector of the i-th coarse level ( ) and the i-th transformed feature vector ( ) can be obtained by combining them through a concatenation function.

[0149] According to the granularity prior, features that are already distinguished at the coarse level should also be distinguished at the fine level, so the i-th feature vector at the fine level ( ) is the feature vector of the i-th coarse level ( ) can be configured to include.

[0150] Granularity prior is a priori assumption that features identifiable at the coarse level should also be maintained at the fine level, which ensures consistency in multi-level learning.

[0151] Referring to FIG. 6b, the processor (180) can generate a first feature field (610) from a coarse level feature vector and a second feature field (620) from a fine level feature vector using a Rasterizer rendering method.

[0152] Rasterizer rendering method can be a method of converting feature vectors in the form of 3D modeling into a 2D image composed of pixels.

[0153] The Rasterizer rendering method can be a method of generating a feature field through the following [Mathematical Formula 4].

[0154] [Equation 4]

[0155]

[0156] represents the feature field, represents the ith feature vector, represents the rasterization weight of the ith 3D Gaussian, Ti can represent the ith transformation matrix, and l can represent the subdivision level.

[0157] Each of the first feature field (610) and the second feature field (620) may be a pixel-wise feature distribution map generated by projecting 3D Gaussians onto a 2D screen using a Rasterizer rendering method.

[0158] Again, Figure 3 is explained.

[0159] The processor (180) can perform contrastive learning on the generated first mask image and the generated first feature field, and can perform contrastive learning on the generated second mask image and the generated second feature field (S305).

[0160] Step S305 may correspond to the Contrastive Learning process (410) of FIG. 4.

[0161] The processor (180) can perform contrastive learning for the first mask image and the first feature field so that pixels having the same segment identifier (segment ID) have similar feature vectors, and pixels having different segment identifiers have different feature vectors.

[0162] The processor (180) can perform contrastive learning for the second mask image and the second feature field so that pixels having the same segment identifier (segment ID) have similar feature vectors, and pixels having different segment identifiers have different feature vectors.

[0163] FIG. 7a and FIG. 7b are diagrams illustrating a process of performing contrastive learning according to an embodiment of the present disclosure.

[0164] Referring to FIG. 7a, a first feature field (610) corresponding to a coarse level and a first mask image (550) corresponding to the coarse level are illustrated.

[0165] The feature vector of the first pixel (p1) included in the first feature field (610) is , and the feature vector of the second pixel (p2) is can be expressed as

[0166] The processor (180) can calculate the cosine similarity of two feature vectors through the following [Mathematical Formula 5].

[0167] [Equation 5]

[0168]

[0169] can represent the cosine similarity between feature vectors corresponding to two pixels (p1, p2).

[0170] The processor (180) determines if the segment IDs of two pixels (p1, p2) within the first mask image (550) of the coarse level are the same. = , positive pair), the loss function of contrastive learning can be calculated through the following [Mathematical Formula 6].

[0171] [Equation 6]

[0172]

[0173] can be a positive loss function to increase the cosine similarity of feature vectors corresponding to pixels belonging to the same segment.

[0174] The processor (180) can perform contrastive learning so that a positive loss function is minimized. Accordingly, feature vectors of pixel pairs belonging to the same segment can be learned to become closer to each other.

[0175] Referring to FIG. 7b, a first feature field (610) corresponding to a coarse level and a first mask image (550) corresponding to the coarse level are illustrated.

[0176] The feature vector of the first pixel (p1) included in the first feature field (610) is , and the feature vector of the second pixel (p2) is can be expressed as

[0177] The processor (180) can calculate the cosine similarity of two feature vectors through the above [Mathematical Formula 5].

[0178] The processor (180) detects when the segment IDs of two pixels (p1, p2) are different within the first mask image (550) of the coarse level. , negative pair), the loss function of contrastive learning can be calculated through the following [Mathematical Formula 7].

[0179] [Equation 7]

[0180]

[0181] can be a negative loss function to reduce the cosine similarity of feature vectors corresponding to pixels belonging to different segments.

[0182] The processor (180) can perform contrastive learning so that a negative loss function is maximized. Feature vectors of pixel pairs belonging to different segments can be learned to become closer to each other.

[0183] Meanwhile, in FIG. 7b, the processor (180) determines whether the cosine similarity of the feature vectors corresponding to two pixels (p1, p2) is a threshold similarity ( ) can be used to perform contrastive learning. At the coarse level, the threshold similarity can be 0.5, and at the fine level, the threshold similarity can be 0.75, but these are just examples.

[0184] Contrast learning according to FIGS. 7a and 7b can also be performed for the fine level.

[0185] The overall loss function of contrastive learning can be expressed as the following [Mathematical Formula 8].

[0186] [Equation 8]

[0187]

[0188] can represent the overall loss function of contrastive learning, Is and It may be a hyperparameter that controls the balance.

[0189] For this type of contrastive learning, depending on the view of the 2D image, any one pixel contained in the mask image may have the same segment ID or different segment IDs.

[0190] Figure 8 is a diagram illustrating that the segment ID to which a pixel belongs may vary depending on the view of the 2D image.

[0191] Referring to FIG. 8, a first image (810) captured from a first view and a second image (830) captured from a different angle from the first view are illustrated.

[0192] Each of the first image (810) and the second image (830) may include a first unit object (811) and a second unit object (812).

[0193] Additionally, a coarse level mask image (820) obtained from the first image (810) through SAM and a coarse level mask image (840) obtained from the second image (830) through SAM are shown.

[0194] In the mask image (820) of the first view, the first unit object (811) and the second unit object (812) have the same segment ID, and therefore can be classified as the same segment (821).

[0195] However, in the mask image (840) of the second view, the first unit object (811) corresponds to the first segment (841), and the second unit object (812) corresponds to the second segment (842), and thus have different segment IDs, and thus can be classified into different segments.

[0196] In this way, mask images generated based on SAM may experience instability in contrastive learning due to view mismatch.

[0197] Accordingly, this disclosure aims to resolve the instability of contrastive learning through global feature-guided learning (GFL).

[0198] Again, Figures 3 and 4 are described.

[0199] The processor (180) can obtain an average feature set including a plurality of average feature vectors corresponding to each of a plurality of views (S307).

[0200] Step S307 may correspond to the Average Pooling process (420) of FIG. 4.

[0201] The processor (180) can obtain multiple global clusters by clustering the obtained average feature set (S309).

[0202] Step S309 may correspond to the Clustering process (430) of FIG. 4.

[0203] The processor (180) can perform contrastive learning between each global feature candidate and feature vector representing each of the extracted multiple global clusters (S311).

[0204] Step S311 may correspond to the global feature candidate extraction process (440) and the global feature induction learning (GFL) process of FIG. 4.

[0205] Steps S307 to S311 may be included in a global feature induction learning (GFL) process.

[0206] The processor (180) can learn a 3D segment model through steps S301 to S311.

[0207] FIGS. 9A to 9D are diagrams for explaining global feature induction learning according to an embodiment of the present disclosure.

[0208] Global feature induction learning may be a process of segmenting an object by extracting global feature candidates representing each segment based on 2D images corresponding to multiple views, and grouping segments corresponding to the same object by clustering the extracted global feature candidates.

[0209] Referring to FIG. 9A, a first mask image (910) of a coarse level obtained from a 2D image corresponding to a first view through SAM and a first feature field (920) of a coarse level obtained based on 3D Gaussians are illustrated. Each of a plurality of segments constituting the first mask image (910) may be assigned a different segment ID. A plurality of pixels constituting each segment may be assigned the same segment ID.

[0210] The processor (180) can average a plurality of feature vectors corresponding to a plurality of pixels included in a first feature vector set (921) corresponding to a first segment (911) included in a first mask image (910) through an average pooling method (Average Pooling).

[0211] The processor (180) can obtain a first average feature vector (901) obtained through an averaging process. The processor (180) can place the first average feature vector (901) on a feature space (900). The feature space (900) may be a high-dimensional feature space.

[0212] Referring to FIG. 9b, the processor (180) can average a plurality of feature vectors corresponding to a plurality of pixels included in a second feature vector set (922) corresponding to a second segment (912) through an average pooling method.

[0213] The processor (180) can obtain a second average feature vector (902) obtained through an averaging process. The processor (180) can place the second average feature vector (902) on a feature space (900).

[0214] Referring to FIG. 9c, a second mask image (930) of a coarse level obtained from a 2D image corresponding to a second view through SAM and a first feature field (940) of a coarse level obtained based on 3D Gaussians are illustrated. Each of a plurality of segments constituting the second mask image (930) may be assigned a different segment ID. A plurality of pixels constituting each segment may be assigned the same segment ID.

[0215] The processor (180) can average a plurality of feature vectors corresponding to a plurality of pixels included in a third feature vector set (941) corresponding to a third segment (931) through an average pooling method.

[0216] The processor (180) can obtain a third average feature vector (903) obtained through an averaging process. The processor (180) can place the third average feature vector (903) on a feature space (900).

[0217] The processor (180) can obtain an average feature set including average feature vectors (901 to 905) obtained from all views, and place the obtained average feature set on a feature space (900), as illustrated in FIG. 9d.

[0218] Meanwhile, the processor (180) can obtain an average feature set through the following [Mathematical Formula 9] by applying the average pooling method.

[0219] [Equation 9]

[0220]

[0221] can be defined as a set of masks for all viewpoints for two levels of masks (e.g., coarse level and fine level). V can be the total number of viewpoints. can be a mask for the viewpoint (v) at level l. is a mean feature set including mean feature vectors, and l={f,c} can represent a granularity level. f can represent a fine level, c can represent a coarse level, and s can identify a segment that identifies an object defined in the mask. s can be a segment ID.

[0222] can represent the average feature vector representing the segment (s) of the viewpoint (v).

[0223] can represent a set of pixels belonging to a segment (s) of a viewpoint (v).

[0224] can represent the feature vector of a pixel (p).

[0225] D can represent the dimension of the feature vector.

[0226] The processor (180) can perform clustering on an average feature set arranged on a feature space (900) using the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) method.

[0227] The HDBSCAN method is a density-based unsupervised learning method that automatically determines the number of clusters by considering the spatial distribution of average feature vectors and can be a clustering method that excludes noise data.

[0228] The processor (180) can generate multiple global clusters (950, 960, 970) according to the HDBSCAN clustering method. The processor (180) can obtain multiple global feature candidates (951, 961, 971) representing each global cluster by applying the HDBSCAN method to the average feature vectors. Each global cluster can be a cluster that identifies one unit object. The multiple global feature candidates can be referred to as

[0229] Each global feature candidate can be the candidate located at the very center of each global cluster.

[0230] The first global feature candidate (951) may represent the first global cluster (950), the second global feature candidate (961) may represent the second global cluster (960), and the third global feature candidate (971) may represent the third global cluster (970).

[0231] After that, the processor (180) selects the i-th feature vector ( at level l among multiple global clusters) through the following [Mathematical Formula 10]. ) or find the global cluster closest to the i-th 3D Gaussian. The i-th feature vector ( ) can be referred to as Gaussian features.

[0232] [Equation 10]

[0233]

[0234] is the ith Gaussian feature at the segmentation level (l). ) can represent the identifier of the global cluster (c) most similar to it.

[0235] is the i-th Gaussian feature ( ) and global feature candidates representing the center of the global cluster ( ) can represent the cosine similarity between them.

[0236] A positive loss function to induce Gaussian features to belong to the global cluster with the highest probability of belonging to it ( ) can be defined as in the following [Mathematical Formula 11].

[0237] [Equation 11]

[0238]

[0239] N can be the number of Gaussian features.

[0240] can represent a positive loss function for global feature induction learning (GFL).

[0241] can be a threshold value to determine whether a particular Gaussian feature belongs to the corresponding global cluster, and can be set to 0.9.

[0242] The closeness between Gaussian features and the nearest global cluster If it is larger, the Gaussian features can be trained to bring the closest global cluster closer together.

[0243] Conversely, a negative loss function that induces Gaussian features to move away from other global clusters can be defined as follows [Mathematical Formula 12].

[0244] [Equation 12]

[0245]

[0246] can represent a negative loss function for global feature induction learning (GFL).

[0247] Gaussian features have a critical similarity ( ) between global clusters other than the nearest global cluster. ) can be learned to move away from it.

[0248] The total loss of contrastive learning can be defined as follows [Mathematical Formula 13].

[0249] [Equation 13]

[0250]

[0251] In this way, global feature-guided learning can be a learning method that maintains viewpoint consistency by utilizing global feature candidates obtained from all viewpoints in a scene. In short, global feature-guided learning can be a learning method that guides the features of each 3D Gaussian to belong to a specific global cluster, while simultaneously distancing them from other global clusters.

[0252] FIG. 10 is a flowchart illustrating an operation method of a device according to another embodiment of the present disclosure.

[0253] Referring to FIG. 10, the processor (180) of the device (100) can obtain a plurality of 2D images corresponding to a plurality of viewpoints (S1001).

[0254] In one embodiment, the plurality of 2D images may be images captured from a plurality of shooting angles. A viewpoint may be referred to as a shooting angle, a viewing angle, or a view.

[0255] In one embodiment, the processor (180) can acquire multiple 2D images captured through the camera (121).

[0256] In another embodiment, the processor (180) can receive multiple 2D images from an external device via a communication interface (110).

[0257] The processor (180) can obtain a 3D image including one or more segmented 3D objects based on the obtained plurality of 2D images and display the obtained 3D image on the display (151) (S1003).

[0258] The processor (180) can obtain a 3D image including one or more 3D objects segmented from a plurality of 2D images through a 3D segment model.

[0259] The 3D segment model may be a model learned according to the embodiment of FIG. 3.

[0260] FIG. 11 is a drawing for explaining a 3D segment model according to one embodiment of the present disclosure.

[0261] In one embodiment, the 3D segment model (1100) may be an artificial neural network-based model learned by the learning processor (130) or processor (180) of the device (100).

[0262] In another embodiment, the 3D segment model (1100) may be a model learned by the server (200), and the processor (180) may receive the 3D segment model (1100) from the server (200) through the communication interface (110) and store it in the memory (170).

[0263] A 3D segment model (1100) may be a model that outputs a 3D image (1120) including one or more objects segmented from a plurality of 2D images (1100-1 to 1100-n) corresponding to a plurality of viewpoints.

[0264] The 3D segment model (1100) may be a model learned according to the embodiment of FIG. 3.

[0265] The 3D segment model (1100) may be a model that segments an object by extracting global feature candidates representing each segment based on a plurality of 2D images corresponding to a plurality of viewpoints, and grouping segments corresponding to the same object by clustering the extracted global feature candidates.

[0266] One or more 3D objects included in the 3D image (1120) may be selectable (or clickable).

[0267] Again, Figure 10 is explained.

[0268] Meanwhile, the processor (180) may display the acquired 3D image in a first region and one or more objects segmented within the 3D image in a second region. The first region and the second region may not overlap.

[0269] The processor (180) can receive a first command to select one of one or more 3D objects included in a 3D image (S1005).

[0270] In one embodiment, the processor (180) may receive a first command to select a 3D object via a pointer corresponding to the movement of a mouse connected via a user input interface (123).

[0271] In another embodiment, the processor (180) may receive a touch command for selecting a 3D object as a first command when the display (151) is implemented as a touch screen.

[0272] The processor (180) can display a UI element corresponding to the selected 3D object on the display (151) according to the received first command (S1007).

[0273] A UI element may be an element that indicates that a 3D object is selected.

[0274] In one embodiment, the UI element may be an element for identifying a selected 3D object.

[0275] For example, the processor (180) can highlight a selected 3D object according to the first command.

[0276] As another example, the processor (180) may display an indicator indicating that a 3D object has been selected in accordance with the first command at a location adjacent to the selected 3D object. The indicator may be an arrow or an animation, but this is merely an example.

[0277] In another embodiment, a UI element may include detail objects that constitute a selected 3D object. The detail objects may be segmented sub-objects at the fine level.

[0278] The processor (180) can receive a second command for editing the selected 3D object (S1009).

[0279] In one embodiment, the second command may be a command for changing properties of the 3D object. For example, the second command may be a command for editing one or more of the name of the 3D object, the color of the 3D object, the size of the 3D object, or the position of the 3D object.

[0280] The processor (180) can display the edited 3D object on the display (151) according to the received second command (S1011).

[0281] Below, the embodiment of Fig. 10 is described in detail.

[0282] FIGS. 12A to 12I are drawings for explaining a process of editing a selected 3D object on a 3D image including a plurality of 3D objects according to an embodiment of the present disclosure.

[0283] The processor (180) can obtain a 3D image (1120) from a plurality of 2D images (1100-1 to 1100-n) corresponding to a plurality of viewpoints through a 3D segment model (1100) as illustrated in FIG. 11.

[0284] Referring to FIGS. 12a to 12i, execution screens of a segment application for editing a 3D image obtained through a 3D segment model (1100) are illustrated.

[0285] Referring to FIG. 12a, the processor (180) of the device (100) can display a first screen (1201) on the display (151) according to the execution of the segment application.

[0286] The first screen (1201) may include a 3D image (1120) obtained from a plurality of 2D images (1100-1 to 1100-n) corresponding to a plurality of viewpoints and an editing menu (1210).

[0287] The edit menu (1210) may include a rendering menu (1211), a 3D segmentation menu (1212), and a 3D object editing menu (1213).

[0288] The rendering menu (1211) may be a menu for settings related to image rendering.

[0289] The rendering menu (1211) may include a mode item (1211-1) for setting the image display mode and a FOV item (1211-2) for setting the FOV (Field Of View).

[0290] The image mode item (1211-1) can be set to either an image mode for displaying a 3D image or a segment mode for displaying a segment image. In Fig. 12a, the mode item (1211-1) can be set to an image mode.

[0291] The rendering menu (1211) may further include an item for setting the background color of the area in which the image is displayed.

[0292] The 3D segmentation menu (1212) may be a menu for setting the segmentation level of a mask image. The 3D segmentation menu (1212) may include a segmentation level item (1212-1) for setting the segmentation level. The segmentation level item (1212-1) may be set to either a coarse level or a fine level. It is assumed that the 3D image (1120) is an image set to a coarse level. The 3D image (1120) may be referred to as a coarse level 3D image.

[0293] The 3D object editing menu (1213) may be a menu for editing the properties of a 3D object selected in a 3D image (1120).

[0294] The 3D object editing menu (1213) may include a text-based editing item (1213-1), a size item (1213-2), a transformation item (1213-3), and a remove item (1213-4).

[0295] A text-based editing item (1213-1) may be an item for editing the properties of a 3D object using text input.

[0296] The size item (1213-2) may be an item for editing the size of a 3D object.

[0297] The transformation item (1213-3) may be an item for transforming the direction of a 3D object or for moving a 3D object. The transformation item (1213-3) may be an item for receiving an input for moving a 3D object in the x, y, and z-axis directions.

[0298] The removal item (1213-4) may be an item for removing a 3D object within a 3D image (1120).

[0299] The 3D image (1120) included in the first screen (1201) may include a plurality of 3D objects segmented by the 3D segment model (1100).

[0300] A pointer (1200, or cursor) may be displayed on the first screen (1201), and the processor (180) may receive input for manipulating the pointer (1200) to select (or click) and edit a 3D object.

[0301] The processor (180) can display a second screen (1202) including a first segment image (1220) when the image display mode is set to segment mode through the mode item (1211-1) and the segmentation level is set to coarse level through the segmentation level item (1212), as shown in FIG. 12b.

[0302] The first segment image (1220) may be an image that identifies selectable 3D objects at a coarse level. The coarse level may be a level that segments the entire area into unit objects. The first segment image (1220) may be a mask image. For example, the first segment image (1220) may correspond to the first mask image (550) of FIG. 5B .

[0303] The processor (180) can display a third screen (1203) including a second segment image (1230) when the image display mode is set to segment mode through the mode item (1211-1) and the segmentation level is set to fine level through the segmentation level item (1212), as shown in FIG. 12c.

[0304] The second segment image (1230) may be an image that identifies selectable 3D objects at a fine level. The fine level may be a level that segments each of a plurality of unit objects into detailed objects. The second segment image (1230) may be a mask image. For example, the second segment image (1230) may correspond to the second mask image (560) of FIG. 5B.

[0305] Referring to FIG. 12d, when the processor (180) receives a command to select a first 3D object (1121) within a 3D image (1120), the processor (180) can display a fourth screen (1204) identifying the first 3D object (1121) selected according to the received command on the display (151).

[0306] The processor (180) can highlight the selected first 3D object (1121) or identify the first 3D object (1121). The processor (180) can further display an indicator (not shown) indicating that the first 3D object (1121) has been selected.

[0307] When the processor (180) receives a command to adjust the size of the first 3D object (1121) selected through the size item (1213-2), as illustrated in FIG. 12e, the processor (180) may display a second screen (1205) including the first 3D object (1121) whose size has been adjusted on the display (151). The command to adjust the size of the first 3D object (1121) may be a command to input a scale value.

[0308] Meanwhile, a second 3D object (1122) may be placed adjacent to the first 3D object (1121).

[0309] After the second 3D object (1122) is selected, the processor (180) can move the second 3D object (1122) from the first point to the second point according to a transformation command that inputs an x-axis value, a y-axis value, and a z-axis value through a transformation item (1213-3). The processor (180) can display a sixth screen (1206) showing the state in which the second 3D object is moved according to the transformation command of the second 3D object (1122) on the display (151).

[0310] The processor (180) can reflect the edited 3D object on the 3D image (1120) in real time according to a command to edit the value of any one of the subdivision level item (1212-1), the size item (1213-2), or the transformation item (1213-3) for setting the position of the selected 3D object.

[0311] In this way, according to the embodiment of the present disclosure, since editing is possible in units of segmented objects on a 3D image, properties of each object can be individually adjusted, thereby providing precise customization and an intuitive interface without affecting the entire scene.

[0312] When the subdivision level item (1212-1) is set to the fine level while the sixth screen (1206) is displayed, the processor (180) can display the seventh screen (1207) including the fine level 3D image (1120-1) on the display (151), as in FIG. 12g.

[0313] A fine level 3D image (1120-1) may be an image in which each of a plurality of 3D objects is subdivided into detailed objects.

[0314] Detailed objects that make up a 3D object in a fine level 3D image (1120-1) can be edited through the edit menu (1210).

[0315] A first 3D object (1121) can be segmented into a first detailed object (1121a) and a second detailed object (1121b). Each of the first detailed object (1121a) and the second detailed object (1121b) can be selected or clicked.

[0316] If the first 3D object (1121) is an animal figure, the first detailed object (1121a) may represent the torso of the animal figure, and the second detailed object (1121b) may represent the legs of the animal figure.

[0317] The processor (180) can edit the size and position of the detailed object selected from among the first detailed object (1121a) or the second detailed object (1121b) according to an editing command of the corresponding detailed object.

[0318] The processor (180) may display an eighth screen (1208) in which a source text "apple" indicating a source and a target text "rainbow color apple" indicating a target are entered on a text-based editing item (1213-1), as illustrated in FIG. 12h. The source text may be a text for selecting a first detailed object (1123a), and the target text may be a text for editing an attribute of the selected first detailed object (1123a).

[0319] A third 3D object (1123) representing a fruit can be segmented into a first detailed object (1123a) representing an apple and a second detailed object (1123b) representing a leaf.

[0320] The processor (180) can select a first detailed object (1123a) of a third 3D object (1123) corresponding to an apple based on the input of the source text. The processor (180) can edit the first detailed object (1123a) of the third 3D object (1123) selected based on the input of the target text to have a rainbow color.

[0321] The processor (180) can switch the eighth screen (1208) including the first detailed object (1123a) having a green color to the ninth screen (1209) including the first detailed object (1123a) having a rainbow color according to the input of the source text and the target text (see FIG. 12i).

[0322] In this way, according to an embodiment of the present disclosure, by segmenting detailed objects constituting a unit object on a 3D image, the properties of each detailed object can be individually adjusted, and the user can perform precise customization at the detailed element level of the object.

[0323] FIGS. 13A to 13D are drawings for explaining a method for providing information on segmented 3D objects in a 3D image according to an embodiment of the present disclosure.

[0324] Referring to FIG. 13A, the processor (180) can display a 3D image (1300) including a plurality of 3D objects (1301, 1302, 1303, 1304) on the display (151). The processor (180) can further display an edit menu (1210) on one side of the 3D image (1300). The display of the edit menu (1210) is omitted.

[0325] The processor (180) can display a 3D object selected from among a plurality of 3D objects (1301, 1302, 1303, 1304) on an area other than an area where a 3D image (1300) is displayed.

[0326] For example, when a first 3D object (1301) is selected, the processor (180) can display the selected first 3D object (1301) on a first area (1311) that is a separate area from the area where the 3D image (1300) is displayed. The processor (180) can display the detailed object (1301a) on the first area (1311) according to a command to select the detailed object (1301a) of the first 3D object (1301) on the first area (1311).

[0327] When a second 3D object (1302) is selected, the processor (180) can display the selected second 3D object (1302) on a second area (1312) that is a separate area from the area where the 3D image (1300) is displayed. The processor (180) can display the detailed object (1302a) on the second area (1312) according to a command for selecting the detailed object (1302a) of the second 3D object (1302) on the second area (1312).

[0328] In another embodiment, the processor (180) may determine that the first 3D object (1301) is selected when the detailed object (1301a) is selected once (or clicked), and may determine that the detailed object (1301a) is selected twice in a row (or double-clicked).

[0329] In this way, according to an embodiment of the present disclosure, since segmented 3D objects and detailed objects of 3D objects on a 3D image are displayed on separate areas, the cognitive burden of manipulating a specific object in the entire scene can be reduced, and the accuracy of user input can be increased.

[0330] Referring to FIG. 13b, the processor (180) can display a 3D image (1300), a coarse level result window (1330), and a fine level result window (1340) on the display (151).

[0331] When a 3D image (1300) is acquired from a plurality of 2D images corresponding to a plurality of viewpoints, the processor (180) can display a 3D image (1300), a coarse level result window (1330), and a fine level result window (1340) on the display (151).

[0332] The coarse level result window (1330) may include a plurality of unit objects segmented from the 3D image (1300). Each of the plurality of segmented unit objects may be selectable and editable.

[0333] The fine level result window (1340) may include detailed objects that subdivide each of the multiple unit objects included in the coarse level result window (1330). Each of the subdivided detailed objects may be selectable and editable.

[0334] In this way, according to an embodiment of the present disclosure, by displaying coarse and fine level results separately and enabling selection and attribute editing for each segment, a user can perform intuitive and precise editing work based on a hierarchical structure even within a complex 3D scene.

[0335] Referring to FIG. 13c, the processor (180) can display a 3D image (1300) and tag information (1350) on the display (151). The tag information (1350) may be information for identifying 3D objects that can be selected and edited. The tag information (1350) may include one or more of the color of the 3D object and the name of the 3D object.

[0336] The processor (180) can further display a plurality of 3D view images (1300-1, 1300-2, 1300-3). Each of the plurality of 3D view images (1300-1, 1300-2, 1300-3) may be an image corresponding to a different view point.

[0337] In this way, according to an embodiment of the present disclosure, tag information for an editable 3D object is provided, so that a user can easily and clearly recognize the editable object.

[0338] Referring to FIG. 13D, the processor (180) can identify a plurality of editable 3D objects within a 3D image (1300). For example, the processor (180) can identify a plurality of editable 3D objects within the 3D image (1300) with different colors. Specifically, the color of a first 3D object (1301) can be expressed as a first color, and the color of a second 3D object (1302) can be expressed as a second color.

[0339] FIG. 14 is a drawing illustrating a method for guiding shooting when shooting a 2D image to create a 3D image according to one embodiment of the present disclosure.

[0340] Referring to FIG. 14, the processor (180) can display a 2D image (1410) corresponding to a specific viewpoint captured by the camera (121) on the display (151).

[0341] Users can capture a scene multiple times with different viewpoints.

[0342] The processor (180) may display a guide on the display (151) indicating that additional shooting is required when additional 2D images are required to generate a 3D image. The guide may be expressed in the form of text or a progress bar.

[0343] The processor (180) may further display a first progress bar (1420) indicating the progress of the generation of a 3D image. The first progress bar (1420) may be a UI indicating that additional shooting for another viewpoint is required for outputting a 3D image through a 3D segment model (1100).

[0344] The processor (180) may display a second progress bar (1421) on one side of the first object (1411) indicating that additional shooting for a different viewpoint is required for a segment of the first object (1411). The second progress bar (1421) may indicate the degree of need for additional shooting for a scene including the first object (1411).

[0345] The processor (180) may display a third progress bar (1422) on one side of the second object (1412) indicating that additional shooting for a different viewpoint is required for a segment of the second object (1412). The third progress bar (1422) may indicate the degree to which additional shooting is required for a scene including the second object (1412).

[0346] Additionally, the processor (180) can distinguish and display objects that can be reconstructed into 3D objects and objects that cannot be reconstructed into 3D objects among a plurality of objects included in a 2D image (1410).

[0347] According to an embodiment of the present disclosure, by visually distinguishing and displaying the progress of segment learning for each object on a 2D image (1410), a user can intuitively recognize which object requires additional shooting, and effectively respond to securing an optimal viewpoint for generating a 3D image.

[0348] Additionally, by distinguishing between objects that can be reconstructed as 3D objects and those that cannot, users can selectively collect data that is valid for 3D modeling, thereby improving the efficiency and accuracy of the entire shooting and learning process.

[0349] FIG. 15 is a drawing illustrating an example of performing furniture arrangement in a home based on a 3D image obtained through a 3D segment model according to one embodiment of the present disclosure.

[0350] Referring to FIG. 15, the processor (180) can obtain a 3D image (1500) from a plurality of 2D images corresponding to a plurality of viewpoints through a 3D segment model (1100). The 3D image (1500) can include a segmented TV object (1510) corresponding to a TV.

[0351] The processor (180) may further display an editing menu (1210) on one side of the 3D image (1500). The processor (180) may change at least one of the size or position of the TV object (1510) according to a first command for selecting a TV object (1510) and a second command for editing the selected TV object (1510).

[0352] By editing the size and placement of the TV object (1510), the user can adjust the visual configuration or placement environment occupied by the TV object (1510) within 3D space to suit the user's intentions. Furthermore, this can enhance the realism and usability of virtual interior design and spatial simulation.

[0353] Meanwhile, when editing the size, color, or texture of a TV object (1510), the processor (180) may provide a link for purchasing similar products. For example, when enlarging the size of a TV object (1510), the processor (180) may display the address of a website selling a TV corresponding to the enlarged size.

[0354] FIG. 16 may be a drawing illustrating a scenario of generating a top view image based on a 3D image and linking the 3D image and the top view image according to one embodiment of the present disclosure.

[0355] Referring to FIG. 16, the processor (180) may generate a 3D image (1610) representing the interior of a room from a plurality of 2D images corresponding to a plurality of viewpoints. The 3D image (1610) may include a segmented first 3D object (1611) representing an air purifier.

[0356] The processor (180) may generate a top view image (1620) corresponding to a top view point based on the 3D image (1610). The top view image (1620) may include a second 3D object (1621) in the form of a top view corresponding to the first 3D object (1611). The first 3D object (1611) and the second 3D object (1621) may represent the same object, such as an air purifier.

[0357] The processor (180) can equally reflect the result of editing according to the editing command for the first 3D object (1611) of the 3D image (1610) to the second 3D object (1621) of the top view image (1620).

[0358] For example, when the processor (180) receives an editing command to move a first 3D object (1611) of a 3D image (1610) from a first position to a second position, the processor (180) may move the first 3D object (1611) from the first position to the second position. At the same time, the processor (180) may move a second 3D object (1621) from the first position to the second position on the top view image (1620).

[0359] The processor (180) can equally reflect the edited result according to the editing command for the second 3D object (1621) of the top view image (1620) to the first 3D object (1611) of the 3D image (1610).

[0360] In this way, according to an embodiment of the present disclosure, the result of editing on a 3D image is automatically reflected on a top view image, and vice versa, so that a user can perform intuitive and consistent editing work at various viewpoints, and accordingly, the precision of object arrangement in 3D space and editing efficiency can be greatly improved.

[0361] According to an embodiment of the present disclosure, a method of operating a device may include: generating a first mask image of a first level and a second mask image of a second level based on a plurality of two-dimensional (2-dimensional) images corresponding to a plurality of viewpoints, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object; generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images; performing contrastive learning on the first mask image and the first feature field, and on the second mask image and the second feature field, respectively; obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints; clustering the obtained average feature set to obtain a plurality of global clusters; and performing contrastive learning between a global feature candidate representing each of the obtained global clusters and a first feature vector corresponding to the first feature field or a second feature vector corresponding to the second feature field.

[0362] The step of generating the first mask image and the second mask image may include a step of identifying a location of an object within an image based on an input, and a step of generating the first mask image of the first level and the second mask image of the second level using a SAM (Segment Anything model) that automatically segments the object based on the identified location.

[0363] The step of generating the first feature field and the second feature field may include the step of classifying a 3D view image generated by the plurality of 2D images into a plurality of 3D Gaussians and the step of generating the first feature field of the first level and the second feature field of the second level through a 3D Gaussian Splatting (3DGS) method of projecting the classified plurality of 3D Gaussians onto a 2D screen.

[0364] The step of obtaining the above average feature set may include a step of obtaining the above average feature set through an average pooling method that averages a plurality of feature vectors corresponding to each of the plurality of viewpoints.

[0365] The step of obtaining the above plurality of global clusters may include a step of performing clustering on the average feature set arranged in the feature space using the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) method.

[0366] The operating method of the above device may further include a step of extracting a plurality of global feature candidates corresponding to the center of each of the plurality of global clusters.

[0367] The first feature vector or the second feature vector and the global candidate feature can perform the contrastive learning based on cosine similarity.

[0368] The functions of the elements disclosed in the present invention may be implemented using circuits or processing circuits including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), existing circuits, and / or combinations thereof. A processor may be defined as a processing circuit or circuits including transistors and other circuits.

[0369] In the present invention, the circuits, units, or means may be hardware designed or programmed to perform the specified functions. The hardware may be the hardware disclosed in the present invention or other known hardware programmed or configured to perform the specified functions. If the hardware is a processor, which may be considered a type of circuit, the circuits, units, or means may be a combination of hardware and software, and the software may constitute the hardware and / or the processor.

[0370] The above-described present disclosure can be implemented as computer-readable code on a program-recorded medium. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. In addition, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. In the method of operating the device, A step of generating a first mask image of a first level and a second mask image of a second level based on a plurality of two-dimensional (2-Dimensional) images corresponding to a plurality of viewpoints, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object; A step of generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images; A step of performing contrastive learning for the first mask image and the first feature field, and for the second mask image and the second feature field, respectively; A step of obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints; A step of obtaining multiple global clusters by clustering the above-obtained average feature set; and A step of performing contrastive learning between a global feature candidate representing each of the acquired global clusters and a first feature vector corresponding to the first feature field or a second feature vector corresponding to the second feature field. How the device operates.

2. In paragraph 1, The step of generating the first mask image and the second mask image is Steps to determine the location of objects within an image based on input; and A step of generating the first mask image of the first level and the second mask image of the second level using a SAM (Segment Anything model) that automatically segments an object according to the identified location. How the device operates.

3. In paragraph 2, The step of generating the first feature field and the second feature field is A step of classifying a 3D view image generated by the above plurality of 2D images into a plurality of 3D Gaussians, and A step of generating the first feature field of the first level and the second feature field of the second level through a 3D Gaussian Splatting (3DGS) method that projects a plurality of classified 3D Gaussians onto a 2D screen. How the device operates.

4. In paragraph 3, The step of obtaining the above average feature set is A step of obtaining the average feature set through an average pooling method that averages a plurality of feature vectors corresponding to each of the plurality of viewpoints. How the device operates.

5. In paragraph 4, The step of acquiring the above multiple global clusters is A step of performing clustering on the average feature set arranged on the feature space using the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) method. How the device operates.

6. In paragraph 5, Further comprising a step of extracting a plurality of global feature candidates corresponding to the center of each of the plurality of global clusters. How the device operates.

7. In paragraph 6, Performing the contrastive learning based on the cosine similarity between the first feature vector or the second feature vector and the global candidate feature. How the device operates.

8. In the device, Based on a plurality of two-dimensional (2-dimensional) images corresponding to a plurality of viewpoints, a first mask image of a first level and a second mask image of a second level are generated, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object. Generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images, For the first mask image and the first feature field, contrastive learning is performed for the second mask image and the second feature field, respectively. Obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints, By clustering the above-obtained average feature set, multiple global clusters are obtained, Performing contrastive learning between the global feature candidates representing each of the acquired global clusters and the first feature vector corresponding to the first feature field or the second feature vector corresponding to the second feature field. Containing one or more processors device.

9. In paragraph 8, One or more of the above processors The step of generating the first mask image and the second mask image is The first mask image of the first level and the second mask image of the second level are generated using SAM (Segment Anything model), which identifies the location of an object within an image based on input and automatically segments the object according to the identified location. device.

10. In paragraph 9, One or more of the above processors A 3D view image generated by the plurality of 2D images is classified into a plurality of 3D Gaussians, and the classified plurality of 3D Gaussians are projected onto a 2D screen through a 3D Gaussian Splatting (3DGS) method to generate the first feature field of the first level and the second feature field of the second level based on the plurality of 2D images. device.

11. In paragraph 10, One or more of the above processors The average feature set is obtained through an average pooling method that averages multiple feature vectors corresponding to each of the multiple viewpoints. device.

12. In paragraph 11, One or more of the above processors Clustering is performed on the average feature set arranged in the feature space using the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) method. device.

13. In paragraph 12, One or more of the above processors Extracting multiple global feature candidates corresponding to the center of each of the multiple global clusters. device.

14. In paragraph 13, One or more of the above processors Performing the contrastive learning based on the cosine similarity between the first feature vector or the second feature vector and the global candidate feature. device.

15. In a non-transitory computer-readable recording medium recording a program for performing a method of operating a device, The above operation method A step of generating a first mask image of a first level and a second mask image of a second level based on a plurality of two-dimensional (2-Dimensional) images corresponding to a plurality of viewpoints, wherein the first level is a level for segmenting a unit object corresponding to a subject, and the second level is a level for segmenting sub-objects constituting the unit object; A step of generating a first feature field of the first level and a second feature field of the second level based on the plurality of 2D images; A step of performing contrastive learning for the first mask image and the first feature field, and for the second mask image and the second feature field, respectively; A step of obtaining an average feature set including a plurality of average feature vectors corresponding to the plurality of viewpoints; A step of obtaining multiple global clusters by clustering the above-obtained average feature set; and A step of performing contrastive learning between a global feature candidate representing each of the acquired global clusters and a first feature vector corresponding to the first feature field or a second feature vector corresponding to the second feature field. Non-transitory recording medium.

Images