Electronic device for recognizing object and control method thereof

Abstract

According to one embodiment, an electronic device includes: a communication module; a memory; and at least one processor operably connected to the communication module, where the at least one processor may acquire a plurality of CAD images based on CAD information related to an object, classify the plurality of CAD images into a plurality of groups based on similarities between the plurality of CAD images, and transmit the plurality of CAD images to the at least one processor. Transmitting a representative CAD image of each of the plurality of groups to an external electronic device, receiving, from the external electronic device, an object image corresponding to the representative CAD image captured by the external electronic device, and creating a learning model by using the received object image as learning data to output the representative CAD image.

Description

Technical Field

[0001] An embodiment relates to an electronic device for identifying an object and a control method thereof. Background Art

[0002] In an industrial field, it is necessary to frequently estimate the posture of an object. The posture of an object on a conveyor changes every time, and the object is often picked up for inspection or assembly. It is also necessary to display the inspection result and the posture for assembly.

[0003] When generating a learning algorithm for estimating the posture of an object, an image of the object is captured, and a worker generates training data by dragging the posture and position of the object with a mouse while viewing the image on a monitor.

[0004] Even after training, when viewing an abnormal result or an assembly guide of an object, the worker should compare the image of the object displayed on the monitor with the actual object on the workbench, view them alternately, and there is a difference between the angle at which the object is captured and the posture of the object actually seen by the worker. Summary of the Invention

[0005]

Technical Solution

[0006] According to an embodiment, an electronic device includes: a communication module and at least one processor operably connected to the communication module.

[0007] According to an embodiment, the at least one processor may obtain a plurality of CAD images based on CAD information of an object.

[0008] According to an embodiment, the at least one processor may classify the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images.

[0009] According to an embodiment, the at least one processor may send a representative CAD image of each group among the plurality of groups to an external electronic device.

[0010] According to an embodiment, the at least one processor may receive an image of the object corresponding to the representative CAD image captured by the external electronic device from the external electronic device.

[0011] According to an embodiment, the at least one processor may use the received image of the object as training data to generate a learning model to output the representative CAD image.

[0012] According to an embodiment, an electronic device includes: a communication module; a memory; and at least one processor operably connected to the memory and the communication module.

[0013] According to an embodiment, the at least one processor may receive an image of an object captured by an external electronic device from the external electronic device.

[0014] According to an embodiment, the at least one processor may use the received image as input data to obtain a representative CAD image from a learning model stored in the memory.

[0015] According to an embodiment, the at least one processor may identify a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on CAD information of the object are classified.

[0016] According to an embodiment, the at least one processor may identify a CAD image mapped to the received image among the plurality of CAD images included in the group.

[0017] According to an embodiment, the at least one processor may send information related to the CAD image mapped to the received image to the external electronic device.

[0018] According to an embodiment, the learning model may be trained using the plurality of CAD images obtained based on the CAD information of the object as training data to output a representative CAD image.

[0019] According to an embodiment, a method of controlling an electronic device may include: obtaining a plurality of CAD images based on CAD information of an object.

[0020] According to an embodiment, the method of controlling the electronic device may include: classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images.

[0021] According to an embodiment, the method of controlling the electronic device may include: sending a representative CAD image of each of the plurality of groups to an external electronic device.

[0022] According to an embodiment, the method of controlling the electronic device may include: receiving an image of the object corresponding to the representative CAD image captured by the external electronic device from the external electronic device.

[0023] According to an embodiment, the method of controlling the electronic device may include: using the received image of the object as training data to generate a learning model to output the representative CAD image.

[0024] According to an embodiment, a method of controlling an electronic device may include: receiving an image of an object captured by an external electronic device from the external electronic device.

[0025] According to an embodiment, the method of controlling an electronic device may include: obtaining a representative CAD image from a learning model stored in a memory using the received image as input data.

[0026] According to an embodiment, the method of controlling an electronic device may include: identifying a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on CAD information of the object are classified.

[0027] According to an embodiment, the method of controlling an electronic device may include: identifying a CAD image mapped to the received image among the plurality of CAD images included in the group.

[0028] According to an embodiment, the method of controlling an electronic device may include: sending information related to the CAD image mapped to the received image to the external electronic device.

[0029] According to an embodiment, the learning model may be trained using the plurality of CAD images obtained based on the CAD information of the object as training data to output a representative CAD image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a block diagram showing an electronic device in a network environment according to an embodiment.

[0031] Figure 2 is a perspective view showing internal components of a wearable electronic device according to an embodiment.

[0032] Figure 3a is a view showing a front surface of a wearable electronic device according to an embodiment.

[0033] Figure 3b is a view showing a rear surface of a wearable electronic device according to an embodiment.

[0034] Figure 4 is another perspective view showing a wearable electronic device according to an embodiment.

[0035] Figure 5 is a flowchart showing operations of training a deep learning model for object recognition in an electronic device according to an embodiment.

[0036] Figure 6 is a view showing operations of training a deep learning model for object recognition in an electronic device according to an embodiment.

[0037] Figure 7 is a view showing a classification structure of object CAD images according to an embodiment.

[0038] Figure 8a It is a diagram showing an operation of displaying a guidance image in a wearable electronic device according to an embodiment.

[0039] Figure 8b It is a diagram showing an operation of displaying a guidance image in a wearable electronic device according to an embodiment.

[0040] Figure 9 It is a flowchart showing an operation of training a deep learning model for object recognition in a wearable electronic device according to an embodiment.

[0041] Figure 10 It is a flowchart showing an operation of recognizing an object using a deep learning model in an electronic device according to an embodiment.

[0042] Figure 11 It is a diagram showing an operation of recognizing an object using a deep learning model in an electronic device according to an embodiment.

[0043] Figure 12 It is a diagram showing an operation of recognizing a mapped CAD image based on a classification structure of an object CAD image in an electronic device according to an embodiment.

[0044] Figure 13 It is a diagram showing an operation of using the recognized CAD image in an electronic device according to an embodiment.

[0045] Figure 14 It is a flowchart showing an operation of recognizing an object using a deep learning model in a wearable electronic device according to an embodiment.

[0046] Figure 15 It is a diagram showing an operation of accurately recognizing an object based on a mapped CAD image in an electronic device according to an embodiment.

[0047] Figure 16 It is a diagram showing an accurate recognition operation of an electronic device according to an embodiment.

[0048] Figure 17 It is a diagram showing an accurate recognition operation of an electronic device according to an embodiment.

[0049] Figure 18 It is a diagram showing an accurate recognition operation of an electronic device according to an embodiment.

[0050] Figure 19 It is a diagram showing an accurate recognition operation of an electronic device according to an embodiment.

[0051] Figure 20 It is a flowchart showing an operation of training a deep learning model for recognizing whether an object is defective based on 3D data in an electronic device according to an embodiment.

[0052] Figure 21 is a flowchart showing an operation of identifying whether an object is defective using a deep learning model trained based on 3D data in an electronic device according to an embodiment. Detailed Description

[0053] Figure 1 is a block diagram showing an electronic device 101 in a network environment 100 according to an embodiment. Refer to Figure 1 , the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection end 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection end 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above components (e.g., the sensor module 176, the camera module 180, or the antenna module 197) may be implemented as a single component (e.g., the display module 160).

[0054] The processor 120 may run software (e.g., program 140), for example, to control at least one other component (e.g., a hardware component or a software component) connected to the processor 120 of the electronic device 101, and may perform various data processing or computations. According to an embodiment, as at least part of the data processing or computation, the processor 120 may store commands or data received from another component (e.g., the sensor module 176 or the communication module 190) in the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or to be dedicated to a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0055] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (instead of the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., the server 108). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, or a combination of two or more of them, but is not limited thereto. Additionally or alternatively, the artificial intelligence model may include a software structure in addition to the hardware structure.

[0056] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0057] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or an application 146.

[0058] The input module 150 may receive commands or data to be used by other components of the electronic device 101 (e.g., the processor 120) from the outside of the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0059] The sound output module 155 may output a sound signal to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing a record. The receiver may be used for receiving an incoming call. According to an embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

[0060] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., to a user). The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the intensity of a force caused by the touch.

[0061] The audio module 170 may convert sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or a headset of an external electronic device (e.g., electronic device 102) directly (e.g., wiredly) or wirelessly connected to the electronic device 101.

[0062] The sensor module 176 may detect an operating state of the electronic device 101 (e.g., power or temperature) or an environmental state outside the electronic device 101 (e.g., a state of a user), and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0063] The interface 177 may support one or more specific protocols used to directly (e.g., wiredly) or wirelessly connect the electronic device 101 to an external electronic device (e.g., electronic device 102). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0064] The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (e.g., electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headset connector).

[0065] The haptic module 179 may convert an electrical signal into a mechanical stimulus (e.g., vibration or motion) or an electrical stimulus that can be recognized by a user via his sense of touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0066] The camera module 180 may capture a still image or a moving image. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0067] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0068] The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a primary non-rechargeable battery, a rechargeable battery, or a fuel cell.

[0069] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently of the processor 120 (e.g., an application processor (AP)), and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). Each of these communication modules may communicate with the external electronic device 104 via a first network 198 (e.g., a short-range communication network, such as Bluetooth TM , Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network, such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or these various types of communication modules may be implemented as multiple separate components (e.g., multiple chips). The wireless communication module 192 may identify or authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an international mobile subscriber identity (IMSI)) stored in the user identification module 196.

[0070] The wireless communication module 192 may support 5G networks and next-generation communication technologies (e.g., New Radio (NR) access technology) after 4G networks. The NR access technology may support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 192 may support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transfer rates. The wireless communication module 192 may support various technologies for ensuring performance on high frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., the electronic device 104), or a network system (e.g., the second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for achieving eMBB (e.g., 20 Gbps or greater), a loss coverage for achieving mMTC (e.g., 164 dB or less), or a U-plane latency for achieving URLLC (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or a round-trip of 1 ms or less).

[0071] The antenna module 197 may transmit a signal or power to the outside of the electronic device 101 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190. Subsequently, a signal or power may be transmitted or received between the communication module 190 and an external electronic device via the selected at least one antenna. According to an embodiment, additional components (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as part of the antenna module 197.

[0072] According to an embodiment, the antenna module 197 may form a millimeter-wave antenna module. According to an embodiment, the millimeter-wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., an array antenna), wherein the RFIC is disposed on a first surface (e.g., a bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the plurality of antennas are disposed on a second surface (e.g., a top surface or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the specified high-frequency band.

[0073] At least some of the above components may be interconnected via an inter-peripheral communication scheme (e.g., a bus, a general-purpose input / output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0074] According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and an external electronic device 104 via a server 108 connected to a second network 199. Each of the external electronic devices 102 or 104 may be a device of the same type as the electronic device 101 or a device of a different type from the electronic device 101. According to an embodiment, all or some of the operations running on the electronic device 101 may be run on one or more of the external electronic devices 102, 104, or 108. For example, if the electronic device 101 is to automatically perform a function or service or is to perform a function or service in response to a request from a user or another device, the electronic device 101 may request one or more of the external electronic devices to perform at least part of the function or service instead of running the function or service, or in addition to running the function or service, the electronic device 101 may also request one or more of the external electronic devices to perform at least part of the function or service. The one or more external electronic devices that receive the request may perform the requested at least part of the function or service, or perform additional functions or additional services related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least part of a reply to the request with or without further processing of the result. For this purpose, for example, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology or IoT-related technology.

[0075] Figure 2 is a perspective view showing internal components of a wearable electronic device according to an embodiment.

[0076] Refer to Figure 2 According to an embodiment, the wearable electronic device 200 may include at least one of a light output module 211, a display member 201, and a camera module 250.

[0077] According to an embodiment, the light output module 211 may include a light source capable of outputting an image and a lens that guides the image to the display member 201. According to an embodiment, the light output module 211 may include at least one of a liquid crystal display (LCD), a digital micromirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro LED.

[0078] According to an embodiment, the display member 201 may include an optical waveguide (e.g., a waveguide). According to an embodiment, the output image incident on one end of the optical waveguide of the light output module 211 may propagate through the optical waveguide and be provided to the user. According to an embodiment, the optical waveguide may include at least one of a diffraction element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a mirror). For example, the optical waveguide may guide the output image of the light output module 211 to the user's eyes by using at least one diffraction element or reflective element.

[0079] According to an embodiment, the camera module 250 may capture a still image and / or video. According to an embodiment, the camera module 250 may be disposed within the spectacle frame and around the display member 201.

[0080] According to an embodiment, the first camera module 251 may capture and / or identify the user's eyes (e.g., pupils or irises) or the trajectory of a gaze. According to an embodiment, the first camera module 251 may periodically or aperiodically send information related to the user's eyes or the trajectory of a gaze (e.g., trajectory information) to a processor (e.g., Figure 1 the processor 120).

[0081] According to an embodiment, the second camera module 253 may capture an external image.

[0082] According to an embodiment, the third camera module 255 may be used for hand detection and tracking and user gesture (e.g., hand movement) recognition. The third camera module 255 according to an embodiment may be used for 3 degrees of freedom (3DoF) or 6DoF head tracking, position (space or environment) recognition, and / or motion recognition. According to an embodiment, the second camera module 253 may also be used for hand detection and tracking and user gesture recognition. According to an embodiment, at least one of the first camera module 251 to the third camera module 255 may be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an IR sensor, and / or a photodiode.

[0083] Figure 3a is a view showing the front surface of a wearable electronic device according to an embodiment.

[0084] Figure 3bIt is a view showing the rear surface of a wearable electronic device according to an embodiment.

[0085] Referring to Figure 3a and Figure 3b , in an embodiment, the camera modules 311, 312, 313, 314, 315, and 316 and / or the depth sensor 317 may be disposed on the first surface 310 of the housing to obtain information related to the surrounding environment of the wearable electronic device 300.

[0086] In an embodiment, the camera modules 311 and 312 may obtain images related to the surrounding environment of the wearable electronic device.

[0087] In an embodiment, the camera modules 313, 314, 315, and 316 may obtain images while the wearable electronic device is worn by a user. The camera modules 313, 314, 315, and 316 may be used for hand detection and tracking and user gesture (e.g., hand movement) recognition. The camera modules 313, 314, 315, and 316 may be used for 3DoF or 6DoF head tracking, position (spatial or environmental) recognition, and / or motion recognition. In an embodiment, the camera modules 311 and 312 may also be used for hand detection and tracking and user gesture recognition.

[0088] In an embodiment, the depth sensor 317 may be configured to send a signal and receive a signal reflected from an object, and is used for the purpose of identifying the distance to the object (such as time-of-flight (TOF)). As an alternative or supplement to the depth sensor 217, the camera modules 213, 214, 215, and 216 may identify the distance to the object.

[0089] According to an embodiment, the face recognition camera modules 325 and 326 and / or the display 321 (and / or lens) may be disposed on the second surface 320 of the housing.

[0090] In an embodiment, the face recognition camera modules 325 and 326 adjacent to the display may be used for the purpose of identifying the user's face, or may identify and / or track the user's both eyes.

[0091] In an embodiment, the display 321 (and / or lens) may be disposed on the second surface 320 of the wearable electronic device 300. In an embodiment, the wearable electronic device 300 may not include the camera modules 315 and 316 among the plurality of camera modules 313, 314, 315, and 316. Although not shown in Figure 3a and 3b , the wearable electronic device 300 may further include Figure 2 at least one component among the components shown in

[0092] As described above, according to an embodiment, the wearable electronic device 300 may have a form factor for wearing on a user's head. The wearable electronic device 300 may further include a band and / or a wearing member to be fixed on a body part of the user. The wearable electronic device 300 may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0093] Figure 4 is another perspective view showing an electronic device according to an embodiment.

[0094] Referring to Figure 4 , the electronic device 400 may be a head-mounted display (HMD) device capable of providing an image in front of a user's eyes. Figure 4 The configuration of the electronic device 400 in Figure 2 may be completely or partially the same as the configuration of the electronic device 200 in

[0095] According to an embodiment, the electronic device 400 may include housings 410, 420, and 430 that may form an outer appearance of the electronic device 400 and provide a space for arranging components of the electronic device 400.

[0096] According to an embodiment, the electronic device 400 may include a first housing 410 that may surround at least a part of a user's head. In an embodiment, the first housing 410 may include a first surface 400a facing the outside of the electronic device 400 (e.g., in the -Y direction).

[0097] According to an embodiment, the first housing 410 may surround at least a part of an internal space I. For example, the first housing 410 may include a second surface 400b facing the internal space I of the electronic device 400 and a third surface 400c opposite to the second surface 400b. According to an embodiment, the first housing 410 may be combined with a third housing 430 to form a closed curved shape surrounding the internal space I.

[0098] According to an embodiment, the first housing 410 may accommodate at least some of the components of the electronic device 400. For example, a light output module and a circuit board may be disposed within the first housing 410.

[0099] According to an embodiment, the electronic device 400 may include a display member 440 corresponding to the left and right eyes. The display member 440 may be disposed in the first housing 410. Figure 4 The configuration of the display member 440 in Figure 2 may be completely or partially the same as the configuration of the display member 201 in

[0100] According to an embodiment, the electronic device 400 may include a second housing 420 that can be mounted on a user's face. According to an embodiment, the second housing 420 may include a fourth surface 400d that can face at least a part of the user's face. According to an embodiment, the fourth surface 400d may be a surface facing the internal space I of the electronic device 400 (e.g., in the +Y direction). According to an embodiment, the second housing 420 may be coupled to the first housing 410.

[0101] According to an embodiment, the electronic device 400 may include a third housing 430 that can be mounted on the back of a user's head. According to an embodiment, the third housing 430 may be coupled to the first housing 410. According to an embodiment, the third housing 430 may accommodate at least some of the components of the electronic device 400. For example, a battery (e.g., Figure 1 the battery 189) may be disposed within the third housing 430.

[0102] Figure 5 is a flowchart illustrating an operation of training a deep learning model for object recognition in an electronic device according to an embodiment.

[0103] Referring to Figure 5 , in operation 510, the electronic device (e.g., Figure 1 the electronic device 101 or Figure 1 the processor 120) may obtain a plurality of CAD images based on computer-aided design (CAD) information of an object.

[0104] According to an embodiment, the electronic device may obtain a plurality of CAD images from CAD data of an object received from a server (e.g., Figure 1 the server 108) and / or CAD data of an object stored in a memory (e.g., Figure 1 the memory 130).

[0105] According to an embodiment, the plurality of CAD images may respectively correspond to objects placed on different surfaces and / or placed in different directions.

[0106] According to an embodiment, in operation 520, the electronic device may classify the plurality of CAD images into a plurality of groups based on the similarity between the plurality of CAD images.

[0107] According to an embodiment, the electronic device may classify the plurality of CAD images into a tree structure based on the similarity between the plurality of CAD images.

[0108] According to an embodiment, an electronic device may calculate a score between multiple CAD images by converting a score between the multiple CAD images into a structural similarity index map (SSIM) and / or a perspective cumulative orientation feature (PCOF) between the multiple CAD images. The SSIM may be calculated by Equation 1 below.

[0109] [Equation 1]

[0110]

[0111] Here, x or y is an index of each image, μ x is the mean of pixel samples of x, μ y is the mean of pixel samples of y, is the variance of x, is the variance of y, and σ xy is the covariance of x and y.

[0112] According to an embodiment, C 1 may be (k 1 L) 2 and C 2 may be (k 2 L) 2 .

[0113] Here, L may be the dynamic range of pixel values. By default, k 1 may be 0.01, and k 2 may be 0.03.

[0114] According to an embodiment, the electronic device may group multiple CAD images (or templates) having a similarity score equal to or greater than a predetermined threshold into the same group. According to an embodiment, the electronic device may generate an upper-level tree by grouping similar CAD images while reducing the resolution.

[0115] For example, the electronic device may classify multiple CAD images into multiple first groups based on similarity, and identify multiple first representative CAD images corresponding to the multiple first groups, respectively.

[0116] According to an embodiment, the electronic device may classify the multiple first representative CAD images into multiple second groups based on similarity, and identify multiple second representative CAD images corresponding to the multiple second groups, respectively. According to an embodiment, the multiple second groups may be upper-level groups of the multiple first groups.

[0117] According to an embodiment, the representative CAD image of the upper-level group including the representative CAD image of the lower-level group may be the final representative CAD image of the multiple grouped CAD images.

[0118] This will be described below with reference to Figure 7Describe a classification structure for a plurality of CAD images according to an embodiment.

[0119] According to an embodiment, in operation 530, the electronic device may send a plurality of groups of representative CAD images to an external electronic device (e.g., Figure 1 electronic device 104, Figure 2 electronic device 200, Figure 3a or Figure 3b electronic device 300, or Figure 4 electronic device 400).

[0120] According to an embodiment, the electronic device may send a plurality of second representative CAD images as representative CAD images to an external electronic device. For example, the electronic device may send the representative CAD images of the upper layer group to an external electronic device.

[0121] According to an embodiment, the electronic device may sequentially send a first guide image including a first CAD image related to a first pose of an object and a second guide image including a second CAD image related to a second pose of the object among the plurality of representative CAD images to an external electronic device.

[0122] For example, when an image of the object corresponding to the first guide image is received after the first guide image is sent to an external electronic device, the electronic device may sequentially send the second guide image to the external electronic device.

[0123] According to an embodiment, the electronic device may send the first guide image and the second guide image to an external electronic device together.

[0124] According to an embodiment, at least one of the first guide image or the second guide image may include a guide for guiding a change in the pose of the object. According to an embodiment, the guide may include a guide phrase, a guide voice, and / or a guide highlight.

[0125] According to an embodiment, the external electronic device may sequentially display the first guide image and the second guide image.

[0126] According to an embodiment, three or more guide images may be sent to the external electronic device.

[0127] According to an embodiment, the external electronic device may be augmented reality (AR) glasses. According to an embodiment, the external electronic device may be an HMD device or an electronic device including a display and a camera.

[0128] According to an embodiment, the operation of displaying a guide image by the external electronic device will be described below with reference to Figure 8a and Figure 8b

[0129] According to an embodiment, in operation 540, the electronic device may receive, from an external electronic device, an image of an object corresponding to a representative CAD image captured by the external electronic device.

[0130] According to an embodiment, the received image of the object may be an image of the object whose pose has been changed by the user based on the display of a guidance image.

[0131] According to an embodiment, in operation 550, the electronic device may use the received image of the object as training data to generate a learning model for outputting a representative CAD image. For example, the learning model may be a deep learning model or a general computing model. According to an embodiment, the “general computing model” may be applied to the operations where the “deep learning model” is applied.

[0132] In this way, the deep learning model can be trained on the initial pose of the object by classifying multiple CAD images into a tree structure, obtaining a representative CAD image, and outputting the representative CAD image by using the object image corresponding to the representative CAD image as input data.

[0133] Figure 6 is a diagram illustrating operations of training a deep learning model for object recognition in an electronic device according to an embodiment.

[0134] Referring to Figure 6 , in operation 610, the electronic device 101 (e.g., Figure 1 the electronic device 101 of Figure 1 or the processor 120 of

[0135] According to an embodiment, the electronic device may obtain CAD data of an object from CAD data of the object received from a server (e.g., Figure 1 the server 108 of Figure 1 and / or CAD data of the object stored in a memory (e.g.,

[0136] According to an embodiment, in operation 620, the electronic device 101 may generate pose templates (or CAD images) and calculate the similarity between the templates.

[0137] According to an embodiment, the electronic device may calculate the similarity between multiple CAD images by converting the similarity between the multiple CAD images into SSIM and / or PCOF.

[0138] According to an embodiment, in operation 630, the electronic device 101 may classify CAD images into a tree structure for accurate pose estimation.

[0139] According to an embodiment, the electronic device may classify a plurality of pose templates into a tree structure based on similarity. For example, the electronic device may classify a plurality of pose templates into a plurality of first groups and respectively identify representative pose templates of the plurality of first groups.

[0140] According to an embodiment, the electronic device may classify a plurality of first pose templates into a plurality of second groups based on similarity and identify a plurality of second representative pose templates respectively corresponding to the plurality of second groups. According to an embodiment, the plurality of second groups may be upper-level groups of the plurality of first groups.

[0141] According to an embodiment, a representative pose template of an upper-level group including a representative pose template of a lower-level group may be a final upper-level pose template of the grouped plurality of pose templates.

[0142] Below will be referred to Figure 7 to describe the classification structure of a plurality of pose templates according to an embodiment.

[0143] According to an embodiment, in operation 640, the electronic device 101 may generate an image acquisition guide using an upper-level template.

[0144] According to an embodiment, the electronic device may generate an image acquisition guide for a final upper-level pose template having a tree structure. According to an embodiment, each image acquisition guide may include a guide for guiding the pose of an object and / or a change in the pose of the object. According to an embodiment, the guide may include a guide phrase, a guide voice, and / or a guide image.

[0145] According to an embodiment, the electronic device may send the image acquisition guide to an external electronic device 104 (e.g., Figure 1 the electronic device 104 of Figure 2 the electronic device 200 of Figure 3a or Figure 3b the electronic device 300 of Figure 4 the electronic device 400 of

[0146] According to an embodiment, in operation 650, the external electronic device 104 (e.g., Figure 1 the electronic device 104 of Figure 2 the electronic device 200 of Figure 3a or Figure 3b the electronic device 300 of Figure 4 the electronic device 400 of

[0147] According to an embodiment, the external electronic device 104 may also display a guide for guiding a change in the pose of an object.

[0148] According to an embodiment, below will be referred to Figure 8a and Figure 8bDescribe the operation of displaying a guidance image by the external electronic device 104.

[0149] According to an embodiment, in operation 660, the external electronic device 104 may acquire an image corresponding to the guidance image.

[0150] According to an embodiment, the external electronic device 104 may use a camera included in the external electronic device 104 to acquire an image of an object corresponding to the guidance image.

[0151] According to an embodiment, in operation 670, the external electronic device 104 may send the acquired image to the electronic device.

[0152] According to an embodiment, in operation 680, the electronic device 101 may train a deep learning model based on the received image.

[0153] According to an embodiment, the deep learning model may use Recursive 6-DoF Object Pose Refinement (RNNPose) with robust correspondence field estimation and pose optimization, a scalable, accurate, and occlusion-robust method for predicting the 3D pose of challenging objects without using depth (BB8), and / or a 6D pose object detector and refiner (DPOD).

[0154] In this way, the guidance image may be displayed on the display of the AR glasses, which may eliminate the need for the user to adjust the pose of the object in the case of alternately viewing the display and the object, thereby shortening the time required to acquire an image to be used as training data.

[0155] Figure 7 Is a diagram showing a classification structure of CAD images of an object according to an embodiment.

[0156] Refer to Figure 7 , an electronic device (e.g., Figure 1 the electronic device 101 of Figure 1 the processor 120 of Figure 2 the electronic device 200 of Figure 3a or Figure 3b the electronic device 300 of Figure 4 or the electronic device 400 of

[0157] According to an embodiment, the electronic device may classify a plurality of CAD images into a first (1-1) group 710, a first (1-2) group 711, a first (1-3) group 712, and / or a first (1-4) group 713 based on similarity, and identify a representative CAD image of each of the first groups 710, 711, 712, and 713.

[0158] According to an embodiment, an electronic device may classify representative CAD images of a first set 710, 711, 712, and 713 into a plurality of second sets 720 and 721 based on similarity. According to an embodiment, the electronic device may classify representative CAD images of each of the first sets 710, 711, 712, and 713 into a (2-1)th set 720 and / or a (2-2)th set 721 based on similarity, and may respectively identify representative CAD images 730 and 731 of the second sets 720 and 721.

[0159] In this way, the electronic device may classify a plurality of CAD images into a tree structure and may search for a mapped CAD image only among the CAD images of the group of representative CAD images of an image including an object, thereby reducing search resources and search time.

[0160] Figure 8a is a diagram illustrating an operation of displaying a guidance image in a wearable electronic device according to an embodiment.

[0161] Referring to Figure 8a , an external electronic device 104 (e.g., Figure 1 the electronic device 104, Figure 2 the electronic device 200, Figure 3a or Figure 3b the electronic device 300, or Figure 4 the electronic device 400) may display a first guidance image 810 received from an electronic device (e.g., Figure 1 the electronic device 101) or generated by the external electronic device 104 on a display. According to an embodiment, a display of the external electronic device 104 may be transparent to light, and a user may view an object 820 located opposite the user's eyes while the display of the external electronic device 104 is between the object 820 and the user's eyes.

[0162] According to an embodiment, the first guidance image 810 may be a representative CAD image of an object.

[0163] According to an embodiment, the external electronic device 104 may display a guidance phrase 811 (e.g., “Take a picture so that the object appears in the pose shown in the area”) for changing a pose of the object 820 together with the first guidance image 810.

[0164] According to an embodiment, after displaying the first guidance image 810 as shown in Figure 8a , the external electronic device 104 may sequentially display a second guidance image 812 as shown in Figure 8b .

[0165] Figure 8b is a diagram illustrating an operation of displaying a guidance image in a wearable electronic device according to an embodiment.

[0166] Reference Figure 8b Figure 8b , the external electronic device 104 may display a second guidance image 812. According to an embodiment, the second guidance image 812 may include a CAD image of an object in a pose different from the pose in the CAD image of the object included in the first guidance image 810.

[0167] According to an embodiment, the external electronic device 104 may display a guidance phrase 813 (e.g., "Take a picture so that the object appears in the pose shown in the area") for changing the pose of the guidance change object 821 together with the second guidance image 812.

[0168] In this way, since the guidance image and the pose of the object to be acquired can be viewed together through the display of the external electronic device 104, post-processing of the data for training the deep learning model is not required, and the user can generate images for training data without leaving the workbench.

[0169] Figure 9 Figure 9 is a flowchart showing an operation of training a deep learning model for object recognition in a wearable electronic device. According to an embodiment, Figure 9 Figure 9 is a diagram showing an operation of training a deep learning model when the electronic device is a wearable electronic device (e.g., AR glasses and / or HMD device).

[0170] Reference Figure 9 Figure 9 , in operation 910, the electronic device (e.g., Figure 1 the electronic device 101 of Figure 1 the processor 120 of Figure 2 the electronic device 200 of Figure 3a or Figure 3b the electronic device 300 of Figure 4 or the electronic device 400 of

[0171] According to an embodiment, operation 910 is the same as operation 510 of Figure 5 Figure 5 , so redundant description is avoided.

[0172] According to an embodiment, in operation 920, the electronic device may classify the plurality of CAD images into a plurality of groups based on the similarity between the plurality of CAD images.

[0173] According to an embodiment, operation 920 is the same as operation 520 of Figure 5 Figure 5 , so redundant description is avoided.

[0174] According to an embodiment, in operation 930, the electronic device may display a representative CAD image of each of the plurality of groups.

[0175] According to an embodiment, an electronic device may display a representative CAD image through a display of the electronic device (e.g., Figure 1 display module 160 of Figure 2 display member 201 of Figure 3b display 321 of Figure 4 or display member 440 of Figure 5 instead of sending the representative CAD image to an external electronic device in operation 530 of

[0176] According to an embodiment, the electronic device may sequentially display, on an external electronic device, a first guidance image including a first CAD image related to a first posture of an object and a second guidance image including a second CAD image related to a second posture of the object among a plurality of representative CAD images.

[0177] According to an embodiment, at least one of the first guidance image or the second guidance image may include a guidance phrase for guiding a change in the posture of the object.

[0178] According to an embodiment, in operation 940, the electronic device may obtain an image of an object corresponding to the representative CAD image.

[0179] According to an embodiment, the electronic device may obtain an image of the object through a camera (e.g., Figure 1 camera module 180 of Figure 2 second camera module 253 of Figure 2 third camera module 255 of Figure 3a or camera modules 311, 312, 313, 314, 315, and 316 of

[0180] According to an embodiment, the electronic device may obtain an image of an object whose posture has been changed by a user to correspond to the first guidance image or the second guidance image.

[0181] According to an embodiment, in operation 950, the electronic device may use the obtained image as training data to train a deep learning model to output a representative CAD image.

[0182] Figure 10 is a flowchart illustrating an object recognition operation using a deep learning model in an electronic device according to an embodiment.

[0183] Referring to Figure 10 , in operation 1010, the electronic device (e.g., Figure 1 electronic device 101 of Figure 1 or processor 120 of Figure 1 electronic device 104 of Figure 2 electronic device 200 of Figure 3a orFigure 3b The electronic device 300, or Figure 4 the electronic device 400) receives an image captured by an external electronic device.

[0184] According to an embodiment, the received image may be captured by a camera of an external electronic device and may be captured after a guidance image sent from the electronic device is displayed on the external electronic device.

[0185] According to an embodiment, in operation 1020, the electronic device may use the received image as input data to obtain a representative CAD image from a learning model stored in a memory (e.g., Figure 1 the memory 130). For example, the learning model may be a deep learning model or a general computing model. According to an embodiment, the “general computing model” may also be applied to the operation of applying the “deep learning model”.

[0186] According to an embodiment, the electronic device may obtain a representative CAD image corresponding to the received image based on the similarity between the received image and each representative CAD image among a plurality of representative CAD images.

[0187] According to an embodiment, in operation 1030, the electronic device may identify a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on object-based CAD information have been classified.

[0188] According to an embodiment, the electronic device may identify an upper-level group including the representative CAD image and each lower-level group included in the upper-level group.

[0189] According to an embodiment, in operation 1040, the electronic device may identify a CAD image mapped to the received image among a plurality of CAD images included in the identified group.

[0190] According to an embodiment, the electronic device may identify a plurality of CAD images including a group including the representative CAD image corresponding to the received image among a plurality of CAD images of an object, and may identify a CAD image mapped to the received image among a plurality of CAD images of a lower-level group including the representative CAD image.

[0191] According to an embodiment, the electronic device may identify a CAD image mapped to the received image among a plurality of CAD images included in the identified group based on the similarity to the received image.

[0192] According to an embodiment, in operation 1050, the electronic device may send information related to the identified CAD image to the external electronic device.

[0193] According to an embodiment, an electronic device may identify a defective part of an object included in a received image based on information related to a CAD image mapped to the received image, and transmit an image including a guide for guiding the identification of the defective part to an external electronic device.

[0194] According to an embodiment, an electronic device may identify a defective part of an object by comparing a CAD image mapped to a received image with the received image. For example, the electronic device may compare the CAD image mapped to the received image with the received image, and when at least one eigenvalue included in the received image differs from at least one eigenvalue included in the CAD image mapped to the received image by a set range or more, the electronic device may identify the at least one eigenvalue as a defective part of the object.

[0195] According to an embodiment, an electronic device may transmit an image including an assembly guide for assembling an object included in a received image as a component to an external electronic device based on information related to a CAD image mapped to the received image.

[0196] Figure 11 is a diagram illustrating an object recognition operation using a deep learning model in an electronic device according to an embodiment.

[0197] Referring to Figure 11 , in operation 1110, an external electronic device 104 (e.g., Figure 1 electronic device 104, Figure 2 electronic device 200, Figure 3a or Figure 3b electronic device 300, or Figure 4 electronic device 400) may acquire an image.

[0198] According to an embodiment, the external electronic device 104 may capture an image after a guide image transmitted from an electronic device 101 (e.g., Figure 1 electronic device 101 or Figure 1 processor 120) to the external electronic device 104 is displayed on the external electronic device 104.

[0199] For example, the external electronic device 104 may capture an image through a user input after displaying the guide image, capture an image when a set time has elapsed after displaying the guide image, or capture an image when it is detected that the posture of the product is similar to the posture in the CAD image included in the guide image after displaying the guide image.

[0200] According to an embodiment, an image captured by the external electronic device 104 may be transmitted to the electronic device 101.

[0201] According to an embodiment, in operation 1120, the electronic device 101 may receive an image from an external electronic device 104.

[0202] According to an embodiment, the electronic device 101 may input the received image into a learning-based pose estimator 1130 (e.g., a deep learning model).

[0203] According to an embodiment, in operation 1140, the electronic device 101 may estimate an initial pose of an object. According to an embodiment, the electronic device 101 may obtain an initial pose (e.g., a representative CAD image) of the object included in the received image through the learning-based pose estimator 1130.

[0204] According to an embodiment, the electronic device 101 may obtain a representative CAD image corresponding to the received image based on the similarity between the received image and each of a plurality of representative CAD images.

[0205] According to an embodiment, in operation 1150, the electronic device 101 may perform tree-based accurate pose (e.g., CAD image) estimation based on the initial pose.

[0206] According to an embodiment, the electronic device 101 may identify a group including the initial pose (e.g., a representative CAD image) corresponding to the received image among a plurality of groups into which a plurality of CAD images obtained based on the CAD information of the object have been classified.

[0207] According to an embodiment, the electronic device 101 may identify an upper group including the initial pose of the object and each lower group included in the upper group.

[0208] According to an embodiment, the electronic device 101 may identify an accurate pose (e.g., a CAD image mapped to the received image) mapped to the received image among a plurality of poses (e.g., CAD images) included in the identified group.

[0209] According to an embodiment, the electronic device 101 may identify a group including a plurality of poses including the initial pose corresponding to the received image among a plurality of poses of the object, and identify a pose mapped to the received image among a plurality of poses in the lower group including the initial pose.

[0210] According to an embodiment, the electronic device 101 may identify a pose mapped to the received image based on the similarity between a plurality of poses included in the identified group and the received image.

[0211] According to an embodiment, the operation of obtaining a pose (e.g., a CAD image) mapped to an image received from the external electronic device 104 will be described in more detail below with reference to Figure 12 The operation of obtaining a pose (e.g., a CAD image) mapped to an image received from the external electronic device 104 will be described in more detail below.

[0212] According to an embodiment, at operation 1160, the electronic device 101 may obtain an accurate attitude result.

[0213] According to an embodiment, at operation 1170, the electronic device 101 may modify the CAD image based on the attitude result.

[0214] According to an embodiment, the electronic device 101 may compare the CAD image mapped to the received image with the received image, and when at least one eigenvalue included in the received image differs from at least one eigenvalue included in the CAD image mapped to the received image by a set range or more, the electronic device may identify the at least one eigenvalue as a defective part of the object included in the received image.

[0215] According to an embodiment, the electronic device 101 may modify the CAD image to display the included defective part in the CAD image mapped to the received image.

[0216] According to an embodiment, the electronic device 101 may modify the CAD image based on information related to the CAD image mapped to the received image to include an assembly guide for assembling the object included in the received image as a component.

[0217] According to an embodiment, the electronic device 101 may send the modified CAD image to the external electronic device 104.

[0218] According to an embodiment, at operation 1180, the external electronic device 104 may display the modified CAD data.

[0219] According to an embodiment, the external electronic device 104 may provide a guide for identifying the defective part of the object or an assembly guide through a display.

[0220] According to an embodiment, hereinafter, reference will be made to Figure 13 Describe the operation of displaying the modified CAD data in the external electronic device 104.

[0221] Since the defective part or the assembly guide is displayed by reflecting the attitude of the object in this way, a more intuitive guide can be provided to the user.

[0222] Figure 12 is a diagram showing an operation of identifying a mapped CAD image based on a classification structure of an object CAD image in an electronic device according to an embodiment.

[0223] Refer to Figure 12 , the electronic device (e.g., Figure 1 the electronic device 101 of Figure 1 the processor 120 of Figure 2The electronic device 200, Figure 3a or Figure 3b the electronic device 300, or Figure 4 the electronic device 400) may recognize a representative CAD image 1210 corresponding to an image of an object.

[0224] According to an embodiment, the electronic device may obtain, as output data, a representative CAD image 1210 corresponding to an image of an object through a trained deep learning model, or may obtain a representative CAD image 1210 corresponding to an image of an object among a plurality of representative CAD images based on similarity.

[0225] According to an embodiment, the electronic device may recognize, based on similarity, an image of an upper layer group 1220 including the representative CAD image 1210, and may recognize a lower layer group 1230 corresponding to an image of an object among a plurality of lower layer groups included in the upper layer group 1220.

[0226] According to an embodiment, the electronic device may recognize, based on similarity, a CAD image 1240 mapped to an image of an object among a plurality of CAD images included in the lower layer group 1230.

[0227] Since the representative CAD image (or initial pose) is recognized in this way, and then only the CAD image mapped to the image of the object is recognized among the plurality of CAD images related to the representative CAD image, the search resources and search time can be reduced compared to searching all pixel regions.

[0228] Figure 13 is a diagram showing an operation of using the recognized CAD image in an electronic device according to an embodiment.

[0229] Referring to Figure 13 , the electronic device (e.g., Figure 1 the electronic device 101, Figure 2 the electronic device 200, Figure 3a or Figure 3b the electronic device 300 or Figure 4 the electronic device 400) may modify the object's CAD image 1310 by displaying a portion 1311 to be inspected in the object's CAD image 1310 mapped to the captured object's image.

[0230] According to an embodiment, the electronic device may display the modified CAD image 1320. According to an embodiment, the electronic device may also display a guiding phrase 1321 for inspection together with the modified CAD image 1320.

[0231] Figure 14 is a flowchart showing an object recognition operation using a deep learning model in a wearable electronic device. According to an embodiment,Figure 14 FIG. is a diagram illustrating an operation of training a deep learning model when an electronic device is a wearable electronic device (e.g., AR glasses and / or an HMD device).

[0232] Reference Figure 14 , in operation 1410, an electronic device (e.g., Figure 1 electronic device 104 of Figure 2 electronic device 200 of Figure 3a or Figure 3b electronic device 300 of Figure 4 or electronic device 400 of

[0233] According to an embodiment, the electronic device may obtain an image of an object. Figure 1 camera module 180 of Figure 2 second camera module 253 of Figure 2 third camera module 255 of Figure 3a or camera modules 311, 312, 313, 314, 315, and 316 of

[0234] According to an embodiment, the image may be an image received by the electronic device from an external electronic device, or an image captured after a guidance image generated by the electronic device is displayed on the electronic device.

[0235] According to an embodiment, in operation 1420, the electronic device may obtain a representative CAD image from a deep learning model stored in a memory (e.g., Figure 1 memory 130 of

[0236] using the obtained image as input data.

[0237] According to an embodiment, the electronic device may obtain a representative CAD image corresponding to the obtained image based on the similarity between the received image and each representative CAD image among a plurality of representative CAD images.

[0238] According to an embodiment, in operation 1430, the electronic device may identify a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on CAD information of an object have been classified.

[0239] According to an embodiment, the electronic device may identify an upper group including the representative CAD image and each lower group included in the upper group.

[0240] According to an embodiment, the electronic device may identify, among a plurality of CAD images of an object, a plurality of CAD images including a group including a representative CAD image corresponding to the acquired image, and may identify, among the plurality of CAD images including the lower-level group of the representative CAD image, a CAD image mapped to the received image.

[0241] According to an embodiment, the electronic device may identify, based on similarity, a CAD image mapped to the acquired image among the plurality of CAD images included in the identified group.

[0242] According to an embodiment, in operation 1450, the electronic device may acquire information related to the identified CAD image.

[0243] According to an embodiment, the electronic device may identify a defective part of an object included in the acquired image based on information related to the CAD image mapped to the acquired image, and may display an image including a guide for guiding the identification of the defective part.

[0244] According to an embodiment, the electronic device may identify a defective part of an object by comparing the CAD image mapped to the acquired image with the acquired image. For example, the electronic device may identify a defective part of an object included in the acquired image by comparing the CAD image mapped to the acquired image with the acquired image, and when at least one eigenvalue included in the acquired image differs from at least one eigenvalue included in the CAD image mapped to the acquired image by a set range or more, the electronic device may identify the at least one eigenvalue as a defective part of the object included in the acquired image.

[0245] According to an embodiment, the electronic device may display an image including an assembly guide for assembling an object included in the acquired image as a component, based on information related to the CAD image mapped to the acquired image.

[0246] Figure 15 is a diagram illustrating an operation of accurately identifying an object based on a mapped CAD image in an electronic device according to an embodiment.

[0247] Refer to Figure 15 , in operation 1510, an external electronic device 104 (e.g., Figure 1 electronic device 104 of Figure 2 electronic device 200 of Figure 3a or Figure 3b electronic device 300 of Figure 4 or electronic device 400 of

[0248] According to an embodiment, in operation 1520, the electronic device 101 may receive an image from the external electronic device 104.

[0249] According to an embodiment, the electronic device 101 may input a received image into a learning-based pose estimator 1530 (e.g., a deep learning model).

[0250] According to an embodiment, in operation 1540, the electronic device 101 may estimate an initial pose.

[0251] According to an embodiment, in operation 1550, the electronic device 101 may input the initial pose into a tree-based accurate pose estimator.

[0252] According to an embodiment, in operation 1560, the electronic device 101 may obtain an accurate pose result. According to an embodiment, the electronic device 101 may obtain a CAD image among a plurality of CAD images of an object that is mapped to the image of the object.

[0253] According to an embodiment, operations 1510 to 1560 are the same as Figure 11 operations 1110 to 1160, and thus, redundant descriptions are avoided.

[0254] According to an embodiment, in operation 1570, respective accurate pose values may be obtained. According to an embodiment, the electronic device may obtain a CAD image that is mapped to the image of the object, and may estimate the accurate poses of a plurality of regions of the object, respectively. For example, when a physical key included in the object includes a printing area, the printing area also moves according to the movement of the physical key. Accordingly, the electronic device 101 may estimate and correct the movement of the physical key based on the CAD image, and identify whether there is a defect in the printing area.

[0255] According to an embodiment, in operation 1580, the electronic device 101 may correct an internal CAD area by reflecting the pose result.

[0256] According to an embodiment, when identifying a defect in a part of an object, the electronic device 101 may modify the CAD image to include a guide for guiding the identification of the defective part or the assembly part.

[0257] According to an embodiment, the electronic device 101 may send the modified CAD image to an external electronic device 104.

[0258] According to an embodiment, in operation 1590, the external electronic device 104 may display the modified CAD data.

[0259] According to an embodiment, the external electronic device 104 may display CAD data including a defective part identification guide or an assembly guide.

[0260] Although the electronic device 101 and the external electronic device 104 are Figure 15Devices shown and described as separate, but according to an embodiment, operations of the electronic device 101 and the external electronic device 104 may be performed in one device. For example, when the electronic device 101 is AR glasses, operations 1510 and 1590 may also be performed by the electronic device 101.

[0261] Figure 16 is a diagram showing an accurate recognition operation of an electronic device according to an embodiment.

[0262] Referring to Figure 16 , an electronic device (e.g., Figure 1 electronic device 104 of Figure 2 electronic device 200 of Figure 3a or Figure 3b electronic device 300 of Figure 4 or electronic device 400 of

[0263] According to an embodiment, the electronic device may obtain a representative CAD image by inputting an image 1620 of an object into a trained deep learning model, and identify a CAD image 1610 that maps to the image 1620 of the object among a plurality of CAD images including the representative CAD image.

[0264] According to an embodiment, the electronic device may identify differences in a detailed area of an object by comparing the CAD image 1610 with the image 1620 of the object. According to an embodiment, the electronic device may compare at least one feature value included in the CAD image 1610 with at least one feature value included in the image 1620 of the object, and identify areas 1611 and 1621 where the difference between the feature values is within a set range or greater as defect areas.

[0265] According to an embodiment, the electronic device may modify the CAD image 1610 to display a guide for guiding the identification of the defect area 1611 of the CAD image 1610, and send the modified CAD image to an external electronic device (e.g., AR glasses) or display it on a display of the electronic device that is AR glasses.

[0266] Figure 17 is a diagram showing an accurate recognition operation of an electronic device according to an embodiment.

[0267] Referring to Figure 17 , an electronic device (e.g., Figure 1 electronic device 104 of Figure 2 electronic device 200 of Figure 3a or Figure 3b electronic device 300 of Figure 4The electronic device 400) can obtain a representative CAD image by inputting an image of the first object 1710 into a trained deep learning model, and identify a CAD image among the CAD images of the group including the representative CAD image that is mapped to the image of the first object 1710.

[0268] According to an embodiment, the electronic device can identify the posture of the first object 1710 based on the CAD image mapped to the image of the first object 1710. According to an embodiment, the electronic device can identify the postures of the first region 1711 and the second region 1712 of the first object 1710 based on the CAD image mapped to the image of the first object 1710.

[0269] According to an embodiment, even if the captured object changes to the second object 1720, the electronic device can obtain a representative CAD image by inputting an image of the second object 1720 into a trained deep learning model, and identify a CAD image among the CAD images of the group including the representative CAD image that is mapped to the image of the second object 1720.

[0270] According to an embodiment, the electronic device can identify the posture of the second object 1720 based on the CAD image mapped to the image of the second object 1720. According to an embodiment, the electronic device can identify the postures of the first region 1721 and the second region 1722 of the second object 1720 based on the CAD image mapped to the image of the second object 1720.

[0271] In this way, even if the object in the CAD image changes, the trained deep learning model can be used to estimate the posture of the object.

[0272] Figure 18 is a diagram showing the precise recognition operation of an electronic device according to an embodiment.

[0273] Referring to Figure 18 , the electronic device (e.g., Figure 1 the electronic device 104 of Figure 2 the electronic device 200 of Figure 3a or Figure 3b the electronic device 300 of Figure 4 or Figure 17 the electronic device 400) can identify the detailed region of the object after estimating the posture of the object as shown in

[0274] According to an embodiment, the electronic device may compare an image of an object with a mapped CAD image to identify whether there is a difference in a detailed area based on the feature value of the object. For example, when the object is a laptop computer, the electronic device may compare the printing area 1810 of the keys included in the CAD image of the object with the printing area 1811 or 1812 of the keys in the object to determine whether there is a defect based on whether the difference in the position and / or interval of the printing area is equal to or greater than a set range.

[0275] Figure 19 is a diagram showing an accurate identification operation of an electronic device according to an embodiment.

[0276] Referring to Figure 19 , the electronic device (e.g., Figure 1 electronic device 104 of Figure 2 electronic device 200 of Figure 3a or Figure 3b electronic device 300 of Figure 4 or electronic device 400 of

[0277] According to an embodiment, the electronic device may display a guide for guiding the identification of an area corresponding to the defective part 1911 of the first object 1910 on the CAD image mapped to the image of the first object 1910, and send the CAD image of the mapped image of the first object 1910 with the guide displayed thereon to an external electronic device (e.g., AR glasses), or display the CAD image on the display of the electronic device serving as AR glasses.

[0278] According to an embodiment, the electronic device may display a guide for guiding the identification of an area corresponding to the defective part 1921 of the second object 1920 on the CAD image mapped to the image of the second object 1920, and send the CAD image of the mapped image of the second object 1920 with the guide displayed thereon to an external electronic device (e.g., AR glasses), or display the CAD image on the display of the electronic device serving as AR glasses.

[0279] Figure 20 is a flowchart showing an operation of training a deep learning model for identifying whether an object is defective based on 3D data in an electronic device according to an embodiment.

[0280] Referring to Figure 20 , in operation 2010, the electronic device (e.g.,Figure 1 The electronic device 101, Figure 1 the processor 120, Figure 2 the electronic device 200, Figure 3a or Figure 3b the electronic device 300, or Figure 4 the electronic device 400) can obtain 3D data of excellent products.

[0281] According to an embodiment, the electronic device may obtain 3D data of excellent products based on images obtained by a camera (e.g., Figure 1 the camera module 180, Figure 2 the second camera module 253, Figure 2 the third camera module 255 or Figure 3a the camera modules 311, 312, 313, 314, 315, and 316), or may obtain 3D data of excellent products based on images obtained by a camera of an external electronic device. According to an embodiment, the 3D data may be point cloud data.

[0282] According to an embodiment, in operation 2020, the electronic device may arrange the 3D data of the excellent product based on coordinates.

[0283] According to an embodiment, the electronic device may arrange the 3D data to have the same coordinate axes, or may reflect the coordinate axes in each data of the 3D data. In this way, even if the pose of the object changes, the defective part can be identified by arranging the 3D data based on coordinates.

[0284] According to an embodiment, the electronic device may enhance the arranged data to improve the transformation invariance of the deep learning model.

[0285] According to an embodiment, in operation 2030, the electronic device may extract feature points.

[0286] According to an embodiment, the electronic device may extract feature points from the 3D data arranged based on coordinates. For example, the electronic device may extract feature points from the point cloud data. For example, the feature points independent of the pose (or rotation) of the object may be fast point feature histograms (FPFH).

[0287] According to an embodiment, in operation 2040, the electronic device may train a deep learning model for identifying defective products based on the feature points.

[0288] According to an embodiment, the electronic device may use a small amount of 3D data of excellent products as training data to enable the deep learning model to learn the features of excellent products, so that when an image of a defective product is input, the defective part is output.

[0289] Figure 21It is a flowchart showing an operation of using a deep learning model trained based on 3D data to identify whether an object is defective in an electronic device according to an embodiment.

[0290] Referring to Figure 21 , in operation 2210, an electronic device (e.g., Figure 1 electronic device 101, Figure 1 processor 120, Figure 2 electronic device 200, Figure 3a or Figure 3b electronic device 300 or Figure 4 electronic device 400) can obtain parameters for identifying defective products from the trained deep learning model.

[0291] According to an embodiment, the parameters obtained from the deep learning model can be eigenvalue of good products.

[0292] According to an embodiment, in operation 2220, the electronic device can obtain test 3D data. According to an embodiment, the electronic device can obtain 3D data of a product to be tested through a camera (e.g., Figure 1 camera module 180, Figure 2 second camera module 253, Figure 2 third camera module 255 or Figure 3a camera modules 311, 312, 313, 314, 315 and 316), or can obtain test 3D data based on an image of a product to be tested obtained through a camera of an external electronic device. According to an embodiment, the 3D data can be point cloud data.

[0293] According to an embodiment, in operation 2230, the electronic device can arrange the test 3D data based on coordinates.

[0294] According to an embodiment, in operation 2240, the electronic device can obtain feature points.

[0295] According to an embodiment, in operation 2250, the electronic device can identify defective products based on the feature points.

[0296] According to an embodiment, the electronic device can arrange the test 3D data based on coordinates and identify defective parts by inputting the arranged test 3D data into a deep learning model to which learned parameters are applied.

[0297] According to an embodiment, the electronic device can identify defective parts by comparing the feature points of 3D data of good products with the feature points of test 3D data based on the deep learning model.

[0298] In this way, even if training is not performed on each defective part but only on data of good products, defective parts can be determined.

[0299] According to an embodiment, an electronic device includes a communication module and at least one processor operatively connected to the communication module.

[0300] According to an embodiment, the at least one processor may obtain a plurality of CAD images based on CAD information of an object.

[0301] According to an embodiment, the at least one processor may classify the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images.

[0302] According to an embodiment, the at least one processor may send a representative CAD image of each of the plurality of groups to an external electronic device.

[0303] According to an embodiment, the at least one processor may receive, from the external electronic device, an image of an object corresponding to the representative CAD image captured by the external electronic device.

[0304] According to an embodiment, the at least one processor may use the received image of the object as training data to train a learning model to output a representative CAD image.

[0305] According to an embodiment, the at least one processor may sequentially send a first guidance image including a first CAD image related to a first pose of an object and a second guidance image including a second CAD image related to a second pose of the object among the plurality of representative CAD images to the external electronic device.

[0306] According to an embodiment, at least one of the first guidance image or the second guidance image may include a guidance for guiding a change in the pose of the object.

[0307] According to an embodiment, the at least one processor may classify the plurality of CAD images into a plurality of first groups based on similarity.

[0308] According to an embodiment, the at least one processor may identify a plurality of first representative CAD images respectively corresponding to the plurality of first groups.

[0309] According to an embodiment, the at least one processor may classify the plurality of first representative CAD images into a plurality of second groups based on similarity.

[0310] According to an embodiment, the at least one processor may identify a plurality of second representative CAD images respectively corresponding to the plurality of second groups.

[0311] According to an embodiment, the at least one processor may send the plurality of second representative CAD images to the external electronic device as representative CAD images.

[0312] According to an embodiment, the external electronic device may be AR glasses.

[0313] According to an embodiment, an electronic device includes a communication module and at least one processor operatively connected to a memory and the communication module.

[0314] According to an embodiment, at least one processor may receive an image of an object captured by an external electronic device from the external electronic device.

[0315] According to an embodiment, at least one processor may use the received image as input data to obtain a representative CAD image from a learning model stored in the memory.

[0316] According to an embodiment, at least one processor may identify a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on CAD information of an object are classified.

[0317] According to an embodiment, at least one processor may identify a CAD image mapped to the received image among the plurality of CAD images included in the group.

[0318] According to an embodiment, at least one processor may send information related to the CAD image mapped to the received image to the external electronic device.

[0319] According to an embodiment, the learning model may be trained using a plurality of CAD images obtained based on CAD information of an object as training data to output a representative CAD image.

[0320] According to an embodiment, at least one processor may identify a CAD image mapped to the received image among the plurality of CAD images included in the group based on similarity to the received image.

[0321] According to an embodiment, at least one processor may identify a defective part of an object included in the received image based on information related to the CAD image mapped to the received image.

[0322] According to an embodiment, at least one processor may send an image including a guide for guiding identification of the defective part to the external electronic device.

[0323] According to an embodiment, at least one processor may compare the CAD image mapped to the received image with the received image, and when at least one eigenvalue included in the received image differs from at least one eigenvalue included in the CAD image mapped to the received image by a set range or more, identify the at least one eigenvalue as a defective part of the object.

[0324] According to an embodiment, at least one processor may send an image including an assembly guide for assembling an object included in a received image as a component to an external electronic device based on information related to a CAD image mapped to the received image.

[0325] According to an embodiment, a method of controlling an electronic device may include: obtaining a plurality of CAD images based on CAD information of an object.

[0326] According to an embodiment, the method of controlling the electronic device may include: classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images.

[0327] According to an embodiment, the method of controlling the electronic device may include: sending a representative CAD image of each of the plurality of groups to an external electronic device.

[0328] According to an embodiment, the method of controlling the electronic device may include: receiving, from the external electronic device, an image of an object corresponding to the representative CAD image captured by the external electronic device.

[0329] According to an embodiment, the method of controlling the electronic device may include: using the received image of the object as training data to train a learning model to output a representative CAD image.

[0330] According to an embodiment, the step of sending a representative CAD image of each of the plurality of groups to an external electronic device may include: sequentially sending a first guide image including a first CAD image related to a first pose of the object and a second guide image including a second CAD image related to a second pose of the object among the plurality of representative CAD images to the external electronic device.

[0331] According to an embodiment, at least one of the first guide image or the second guide image may include a guide for guiding a change in the pose of the object.

[0332] According to an embodiment, the step of classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images may include: classifying the plurality of CAD images into a plurality of first groups based on similarity.

[0333] According to an embodiment, the step of classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images may include: identifying a plurality of first representative CAD images respectively corresponding to the plurality of first groups.

[0334] According to an embodiment, the step of classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images may include: classifying the plurality of first representative CAD images into a plurality of second groups based on similarity.

[0335] According to an embodiment, the step of sending representative CAD images of each of a plurality of groups to an external electronic device may include: identifying a plurality of second representative CAD images respectively corresponding to the plurality of second groups.

[0336] According to an embodiment, the step of sending representative CAD images of each of a plurality of groups to an external electronic device may include: sending the plurality of second representative CAD images to the external electronic device as the representative CAD images.

[0337] According to an embodiment, the external electronic device may be AR glasses.

[0338] According to an embodiment, a method of controlling an electronic device may include: receiving, from an external electronic device, an image of an object captured by the external electronic device.

[0339] According to an embodiment, the method of controlling the electronic device may include: using the received image as input data to obtain a representative CAD image from a learning model stored in a memory.

[0340] According to an embodiment, the method of controlling the electronic device may include: identifying a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on CAD information of an object are classified.

[0341] According to an embodiment, the method of controlling the electronic device may include: identifying a CAD image mapped to the received image among the plurality of CAD images included in the group.

[0342] According to an embodiment, the method of controlling the electronic device may include: sending information related to the CAD image mapped to the received image to the external electronic device.

[0343] According to an embodiment, the learning model may be trained using a plurality of CAD images obtained based on CAD information of an object as training data to output a representative CAD image.

[0344] According to an embodiment, the step of identifying a CAD image mapped to the received image among the plurality of CAD images included in the group may include: identifying, based on similarity to the received image, a CAD image mapped to the received image among the plurality of CAD images included in the group.

[0345] According to an embodiment, the method of controlling the electronic device may further include: identifying a defective part of an object included in the received image based on information related to the CAD image mapped to the received image.

[0346] According to an embodiment, the step of sending information related to a CAD image mapped to a received image may include: sending an image including a guide for guiding the identification of a defective part to an external electronic device.

[0347] According to an embodiment, the step of identifying a defective part of an object may include: comparing a CAD image mapped to a received image with the received image, and when at least one eigenvalue included in the received image differs from at least one eigenvalue included in the CAD image mapped to the received image by a set range or more, identifying the at least one eigenvalue as a defective part of the object.

[0348] According to an embodiment, the step of sending information related to a CAD image mapped to a received image to an external electronic device may include: sending an image including an assembly guide for assembling an object included in the received image as a component to the external electronic device based on the information related to the CAD image mapped to the received image.

[0349] According to an embodiment, at least one program in a non-transitory computer-readable storage medium storing at least one program may include instructions that enable an electronic device to obtain a plurality of CAD images based on CAD information of an object.

[0350] According to an embodiment, at least one program may include instructions that enable an electronic device to classify a plurality of CAD images into a plurality of groups based on the similarity between the plurality of CAD images.

[0351] According to an embodiment, at least one program may include instructions that enable an electronic device to send a representative CAD image of each of the plurality of groups to an external electronic device.

[0352] According to an embodiment, at least one program may include instructions that enable an electronic device to receive, from an external electronic device, an image of an object corresponding to a representative CAD image captured by the external electronic device.

[0353] According to an embodiment, at least one program may include instructions that enable an electronic device to use the received image of the object as training data to train a learning model to output a representative CAD image.

[0354] According to an embodiment, at least one program may include instructions that enable an electronic device to sequentially send a first guide image including a first CAD image related to a first pose of an object and a second guide image including a second CAD image related to a second pose of the object among a plurality of representative CAD images to an external electronic device.

[0355] According to an embodiment, at least one of the first guiding image or the second guiding image may include a guidance for guiding a change in the posture of an object.

[0356] According to an embodiment, at least one program may include instructions that enable the electronic device to classify a plurality of CAD images into a plurality of first groups based on similarity.

[0357] According to an embodiment, at least one program may include instructions that enable the electronic device to identify a plurality of first representative CAD images respectively corresponding to the plurality of first groups.

[0358] According to an embodiment, at least one program may include instructions that enable the electronic device to classify the plurality of first representative CAD images into a plurality of second groups based on similarity.

[0359] According to an embodiment, at least one program may include instructions that enable the electronic device to identify a plurality of second representative CAD images respectively corresponding to the plurality of second groups.

[0360] According to an embodiment, at least one program may include instructions that enable the electronic device to send the plurality of second representative CAD images to an external electronic device as representative CAD images.

[0361] According to an embodiment, the external electronic device may be AR glasses.

[0362] According to an embodiment, at least one program in a non - transitory computer - readable storage medium storing at least one program may include instructions that enable the electronic device to receive an image of an object captured by the external electronic device from the external electronic device.

[0363] According to an embodiment, at least one program may include instructions that enable the electronic device to use the received image as input data to obtain a representative CAD image from a learning model stored in a memory.

[0364] According to an embodiment, at least one program may include instructions that enable the electronic device to identify a group including the obtained representative CAD image among the plurality of groups into which a plurality of CAD images obtained based on CAD information of an object are classified.

[0365] According to an embodiment, at least one program may include instructions that enable the electronic device to identify a CAD image mapped to the received image among the plurality of CAD images included in the group.

[0366] According to an embodiment, at least one program may include instructions that enable the electronic device to send information related to the CAD image mapped to the received image to an external electronic device.

[0367] According to an embodiment, a learning model may be trained using a plurality of CAD images obtained based on object-based CAD information as training data to output a representative CAD image.

[0368] According to an embodiment, at least one program may include instructions that enable the electronic device to identify a CAD image mapped to a received image among a plurality of CAD images included in the group based on similarity to the received image.

[0369] According to an embodiment, at least one program may include instructions that enable the electronic device to identify a defective part of an object included in the received image based on information related to the CAD image mapped to the received image.

[0370] According to an embodiment, at least one program may include instructions that enable the electronic device to send an image including guidance for guiding the identification of a defective part to an external electronic device.

[0371] According to an embodiment, at least one program may include instructions that enable the electronic device to compare the CAD image mapped to the received image with the received image, and when at least one eigenvalue included in the received image differs from at least one eigenvalue included in the CAD image mapped to the received image by a set range or more, identify the at least one eigenvalue as a defective part of the object.

[0372] According to an embodiment, at least one program may include instructions that enable the electronic device to send an image including assembly guidance for assembling an object included in the received image as a component to an external electronic device based on information related to the CAD image mapped to the received image.

[0373] An electronic device according to an embodiment of the present disclosure may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the above-described electronic devices.

[0374] It should be understood that the embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the drawings, like reference numerals may be used to refer to like or related elements. It will be understood that a singular noun corresponding to a term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include all possible combinations of the items listed together in the corresponding one of the plurality of phrases. As used herein, terms such as "first" and "second" or "1st" and "2nd" may be used to simply distinguish the corresponding components from another component, and do not limit the components in other respects (e.g., importance or order). It will be understood that in the case where the term "operably" or "communicatively" is used or where the term "operably" or "communicatively" is not used, if an element (e.g., a first element) is referred to as "coupled to" or "connected to" another element (e.g., a second element), it means that the one element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0375] As used in connection with the embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware and may be used interchangeably with other terms (e.g., "logic", "logic block", "portion", or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0376] The embodiments described herein can be implemented as software (e.g., program 140) including one or more instructions that are machine-readable (e.g., by an electronic device 101) and stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke at least one of the one or more instructions stored in the storage medium and run the at least one instruction, with or without using one or more other components. This enables the machine to operate to perform at least one function in accordance with the at least one instruction invoked. The one or more instructions can include code generated by a compiler or code that can be run by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Herein, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being stored temporarily in the storage medium.

[0377] According to an embodiment, a method according to various embodiments of the present disclosure can be included and provided in a computer program product. The computer program product can be traded between a seller and a purchaser as a product. The computer program product can be distributed (e.g., downloaded or uploaded) online via an app store (e.g., Play Store TM ), or can be directly distributed (e.g., downloaded or uploaded) between two user devices (e.g., smartphones). If it is distributed online, at least part of the computer program product can be generated temporarily, or at least part of the computer program product can be stored at least temporarily in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).

[0378] According to an embodiment, each of the above components (e.g., a module or a program) may include a single entity or multiple entities. Some of the multiple entities may be separately provided in different components. According to an embodiment, one or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to an embodiment, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding one of the multiple components performed one or more functions before integration. According to an embodiment, the operations performed by a module, a program, or another component may be performed sequentially, in parallel, repeatedly, or in a heuristic manner, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device, comprising: a communication module; and at least one processor operably connected to the communication module, wherein the at least one processor is configured to: obtain a plurality of CAD images based on CAD information of an object, classify the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images, send a representative CAD image of each of the plurality of groups to an external electronic device, receive, from the external electronic device, an image of the object corresponding to the representative CAD image captured by the external electronic device, and generate a learning model using the received image of the object as training data to output the representative CAD image.

2. The electronic device according to claim 1, wherein the at least one processor is configured to: sequentially send a first guide image and a second guide image among a plurality of representative CAD images to the external electronic device, the first guide image including a first CAD image related to a first pose of the object, and the second guide image including a second CAD image related to a second pose of the object.

3. The electronic device according to claim 2, wherein at least one of the first guide image or the second guide image includes a guide for guiding a change in the pose of the object.

4. The electronic device according to any one of claims 1 to 3, wherein the at least one processor is configured to: classify the plurality of CAD images into a plurality of first groups based on similarity; identify a plurality of first representative CAD images respectively corresponding to the plurality of first groups; classify the plurality of first representative CAD images into a plurality of second groups based on similarity; identify a plurality of second representative CAD images respectively corresponding to the plurality of second groups; and send the plurality of second representative CAD images to the external electronic device as the representative CAD images.

5. The electronic device according to any one of claims 1 to 4, wherein the external electronic device is an augmented reality (AR) glasses.

6. An electronic device, comprising: a communication module; a memory; and at least one processor operably connected to the memory and the communication module, wherein the at least one processor is configured to: receive, from an external electronic device, an image of an object captured by the external electronic device, use the received image as input data to obtain a representative CAD image from a learning model stored in the memory, identify a group including the obtained representative CAD image among a plurality of groups into which a plurality of CAD images obtained based on CAD information of the object are classified, identify a CAD image mapped to the received image among the plurality of CAD images included in the group, and send information related to the CAD image mapped to the received image to the external electronic device, wherein the learning model is trained using the plurality of CAD images obtained based on CAD information of the object as training data to output a representative CAD image.

7. The electronic device according to claim 6, wherein The at least one processor is configured to: identify, based on similarity to the received image, the CAD image among the plurality of CAD images included in the group that is mapped to the received image.

8. The electronic device according to claim 6 or 7, wherein, the at least one processor is configured to: identify a defective part of the object included in the received image based on information related to the CAD image mapped to the received image, and send an image including guidance for guiding the identification of the defective part to the external electronic device.

9. The electronic device according to claim 8, wherein, the at least one processor is configured to: compare the CAD image mapped to the received image with the received image, and when at least one eigenvalue included in the received image differs from at least one eigenvalue included in the CAD image mapped to the received image by a set range or more, identify the at least one eigenvalue as a defective part of the object.

10. The electronic device according to any one of claims 6 to 9, wherein, the at least one processor is configured to: send an image including assembly guidance for assembling the object included in the received image as a component to the external electronic device based on information related to the CAD image mapped to the received image.

11. A method of controlling an electronic device, comprising: acquiring a plurality of CAD images based on CAD information of an object; classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images; sending a representative CAD image of each of the plurality of groups to an external electronic device; receiving, from the external electronic device, an image of the object corresponding to the representative CAD image captured by the external electronic device; and using the received image of the object as training data to generate a learning model for outputting the representative CAD image.

12. The method according to claim 11, wherein, the step of sending a representative CAD image of each of the plurality of groups to an external electronic device includes: sequentially sending a first guidance image and a second guidance image among the plurality of representative CAD images to the external electronic device, the first guidance image including a first CAD image related to a first posture of the object, and the second guidance image including a second CAD image related to a second posture of the object.

13. The method according to claim 12, wherein, at least one of the first guidance image or the second guidance image includes guidance for guiding a change in the posture of the object.

14. The method according to any one of claims 11 to 13, wherein, the step of classifying the plurality of CAD images into a plurality of groups based on similarity between the plurality of CAD images includes: classifying the plurality of CAD images into a plurality of first groups based on similarity, identifying a plurality of first representative CAD images respectively corresponding to the plurality of first groups, and classifying the plurality of first representative CAD images into a plurality of second groups based on similarity, and wherein the step of sending the representative CAD image of each of the plurality of groups to an external electronic device includes: identifying a plurality of second representative CAD images respectively corresponding to the plurality of second groups, and sending the plurality of second representative CAD images to the external electronic device as the representative CAD images.

15. The method according to any one of claims 11 to 14, wherein the external electronic device is an augmented reality (AR) glasses.

Images