Electronic device for identifying external objects and operating method thereof

Abstract

Provided are an electronic device for identifying external objects based on relevant information and an operating method thereof. The electronic device includes a memory configured to store an object list and object image learning data, and at least one processor. The at least one processor can be configured to acquire an image including a specific object, update the object list based on user purchase history information, collect image data related to the object based on the updated object list to update the object image learning data, learn an object classification model based on the updated object list and the updated object image learning data, and identify the specific object using the learned object classification model.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for identifying an external object based on relevant information and an operating method thereof. Background Art

[0002] With the development of information and communication technologies and semiconductor technologies, electronic devices have evolved into multifunctional devices that provide various services based on user space and objects. For example, based on wireless communication services, electronic devices can provide various services such as product purchase and management services, control and management services for various external electronic devices, and user-specific information provision and management services.

[0003] Recently, as vision systems have developed to provide the ability to recognize objects in images captured by cameras, electronic devices have been used in a wider range of fields using this capability. For example, electronic devices can recognize objects based on images captured by cameras and provide various services related to the recognized objects.

[0004] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the Invention

[0005] Generally, in an object recognition function, an object is detected and recognized using only object information included in an image, but there are limitations in recognition accuracy and identifiable object types, and a method of improving recognition accuracy of various objects is needed.

[0006] Aspects of the present disclosure are to address at least the above-mentioned problems and / or disadvantages and provide at least the advantages described below. Therefore, one aspect of the present disclosure provides an electronic device and an operating method thereof to improve recognition accuracy of external objects by utilizing various relevant information.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0008] According to one aspect of the present disclosure, an electronic device is provided. The electronic device includes a memory configured to store an object list and object image learning data, and at least one processor operably connected to the memory, wherein the at least one processor is configured to: acquire an image including a specific object; update the object list based on user purchase history information; collect image data related to the object based on the updated object list to update the object image learning data; learn an object classification model based on the updated object list and the updated object image learning data; and recognize the specific object using the learned object classification model.

[0009] According to one aspect of the present disclosure, a method for operating an electronic device is provided. The method includes acquiring an image including a specific object, updating an object list based on user purchase history information, collecting image data related to the object based on the updated object list to update object image learning data, learning an object classification model based on the updated object list and the updated object image learning data, and recognizing the specific object using the learned object classification model.

[0010] According to various embodiments, the electronic device may acquire image information and related information about an external object, and improve a recognition rate of the external object based on the acquired information.

[0011] According to various embodiments, the electronic device may acquire relevant information for external object recognition to add learning data and learn an object classification model, thereby improving image recognition performance of the external object.

[0012] According to various embodiments, the electronic device may appropriately limit the range of objects to be compared by acquiring relevant information for external object recognition based on the location of the external object, thereby saving resources and time for external object recognition.

[0013] According to various embodiments, the electronic device can quickly and accurately recognize an external object with a small amount of resources, thereby immediately providing various user services associated with the external object.

[0014] Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 is a block diagram of an electronic device in a network environment according to an embodiment of the present disclosure;

[0017] Figure 2 is a functional block diagram of an electronic device according to an embodiment of the present disclosure;

[0018] Figure 3 is a flowchart illustrating an operation for identifying an external object in an electronic device according to an embodiment of the present disclosure;

[0019] Figure 4 is another flow chart illustrating an operation for identifying an external object in an electronic device according to an embodiment of the present disclosure;

[0020] Figure 5 is a flowchart illustrating an operation for identifying an external object in an electronic device according to an embodiment of the present disclosure;

[0021] Figure 6 is a flowchart illustrating an operation of updating an object classification model for recognizing an external object in an electronic device according to an embodiment of the present disclosure;

[0022] Figure 7A is a diagram for describing a service providing operation according to external object recognition of an electronic device according to an embodiment of the present disclosure;

[0023] Figure 7B is a diagram for describing a service providing operation according to external object recognition of an electronic device according to an embodiment of the present disclosure; and

[0024] Figure 7C is a diagram for describing a service providing operation according to external object recognition of an electronic device according to an embodiment of the present disclosure.

[0025] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION

[0026] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist understanding, but these are merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0027] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and is not intended to limit the present disclosure as defined by the appended claims and their equivalents.

[0028] It will be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0029] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to an embodiment of the present disclosure.

[0030] Reference Figure 1 , the electronic device 101 in the network environment 100 can communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 can communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190 (e.g., a communication circuit or a transceiver), a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented as embedded in the display device 160 (eg, a display).

[0031] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component of the electronic device 101 connected to the processor 120 (e.g., a hardware component or a software component), and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load commands or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. Depending on the embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121 .

[0032] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) together with the main processor 121. Depending on the embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123.

[0033] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0034] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0035] The input device 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by another component of the electronic device 101 (e.g., the processor 120). The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).

[0036] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0037] The display device 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. Depending on the embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit adapted to measure the strength of the force caused by the touch (e.g., a pressure sensor).

[0038] The audio module 170 can convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0039] The sensor module 176 can detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0040] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). Depending on the embodiment, the interface 177 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0041] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. Depending on the embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0043] The camera module 180 may capture still images or moving images. Depending on the embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0044] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0045] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0046] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. Depending on the embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0047] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., an external electronic device). Depending on the embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). Depending on the embodiment, antenna module 197 may include multiple antennas. In this case, at least one antenna suitable for the communication scheme used in a communication network (such as first network 198 or second network 199) may be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. Depending on the embodiment, additional components other than the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may also be formed as part of antenna module 197.

[0048] At least some of the above components can be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general-purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0049] According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type than the electronic device 101. According to an embodiment, all or some operations to be executed on the electronic device 101 may be executed on one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or execute another function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial response to the request, either by further processing the result or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0050] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those described above.

[0051] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but rather include various changes, equivalents or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding phrase in the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish corresponding components from another component, and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., a second element)”, “combined to another element (e.g., a second element)”, “connected with another element (e.g., a second element)”, or “connected to another element (e.g., a second element)”, when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0052] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, depending on an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0053] The various embodiments described herein can be implemented as software (e.g., program 140) comprising one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. The term "non-transitory storage medium" is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0054] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™), or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated, or at least a portion of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0055] According to various embodiments, each component (for example, module or program) in the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (for example, module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operations performed by module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0056] Figure 2 is a functional block diagram of an electronic device according to an embodiment of the present disclosure. The functional configuration of such an electronic device may be included in Figure 1 In the electronic device 101 shown.

[0057] refer to Figure 2 , the electronic device 101 may include a memory 130 , a camera module 180 and an object recognition module 200 .

[0058] According to various embodiments, the object recognition module 200 may represent an example of a functional processing module related to identifying an object by processing an acquired image to obtain image information and extracting an object region based on the image information. In various embodiments, the object recognition module 200 may be included as a hardware module in a processor including a processing circuit (e.g., Figure 1 120) or included as a software module.

[0059] According to various embodiments, the memory 130 may store instructions, control command codes, control data, or related data for controlling an object recognition operation of the electronic device 101 .

[0060] According to various embodiments, the memory 130 may include a domain object list 131 , a purchase record (history) 133 , and / or learning data 135 as a database.

[0061] For example, the domain object list 131 of the memory 130 may include domain information for classifying multiple objects, as well as a list of object names for each of the multiple object types included in the domain information. The domain information can be used to classify multiple objects into different domains based on various locations (such as residence or business) classified according to various criteria. If desired, for certain domains, the domain information can be reclassified into multiple subdomains based on various criteria, such as object function or purpose of use. For example, the domain object list 131 can be categorized into subtypes that include at least some of the multiple object types included in the type information. For example, the domain object list 131 may classify detailed locations (such as living room, room, kitchen, or bathroom) as subdomains of the residential domain (e.g., based on the "residence" item in the location domain based on the classified domain information) and may classify object types (such as electronic devices, furniture, tools, etc.) as being included in a subdomain (such as, for example, kitchen). Refrigerators, microwave ovens, coffee makers, etc. may be included as electronic device types, while dining tables, chairs, shelves, etc. may be included as furniture types.

[0062] For example, the purchase record (history) 133 of the memory 130 may include purchase product information and / or related information about the product based on the purchase history of each user's actual product. In addition, the purchase record (history) 133 may include expected purchase products and / or related information about the product based on the user's network search history.

[0063] For example, the product information and / or product-related information stored in the purchase record (history) 133 may be obtained from the electronic device 101, or other electronic devices existing in the corresponding space, or other electronic devices used by the user (e.g., Figure 1 For example, the product information and / or related information about the product stored in the purchase record (history) 133 may be obtained from an external device (e.g., a product search website such as Amazon, Alibaba, or Taobao) Figure 1 electronic device 102 or 104 or server 108) to obtain.

[0064] According to various embodiments, whenever a purchase or web search record occurs, the object recognition module 200 may acquire product information and / or related information about the product from, for example, an external device including a product search website.

[0065] According to various embodiments, whenever a purchase history occurs, the object recognition module 200 may receive corresponding purchase and product information and / or related information about the product through, for example, a push alarm from an external device including a product search website.

[0066] According to various embodiments, the object recognition module 200 may, for example, check whether a purchase history or product search occurs every predetermined period (e.g., once every two days), and if so, may obtain product information and / or related information about the product from an external device including a product search website.

[0067] For example, the learning data 135 of the memory 130 may include various image data and related data required to recognize an object based on the domain object list 131. For example, the learning data 135 may be obtained from the learning data management server 108 and stored in the memory 130. For example, the learning data 135 may include learning data including various information, such as an object image, for classifying a specific domain (such as a place).

[0068] For example, the learning data 135 may include type information of each object type belonging to each domain. The type information may be information generated based on a common point of a plurality of objects, which may be classified as a common type. For example, the learning data 135 may include information about key points of each object type belonging to each domain. The key point information may be information obtained by image processing. For example, an object corresponding to each type may correspond to a shape including a common feature corner point. The learning data 135 may store information about the characteristic shape of each type as key point information.

[0069] According to various embodiments, the object recognition module 200 may include an image processing module 210 , an object extraction module 220 , a domain selection module 230 , an object classification module 240 , and / or a learning data management module 250 .

[0070] According to various embodiments, the image processing module 210 may process, for example, an original image including an object obtained from, for example, the camera module 180 or an external device (not shown) and output image information that has been pre-processed (such as lens distortion compensation or noise removal) to improve an image recognition rate.

[0071] According to various embodiments, the object extraction module 220 can analyze image information and identify to extract objects (or object areas). For example, the object extraction module 220 can output an area composed of similar feature points from the image information as an object area. For example, the object extraction module 220 can use various recognition algorithms (such as the Scalar Invariant Feature Transform (SIFT) algorithm and the Speeded Up Robust Features (SURF) algorithm) to extract feature points of edges, corners, and contours. This is merely an example, and the embodiments of the present disclosure are not limited thereto, and reference can be made to identifying objects in combination with various known technologies. For example, the object extraction module 220 can identify an object by generating a point cloud based on a signal reflected from the object, where the point cloud is a collection of points defining an object in the form of voxels.

[0072] According to various embodiments, the domain selection module 230 can select a domain for object classification and load an object list of the corresponding domain from the domain object list 131 of the memory 130. For example, the domain selection module 230 can select a specific space (such as a home or company) where the electronic device 101 is located as the domain for object classification based on the location identification of the electronic device 101. To this end, the domain selection module 230 can use, for example, identification information of various access providers (APs) (such as wireless fidelity (WIFI) repeaters) communicating with the electronic device 101, location identification information (such as global positioning system (GPS) information) for location identification, and various applications for identifying locations based on various identifiers of location identification. Alternatively or in addition, the domain selection module 230 can extract other object information or other location-related information that can identify a location from an image including an object and select a domain based on this.

[0073] According to various embodiments, the object classification module 240 can identify objects based on the learning data 135. For example, based on the object list included in the domain selected by the domain selection module 230, the object classification module 240 can use the learning data 135 to output the name of the object corresponding to the object or object region extracted from the object extraction module 220. For example, based on the object list included in the domain selected by the domain selection module 230, the object classification module 240 can compare object information (such as image information or feature point information of the object or object region extracted by the object extraction module 220) with the data stored in the learning data 135, search for data corresponding to the domain object list, determine data that matches the object information, and output the object name corresponding to the data. To this end, for example, the object classification module 240 can calculate a similarity indicating the degree of similarity with the feature point information of the extracted object for each feature point information stored in the learning data 135, and can check for at least one feature point information that exceeds a reference similarity. To this end, the object classification module 240 can include an artificial intelligence module and / or an advanced image recognition framework (AIR framework). The object classification module 240 may analyze the object area based on feature points of the object area as object information, calculate analysis information, and thereby calculate keywords or metadata.

[0074] According to various embodiments, when the object classification module 240 cannot identify an object corresponding to object information exceeding a reference similarity through analysis and / or matching, the object classification module 240 may update the domain object list 131 through the purchase record (history) 133 .

[0075] According to various embodiments, the object classification module 240 may update the learning data through the learning data management module 250 according to the update of the domain object list 131 .

[0076] According to various embodiments, based on the object list of the domain selected using the updated domain object list 131 , the object classification module 240 may perform the object recognition operation again using the updated learning data.

[0077] According to various embodiments, when necessary (such as when object recognition fails), the object classification module 240 may output a method for guiding the user to identify the object through an input device (eg, Figure 1 The information inputted by the input device 150 is displayed, for example, by a display device (e.g., Figure 1 The display device 160) receives the object name.

[0078] According to various embodiments, the object classification module 240 can send object-related information (such as the object name based on the purchase record (history) 133) to the external server 108, and can update the learning data 135 by downloading the learning data updated by adding learning data related to the object based on the purchase record (history) 133 from the external server 108.

[0079] According to various embodiments, the domain selection module 230 or the object classification module 240 may check the purchase record (history) 133 and update the domain object list 131 based on user purchase activity information including product purchase information and / or product search information according to purchase history.

[0080] According to various embodiments, the learning data management module 250 may acquire object information (such as an image of an object corresponding to the purchase record (history) 133) and update the learning data 135 by adding it to the learning data 135, and the object classification module 240 may classify the object based on the updated domain object list 131 and the updated learning data 135. The object image may include, for example, a high-resolution image for merchandise sales, and may be obtained, for example, through a product search website such as Amazon, Alibaba, or Taobao.

[0081] According to various embodiments, an electronic device (e.g., Figure 1 and Figure 2 The electronic device 101 may include a memory (eg, Figure 1 and Figure 2 130), and a processor operatively connected to the memory (e.g., Figure 1 and Figure 2 processor 120 and / or 200 in the system).

[0082] According to various embodiments, the processor may acquire an image including a specific object, update an object list based on user purchase history information, collect image data related to the object based on the updated object list to update object image learning data, learn an object classification model based on the updated object list and the updated object image learning data, and identify the specific object through the learned object classification model.

[0083] According to various embodiments, the processor may obtain user purchase history information from information generated by purchasing or searching for a specific product.

[0084] According to various embodiments, the processor may obtain user purchase history information from an application for purchasing or searching for a specific product.

[0085] According to various embodiments, the electronic device may further include a communication module (eg, Figure 1 Communication module 190).

[0086] According to various embodiments, the processor may acquire image data from a website related to purchase or search of a specific product through the communication module based on user purchase history information.

[0087] According to various embodiments, based on information generated by searching for or purchasing a specific product, the processor may acquire similar image data of a product similar to the specific product as image data through the communication module.

[0088] According to various embodiments, the processor may acquire object-related management information based on information generated by searching for or purchasing a specific product, and provide an object-related management service based on the object-related management information.

[0089] According to various embodiments, the processor may attempt to recognize a specific object based on the object list, update the object list when the attempt fails, preliminarily learn an object classification model based on the updated object list, and recognize the specific object through the preliminarily learned object classification model.

[0090] According to various embodiments, when the primary learning object classification model fails to recognize a specific object, the processor can obtain image data related to the object from an external device through a communication module based on user purchase history information, and perform secondary learning on the primary learning object classification model using the learning data updated by adding the obtained image data.

[0091] According to various embodiments, the processor may recognize a specific object through the secondary learned object classification model, and when recognition of the specific object through the secondary learned object classification model fails, may acquire information related to the object from the user.

[0092] According to various embodiments, the processor may acquire location information where a specific object is located, and may identify the specific object in the object list by limiting the object list based on the location information.

[0093] Figure 3 is a flowchart illustrating an operation for recognizing an external object in an electronic device according to an embodiment of the present disclosure.

[0094] According to various embodiments, Figure 3 The operations shown can be performed by an electronic device (e.g., Figure 1 A processor (eg, Figure 1 Processor 120 or Figure 2 The object recognition module 200, hereinafter referred to as the "processor") is executed.

[0095] refer to Figure 3 According to various embodiments, in operation 311, the processor may obtain object information that needs to be identified and object-related information.

[0096] For example, the object information may include image information about the object or an image including the object. For example, the object may include a person, an object, or a background. In addition, the object image information may include image information about the object or an object area extracted from, for example, the image information including the object. Based on the position coordinates of the object from the image including the object, the object area may mean, for example, an area consisting of a boundary separated from other objects (for example, the background). The image information including the object may be obtained by, for example, an image sensor (for example, Figure 1 or 2) or can be obtained from an external device (eg, Figure 1 electronic device 102 or 104) receives.

[0097] For example, object-related information may include information about the location of the object. The information about the location of the object may be one of various predefined locations classified according to various criteria, such as a residence or a company. The information about the location of the object may include, for example, a specific location as an upper domain and a detailed location of a specific location as a lower domain. The location information may be obtained, for example, by using identification information of various access providers (APs) that communicate with the electronic device 101 (such as a Wi-Fi repeater), location identification information for location identification (such as global positioning system (GPS) information), and various applications for identifying locations based on various identifiers for location identification. Alternatively or in addition, information about the location may be identified by extracting other object information or other location-related information that can identify the location from an image containing the object, and based on this, using information stored in a memory (for example, Figure 1 or Figure 2 Memory 130) in the learning data (e.g., Figure 2 learning data 135).

[0098] According to various embodiments, in operation 313 , the processor may select an object list for identifying the object based on the object information and / or object-related information.

[0099] For example, the object list may include a list of objects included in a domain related to the object. The object list may include a list of objects included in a domain corresponding to a location where the object is located.

[0100] According to various embodiments, the object list included in the domain corresponding to the place where the object is located may be, for example, a preconfigured object list of objects pre-registered for the place.

[0101] According to various embodiments, the object list included in the domain corresponding to the location where the object is located may include a list of objects pre-registered at the location through a general object recognition process, such as, for example, an environment recognition process. Furthermore, the object list may also include a list of objects that were recognized or modified at the location by a user request, in addition to or in addition to the general object recognition process.

[0102] According to various embodiments, in operation 315, the processor may identify an object corresponding to the object information based on a classification model. In this case, the object corresponding to the object information may be identified as any one of the (plural) selected domain object lists. The classification model may be a learning model for object recognition learned using the learning data 135 stored in the memory 130. The classification model may be established by learning (such as deep learning) using the learning data 135 in the electronic device 101. In the classification model, the collection of the learning data 135 and the object recognition learning may be performed by an external server (e.g., Figure 1 In this case, the electronic device 101 may receive the learning data 135 and the classification model from the external server 108 and store them in the memory 130.

[0103] According to various embodiments, when the learning data described above needs to be updated, the processor may update the learning data and, in response to the updated learning data, update the classification model. For example, when the processor tracks a user's product purchase history or web search history associated with a location corresponding to a domain and obtains relevant information, the processor may extract object information or object-related information (such as product information and / or purchase or search information) from the purchase history or web search record and add it to the domain object list. The processor may update the domain object list at predetermined intervals or whenever a specific purchase history or web search record appears.

[0104] The processor may update the learning data based on the updated domain object list and relearn the classification model accordingly. Therefore, in operation 315, based on the updated classification model, the processor may output the name of a specific object among the objects included in the updated domain object list as an object corresponding to the obtained object information.

[0105] According to various embodiments, the scope of target objects for object recognition is limited to a list of objects in a selected domain, thereby reducing the problem of misidentification of an object or a product corresponding to the object itself that occurs in existing deep learning models, which is caused by applying a general name without restricting the object name, and thus improving the object recognition rate.

[0106] According to various embodiments, the processor may send object information and / or object-related information to an external server 108, and the server 108 may identify the object corresponding to the object information based on the classification model and send the identified object name to the electronic device 101. In this case, the processor may send an image of the object or an image including the object as the object information to the server 108 for analysis. In addition, in addition to the image, the processor may also send information such as the location of the object as the object-related information to the server 108.

[0107] Figure 4 is another flowchart illustrating an operation for recognizing an external object in an electronic device according to an embodiment of the present disclosure.

[0108] According to various embodiments, Figure 4 The operations shown can be performed by an electronic device (e.g., Figure 1 A processor (eg, Figure 1 Processor 120 or Figure 2 The object recognition module 200, hereinafter referred to as the "processor") is executed.

[0109] refer to Figure 4 According to various embodiments, in operation 411, a processor may acquire an image including an object to be recognized. The object may include, for example, a person, an object, or a background.

[0110] Image information including the object may be obtained by, for example, an image sensor (e.g., Figure 1 The image may be acquired from a camera module 180 of the electronic device 101 or may be acquired from an external device (eg, Figure 1 electronic device 102 or 104) receives.

[0111] According to various embodiments, in operation 413, the processor may obtain information about a location where the object is located. The information about the location where the object is located may be one of various predefined locations classified according to various criteria, such as a residence or a company. The information about the location may be identified based on extraction of other object information or other location-related information that is capable of identifying the location from an image containing the object. For example, the information about the location may be identified based on predetermined reference information, such as the presence of specific furniture (e.g., a bed or a sink) or the number of specific furniture (e.g., the number of tables is 4 or more). For example, for all objects extracted from an image containing an object, learning data stored in a memory (e.g., Figure 2 The learning data 135 of the memory 130 is used to identify information about the location.

[0112] According to various embodiments, the processor may extract a target object or an object region from an image including an object in operation 415. The object region may mean, for example, a region consisting of a boundary separated from other objects (e.g., background) based on the position coordinates of the object from the image including the object.

[0113] According to various embodiments, in operation 417 , the processor may select an object list for identifying the extracted object based on the location information.

[0114] For example, the selected object list may include a list of objects included in the domain corresponding to the location where the object exists. For example, the object list of the domain corresponding to the location where the object exists may be a list of objects pre-registered at the specific location where the object currently exists. For example, the object list of the domain corresponding to the location where the object exists may include a list of objects pre-registered through a general object recognition process, such as an environment recognition process. Furthermore, the object list of the domain may also include a list of objects that have been recognized or modified at the location by a user request, in addition to or in addition to the general object recognition process.

[0115] According to various embodiments, the processor may identify an object corresponding to the object information based on a classification model for object recognition in operation 419. In this case, the object corresponding to the object information may be identified as any one of the selected domain object list(s).

[0116] According to various embodiments, as a result of comparing the learning data with the object information according to the classification model, the processor may display the object information, for example, via a display device (e.g., Figure 1 The processor may also provide a display device 160 of the user's computer to provide a similarity (e.g., 60%, 80%, or 85%) between the object and the recognized object name. Additionally, the processor may prompt the user to provide feedback on the object-specific results and similarities.

[0117] The classification model may be a learning model for object recognition learned using the learning data 135 stored in the memory 130. The classification model may be established through learning (such as deep learning) using the learning data 135 in the electronic device 101. Alternatively, in the classification model, the collection of the learning data 135 and the object recognition learning process may be performed by an external server (e.g., Figure 1 In this case, the electronic device 101 may receive the learning data 135 and the classification model from the external server 108 and store them in the memory 130.

[0118] According to various embodiments, when object recognition corresponding to object information in the domain object list fails, the processor may determine that the object does not exist in the domain object list based on the classification model, and update the domain object list by collecting information related to the object in addition to the image of the object. Information about the object may be obtained from, for example, purchase history or web search records of users (e.g., residents, employees, or registrants) associated with the place corresponding to the domain.

[0119] Figure 5 is a flowchart illustrating an operation for recognizing and identifying an external object in an electronic device according to an embodiment of the present disclosure.

[0120] According to various embodiments, Figure 5 The operations shown in FIG are used to describe in more detail Figure 4 Operation 419 and may be performed by an electronic device (e.g., Figure 1 A processor (eg, Figure 1 Processor 120 or Figure 2 The object recognition module 200, hereinafter referred to as the "processor") is executed.

[0121] According to various embodiments, for an image including an object, the processor may identify the object according to an image object classification model using pre-built learning data based on the location of the object or the location of a list of objects classified into the domain where the object is located.

[0122] refer to Figure 5 According to various embodiments, in operation 511, when object recognition fails, the processor may obtain object-related information. For example, when object recognition fails, the processor may determine a domain object list that needs to be updated. Thus, for example, for a domain where an object is located, the processor may obtain object information or object-related information from information generated based on correlations between various objects of a user associated with the corresponding location, such as a purchase history or web search history of a specific object.

[0123] For example, when the domain is selected as "residence", object information or object-related information can be obtained based on the purchase history or network search record of the first user who is a resident of the residence. That is, if the first user searches for a coffee machine in an electronic device through a network search, or even purchases a coffee machine, the corresponding product is likely to be newly introduced to the domain "residence" and corresponds to the object currently to be recognized. Therefore, the processor can improve the recognition rate of the object by adding the corresponding product to the domain object list. The processor can include various product information, such as product name, product type, manufacturer, size, identification and image of the corresponding product, and / or related purchase information (such as search or purchase date from, for example, an electronic device (such as the first user's smartphone)).

[0124] According to various embodiments, in operation 513 , the processor may add the object identified based on the obtained object-related information to the object list of the domain.

[0125] In the example above, the processor could add a coffee machine to the domain's list of objects.

[0126] According to various embodiments, in operation 515 , the processor may acquire image information about the object added to the object list of the domain and update the learning data.

[0127] According to various embodiments, the processor may acquire high-resolution image information, such as images of goods for sale or advertisements, based on purchase history or search history, and add it as learning data. The added object image information may be obtained, for example, from various websites (e.g., Amazon, Alibaba, or Taobao) that sell objects as products. Furthermore, the processor may acquire images based on product information and images of various products corresponding to the same object name based on purchase history, and add them as learning data.

[0128] In operation 517 , the processor may learn and update the object classification model based on the updated domain object list and the updated learning data.

[0129] According to various embodiments, in operation 519, the processor may perform an object recognition operation according to the classification model based on the updated domain object list and the updated learning data. In this case, for example, when the object region extracted from the image including the object is compared using the learning data according to the image object classification model, and the similarity is greater than or equal to a predetermined reference value (e.g., 95% or more), the processor may determine that the object is recognized and thereby recognize the object. On the other hand, even when the similarity is greater than or equal to the predetermined reference value, the processor may determine that the object recognition has failed, for example, when there is negative feedback depending on feedback from the user. In addition, even when the similarity is less than or equal to the predefined reference value, the processor may determine that the object recognition is successful when there is positive feedback depending on, for example, user feedback.

[0130] Figure 6 is a flowchart illustrating an operation of updating an object classification model for recognizing an external object in an electronic device according to an embodiment of the present disclosure.

[0131] According to various embodiments, Figure 6 The operations shown in FIG are used to describe in more detail Figure 5 The object classification model update operation may be performed by an electronic device (e.g., Figure 1 A processor (eg, Figure 1 Processor 120 or Figure 2 The object recognition module 200, hereinafter referred to as the "processor") is executed.

[0132] refer to Figure 6 According to various embodiments, at operation 611, the processor may acquire an image of the object based on the updated domain object list, add it to the learning data, and perform preliminary learning of the classification model. The image of the object may be acquired, for example, by the electronic device 101 or an external device (e.g., a home network or a wired / wireless local area network) connected via a wired / wireless local area network. Figure 1 electronic device 102 or 104) to be received by electronic device 101.

[0133] According to various embodiments, the learning data may be preconfigured to include sample images capable of distinguishing features of each object in the domain object list, or the preconfigured learning data may be received from the server 108 and stored in a memory of the electronic device 101 (e.g., Figure 1 or Figure 2 in the memory 130).

[0134] According to various embodiments, at operation 613, the processor may determine whether the object is recognized. The processor may recognize the object by, for example, determining the degree of similarity or difference between images included in the learning data based on a classification model and determining the most similar image. As a result of the similarity determination, if the similarity is less than a reference value, the processor may determine that no similar images exist. The processor may apply, for example, meta-learning techniques to improve the image recognition rate.

[0135] According to various embodiments, when it is determined in operation 613 that the object is not recognized, in operation 615 the processor may collect object data for updating the classification model.

[0136] For example, the processor may extract or receive high-resolution images such as product sales images and various product information (such as product names and manufacturers of objects) from purchase or search websites, and collect object information or object-related information (such as residents of a residence in a specific location or registered users) obtained based on domain-based purchase history or network search records as object data. The collected object data may be added to the learning data.

[0137] According to various embodiments, in operation 617 , the processor may perform secondary learning on the classification model using the added learning data based on the updated domain object list.

[0138] According to various embodiments, in operation 619 , the processor may perform an object recognition operation using the object list of the domain updated by the newly added object based on the secondary learned classification model, thereby determining whether the object is recognized.

[0139] According to various embodiments, when the classification model based on secondary learning fails in object recognition, in operation 621 , the processor may collect similar data for updating the classification model.

[0140] According to various embodiments, similarity data may include, for example, related images collected by web crawling based on keyword searches using object-related information (such as product information obtained based on the object's purchase history or web search history). In addition, the processor may generate extended data and add it as learning data by applying image processing techniques such as rotation, black and white, inversion, or tilting to the additionally obtained similarity data.

[0141] According to various embodiments, in operation 623 , the processor may perform tertiary learning on the classification model using learning data to which the collected and / or generated data is added based on the added domain object list.

[0142] According to various embodiments, in operation 625 , the processor may perform an object recognition operation using the object list of the domain updated by the newly added object based on the three-level learned classification model, thereby determining whether the object is recognized.

[0143] According to various embodiments, when recognition of an object fails according to the classification model of the three-level learning, the processor may acquire the name of the object by asking the user through the electronic device 101 in operation 627 .

[0144] According to various embodiments, the processor may output a guide requesting the user to select a desired mode of operation through a display device (eg, Figure 1 display device 160) to enter the object name.

[0145] According to various embodiments, the processor may collect object data based on the acquired object name in operation 629. The object data may include related images collected by crawling the web through a keyword-based search using the acquired object name.

[0146] According to various embodiments, in operation 631 , the processor may add the collected object data as learning data and perform quaternary learning on the classification model based on the added learning data.

[0147] According to various embodiments, in operation 633 , the processor may perform an object recognition operation using the object list of the domain updated by the newly added object based on the classification model learned in four stages, thereby determining whether the object is recognized.

[0148] According to various embodiments, when it is determined in operation 633 that the object is recognized, in operation 635, the processor may store the updated learning data in a memory (eg, Figure 1 or Figure 2 in the memory 130).

[0149] Figure 7A is a diagram for describing a service providing operation according to external object recognition of an electronic device according to an embodiment of the present disclosure.

[0150] Figure 7B is a diagram for describing a service providing operation according to external object recognition of an electronic device according to an embodiment of the present disclosure.

[0151] Figure 7C is a diagram for describing a service providing operation according to external object recognition of an electronic device according to an embodiment of the present disclosure.

[0152] 7A to 7C The operations shown can be performed by an electronic device (e.g., Figure 1 A processor (eg, Figure 1 Processor 120 or Figure 2 The object recognition module 200, hereinafter referred to as the "processor") is executed.

[0153] refer to Figure 7A For example, the processor may receive data from the electronic device 101 or an external device (e.g., Figure 1 electronic device 102 or 104) of the camera module (eg, Figure 1 or Figure 2 The camera module 180) captures an image including the object 711 as object information.

[0154] For example, an object image may include various image information about the object. The object information may include, for example, object-related information, such as a product name (e.g., C beverage) and / or product management information, such as a shelf life. The object image may also include a background image or the surrounding environment of the object, or another object. Furthermore, the processor may estimate the object name by recognizing, for example, text information from the object or object region extracted from the object image.

[0155] According to various embodiments, the processor may extract an object or an object region from the object image, and extract feature point information of the object from the object region.

[0156] According to various embodiments, the processor may extract information about the location of the object from the object image. For example, the location of the object may be identified from the object image, such as on a refrigerator shelf. In this case, the processor may select refrigerator as a subdomain. Furthermore, the processor may extract information that can be used to estimate a higher-level domain (e.g., "kitchen" or "house") by analyzing other images acquired before or after acquiring the object image.

[0157] According to various embodiments, the processor can obtain the location of the electronic device 101 using, for example, identification information of various access providers (APs) (such as, WIFI repeaters) that communicate with the electronic device 101, location identification information for location identification (such as, Global Positioning System (GPS) information), and various applications for identifying locations based on various identifiers of location identification.

[0158] According to various embodiments, the processor may select a domain for identifying an object, for example, a residence, based on object information and / or object-related information. The processor may select the domain for identifying an object as "residence" as an upper domain and "refrigerator" as a lower domain.

[0159] According to various embodiments, the processor may recognize the object according to the classification model using the object image based on the object list of the domain selected for object recognition.

[0160] According to various embodiments, the object list included in a domain may include, for example, a preconfigured object list of objects pre-learned or registered for the domain. A domain's object list may include, for example, a list of objects pre-identified and registered through an object recognition process. Furthermore, a domain's object list may also include a list of objects identified or modified by a request from a user registered with the domain. For example, a domain's object list may include a list of objects such as various beverages, foods, and food containers in a "refrigerator" as a lower-level domain for the upper-level domain "residence."

[0161] According to various embodiments, when an object cannot be identified based on the domain object list, the processor can track the purchase history or web search history of a relevant person (such as a resident of the domain or a registered user), obtain product-related information, extract the object information therefrom, and add it to the domain object list. The processor can track the purchase history or web search history and obtain the product-related information each time the purchase history or web search history of the relevant person appears or at predetermined intervals, even if the object cannot be identified. Thus, the object information can be extracted and added to the domain object list.

[0162] According to various embodiments, the processor may update learning data for recognizing an object by obtaining image data and / or object-related information corresponding to an object list added in response to a domain object list update.

[0163] The processor may update the learning data based on the updated domain object list and perform learning on the classification model to recognize the object accordingly. Therefore, the processor may output the name of a specific object (e.g., soft drink) among the objects included in the updated domain object list as the object corresponding to the obtained object information based on the updated classification model.

[0164] In addition, the processor can output object management information corresponding to the output object information. The object management information can include, for example, information such as the manufacturer of the object, product name, expiration date, purchase date, purchase website, calories, and nutritional information.

[0165] According to various embodiments, the processor may provide object-based services to the user based on the object information and / or object management information. For example, when the expiration date of an object approaches, the processor may provide the user with information about the expiration date of the object through the electronic device 101.

[0166] In addition, for example, when an object is provided to a user and consumed and is no longer recognized at the object's location, the processor may provide, through the electronic device 101, guidance information for asking the user to reorder the objects based on the object information and / or object management information.

[0167] According to various embodiments, the processor may configure a separate domain object list update scenario for, for example, the lower domain "refrigerator" included in the upper domain "residence." For example, whenever a new item is introduced into the "refrigerator" and / or leaked, the processor may separately manage the list of objects for the "refrigerator," obtain object information (such as the location where the item is placed in the "refrigerator," the product type, and the product name), as well as object-related information (such as the purchase date, purchase website, calories, and nutritional information) or object management information, to update the object list, and provide the aforementioned services based on this information.

[0168] refer to Figure 7B , for example, the processor may acquire an image including the object 713 as the object information from the camera module 180 of the electronic device 101 or an external device.

[0169] According to various embodiments, the object image may include a specific product image, such as a smartphone.

[0170] According to various embodiments, the processor may extract an object or an object region from the object image, and extract feature point information of the object from the object region.

[0171] According to various embodiments, the processor may extract information about the location of the object from the object image. For example, when another object is located in the background of the object image, information that can be used to infer the location of the object may be extracted from the other object. In addition, the processor may analyze another image acquired before or after acquiring the object image and identify the location of the object, such as a bedroom, from a portion of another object (e.g., a side table) included in the other image. In this case, the processor may select the domain "residence" as the domain corresponding to the object based on the location of the object.

[0172] According to various embodiments, the processor can obtain the location of the electronic device 101 using, for example, identification information of various access providers (APs) (such as, WIFI repeaters) that communicate with the electronic device 101, location identification information for location identification (such as, Global Positioning System (GPS) information), and various applications for identifying locations based on various identifiers of location identification.

[0173] According to various embodiments, the processor may load an object list of the domain based on the selected domain "residence" and identify the object according to the classification model using the object image.

[0174] According to various embodiments, the list of objects included in the domain may include, for example, objects that may generally be included in the domain, and may be obtained from, for example, a server (e.g., Figure 1 The server 108) receives the domain object list in advance, for example, and stores it in the memory 130. In addition, the processor can pre-identify and learn the domain object list and learning data through the object recognition process in the domain, or update the object through the user's registration process.

[0175] According to various embodiments, when an object is first identified in the domain, when there is a user request for any related service, or when a purchase history or search history related to the object appears, the processor may check the domain object list and update the domain object list by collecting object-related information.

[0176] According to various embodiments, for example, the processor may check the purchase history or search history and obtain object-related information such as the type of product purchased, the product name, the product manufacturer, the purchase or search date, and the purchase price through various application information of the electronic device 101, such as a web browser, a communication (messenger) application, a text message application, and a payment application. For example, the processor may check when purchasing or searching for a Samsung Electronics Galaxy S10 smartphone manufactured on May 3, 2019, through a payment application.

[0177] According to various embodiments, the processor may obtain image data and / or product-related information of related products from purchase information or search information and update the learning data for identifying objects. To this end, the processor may obtain high-resolution images, such as images for product sales, from, for example, Samsung Electronics' product promotion website, an online sales website for purchasing or searching for Galaxy S10 products, a website for general product sales (such as Amazon), or a general search website (such as Google). The processor may update the acquired images by adding the acquired images to the learning data and learning the classification model.

[0178] In addition, the processor can collect product-related information related to the product information based on the product information according to the purchase history. For example, the product-related information may include price information of various websites selling the product, information about series products of the product, or accessory-related information about the product.

[0179] According to various embodiments, the processor can provide product or object-based services to users based on product-related information. When an event such as a series of product releases, a discount event, or an object's related attachment release occurs, the processor can provide guidance information to the user.

[0180] refer to Figure 7C , for example, the processor may acquire an image including the object 715 as the object information from the electronic device 101 or the camera module 180 of the external device.

[0181] According to various embodiments, the object image may include a product image of a specific electronic device (eg, an air purifier).

[0182] According to various embodiments, the processor may extract an object or an object region from the object image, and extract feature point information of the object from the object region.

[0183] According to various embodiments, the processor may extract information about the location of the object from an object image. For example, when a portion of another object (e.g., a wall-mounted television (TV)) is included in the background of the object image, information that can be used to infer the location of the object (e.g., a living room) may be extracted from the other object. In this case, the processor may select the "residence" domain as the domain based on the location of the object.

[0184] According to various embodiments, the processor may acquire and preselect the location where the electronic device 101 is located using various applications for identifying a location based on various identifiers used for the above-mentioned location identification.

[0185] According to various embodiments, the processor may load an object list of the domain based on the selected domain "residence" and identify the object according to the classification model using the object image.

[0186] According to various embodiments, when an object is first identified in the domain, when there is a user request for any related service related to the object, or when a purchase history or search history related to the object appears, the processor may check the domain object list and collect object-related information to update the domain object list.

[0187] According to various embodiments, for example, the processor may check the purchase history or search history and obtain object-related information such as the type of product purchased, the product name, the manufacturer of the product, the purchase or search date, and the purchase price through various application information of the electronic device 101, such as a web browser, a communication (messenger) application, a text message application, and a payment application. For example, the processor may check when purchasing or searching for the Blue Sky air purifier manufactured by Samsung Electronics on May 8, 2019.

[0188] According to various embodiments, the processor can obtain image data and / or product-related information of related products from purchase information or search information, and update the learning data for identifying objects. To this end, the processor can obtain high-resolution images, such as images for product sales, through, for example, Samsung Electronics' product promotion website, an online sales website for purchasing or searching for Clevo air purifier products, a website for general product sales (such as Amazon), or a general search website (such as Google). The processor can update the acquired images by adding the acquired images to the learning data and learning the classification model.

[0189] In addition, the processor may collect product-related information related to the product information based on the product information according to the purchase history or search history. For example, the product-related information may include various information such as price information of various websites selling the product, or replacement cycles, or price information about replacement parts (such as filters for the product).

[0190] According to various embodiments, the processor can provide the user with a product or object-based service based on the product-related information. The processor can provide the user with various management information, such as the object's A / S expiration date, filter replacement time, filter purchase price, and / or sales website information.

[0191] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device comprising: monitor; Communication circuits; a memory storing an object list and object image learning data used to recognize objects in an image; and at least one processor, Wherein, the at least one processor is configured to: Acquire an image that includes a specific object, acquiring location information of a specific object based on at least one of location information of the electronic device or information extracted from the image, Recognize a specific object from a partial list and primary learned object image learning data, wherein the partial list is selected from the object list by location information, In response to a failure to identify the particular object, updating the object list by adding at least one object to the object list, wherein the at least one object is identified based on the location information and user purchase history information, wherein the user purchase history information includes product-related information about the product according to at least one purchase history of at least one user, collecting image data related to the at least one object based on the updated object list, Based on the collected image data, update the object image learning data, Based on the updated object list and updated object image learning data, the second level learns the object classification model, Recognize specific objects through a secondary learned object classification model, and Information regarding the identified specific object is displayed via the display.

2. The electronic device according to claim 1, wherein The at least one processor is further configured to obtain user purchase history information from information generated by purchasing or searching for a specific product.

3. The electronic device according to claim 1, wherein The at least one processor is further configured to obtain user purchase history information from an application used to purchase or search for a specific product.

4. The electronic device according to claim 1, in, The at least one processor is further configured to obtain image data from a website related to purchasing or searching for a specific product through the communication circuit based on user purchase history information.

5. The electronic device according to claim 4, wherein The at least one processor is further configured to acquire, as the image data, similar image data of a product similar to the specific product through the communication circuit based on information generated by searching for or purchasing the specific product. The electronic device according to claim 4 , wherein: The at least one processor is further configured to: Acquiring object-related management information based on information generated by searching for or purchasing a specific product, and Object-related management services are provided based on the object-related management information.

7. The electronic device according to claim 1, wherein The at least one processor is further configured to: In response to a failure in recognizing the specific object through the secondary learned object classification model, displaying a screen for requesting information related to the specific object via the display, and Information related to a particular object is obtained from user input to the screen.

8. A method of operating an electronic device, the method comprising: acquiring an image including a specific object; acquiring location information of a specific object based on at least one of location information of the electronic device or information extracted from the image, Recognize specific objects from a list of parts and images of objects learned by the primary learner, where Some lists are selected from the object list through location information. in response to a failure to identify the particular object, updating the object list by adding at least one object to the object list, wherein the at least one object is identified based on the location information and user purchase history information, wherein the user purchase history information includes product-related information about the product according to at least one purchase history of at least one user; collecting image data related to the at least one object based on the updated object list, updating the object image learning data based on the collected image data; Based on the updated object list and the updated object image learning data, the second level learns the object classification model; Identify specific objects through a secondary learned object classification model; and Information regarding the identified specific object is displayed via the display.

9. The method according to claim 8, wherein Obtain user purchase history information from information generated by purchasing or searching for specific products.

10. The method according to claim 8, wherein Get user purchase history information from apps used to purchase or search for specific products.

11. The method according to claim 8, wherein Based on user purchase history information, image data is obtained from websites related to purchasing or searching for specific products.

12. The method according to claim 11, wherein The image data includes similar image data of products similar to the specific product acquired based on information generated by searching for or purchasing the specific product.

13. The method according to claim 11, further comprising: Acquiring object-related management information based on information generated by searching for or purchasing a specific product, and Provide object-related management services based on object-related management information.

14. The method according to claim 8, in, In response to a failure in recognizing the specific object by the secondary learned object classification model, the method further includes: displaying, via the display, a screen for requesting information related to a specific object, and Information related to a particular object is obtained from user input to the screen.

Images