Electronic Device for Object Identification and Control Method Thereof

Abstract

The present disclosure relates to an electronic device. The electronic device includes a sensor, a camera, a memory, a camera, and a processor. The memory stores multiple artificial intelligence models trained to recognize objects and stores information about a map. A first processor provides a second processor with regional information about an area where the electronic device is located, determined from multiple areas included in a map based on sensing data obtained from the sensor. The second processor loads at least one of the multiple artificial intelligence models into a volatile memory based on the regional information, and recognizes an object by inputting an image obtained by the camera into the loaded artificial intelligence model.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for object identification. More particularly, the present disclosure relates to an electronic device configured to perform object identification using an artificial intelligence model corresponding to the location of the electronic device. Background Art

[0002] Recently, the object identification capabilities of electronic devices have been greatly improved as artificial intelligence models have been used in object identification technology.

[0003] However, as more advanced forms of artificial intelligence models are used to more accurately identify more objects, the amount of processing performed increases proportionally, and electronic devices require very large memory capacities or memory speeds.

[0004] Therefore, there are limitations in overcoming the object identification capabilities of user devices by simply improving the functionality of the artificial intelligence model itself, especially in terms of memory, processing power, and communication capabilities. Summary of the Invention

[0005] Technical issues

[0006] The present disclosure provides an electronic device configured to perform object identification using an artificial intelligence model corresponding to a location of the electronic device and an operating method thereof.

[0007] The present disclosure relates to an electronic device that performs efficient object identification by selectively using only an artificial intelligence model that is appropriate for a situation.

[0008] More specifically, the present disclosure provides an electronic device that is capable of identifying various types of objects while being able to identify the objects relatively quickly compared to its limited processing capabilities.

[0009] Technical Solution

[0010] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0011] According to one aspect of the present disclosure, an electronic device includes: a sensor; a camera; a storage for storing multiple artificial intelligence models trained to recognize objects and for storing information about a map; a first processor configured to control the electronic device; and a second processor configured to identify the object, wherein the first processor is configured to determine an area in which the electronic device is located from multiple areas included in the map based on sensing data obtained from the sensor, and provide area information about the determined area to the second processor, and wherein the second processor includes a volatile memory and is configured to: load an artificial intelligence model from the multiple artificial intelligence models stored in the storage into the volatile memory based on the area information provided by the first processor, and input an image obtained by the camera into the loaded artificial intelligence model to recognize the object.

[0012] Each of the plurality of artificial intelligence models may include a convolutional layer and a fully-connected layer trained to recognize an object based on feature information extracted from the convolutional layer. The convolutional layer may be a common layer in the plurality of artificial intelligence models, and the second processor may be further configured to load the convolutional layer and the fully-connected layer of the artificial intelligence model associated with the determined area into a volatile memory.

[0013] Here, the multiple artificial intelligence models may include a first model corresponding to a first area of the multiple areas and a second model corresponding to a second area of the multiple areas. In addition, the second processor may be further configured to load the fully connected layer and the convolutional layer corresponding to the first model into the volatile memory based on the electronic device being located in the first area, and load the fully connected layer and the convolutional layer corresponding to the second model into the volatile memory based on the electronic device being located in the second area.

[0014] The information about the map may include information about structures of the plurality of regions. Furthermore, the first processor may be further configured to compare the information about the structure with sensing data obtained from the sensor to determine the region where the electronic device is located from the plurality of regions.

[0015] The first processor may also be configured to obtain information about the structure of the place where the electronic device is located based on the sensing data obtained from the sensor. In this case, the first processor may divide the place into a plurality of areas based on the obtained information about the structure, and generate information about a map including information about the structure of each of the divided plurality of areas and store the generated information about the map in a memory.

[0016] The second processor may also be configured to, when the electronic device is located in a specific area among the multiple areas, input the image obtained by the camera into at least one of the multiple stored artificial intelligence models to identify objects present in the specific area. Furthermore, the second processor may provide information about the identified objects to the first processor. In this case, the first processor may obtain an artificial intelligence model corresponding to the specific area based on the provided information about the identified objects.

[0017] Here, each of the stored multiple artificial intelligence models may include a convolutional layer and a fully connected layer trained to identify multiple objects based on feature information extracted from the convolutional layer. Furthermore, the first processor may obtain a first model including a convolutional layer and a first part of a fully connected layer trained to identify the first object based on the first object among the multiple objects being identified as existing in the first area based on information about objects existing in the first area. Furthermore, the first processor may obtain a second model including a convolutional layer and a second part of a fully connected layer trained to identify the second object based on the second object among the multiple objects being identified as existing in the second area based on information about objects existing in the second area.

[0018] When the electronic device is located in an area among the plurality of areas, the second processor may input an image obtained by the camera into at least one of the plurality of artificial intelligence models loaded into the volatile memory to recognize an object present in the area, and provide information about the recognized object to the first processor. Furthermore, the first processor may update the artificial intelligence model corresponding to the area based on the information about the recognized object provided by the second processor.

[0019] In this case, the memory may store information about an object existing at each of the plurality of areas.

[0020] Here, the first processor may determine at least one object present in the region from among the plurality of regions based on information about objects present in each of the plurality of regions stored in the memory. Furthermore, the first processor may determine an object that is not recognized in the region from among the determined at least one object based on information about the recognized objects provided by the second processor, and update the artificial intelligence model corresponding to the region by removing a portion trained to recognize the determined unrecognized object from the artificial intelligence model corresponding to the region.

[0021] On the other hand, the first processor may determine at least one object present in the region based on information about objects present in each of the plurality of regions stored in the memory. In this case, the first processor may determine whether the recognized object is not included in the at least one determined object present in the region based on the information about the recognized object provided from the second processor, and based on the recognition that the recognized object is not included in the at least one determined object present in the region, add a training portion for recognizing the recognized object to the artificial intelligence model corresponding to the region to update the artificial intelligence model corresponding to the region.

[0022] According to another aspect of the present disclosure, an electronic device includes: a camera; a sensor; a storage for storing multiple artificial intelligence models trained to recognize objects and for storing information about a map; and a processor configured to control the electronic device, wherein the processor includes a volatile memory, and wherein the processor is configured to: determine an area where the electronic device is located from multiple areas included in a map based on sensing data obtained from the sensor, load an artificial intelligence model from multiple artificial intelligence models stored in the storage based on the determined area, and input an image obtained by the camera to the loaded artificial intelligence model to recognize the object.

[0023] According to another aspect of the present disclosure, a control method of an electronic device using an object identification model includes: identifying multiple areas included in a map based on information about the map stored in a memory of the electronic device; determining an area in which the electronic device is located from multiple areas based on sensing data obtained from a sensor; loading an artificial intelligence model from multiple artificial intelligence models stored in the memory into a volatile memory based on the determined area; and identifying an object by inputting an image obtained through a camera into the loaded artificial intelligence model.

[0024] Each of the multiple artificial intelligence models may include a convolutional layer and a fully connected layer trained to recognize objects based on feature information extracted from the convolutional layer. Here, the convolutional layer may be a common layer in the multiple artificial intelligence models, and the fully connected layer may be a layer provided separately to each of the multiple artificial intelligence models. In such a case, loading into the volatile memory includes: based on the electronic device being located in a first area of the multiple areas, loading the fully connected layer and the convolutional layer corresponding to the first model of the multiple artificial intelligence models into the volatile memory, and based on the electronic device being located in a second area of the multiple areas, loading the fully connected layer and the convolutional layer corresponding to the second model of the multiple artificial intelligence models into the volatile memory.

[0025] The information about the map includes information about the structure of each of the plurality of regions. Here, determining the region where the electronic device is located may include comparing the information about the structure with sensing data obtained from the sensor to determine the region where the electronic device is located from the plurality of regions.

[0026] The control method may also include: obtaining information about the structure of a place where the electronic device is located based on sensing data obtained from a sensor; dividing the place into multiple areas based on the obtained information about the structure; and generating information about a map and storing the generated information about the map in a storage, wherein the information about the map includes information about the structure of each of the multiple divided areas.

[0027] Here, the control method may further include: when the electronic device is located in a specific area among multiple areas, inputting an image obtained by a camera into at least one of the stored artificial intelligence models to identify an object present in the specific area, and obtaining an artificial intelligence model corresponding to the specific area based on information about the identified object.

[0028] In such a case, each of the stored multiple artificial intelligence models may include a convolutional layer and a fully connected layer trained to identify multiple objects based on feature information extracted from the convolutional layer. In addition, obtaining the artificial intelligence model may include, based on a first object among the multiple objects being identified as existing in the first area based on information about objects existing in the first area, obtaining a first model including a convolutional layer and a first part of the fully connected layer trained to identify the first object of the fully connected layer, and based on a second object among the multiple objects being identified as existing in the second area based on information about objects existing in the second area, obtaining a second model including a convolutional layer and a second part of the fully connected layer trained to identify the second object.

[0029] The control method may further include: when the electronic device is located in an area among multiple areas, inputting an image obtained by the camera into at least one of the multiple artificial intelligence models loaded into a volatile memory to identify objects existing at the area, and updating the artificial intelligence model corresponding to the area based on information about the identified objects.

[0030] Also, the storage may store information about an object existing at each of the plurality of areas.

[0031] In such a case, updating the artificial intelligence model may include, based on information stored in a storage about objects present at each of the multiple regions, determining at least one object present in the region from the multiple regions, based on information about the identified objects, determining an object that is not identified in the region from the determined at least one object, and removing a portion trained to identify the determined unidentified object from the artificial intelligence model corresponding to the region.

[0032] On the other hand, updating the artificial intelligence model may include determining at least one object present in the area based on information stored in a storage about objects present in each of a plurality of areas, determining whether the identified object is not included in the at least one determined object present in the area based on information about the identified object, and adding a portion trained to recognize the identified object to the artificial intelligence model corresponding to the area based on the identified object being determined to be not included in the at least one determined object present in the area.

[0033] According to another aspect of the present disclosure, an electronic device includes: a storage; and at least one processor, including a volatile memory, and is configured to: determine, based on the determined area, an area where the electronic device is located from multiple areas included in a map, load an artificial intelligence model corresponding to the determined area from multiple artificial intelligence models stored in the storage, and input an image obtained by a camera into the loaded artificial intelligence model to recognize an object.

[0034] Beneficial effects

[0035] The electronic device and the control method of the electronic device described in this article can not only use all artificial intelligence models to identify various types of objects, but also have the effect of quickly and accurately identifying objects.

[0036] Specifically, because the electronic device and control method according to one or more embodiments selectively load only the artificial intelligence model suitable for the area where the electronic device is located for object identification, the electronic device and control method according to one embodiment have the advantage that many types of objects can be identified in a very short period of time by processing only a relatively small amount.

[0037] An advantage of the electronic device and control method according to one or more embodiments is that the artificial intelligence model of each area can be updated according to the situation to quickly maintain or further improve the object identification rate.

[0038] The electronic device according to one or more embodiments is not only capable of performing a wide range of types of object identification using only a self-stored artificial intelligence model without considering communication with a server, but also has the advantage that relatively fast object identification is possible despite limited processing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0040] Figure 1 is a diagram illustrating an example of an electronic device that recognizes an object using a different artificial intelligence model for each area (e.g., a living room and a kitchen) according to an embodiment;

[0041] Figure 2 is a block diagram showing a configuration of an electronic device according to an embodiment;

[0042] Figure 3a is a diagram illustrating an embodiment of an electronic device that determines an area in which the electronic device is located from a plurality of areas;

[0043] Figure 3b is a diagram illustrating an embodiment of an electronic device that determines an area in which the electronic device is located from a plurality of areas;

[0044] Figure 4 is a diagram showing a table of examples of a plurality of artificial intelligence models stored in a memory of an electronic device;

[0045] Figure 5 is a diagram illustrating a specific example of selectively loading an artificial intelligence model corresponding to a determined region from a plurality of artificial intelligence models composed of convolutional layers and fully connected layers;

[0046] Figure 6a is a diagram illustrating an example of an electronic device that generates a first-dimensional map using a LiDAR sensor;

[0047] Figure 6b is a diagram illustrating an example of an electronic device that divides a first-dimensional map into a plurality of areas using a result of recognizing walls, doors, etc.;

[0048] Figure 6c is a diagram illustrating an example of identifying an object in each divided area to identify an electronic device used in each of a plurality of identified areas;

[0049] Figure 6d is a diagram showing an example of an electronic device that identifies a target for each of a plurality of divided areas through communication with an external device;

[0050] Figure 7ais a diagram illustrating an example of an electronic device that uses an object located in each of a plurality of areas to obtain an artificial intelligence model corresponding to each of the plurality of areas;

[0051] Figure 7b is a diagram illustrating an example of an electronic device that uses an object located in each of a plurality of areas to obtain an artificial intelligence model corresponding to each of the plurality of areas;

[0052] Figure 7c is a diagram illustrating an example of an electronic device that uses an object located in each of a plurality of areas to obtain an artificial intelligence model corresponding to each of the plurality of areas;

[0053] Figure 8a is a diagram illustrating an example in which, when an object located in one of a plurality of areas is recognized as no longer located in the related area, an electronic device updates an artificial intelligence model corresponding to the related area;

[0054] Figure 8b is a diagram illustrating an example in which, when an object located in one of a plurality of areas is recognized as no longer located in the related area, an electronic device updates an artificial intelligence model corresponding to the related area;

[0055] Figure 8c is a diagram illustrating an example in which, when an object located in one of a plurality of areas is recognized as no longer located in the related area, an electronic device updates an artificial intelligence model corresponding to the related area;

[0056] Figure 9a is a diagram illustrating an example in which, when a new object is recognized as being added to one of a plurality of areas, an electronic device updates an artificial intelligence model corresponding to the relevant area;

[0057] Figure 9b is a diagram illustrating an example in which, when a new object is recognized as being added to one of a plurality of areas, an electronic device updates an artificial intelligence model corresponding to the relevant area;

[0058] Figure 9c is a diagram illustrating an example in which, when a new object is recognized as being added to one of a plurality of areas, an electronic device updates an artificial intelligence model corresponding to the relevant area;

[0059] Figure 10 is a block diagram illustrating a configuration of an electronic device according to various embodiments;

[0060] Figure 11 are diagrams illustrating various embodiments of an electronic device that performs object identification based on communication with external devices including a server device and an external terminal device;

[0061] Figure 12a is a block diagram showing a configuration of an electronic device including a processor;

[0062] Figure 12b is a block diagram showing a configuration of an electronic device including a processor;

[0063] Figure 13 is a flowchart illustrating a method for controlling an electronic device according to an embodiment;

[0064] Figure 14 is a flowchart illustrating an embodiment of a method for controlling an electronic device that generates information about a map and identifies an object existing in each of a plurality of areas to obtain an artificial intelligence model corresponding to each of the plurality of areas according to an embodiment; and

[0065] Figure 15 is a diagram illustrating an algorithm of an example of a method of controlling an electronic device of updating an artificial intelligence model corresponding to each of a plurality of areas according to a result of recognizing an object existing in each of the plurality of areas according to an embodiment. DETAILED DESCRIPTION

[0066] Before describing the following embodiments, the description method of the present disclosure and the drawings is described.

[0067] First, the terms used in this disclosure are general terms identified in consideration of the functions of the various embodiments of the disclosure. However, these terms may vary depending on the intentions of those skilled in the relevant art, legal or technical interpretations, the emergence of new technologies, etc. In addition, the applicant may select some arbitrary terms. These terms may be interpreted based on the meanings defined herein, and in the absence of a specific definition of the term, may be interpreted based on the overall content of this disclosure and the technical common sense of those skilled in the relevant art.

[0068] In addition, the same reference numerals or characters disclosed in each of the accompanying figures herein indicate components or elements that perform substantially the same function. For ease of description and understanding, the same reference numerals or characters may be used to describe different embodiments. That is, even if elements with the same reference numerals are shown in all multiple figures, the multiple figures do not necessarily refer to only one embodiment.

[0069] In addition, terms including ordinal numbers such as "first" and "second" may be used to distinguish elements in the present disclosure. These ordinal numbers are used only to distinguish identical or similar elements and should not be construed as limiting the meaning of the terms due to their use. For example, elements associated with ordinal numbers should not be limited in the order or sequence in which the numbers are used. Each ordinal number can be used interchangeably if necessary.

[0070] Singular expressions in the present disclosure include plural expressions unless the context clearly indicates otherwise. It should be understood that terms such as "comprising" or "consisting of..." are used herein to indicate the presence of a feature, number, step, operation, element, component, or a combination thereof, and do not exclude the presence or possibility of adding one or more other features, numbers, steps, operations, elements, components, or a combination thereof.

[0071] In the specification, terms such as "module," "unit," and "part" may be used to refer to elements that perform at least one function or operation, and these elements may be implemented as hardware or software, or a combination of hardware and software. In addition, except when each of multiple "modules," "units," "parts," etc. needs to be implemented in separate hardware, the components may be integrated into at least one module or chip and may be implemented in at least one processor.

[0072] In addition, in the embodiments, when any part is indicated as being connected to other parts, this includes not only direct connection but also indirect connection through other media. In addition, when a specific part includes a specific element, it may further include another element rather than excluding the other element unless otherwise specified.

[0073] Figure 1 is a diagram illustrating an example of an electronic device that recognizes an object using a different artificial intelligence model for each area (eg, a living room and a kitchen) according to an embodiment.

[0074] Figure 1 An object identification process of the electronic device 10 implemented as a robot cleaner according to an embodiment is shown. Figure 1 It is shown that multiple artificial intelligence models for identifying various objects such as a television (TV), a sofa, a bed, a closet, clothes, foreign objects, a chair, a sink, and an air conditioner are stored in the non-volatile memory or storage of the electronic device 10.

[0075] refer to Figure 1 , the electronic device 10 as a robot cleaner can be used to capture images of various objects. Figure 1 The camera moves around areas 1-1, 1-2, and 1-3 on the image, and the captured images can be input into multiple artificial intelligence models to identify objects within the image.

[0076] In such a case, the capacity or processing rate of the volatile memory of the electronic device 10 may be insufficient for the electronic device 10 to load a plurality of artificial intelligence models stored in a non-volatile memory such as a storage and perform object identification.

[0077] Therefore, the electronic device 10 can Figure 1The area on the image determines the area where the electronic device 10 is located, and loads only the artificial intelligence model corresponding to the area where the electronic device 10 is located from the stored multiple artificial intelligence models to the volatile memory to identify the object in the image of the captured object.

[0078] To this end, each of the plurality of artificial intelligence models stored for different regions may be pre-stored in the electronic device 10 .

[0079] For example, reference Figure 1 According to Table (2) in the figure, the living room model 2-1 can identify the air conditioner, TV, etc. which are usually located in the living room. On the other hand, the living room model 2-1 may not be able to identify the refrigerator, bed, etc. which are not usually located in the living room.

[0080] In addition, referring to Table (2), the bedroom model 2-2 may not be able to identify the refrigerator when identifying the air conditioner, TV, bed, etc.

[0081] Therefore, if the electronic device 10 is located in the living room 1-1, the electronic device 10 can selectively load only the living room model 2-1 from the stored multiple artificial intelligence models into the volatile memory of the processor to identify the air conditioner, TV, etc., and use the loaded living room model 2-1 to perform identification on objects in the image taken from the living room 1-1.

[0082] On the other hand, if the electronic device 10 is located in the bedroom 1-2, the electronic device 10 can selectively load only the bedroom model 2-2 from the stored multiple artificial intelligence models into the volatile memory of the processor to identify the air conditioner, TV, bed, etc., and use the loaded bedroom model 2-2 to perform identification on objects in the image taken from the bedroom 1-2.

[0083] The configuration and operation of the electronic devices 10 , 100 according to various embodiments are described in more detail below.

[0084] Figure 2 is a block diagram showing a configuration of an electronic device 100 according to an embodiment.

[0085] refer to Figure 2 , the electronic device 100 may include a sensor 110, a camera 120, a storage 130, a first processor 140-1, and a second processor 140-2. The electronic device 100 may be a mobile robot provided with a mobile mechanism or device, or an auxiliary device capable of connecting and disconnecting a mobile device. The electronic device 100 may be implemented as various types of wearable devices. In addition, the electronic device 100 may be implemented as various terminal devices, such as a smartphone, a tablet personal computer (PC), a notebook PC, etc.

[0086] As a configuration for determining the position of the electronic device 100 , the sensor 110 may be implemented as a light detection and ranging (LiDAR) sensor, an ultrasonic sensor, or the like, but is not limited thereto.

[0087] The camera 120 is a configuration for obtaining or capturing one or more images around the electronic device 100. The camera 120 may be implemented as a red / green / blue (RGB) camera, a three-dimensional (3D) camera, or the like.

[0088] The storage 130 is a configuration for variably storing various information related to the functions of the electronic device 100. The storage 130 may be implemented as a nonvolatile memory such as a hard disk, a solid state drive (SSD), and a flash memory (e.g., a NOR-type flash memory or a NAND-type flash memory).

[0089] The storage 130 may store information 131 related to a map. A map may refer to data indicating the physical topography of a location where the electronic device 100 is operated. Although the map may be stored in the storage 130 in an image form, it should be understood that one or more other embodiments are not limited thereto.

[0090] The information about the map or the map itself may include topographic information of the place where the electronic device 100 is operated. Area information of each of a plurality of areas included in the map, additional information related to the map, etc. may also be included.

[0091] The terrain information may include information about the structure (e.g., shape and / or size) of the location, information about the structure (e.g., shape and / or size) of each of multiple areas included in the space, information about the position within the location of each of the multiple areas, and the like.

[0092] The area information may refer to information used to identify each of the multiple areas. The area information may include or include an identification name, identification number, etc. indicating each of the multiple areas. Furthermore, the area information may include information regarding the usage of each of the multiple areas, and each of the multiple areas according to the area information may be defined as, for example, a living room, bathroom, bedroom, etc.

[0093] The additional information may include information about a place (eg, home, work, gym, etc.), a location, a name, a user, etc., and usage of image data obtained by the camera 120 in each of a plurality of areas.

[0094] The storage 130 may store one or more artificial intelligence models 132. Specifically, according to an embodiment, a plurality of artificial intelligence models 132 trained to recognize an object may be stored in the storage 130. For example, an artificial intelligence model trained to recognize an object included in an input image may be stored in plurality.

[0095] Identifying an object can be understood as obtaining information about the object, such as the name and type of the object. In this case, the information about the object can be information about the identified object output by multiple artificial intelligence models of the identified related objects.

[0096] The first processor 140-1 may be connected to the sensor 110 and the storage 130 to control the electronic device 100. In addition, the first processor 140-1 may be connected to the second processor 140-2 as a main processor to control the second processor 140-2.

[0097] The second processor 140 - 2 may be connected to the camera 120 , the storage 130 , and the first processor 140 - 1 to perform an object identification function.

[0098] refer to Figure 2 , the first processor 140-1 may determine the region where the electronic device 100 is located from a plurality of regions included in the map (operation S110). Specifically, the first processor 140-1 may use information about the map divided by a plurality of regions to identify a plurality of regions included in the map, and use sensing data obtained by the sensor 110 to determine the region where the electronic device 100 is located from the plurality of regions on the map.

[0099] The first processor 140-1 may transmit region information about the determined region to the second processor 140-2 connected to the first processor 140-1 (operation S120). The first processor 140-1 may transmit the region information to the second processor 140-2 in the form of an electrical signal or data.

[0100] Second processor 140-2 may load at least one artificial intelligence model (i.e., Model 1, Model 2) from among the plurality of artificial intelligence models 132 stored in storage 130 to volatile memory 145 based on the region information transmitted from first processor 140-1. Specifically, second processor 140-2 may load an artificial intelligence model corresponding to the determined region from the plurality of artificial intelligence models 132.

[0101] In this case, the second processor 140 - 2 may identify and load an artificial intelligence model mapped to the determined region using logical mapping information between the plurality of artificial intelligence models 132 and the plurality of regions stored in the storage 130 .

[0102] The logical mapping information may be information for mapping one or more artificial intelligence models to each of the plurality of regions. The logical mapping information may include information about parameters for outputting information indicating at least one of the plurality of artificial intelligence models 132 from information indicating each of the plurality of regions. The logical mapping information may include information about the address of the artificial intelligence model corresponding to each region stored in the storage 130.

[0103] The logical mapping information may be preset by a user and / or may be generated and stored by the first processor 140-1, which may be configured to perform the following operations according to the following example: Figures 7a to 7c The embodiment of the invention obtains an artificial intelligence model corresponding to (mapped to) each of the plurality of regions. In addition, based on the following description, for example Figures 8a to 8c and Figures 9a to 9c As a result of updating the artificial intelligence model corresponding to each of the multiple regions of the embodiment, the logical mapping information may also be updated.

[0104] The first processor 140-1 may use the logical mapping information on the determined area to identify the artificial intelligence model corresponding to the determined area. When (or based on) information about the identified artificial intelligence model is transmitted to the second processor 140-2, the second processor 140-2 may load the corresponding artificial intelligence model.

[0105] The second processor 140-2 may input the image obtained by the camera 120 to the loaded artificial intelligence model to recognize the object (operation S140). In this case, the second processor 140-2 may recognize the object included in the image obtained by the camera 120 using the output of the loaded model.

[0106] Therefore, because the second processor 140-2 can load only the artificial intelligence model corresponding to the determined area from the plurality of artificial intelligence models 132 stored in the storage 130 for object identification, relatively accurate and fast object identification can be performed within the processing capability of the second processor 140-2.

[0107] In this regard, if the area where the electronic device 100 is located changes, when loading a different artificial intelligence model corresponding to the changed area to perform object identification, the second processor 140-2 can remove the artificial intelligence model loaded in the volatile memory 145 from the volatile memory 145 before the location change. That is, the second processor 140-2 can load only the artificial intelligence model required for each area where the electronic device 100 is located from at least the plurality of artificial intelligence models 132 and use the artificial intelligence model.

[0108] The first processor 140 - 1 and the second processor 140 - 2 may be implemented as one processor or a plurality of processors to perform operations.

[0109] Specifically, refer to Figure 2 , the processor 141 including the first processor 140-1 and the second processor 140-2 can determine the area where the electronic device 100 is located from multiple areas on the map, load at least one model of the multiple artificial intelligence models 132 based on the area information, and use the loaded model to recognize objects in the image obtained by the camera 120.

[0110] Figure 3a and Figure 3b is a diagram illustrating an example of the electronic device 100 identifying a region in which the electronic device 100 is located from a plurality of regions.

[0111] The first processor 140 - 1 may determine an area in which the electronic device 100 is located using information on a map and sensing data stored in the storage 130 .

[0112] As a specific example, if the sensor 110 is a LiDAR sensor, the first processor 140 - 1 may compare sensing data received from the sensor 110 and information about a map stored in the storage 130 to determine an area where the electronic device 100 is located.

[0113] The sensing data may include information about a structure around the electronic device 100. The information about the surrounding structure may include information about a shape and / or size of a structured object or surroundings.

[0114] In such a case, the first processor 140-1 may compare the information about the structure (i.e., shape and / or size) around the electronic device 100 included in the sensing data with the information about the structure (i.e., shape and / or size) of each of a plurality of areas on the map included in the map information to determine the area in which the electronic device 100 is located from among the plurality of areas on the map.

[0115] refer to Figure 3aThe electronic device 100 implemented as a robot cleaner can use the sensing data received from the sensor 110 implemented as a LiDAR sensor to identify the surrounding structures 301 based on the distance from the surrounding structured objects or the distance from each point (or each of multiple points) in the object. In addition, the electronic device 100 can identify points on the map 300 that coincide with the identified surrounding structures 301 to determine the position of the electronic device 100 on the map 300. The electronic device 100 can use the determined position and the position of each of the multiple areas 300-10, 300-20, 300-30, and 300-40 on the map 300 to determine whether the electronic device 100 is located in the kitchen 300-30 among the multiple areas 300-10, 300-20, 300-30, and 300-40.

[0116] The first processor 140 - 1 may also determine in which region of the plurality of regions the electronic device 100 is located using data on surrounding images obtained through the camera 120 .

[0117] For example, if the information about the map includes data about a three-dimensional (3D) image of each of a plurality of areas, the first processor 140-1 may determine the area in which the electronic device 100 is located using a result of comparing the image of each of the plurality of areas included in the information about the map with a 3D image obtained by the camera 120 implemented as a 3D camera.

[0118] refer to Figure 3b , the electronic device 100 implemented as a robot cleaner may compare the image 302 obtained through the camera 120 with the interior images of the plurality of areas 300-10, 300-20, 300-30, and 300-40 stored in the storage 130 to determine whether the electronic device 100 is located in the bedroom 300-20.

[0119] Optionally, the first processor 140 - 1 may further identify one or more objects in the image obtained by the camera 120 from the area where the electronic device 100 is located to determine the area where the electronic device 100 is located.

[0120] As a specific example, the first processor 140-1 may input an image captured by the camera 120 from the area where the electronic device 100 is located into at least one of the plurality of stored artificial intelligence models 132 to identify objects in the image. If the identified objects are a sofa and a television, the first processor 140-1 may use preset information of one or more objects in each area to identify that the sofa and the television correspond to the "living room."

[0121] In addition, the first processor 140-1 can use the sensor 110 including an inertial sensor, an acceleration sensor, etc. to determine the point where the electronic device 100 is located on the map 300, and determine an area including the determined point from multiple areas on the map as the area where the electronic device 100 is located.

[0122] The process in which the first processor 140 - 1 determines the area where the electronic device 100 is located is not limited to the above-described embodiment and may be performed through various other methods.

[0123] Figure 4 is a diagram showing a table of examples of a plurality of artificial intelligence models stored in the storage 130 of the electronic device 100 .

[0124] refer to Figure 4 In the storage 130, one or more trained artificial intelligence models may be stored, such as an air conditioner model 401 trained to identify an air conditioner and a refrigerator model 402 trained to identify a refrigerator, to respectively identify corresponding objects.

[0125] In the storage 130, one or more artificial intelligence models trained to recognize multiple objects may be stored. Figure 4 , the bedroom model 410 of the plurality of artificial intelligence models stored in the storage 130 may be a model capable of recognizing objects such as an air conditioner, a television, a bed, a chair, a cup, and a glass bottle. Figure 4 , the kitchen model 430 among the plurality of artificial intelligence models stored in the storage 130 may be a model capable of recognizing objects such as an air conditioner, a refrigerator, a chair, a cup, a glass bottle, and a plate.

[0126] When the first processor 140-1 determines that the electronic device 100 is located in the bedroom, the second processor 140-2 may load the bedroom model 410 into the volatile memory 145. On the other hand, if the first processor 140-1 determines that the electronic device 100 is located in the kitchen, the second processor 140-2 may load the kitchen model 430 into the volatile memory 145.

[0127] although Figure 4 Artificial intelligence models 401, 402 trained to recognize objects related to "home" and artificial intelligence models 401, 402, 430, 440... stored corresponding to each of multiple areas in "home" are shown. The artificial intelligence model is used to recognize one or more objects included in various places other than home (i.e., libraries, museums, squares, sports fields, etc.), and multiple artificial intelligence models stored corresponding to each of multiple areas included in places other than home.

[0128] The artificial intelligence model stored in the storage 130 may be composed of or include multiple neural network layers. Each layer may include multiple weighted values, and the calculation of the layer is performed by the calculation result of the previous layer and the calculation of the multiple weighted values. Examples of neural networks may include convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN) and deep Q networks. In addition, unless otherwise stated, the neural networks in the present disclosure are not limited to the above examples.

[0129] Artificial intelligence models may consist of or include ontology-based data structures in which knowledge of various concepts, conditions, relations, or conventions is represented in a computer-processable form.

[0130] The artificial intelligence model stored in the storage 130 can be trained by various learning algorithms through the electronic device 100 or a separate server / system. The learning algorithm can be a method of training a predetermined target device (e.g., a robot) so that the predetermined target device can determine or predict on its own using multiple training data. Examples of learning algorithms may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. It should be understood that unless otherwise stated, the learning algorithms in the present disclosure are not limited to the above examples.

[0131] The form / type of the artificial intelligence model is not limited to the above examples.

[0132] Each of the plurality of artificial intelligence models stored in the storage 130 may include a convolutional layer and a fully connected layer trained to recognize at least one object based on feature information extracted by the convolutional layer.

[0133] In such a case, the convolution layer may be a common layer in the plurality of artificial intelligence models 132 stored in the storage 130 , and the fully connected layer may be a layer provided separately to each of the plurality of artificial intelligence models 132 .

[0134] Each fully connected layer constituting or included in the plurality of artificial intelligence models may be a layer trained to recognize at least one object based on feature information output by a convolutional layer. In this case, each fully connected layer may also output a probability value of at least one object included in an image associated with each object.

[0135] As a specific example of the electronic device 100 that performs object identification using an artificial intelligence model including a convolutional layer and a fully connected layer, the second processor 140-2 may input an image obtained by the camera 120 into the convolutional layer. When feature information output by the convolutional layer is then input to the fully connected layer, the second processor 140-2 may use the data output by the fully connected layer to obtain a probability that a predetermined object is included in the image.

[0136] When comparing the obtained probability with the threshold, the first processor 140 - 1 and the second processor 140 - 2 may recognize that a predetermined object is included in the input image based on the obtained probability being greater than the threshold.

[0137] Figure 5 is a diagram showing a specific example of selectively loading an artificial intelligence model corresponding to an identified area from multiple artificial intelligence models composed of or including convolutional layers and fully connected layers.

[0138] exist Figure 5 In the example, it can be assumed that the data stored in the memory 130 is Figure 4 Multiple artificial intelligence models are shown and described. Figure 4 , the multiple artificial intelligence models stored in the storage 130 may include a bedroom model 410, a living room model 420, a kitchen model 430, and the like.

[0139] refer to Figure 5 , the multiple artificial intelligence models 500 stored in the storage 130 can be composed of or include a convolutional layer 501 and a fully connected layer 502, the convolutional layer 501 extracts feature information when image data is input (or based on image data), and the fully connected layer 502 is trained to recognize one or more objects when the extracted feature information is input.

[0140] refer to Figure 5 , the fully connected layer 502 can be divided into multiple independent fully connected layers 510, 520, 530.... The independent fully connected layers 510, 520, 530... can be layers trained to recognize one or more different objects when inputting (or based on) the extracted feature information.

[0141] Figure 4 The bedroom model 410 may include Figure 5 The convolutional layer 501 and the fully connected layer 510, Figure 4 The living room model 420 can be made by Figure 5 The convolutional layer 501 and the fully connected layer 520 are composed of Figure 4 The kitchen model 430 can be made of Figure 5 The convolutional layer 501 and the fully connected layer 530 are composed of

[0142] That is to say, Figure 4 The multiple artificial intelligence models 410, 420, 430... can use the convolutional layer 501 in common and be distinguished from each other by the fully connected layers 510, 520, 530... that are trained to recognize objects that are different from each other.

[0143] The second processor 140 - 2 may load the convolutional layer and the fully connected layer of the artificial intelligence model corresponding to the determined area to the volatile memory 145 .

[0144] For example, based on determining that the electronic device 100 is located in the “bedroom”, the second processor 140 - 2 may load the convolutional layer 501 and the fully connected layer 510 to the volatile memory 145 to use the bedroom model 410 .

[0145] It can be assumed that the plurality of artificial intelligence models 500 include a first model corresponding to a first region among the plurality of regions and a second model corresponding to a second region among the plurality of regions.

[0146] The second processor 140-2 can load the fully connected layers and convolutional layers corresponding to the first model into the volatile memory 145 based on the electronic device 100 being located in the first area, and load the fully connected layers and convolutional layers corresponding to the second model into the volatile memory 145 based on the electronic device 100 being located in the second area.

[0147] For example, based on determining that the electronic device 100 is located in the "bedroom," the second processor 140-2 may load the convolutional layer 501 and the fully connected layer 510. If the location of the electronic device 100 is determined to have changed to the "living room," the second processor 140-2 may keep the convolutional layer 501 loaded and load a new fully connected layer 520 after removing the fully connected layer 510 from the volatile memory 145.

[0148] The information of the map stored in the storage 130 may be generated / updated by the first processor 140 - 1 .

[0149] Figures 6a to 6d are diagrams illustrating various examples of the electronic device 100 that generates information about a map.

[0150] The first processor 140 - 1 may obtain information about a structure of a place where the electronic device 100 is located based on the sensing data received from the sensor 110 .

[0151] 6 , the first processor 140-1 may control a moving device or mechanism of the electronic device 100 to move the electronic device 100 within a relevant location. When the electronic device 100 is moving, the first processor 140-1 may then use sensing data received by the sensor 110 (which may be a LiDAR sensor) to obtain information about the structure of the relevant location.

[0152] In such a case, the first processor 140-1 can use the sensing data received by the sensor 110 (which can be a LiDAR sensor) to identify the distance from the surrounding structured objects of the electronic device 100 and the distance from the points in the objects, and obtain information about the surrounding structure (i.e., shape and / or size) based on the identified distance.

[0153] In addition, when the electronic device 100 is moving, the first processor 140-1 can obtain data about an image captured by the camera 120, which can be implemented as a 3D camera. The first processor 140-1 can then use the obtained image to obtain information about the structure (i.e., shape and / or size) of the relevant place.

[0154] refer to Figure 6a , the first processor 140 - 1 may use the information about the obtained structure to obtain image data associated with a first dimensional map 600 that appears as if a portion or all of the relevant place is viewed from a specific direction.

[0155] The first processor 140 - 1 may divide the place where the electronic device 100 is located into a plurality of areas based on the information about the obtained structure.

[0156] In this case, the first processor 140-1 may use various algorithms to divide the map into multiple areas. For example, the first processor 140-1 may identify points where a dividing line or protrusion (or threshold) exists on the floor, points where the movable width narrows, points where a wall exists, points where a wall ends, points where a door exists, etc., using sensing data obtained by the sensor 110 (which may be a LiDAR sensor) and / or images obtained by the camera 120. The first processor 140-1 may divide each area on the map by using the identified points as the boundaries between the areas.

[0157] exist Figure 6b , the first processor 140 - 1 can determine the position of the electronic device 100 on the first dimension map 600 through an image (e.g., a 3D image) obtained by sensing data obtained by a LiDAR sensor included in the sensor 110 and / or the camera 120.

[0158] The first processor 140-1 may then use the sensing data obtained by the LiDAR sensor to identify the “point where the wall ends.” Furthermore, the first processor 140-1 may control the second processor 140-2 to input the image 650 (e.g., RGB image) obtained by the camera 120 into at least one of the plurality of artificial intelligence models stored in the storage 130 to identify the “point where the wall ends” 651, the “door” 652, etc. included in the image 650.

[0159] The first processor 140-1 can use the above-mentioned points 651 and 652 and the structure of the wall to divide / define an independent area 600-20 including the point where the electronic device 100 is currently located on the first-dimensional map 600. The first processor 140-1 can then use various other algorithms to respectively divide the remaining areas 600-10, 600-30, and 600-40 on the first-dimensional map 600.

[0160] However, it should be understood that dividing the location where the electronic device 100 is located into multiple areas is not limited to the above embodiment, and various other methods and / or devices can be implemented in one or more other embodiments.

[0161] The first processor 140 - 1 may generate information about a map including information about a structure of each of the divided plurality of areas, and store (or control storage of) the information about the generated map in the storage 130 .

[0162] The first processor 140-1 may generate / store information about a map including an internal image of each divided area and information about features of each divided area. The features of each divided area may involve an object, a size, and the like of each area.

[0163] When the electronic device 100 is located in each of the divided plurality of areas, the first processor 140 - 1 may add an internal image of each of the plurality of areas obtained through the camera 120 to the information on the map.

[0164] For example, the first processor 140 may obtain multi-angle images obtained by the camera 120 each time the electronic device 100 is located at respective points on a plurality of areas, and may store the obtained multi-angle images as information on a map.

[0165] The second processor 140-2 may input an image obtained by the camera 120 to at least one of the plurality of artificial intelligence models stored in the storage 130 when the electronic device 100 is located in each of the divided plurality of areas to identify an object located in each of the plurality of areas. The first processor 140-1 may then obtain information about the object identified as being located in each of the plurality of areas as information about the object existing in each of the plurality of areas, and store the obtained information in the storage 130.

[0166] The information about the object present in each of the plurality of regions may be information related to at least one output of the plurality of artificial intelligence models. That is, the information about the object may include the result of at least one artificial intelligence model among the plurality of artificial intelligence models 132 stored in the storage 130 (i.e., the name, size, type, etc. of the recognized object), recognize the object by receiving input of images obtained from the plurality of regions, and output information about the recognized object. If the object is a person, the information about the object may also include information about the person's identity.

[0167] Since information about objects can be pre-stored for each of the multiple regions, information about objects existing in each of the multiple regions can be stored / matched to match information output by multiple artificial intelligence models, thereby identifying at least one object. Information about objects existing in each of the multiple regions can be categorized and managed by categories such as name and type (i.e., home appliances, furniture, fitness equipment, etc.) to facilitate search / processing by each artificial intelligence model.

[0168] The first processor 140 - 1 may recognize a target for each of the plurality of areas using information about an object existing at each of the plurality of areas.

[0169] refer to Figure 6c When the electronic device 100 is located in the relevant area 600 - 10 , the first processor 140 - 1 may control the second processor 140 - 2 to input one or more images of the area 600 - 10 obtained by the camera 120 into one or more artificial intelligence models stored in the storage 130 .

[0170] Therefore, the first processor 140-1 and the second processor 140-2 can recognize that the TV 661 and the sofa 662 are located in the area 600-10. The first processor 140-1 can then use the object information of each pre-stored area so that one or more objects correspond to each of one or more areas such as "living room", "kitchen" and "bedroom" to identify the area corresponding to the TV 661 and the sofa 662 as the "living room". The first processor 140-1 can then recognize that the object or identity of the area 600-10 is the "living room".

[0171] The first processor 140-1 may then use the objects identified in each of the remaining areas 600-20, 600-30, and 600-40 to identify the targets or identities of the remaining areas 600-20, 600-30, and 600-40 as "bedroom," "kitchen," and "bathroom," respectively.

[0172] The first processor 140 - 1 may divide each region or obtain information about each region according to a user input received from the electronic device 100 .

[0173] For example, when an image corresponding to a first dimensional map generated by sensing data of the sensor 110 serving as a LiDAR sensor is displayed on the display of the electronic device 100, the first processor 140-1 may divide the image corresponding to the first dimensional map into a plurality of regions based on a user's touch input for at least some regions included in the image of the displayed first dimensional image.

[0174] In addition, when an image corresponding to a first dimensional map divided into multiple areas is displayed on the display of the electronic device 100, the first processor 140-1 can identify the target or identity of at least one area among the multiple areas divided based on the touch input of the user selecting at least one of the multiple areas divided and the touch input of the user selecting / inputting the target of the selected area.

[0175] The user input for dividing each area or defining information about each area (e.g., a target) can be received not only directly in the electronic device 100 but also indirectly through an external device such as a smartphone and a PC. In this case, the information about the user input received through the external device can be received by the electronic device 100 from the external device.

[0176] For example, Figure 6d 2 is a diagram illustrating that the electronic device 100 recognizes targets of a plurality of areas on a first-dimensional map according to a user input received through the external device 200 .

[0177] refer to Figure 6d, the electronic device 100 as a robot cleaner can perform the following Figure 6a and Figure 6b After the processing in , information about the first dimension map 600 divided into a plurality of areas is transmitted to the external device 200.

[0178] refer to Figure 6d , the external device 200 can display the first dimension map 600 received in the first area 201 on the screen.

[0179] like Figure 6d As shown, when (or based on) a user 670 touches some areas 671 of the area 600-10 among the plurality of areas 600-10, 600-20, 600-30, and 600-40 divided on the first dimension map 600, the external device 200 may recognize that the area 600-10 has been selected. In this case, the external device 200 may visually indicate that the relevant area 600-10 has been selected by adjusting the color of the area included in the selected area 600-10.

[0180] Then, the external device 200 may display a graphical user interface (GUI) for receiving an input of a target of the selected area 600 - 10 of the second area 202 on the screen.

[0181] refer to Figure 6d , the GUI on the second area 202 may include a plurality of menu items 681 that the user can select for the target of the area 600-10. In addition, the GUI on the second area 202 may further include an item 682 for the user to directly input the target of the area 600-10. When the user touches the relevant item 682, a keyboard for the user to input text may be displayed on the screen of the external device 200.

[0182] The external device 200 may then transmit information about the selected area 600 - 10 and information about a target selected / input by the GUI to the electronic device 100 .

[0183] The electronic device 100 may then identify an object of the selected area 600 - 10 among a plurality of areas divided on the first dimension map 600 through the received information.

[0184] pass Figure 6d The configuration of the screen shown and described, the form of receiving user input (ie, touch), etc. are merely examples, and in addition to these, various technical methods generally known may also be applicable. Figure 6dIt is shown that the user input is received when the first dimension map 600 has been divided into a plurality of regions, but in various embodiments, it is also possible to receive the user input for dividing the first dimension map 600 into a plurality of regions through an external device.

[0185] Electronic devices can be newly defined / obtained and Figure 6b and / or Figure 6c To this end, the electronic device can use an artificial intelligence model that is trained to recognize multiple objects pre-stored in a memory.

[0186] Specifically, when an artificial intelligence model trained to recognize multiple objects is stored in the storage 130, the second processor 140-2 can input an image obtained by the camera 120 to the artificial intelligence model when the electronic device 100 is located in each of the multiple areas to recognize the objects present in each of the multiple areas. Figure 6c The process is performed separately from the process of identifying the object at the location of the electronic device, but the process can also be performed in conjunction with Figure 6c The process is executed together.

[0187] In addition, the second processor 140-2 may transmit (or provide) information about the recognized object to the first processor 140-1. In this case, the first processor 140-1 may obtain an artificial intelligence model corresponding to each of the plurality of regions from the stored artificial intelligence models based on the information about the recognized object transmitted from the second processor 140-2.

[0188] Figures 7a to 7c is a diagram illustrating an example of the electronic device 100 that obtains an artificial intelligence model corresponding to each of a plurality of areas using an object recognized in each of the plurality of areas.

[0189] Figure 7a is a diagram briefly showing information about objects recognized at each of a plurality of areas stored in the storage 130 as a table of information about objects existing at each of the plurality of areas.

[0190] refer to Figure 7a , an air conditioner 11 , a television 13 , a bed 21 , a chair 23 , etc. exist in a bedroom 51 , and an air conditioner 11 , a television 13 , a sofa 22 , etc. exist in a living room 52 .

[0191] Figure 7b is a diagram illustrating a data structure of the artificial intelligence model 700 stored in the storage 130 before obtaining a plurality of artificial intelligence models corresponding to each of a plurality of regions.

[0192] refer to Figure 7b , the artificial intelligence model 700 can be composed of a fully connected layer 702, which is used to identify multiple objects using the convolutional layer 701 and feature information extracted from the convolutional layer 701.

[0193] Figure 7b Node 711 outputs the probability that an air conditioner is included in the input image, node 712 outputs the probability that a refrigerator is included in the input image, node 713 outputs the probability that a television is included in the input image, node 721 outputs the probability that a bed is included in the input image, node 722 outputs the probability that a sofa is included in the input image, and node 723 outputs the probability that a chair is included in the input image. Nodes 731, 732, and 733 are nodes associated with a cup, a glass bottle, and a plate, respectively.

[0194] The first processor 140 - 1 may recognize that the first object exists at the first region based on information about the object existing at the first region among information of objects existing at each of the plurality of regions stored in the storage 130 .

[0195] In this case, the first processor 140 - 1 may identify a portion of the fully connected layer, which is trained to recognize the first object, among the fully connected layers of the artificial intelligence model stored in the storage 130 .

[0196] The first processor 140-1 may then obtain (define) a first model including a convolutional layer of the artificial intelligence model stored in the storage 130 and a portion of the identified fully connected layer, and store the first model in the storage 130. In this case, the first processor 140-1 may generate logical mapping information connecting (matching) the first model with the first region, and store the generated information in the storage 130.

[0197] The first processor 140 - 1 may recognize that the second object exists at the second region based on information about the object existing at the second region among information of objects existing at each of the plurality of regions stored in the storage 130 .

[0198] In this case, the first processor 140 - 1 may identify different parts of the fully connected layer of the artificial intelligence model stored in the storage 130 that are trained to recognize the second object.

[0199] The first processor 140-1 may then obtain (define) a second model including the convolutional layer of the artificial intelligence model stored in the storage 130 and different parts of the identified fully connected layer, and store the second model in the storage 130. In this case, the first processor 140-1 may generate logical mapping information connecting (matching) the second model with the second region, and store the generated information in the storage 130.

[0200] For example, the first processor 140-1 may use Figure 7a The information in is used to identify that the air conditioner 11, TV 13, bed 21, chair 23, etc. exist in the "bedroom" 51.

[0201] The first processor 140-1 may define a new fully connected layer 751', including nodes 711, 713, 721, and 723 related to the air conditioner 11, the television 13, the bed 21, and the chair 23, as well as a portion used in the inference process, to obtain the Figure 7b The fully connected layer 702 shown generates outputs of corresponding nodes 711 , 713 , 721 and 723 .

[0202] Although the fully connected layer 751' may be related to the inference process for generating the output of the node 712 associated with the refrigerator of the fully connected layer 702, the fully connected layer 751' may not include a part that is not related to the inference process for generating the output of the nodes 711 and 713 associated with the air conditioner 11 and the television 13.

[0203] In addition, reference Figure 7c , the first processor 140 - 1 may obtain a new bedroom model 751 including a convolutional layer 701 and a fully connected layer 751 ′.

[0204] For example, the first processor 140-1 may use Figure 7a The information is used to identify that the air conditioner 11, the TV 13, the sofa 22, etc. exist in the "living room" 52.

[0205] The first processor 140-1 may define / obtain a new fully connected layer 752', which includes nodes 711, 713, and 722 related to the air conditioner 11, the television 13, and the sofa 22, as well as a portion used in the inference process to obtain the new fully connected layer 752'. Figure 7b The fully connected layer 702 shown generates outputs of corresponding nodes 711 , 713 and 722 .

[0206] Although the fully connected layer 752' may be related to the inference process for generating the output of the node 721 associated with the bed 21 of the fully connected layer 702, the fully connected layer 752' may not include a portion that is not related to the inference process for generating the output of the node 722 associated with the sofa 22.

[0207] refer to Figure 7c , the first processor 140 - 1 can obtain a living room model 752 including a convolutional layer 701 and a fully connected layer 752 ′.

[0208] The first processor 140 - 1 may then store the obtained artificial intelligence model among the artificial intelligence models corresponding to each region in the storage 130 .

[0209] The first processor 140-1 may store, in the storage 130, logical mapping information of the mapping between the fully connected layer 751′ included in the bedroom model 751 and the bedroom 51 in the fully connected layer 702. Furthermore, the first processor 140-1 may store, in the storage 130, logical mapping information of the mapping between the fully connected layer 752′ included in the living room model 752 and the living room 52 in the fully connected layer 702.

[0210] For example, if the electronic device 100 is determined to be located in the bedroom 51, the second processor 140-2 may load only the bedroom model 751 of the artificial intelligence model 700 stored in the storage 130 into the volatile memory 145 under the control of the first processor 140-1. Specifically, the second processor 140-2 may load the fully connected layer 751', which is mapped to the convolutional layer 701 together with the logical mapping information related to the bedroom 51.

[0211] Alternatively, for example, if the electronic device 100 is determined to be located in the living room 52, the second processor 140-2 may load only the living room model 752 of the artificial intelligence model 700 stored in the storage 130 into the volatile memory 145 under the control of the first processor 140-1. Specifically, the second processor 140-2 may load the fully connected layer 752′, which is mapped to the convolutional layer 701 together with the logical mapping information related to the living room 52.

[0212] In addition to identifying an object existing at a location where the electronic device 100 is located when generating information about a map, Figure 6c In addition to the process of scanning, the first processor 140-1 may perform scanning to identify an object existing in each of the plurality of areas. In this case, the first processor 140-1 may control a moving device or mechanism (e.g., wheels) of the electronic device 100 to move around the plurality of areas, and then control the second processor 140-2 to identify an object existing in each of the plurality of areas from an image obtained at each of the plurality of areas by the camera 120.

[0213] The first processor 140 - 1 may perform the scanning according to a received user command, or may perform the above-mentioned scanning according to a preset interval regardless of the user command.

[0214] In addition, the first processor 140-1 may perform the above-mentioned scanning only when there is no user in a place including multiple areas. In such a case, the first processor 140-1 can recognize that there is no user in the corresponding place through the received user input. In addition, the first processor 140-1 can control the second processor 140-2 to identify whether a user is present by images obtained in multiple areas by the camera 120, and recognize that there is no user in the corresponding place based on the output of the artificial intelligence model loaded by the second processor 140-2. In such a case, the artificial intelligence model can be an artificial intelligence model trained to recognize whether a user is included in the input image.

[0215] The electronic device 100 may update the artificial intelligence model corresponding to each of the plurality of areas according to the scanning results on the plurality of areas.

[0216] Specifically, the second processor 140-2 may input an image obtained by the camera 120 to at least one of the plurality of artificial intelligence models loaded in the volatile memory 145 when the electronic device 100 is located in one of the plurality of areas, so as to recognize an object existing in the corresponding area, and may transmit information about the recognized object to the first processor. In such a case, the plurality of artificial intelligence models may not be loaded into the volatile memory 145 at the same time, but may be loaded one or two models sequentially at a time.

[0217] The first processor 140 - 1 may then update the artificial intelligence model corresponding to the relevant area based on the information of the recognized object transmitted from the second processor 140 - 2 .

[0218] For example, when information about objects existing at each of multiple areas is stored, the first processor 140-1 can determine at least one object existing at one of the multiple areas based on the information about objects existing at each of the multiple areas stored in the storage 130, and determine unidentified objects in the corresponding area from the determined objects based on information about identified objects from the corresponding area transmitted from the second processor 140-2.

[0219] The first processor 140 - 1 may then remove a portion trained to recognize an unrecognized object from the artificial intelligence model corresponding to the associated region of the plurality of artificial intelligence models to update the artificial intelligence model corresponding to the associated region.

[0220] Figures 8a to 8c is a diagram illustrating an example in which, when an object existing in one of a plurality of areas is recognized as no longer existing in the relevant area, the electronic device 100 updates an artificial intelligence model corresponding to the relevant area.

[0221] Figures 8a to 8c An example of information about an object existing at each of a plurality of areas is shown. Figure 7a The information about the object existing at each of the plurality of areas stored in the storage 130 may be preset information, or may be information about the object existing at each of the plurality of areas stored in the storage 130. Figure 6c The process of identifying the information of the object in each of the plurality of regions. Figures 8a to 8c Shown with Figure 7c An example of multiple artificial intelligence models corresponding to each of the multiple areas constructed and stored in the storage 130 as described above.

[0222] refer to Figure 8a , the first processor 140-1 of the electronic device 100 implemented as a robot cleaner can control the moving device or mechanism (device) of the electronic device 100 so that the electronic device 100 can move around multiple areas 800-10, 800-20, 800-30 and 800-40 on the location indicated by the map 800.

[0223] In addition, when the electronic device 100 is located in each of the multiple areas 800-10, 800-20, 800-30 and 800-40, the second processor 140-2 can load at least one artificial intelligence model from the multiple artificial intelligence models stored in the storage 130, and input the image obtained through the camera 120 to the loaded artificial intelligence model to recognize the object located in each of the multiple areas.

[0224] To this end, the first processor 140 - 1 may control the movement of the electronic device 100 to pass through all of the plurality of areas 800 - 10 , 800 - 20 , 800 - 30 , and 800 - 40 at least once or a plurality of times.

[0225] 8, the electronic device 100 can identify the "air conditioner" 11 and the "television" 13 in the "living room" 800-10. Figure 8a , because the “sofa” 22 that previously existed in the “living room” 800 - 10 no longer exists in the “living room” 800 - 10 , the electronic device 100 may no longer be able to recognize the “sofa” 22 in the “living room” 800 - 10 .

[0226] When the “sofa” 22 is no longer recognized in the “living room” 800-10, for example, when the “sofa” 22 is not recognized in the “living room” 800-10 for a threshold time (for example, the threshold time can be differently preset to two days, a week, etc.), the first processor 140-1 can update the information of the objects existing in the “living room” 52 based on the information of the objects existing in each of the multiple areas stored in the storage 130.

[0227] Therefore, reference Figure 8b , the information about the objects existing in the “living room” 52 stored in the storage 130 may be updated so as to no longer include the “sofa” 22 .

[0228] In such cases, refer to Figure 8c , the first processor 140 - 1 can obtain an artificial intelligence model 852 , in which the portion of the fully connected layer 702 trained to recognize “sofa” 22 is removed or obtained from the artificial intelligence model 752 corresponding to “living room” 52 .

[0229] The portion trained to recognize "sofa" 22 may refer to the portion of fully connected layer 702 that is used during inference to generate the output of node 722 for "sofa" 22. However, although this portion may be used during inference to generate the output of node 722, if this portion is also used during inference to generate the output of nodes related to "air conditioner" 11 and "television" 13, first processor 140-1 may not remove the relevant portion.

[0230] The first processor 140 - 1 may then update (or remove) the artificial intelligence model 852 obtained from the artificial intelligence model 752 corresponding to the “living room” 52 for storage in the storage 130 .

[0231] When (or based on) information about objects existing in each of the plurality of regions is stored in storage 130, first processor 140-1 may determine at least one object existing in one of the plurality of regions based on the information about objects existing in each of the plurality of regions stored in storage 130. Furthermore, based on the information about the identified objects from the relevant region transmitted from second processor 140-2, first processor 140-1 may determine an object not included in the determined at least one object among the identified objects from the relevant region.

[0232] In such a case, the first processor 140-1 may add a fully connected layer trained to recognize objects not included in the at least one determined object to the artificial intelligence model corresponding to the relevant area among the multiple artificial intelligence models to update the artificial intelligence model corresponding to the relevant area.

[0233] Figures 9a to 9c is a diagram illustrating an example of updating an artificial intelligence model corresponding to a relevant area when a new object is recognized as being added to one of a plurality of areas.

[0234] Figures 9a to 9c The embodiment describes as stored in the memory 130 Figure 8bIn addition, it can be assumed that the information about the objects in the "living room" 52 in the right table is stored as Figure 8c The artificial intelligence model corresponding to the "living room" 52 of the living room model 852.

[0235] refer to Figure 9a , the electronic device 100 can identify the "air conditioner" 11 and the "television" 13 in the "living room" 800-10. Figure 9a , because the “chair” 23 that did not previously exist in the “living room” 800 - 10 now exists in the “living room” 800 - 10 , the electronic device 100 can also recognize the “chair” 23 in the “living room” 800 - 10 .

[0236] When “chair” 23 that did not previously exist in “living room” 800 - 10 is newly recognized, the first processor 140 - 1 may update information of objects existing in “living room” 52 according to information of objects existing at each of a plurality of areas stored in the storage 130 .

[0237] Therefore, reference Figure 9b , the information about the objects existing in the “living room” 52 stored in the storage 130 may be updated to include the “chair” 23 in addition to the “air conditioner” 11 and the “television” 12 .

[0238] In such cases, refer to Figure 9c , the first processor 140-1 may add the trained portion (e.g., including the node 723) to the artificial intelligence model 852 corresponding to the “living room” 52 to recognize the “chair” 23 of the fully connected layer 702, to obtain an artificial intelligence model 952. The portion trained to recognize the “chair” 23 may refer to a portion used to generate an output of the node 723 related to the “chair” 23 of the fully connected layer 702 during inference.

[0239] Although all nodes of "bed" 21, "chair" 23 and "sofa" 22 are in Figure 7b 23 is shown as being included in an independent fully connected layer, but according to other embodiments, the node for recognizing “chair” may be included in a separate independent fully connected layer. In this case, the portion trained to recognize “chair” 23 may be an independent fully connected layer including a node for recognizing “chair” 23.

[0240] In addition, the first processor 140 - 1 may update the artificial intelligence model corresponding to the “living room” 52 to the obtained artificial intelligence model 952 and store it in the storage 130 .

[0241] When the artificial intelligence or electronic device 100 is located in one of multiple areas, if an object of (or based on) the relevant area is not recognized, even when the second processor 140-2 loads the artificial intelligence model corresponding to the relevant area to the volatile memory 145, the second processor 140-2 can also sequentially load different artificial intelligence models among the multiple artificial intelligence models to recognize the relevant object.

[0242] If (or based on) the identification of a related object using a different artificial intelligence model, the first processor 140-1 may use the information about the identified object to change the information about the object existing in each of the plurality of regions. In addition, the first processor 140-1 may use the information about the changed object to update the artificial intelligence model corresponding to the related region. As a specific example, a fully connected layer trained to identify the identified object may be added to the artificial intelligence model corresponding to the related region.

[0243] The information about the objects existing at each of the plurality of regions may be generated and updated through user input received by the electronic device 100 and / or data received by the electronic device 100 from an external device and stored in the storage 130. In this case, the first processor 140-1 may also use the generated / updated “information about the objects existing at each of the plurality of regions” to update the artificial intelligence model corresponding to each of the plurality of regions.

[0244] According to one or more embodiments, the acquisition and / or update of the artificial intelligence model corresponding to each of the multiple regions can be performed based on changes in information of only "fixed type objects" (rather than non-fixed type objects) existing at each of the multiple regions.

[0245] Fixed type objects may refer to objects that are rarely moved in people's real lives, such as beds, sofas, and televisions, while non-fixed type objects may refer to objects that are often moved in people's real lives, such as cups, plates, balls, and toys.

[0246] Even if the electronic device 100 is located in any of the plurality of areas, the second processor 140-2 may always load the artificial intelligence model trained to recognize non-fixed type objects to the volatile memory 145. In this case, the plurality of artificial intelligence models corresponding to each of the plurality of areas stored in the storage 130 may be artificial intelligence models trained to recognize fixed type objects.

[0247] Figure 10 is a block diagram illustrating a detailed configuration of the electronic device 100 including the first processor 140 - 1 and the second processor 140 - 2 according to an embodiment.

[0248] refer to Figure 10 In addition to the sensor 110, the camera 120, the storage 130, the first processor 140-1 and the second processor 140-2, the electronic device 100 may also include a first memory 150-1, a second memory 150-2, a communicator including a circuit 160, a user input 170 (or a user input device), an output 180 (or an output device) and at least one of a drive controller 190.

[0249] The sensor 110 may be implemented as a light detection and ranging (LiDAR) sensor, an ultrasonic sensor, or the like. When the sensor 110 is implemented as a LiDAR sensor, the sensing data generated based on the sensing result of the sensor 110 may include information about the distance between a structured object and / or at least a portion of things (or objects) present in the surroundings and the electronic device 100. The information about the distance may form or be the basis for information about the structure (i.e., shape and / or size) of the structured object / thing present in the surroundings of the electronic device 100.

[0250] The camera 120 may be implemented as an RGB camera, a 3D camera, or the like. The 3D camera may be implemented as a time-of-flight (TOF) camera including a TOF sensor and infrared (IR) light. The 3D camera may include an IR stereo sensor. The camera 120 may include sensors such as, but not limited to, a charge-coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS). If the camera 120 includes a CCD, the CCD may be implemented as a red / green / blue (RGB) CCD, an IR CCD, or the like.

[0251] The information about the map stored in the storage 130 may include information about the target of each of the plurality of areas. If the map including the plurality of areas relates to a map of, for example, "home", the target of each of the plurality of areas may relate to "living room", "bedroom", "kitchen", "bathroom", etc.

[0252] In the storage 130 , in addition to the plurality of artificial intelligence models and information about the map, information about objects existing in each of a plurality of areas on the map may also be stored.

[0253] The information about the objects present in each of the plurality of areas may include the name, type, etc. of the objects present in each of the plurality of areas. If the objects are people, the information about the objects may include information about the identity of the people. The information about the objects present in each of the plurality of areas may be stored / managed to match the information output by the plurality of artificial intelligence models to identify at least one object from the images obtained by the camera 120 in the plurality of areas.

[0254] Information about an object existing at each of the plurality of areas may be pre-stored or obtained by the electronic device 100 that performs object identification at each of the plurality of areas.

[0255] The first processor 140-1 may be composed of or include one or more processors. The one or more processors may be general-purpose processors such as a central processing unit (CPU) and an application processor (AP), as well as graphics-specific processors such as a graphics processing unit (GPU) and a visual processing unit (VPU).

[0256] The first processor 140 - 1 may control various configurations included in the electronic device 100 by executing at least one instruction stored in the first memory 150 - 1 or the storage 130 connected to the first processor 140 - 1 .

[0257] To this end, information or instructions for controlling various configurations included in the electronic device 100 may be stored in the first memory 150 - 1 .

[0258] The first memory 150-1 may include a read-only memory (ROM), a random access memory (RAM), such as a dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), and a double data rate SDRAM (DDR SDRAM), and may be implemented on a chip 1001 together with the first processor 140-1.

[0259] The second processor 140-2 may also be implemented as one or more processors. The second processor 140-2 may be implemented as an artificial intelligence dedicated processor, such as a neural processing unit (NPU), and may include a volatile memory 145 for loading at least one artificial intelligence model. The volatile memory 145 may be implemented as one or more state RAMs (SRAMs).

[0260] The second memory 150-2 may store information or instructions for controlling the function for object identification performed by the second processor 140-2. The second memory 150-2 may also include ROM, RAM (e.g., DRAM, SDRAM, DDR SDRAM), etc., and may be implemented on a single chip 1002 together with the second processor 140-2.

[0261] The communicator including the circuit 160 is a configuration for the electronic device 100 to transmit and receive signals / data by performing communication with at least one external device.

[0262] The communicator including the circuit 160 may include a wireless communication module, a wired communication module, and the like.

[0263] The wireless communication module may include at least one of a Wi-Fi communication module, a Bluetooth module, an infrared data communication (IrDA) communication module, a third generation (3G) mobile communication module, a fourth generation (4G) mobile communication module, and a 4G long term evolution (LTE) communication module to receive content from an external server or an external device.

[0264] The wired communication module may be implemented as a wired port, such as, for example, a Thunderbolt port, a USB port, or the like.

[0265] The first processor 140 - 1 may generate / update information on a map using data received from the outside through the communicator including the circuit 160 .

[0266] The first processor 140 - 1 and / or the second processor 140 - 2 may generate / update an artificial intelligence model corresponding to each of the plurality of regions using data received from the outside through the communicator including the circuit 160 .

[0267] Based on a control signal received through a communicator including circuit 160, first processor 140-1 may control second processor 140-2 to start / stop identifying an object located in at least one of the plurality of areas. At this time, the control signal may have been received from a remote control for controlling electronic device 100 or a smartphone having a remote control application stored on electronic device 100.

[0268] At least a portion of the plurality of artificial intelligence models stored in the storage 130 may be artificial intelligence models included in data received to the electronic device 100 from an external device such as a server device through a communicator including the circuit 160 .

[0269] When the second processor 140-2 cannot identify an object included in an image obtained through the camera 120 despite using all multiple artificial intelligence models stored in the storage 130, the first processor 140-1 may transmit data about the obtained image to the server device through a communicator including the circuit 160.

[0270] At this time, the electronic device 100 may receive data regarding a result of identifying the object included in the obtained image from the server device through the communicator including the circuit 160 .

[0271] In addition, the first processor 140 - 1 may receive data of an artificial intelligence model trained to recognize an identified object from an external device through a communicator including the circuit 160 , and may store the received artificial intelligence model in the storage 130 .

[0272] When data indicating the location of the electronic device 100 is received from an external device through the communicator including the circuit 160 , the first processor 140 - 1 may determine in which area the electronic device 100 is located using the received data.

[0273] The first processor 140-1 may update information on the map and / or at least a portion of information on objects existing in at least one of the plurality of areas based on user input received through the user inputter 170. In addition, the processor 140-1 may generate information on the map using data received according to the user input.

[0274] Based on user input received through the user inputter 170 , the first processor 140 - 1 may control a moving device or mechanism of the electronic device 100 to move around at least one of the plurality of areas and may control the second processor 140 - 2 to start / end object identification.

[0275] When a user input indicating the location of the electronic device 100 is received through the user inputter 170 , the first processor 140 - 1 may determine in which area the electronic device 100 is located using the received user input.

[0276] The user input 170 may include one or more of a button, a keyboard, a mouse, etc. In addition, the user input 170 may include a touch panel implemented together with a display or a separate touch panel.

[0277] The user inputter 170 may include a microphone to receive a voice input of a user command or information, and may be implemented together with the camera 120 to recognize a user command or information in the form of a motion or gesture.

[0278] The outputter 180 may be a configuration for the electronic device 100 to provide the obtained information to the user.

[0279] For example, the outputter 180 may include a display, a speaker, an audio terminal, etc., to provide the object identification result to the user visually / audibly.

[0280] As a configuration for controlling the moving device or mechanism of the electronic device 100, the driving controller 190 may include an actuator for providing power to the moving device or mechanism of the electronic device 100. The first processor 140-1 may control the moving device or mechanism of the electronic device 100 through the driving controller 190 to move the electronic device 100.

[0281] In addition, the electronic device 100 may further include Figure 10 Various configurations not shown in FIG.

[0282] The above embodiments have been described based on multiple artificial intelligence models stored in the storage 130, but it should be understood that one or more other embodiments are not limited thereto. For example, according to one or more other embodiments, multiple artificial intelligence models can be stored in an external server device, and object identification can be performed for each area by the electronic device 100.

[0283] Furthermore, the electronic device may perform object identification for each area through communication with an external terminal device implemented as a smartphone, a tablet PC, or the like.

[0284] Figure 11 1 and 2 are diagrams illustrating various embodiments of the electronic device 100 that performs object identification based on communication with external devices including a server device 300 and external terminal devices 200 - 1 and 200 - 2 .

[0285] refer to Figure 11 , the electronic device 100, which is a robot cleaner in this example, can perform communication with external devices 200-1 and 200-2 such as a smartphone and a server device 300. In this case, the electronic device 100 can also perform communication with the external devices 200-1, 200-2, and 300 through a relay device 400 configured with a router or the like.

[0286] The electronic device 100 may identify objects existing in a plurality of areas based on the control signal received from the external device 200-1 or the external device 200-2 as a smartphone. Furthermore, the electronic device 100 may transmit information about the identified objects to the external device 200-1 and / or the external device 200-2.

[0287] Figure 11Each of a plurality of artificial intelligence models is shown that is trained to identify at least one object stored in the server device 300 instead of (or not necessarily in) the storage 130 of the electronic device 100 .

[0288] In such a case, information about multiple artificial intelligence models stored in the server device 300 (i.e., information about objects that can be identified by each of the multiple artificial intelligence models) can be received by the electronic device 100 from the server device 300 via a communicator including the circuit 160.

[0289] The processor 140 - 1 may then select an artificial intelligence model corresponding to the area determined to be where the electronic device 100 is located from a plurality of artificial intelligence models stored in the server device 300 .

[0290] The first processor 140 - 1 may then control the communicator including the circuit 160 to transmit information about the selected artificial intelligence model to the server device 300 .

[0291] When data about the selected artificial intelligence model is received from the server device 300 through the communicator including the circuit 160, the first processor 140-1 may control the second processor 140-2 to load the selected artificial intelligence model (data) into the volatile memory 145. The second processor 140-2 may then perform object recognition by inputting the image obtained through the camera 120 into the loaded artificial intelligence model.

[0292] In this case, the first processor 140 - 1 may store data about the received artificial intelligence model to the storage 130 .

[0293] According to another embodiment, the electronic device 100 may include a processor.

[0294] Figure 12a and Figure 12b is a block diagram illustrating a configuration of the electronic device 100 including the processor 140 ′.

[0295] refer to Figure 12a , the electronic device 100 may include a processor 140' that controls the electronic device 100 and is connected to the sensor 110', the camera 140', and the storage 130'. In addition, the electronic device 100 may include the sensor 110', the camera 140', and the storage 130'.

[0296] The processor 140' can be implemented as a general-purpose processor such as a CPU and AP, a graphics-specific processor such as a GPU and a visual processing unit (VPU), an artificial intelligence-specific processor such as an NPU, etc., and can include a volatile memory for loading at least one artificial intelligence model.

[0297] The processor 140 ′ may perform operations performed by the first processor 140 - 1 or the second processor 140 - 2 , as in the various embodiments described above.

[0298] Specifically, the processor 140' can identify a plurality of areas included in the map based on information about the map stored in the storage 130', determine the area where the electronic device 100 is located from the plurality of areas based on the sensing data received from the sensor 110', and load an artificial intelligence model corresponding to the determined area from the plurality of artificial intelligence models stored in the storage 130 into the volatile memory. The processor 140' can input an image obtained by the camera 120' into the loaded artificial intelligence model to recognize an object.

[0299] refer to Figure 12b The electronic device 100 including the processor 140' may further include a memory 150', including Figure 10 The circuits shown connected to the processor 140' include a communicator 160', a user inputter 170', an outputter 180', a drive controller 190', and the like.

[0300] The memory 150' is a configuration for storing an operating system (OS) for controlling overall operations of elements of the electronic device 100 and data related to elements of the electronic device 100. The memory 150' may include at least one instruction related to one or more elements of the electronic device 100.

[0301] The memory 150 ′ may include ROM, RAM (eg, DRAM, SDRAM, and DDR SDRAM), etc., and may be implemented to be connected with the processor 140 ′ in one chip 1201 .

[0302] Figures 13 to 15 A control method of the electronic device 100 according to one or more embodiments is described.

[0303] Figure 13 is a flowchart illustrating a control method of the electronic device 100 using an object identification model according to an embodiment of the present disclosure.

[0304] refer to Figure 13, the control method may identify a plurality of areas included in the map based on information about the map stored in the memory of the electronic device 100 (operation S1310). The information about the map may include at least one of information about a place structure, information about a structure of each of the plurality of areas included in the map, information about a position of each of the plurality of areas on the map, information about a target of each of the plurality of areas, and the like.

[0305] Then, a region in which the electronic device 100 is located among the plurality of regions may be determined based on the sensing data received from the sensor (operation S1320 ).

[0306] In this case, the map information stored in the memory and the sensing data received by the sensor can be used to determine the area where the electronic device 100 is located. As a specific example, if the sensor is a LiDAR sensor, the sensing data received from the sensor and the information on the map stored in the storage can be compared to determine the area where the electronic device 100 is located.

[0307] At this time, the information about the structure (i.e., shape and / or size) around the electronic device 100 included in the sensing data can be compared with the information about the structure (i.e., shape and / or size) of each of the multiple areas on the map included in the information about the map, and the area where the electronic device 100 is located can be determined from the multiple areas on the map.

[0308] By using the data about the surrounding image obtained by the camera, it is possible to determine which area of the multiple areas the electronic device 100 is located in. For example, if the information about the map includes data about 3D images of multiple areas, the area where the electronic device 100 is located can be determined by comparing the image of each of the multiple areas included in the information about the map with the 3D image obtained by the camera implemented as a 3D camera.

[0309] Furthermore, the area in which the electronic device 100 is located may be determined by recognizing one or more objects from an image obtained through a camera in the area in which the electronic device is located.

[0310] As a specific example, an image captured by a camera in the area where the electronic device 100 is located can be input into at least one of the stored multiple artificial intelligence models to identify an object in the image. If the identified object is a bed, one or more objects in each area can use pre-stored information to identify the bed corresponding to the "bedroom." In addition, the area where the electronic device 100 is located can be determined as a "bedroom." The area where the electronic device 100 is located can then be determined as a "bedroom."

[0311] In addition, an inertial sensor, an acceleration sensor, or the like may be used to determine a point on a map where the electronic device 100 is located, and an area including the determined point among a plurality of areas on the map may be determined as the area where the electronic device 100 is located.

[0312] The process of determining the area where the electronic device 100 is located is not limited to the above-described embodiment, and other various methods may be applied.

[0313] The control method may include loading an artificial intelligence model corresponding to the determined area from among a plurality of artificial intelligence models stored in a memory to a volatile memory (operation S1330 ).

[0314] Each of the multiple artificial intelligence models may include a convolutional layer and a fully connected layer trained to recognize objects based on feature information extracted from the convolutional layer. In this case, the convolutional layer may be a common layer for the multiple artificial intelligence models, and the fully connected layer may be a layer provided separately to each of the multiple artificial intelligence models.

[0315] If (or based on) the electronic device 100 is located in the first area, the fully connected layers and convolutional layers corresponding to the first model of the multiple artificial intelligence models may be loaded into the volatile memory. If (or based on) the electronic device 100 is located in the second area, the fully connected layers and convolutional layers corresponding to the second model of the multiple artificial intelligence models may be loaded into the volatile memory.

[0316] The first model may correspond to a first area among the plurality of areas, and the second model may correspond to a second area among the plurality of areas. In this case, logical mapping information mapping the first model in the first area and logical mapping information mapping the second model in the second area may be stored in a memory, and the control method may load the artificial intelligence model corresponding to each area using the logical mapping information stored in the memory.

[0317] The image obtained by the camera can then be input to the loaded artificial intelligence model to recognize the object (operation S1340). Specifically, because the information about the object output by the loaded artificial intelligence model can be obtained, the information about the object can vary according to artificial intelligence model attributes such as the name and type of the object.

[0318] The control method may generate information about a map of the location where the electronic device 100 is located to store in a memory. In addition, the control method may newly obtain / define an artificial intelligence model corresponding to each of a plurality of areas included in the map.

[0319] Figure 14is a flowchart illustrating an embodiment of a control method of the electronic device 100 that generates information about a map and identifies objects existing in each of a plurality of areas to obtain an artificial intelligence model corresponding to each of the plurality of areas according to an embodiment.

[0320] refer to Figure 14 , the control method may obtain information about the structure of a place where the electronic device 100 is located based on the sensing data received from the sensor (operation S1410).

[0321] In an example, information about the structure (ie, shape and / or size) of a location where the electronic device is located may be obtained using sensing data received from a sensor implemented as a LiDAR sensor.

[0322] Then, based on the information about the obtained structure, the place where the electronic device 100 is located may be divided into a plurality of areas (operation S1420).

[0323] In such a case, a first algorithm can be used to divide the map into multiple areas. For example, points where there are dividing lines or protrusions (or thresholds) on the floor, points where the movable width narrows, points where there are walls, points where walls end, points where there are doors, etc. can be identified using sensor data obtained by sensors and / or images obtained by cameras. In addition, each area on the map can be divided using the identified points as boundaries between areas. However, other methods can also be applied.

[0324] Then, information on a map including information on a structure of each of the divided plurality of areas may be generated, and the generated information on the map may be stored in a memory (operation S1430).

[0325] When an artificial intelligence model trained to recognize multiple objects is stored in the memory of the electronic device 100, the control method can input an image obtained by the camera when the electronic device 100 is located in each of the multiple areas into the stored artificial intelligence model, and recognize objects existing in each of the multiple areas (operation S1440).

[0326] The stored artificial intelligence model may include a convolutional layer and a fully connected layer trained to recognize multiple objects based on feature information extracted from the convolutional layer.

[0327] Then, a first object may be identified in a first area of the plurality of areas, and a second object may be identified in a second area of the plurality of areas.

[0328] Then, an artificial intelligence model corresponding to each of the plurality of regions may be obtained from stored artificial intelligence models based on information about the recognized object (operation S1450 ).

[0329] Specifically, when (or based on) a first object among multiple objects is identified as existing in a first area based on information of an object existing in a first area of multiple areas, a first model including a part trained to recognize the first object from a convolutional layer of a stored artificial intelligence model and a fully connected layer of a stored artificial intelligence model can be obtained.

[0330] In addition, when a second object among multiple objects is identified as existing in the second area based on information of an object existing at the second area of the multiple areas, a second model can be obtained including a convolutional layer of the stored artificial intelligence model and a fully connected layer of the stored artificial intelligence model that is trained to identify other parts of the second object.

[0331] The control method may identify an object existing at each of a plurality of areas according to a preset interval, a user input, etc., and update an artificial intelligence model corresponding to each of the plurality of areas using information about the identified object.

[0332] Specifically, when the electronic device 100 is located in one of the multiple areas, the image obtained by the camera can be input to the multiple artificial intelligence models loaded in the volatile memory, and the object existing in the relevant area can be identified. In this case, at least one of the multiple artificial intelligence models in the volatile memory can be loaded sequentially, and the image obtained from the relevant area can be input to the loaded artificial intelligence model. Based on the information about the identified object, the artificial intelligence model corresponding to the relevant area can be updated.

[0333] For example, when information about objects existing in each of multiple areas is stored in a memory of an electronic device, the control method can determine at least one object existing in one of the multiple areas based on the information about objects existing in each of the multiple areas stored in the memory.

[0334] In such a case, based on information about objects recognized in the relevant area, objects that were not recognized in the relevant area can be determined from the determined objects. In the artificial intelligence model corresponding to the relevant area among the multiple artificial intelligence models, the part that was trained to recognize the object that was previously determined to be not recognized can be removed.

[0335] In another example, when information about objects existing in each of a plurality of areas is stored in the memory of the electronic device 100, the control method can determine at least one object existing in one of the plurality of areas based on the information about objects existing in each of the plurality of areas stored in the memory.

[0336] In this case, based on information about the objects recognized in the relevant region, an object not included in the at least one determined object of the objects recognized in the relevant region can be determined. At this time, a fully connected layer trained to recognize objects not included in the at least one determined object can be added to the artificial intelligence model corresponding to the relevant region among the multiple artificial intelligence models.

[0337] Figure 15 is a diagram illustrating an algorithm of an example of a method of controlling the electronic device 100 of updating an artificial intelligence model corresponding to each of a plurality of areas according to a result of recognizing an object existing in each of the plurality of areas according to an embodiment.

[0338] refer to Figure 15 , the control method may input an image obtained by a camera in one of the plurality of areas into at least one of the plurality of artificial intelligence models to recognize an object (operation S1510 ).

[0339] Then, pre-stored object information about the relevant area may be compared with the information about the recognized object (operation S1520).

[0340] If the information about the recognized object matches the information about the pre-stored object (operation S1530 -Yes), the artificial intelligence model stored to correspond to the relevant area may not be updated.

[0341] If the information about the recognized object does not match the information about the pre-stored object (operation S1530-No), and if information about the newly added object and information about the pre-stored object are included in addition to the information about the recognized object (operation S1540-Yes), the artificial intelligence model corresponding to the relevant area can be updated so that the artificial intelligence model stored as corresponding to the relevant area can also recognize the added object (operation S1550).

[0342] For example, although the objects recognized in the relevant area can be a TV and a sofa, if the object pre-stored on the relevant area is a TV, the artificial intelligence model corresponding to the relevant area can update the relevant artificial intelligence model to recognize the sofa in addition to the TV. In this case, a separate fully connected layer trained to recognize the sofa can be added to the fully connected layer of the relevant artificial intelligence model.

[0343] Even if the information about the recognized object does not match the information about the pre-stored object (operation S1530-No), and there is an object from the pre-stored information about the object that is not included in the information about the recognized object (operation S1540-No), it may be necessary or performed to update the artificial intelligence model corresponding to the relevant area. That is, because the object that existed at the previously relevant area no longer exists in the relevant area, the part that was trained to recognize the object that no longer exists in the relevant area may be removed (operation S1560).

[0344] For example, although the objects recognized in the relevant area may be a television and a sofa, if the objects pre-stored on the relevant area are a television, a sofa, and a chair, the part trained to recognize the chair in the artificial intelligence model corresponding to the relevant area can be removed. Specifically, the part used to generate the output of the node indicating the possibility of the existence of a chair during the inference process can be removed from the fully connected layer of the relevant artificial intelligence model. However, even if the part is related to the inference process for generating the output of the node indicating the possibility of the existence of a chair, if the part is related to the inference process for generating the output of the node indicating the possibility of the existence of a television or a sofa, the part may not be removed.

[0345] Reference above Figures 13 to 15 The control method of the electronic device described above can be referred to Figure 2 and Figure 11 The electronic device 100 shown and described above with reference to Figure 12a and Figure 12b The electronic device 100 shown and described is implemented.

[0346] Reference above Figures 13 to 15 The described control method of the electronic device may be implemented by the electronic device 100 and a system including one or more external devices.

[0347] The various embodiments described above may be implemented in a recordable medium that can be read by a computer or a computer-like device using software, hardware, or a combination of software and hardware.

[0348] By hardware implementation, the embodiments described in the present disclosure may be implemented using, for example but not limited to, at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and electrical units for performing other functions.

[0349] In some cases, the embodiments described herein can be implemented by a processor itself. According to software implementation, embodiments such as the processes and functions described herein can be implemented with separate software modules. Each of the above software modules can perform one or more functions and operations described herein.

[0350] Computer instructions for executing processing operations in the electronic device 100 according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When executed by a processor of a specific device, the computer instructions stored in the non-transitory computer-readable medium cause the specific device to execute processing operations in the electronic device 100 according to the various embodiments described above.

[0351] Non-transitory computer-readable media may refer to, for example, a medium that stores data semi-permanently rather than for a short period of time, such as a register, a cache, and a memory, and can be read by a device. Specific examples of non-transitory computer-readable media may include compact discs (CDs), digital versatile discs (DVDs), hard disks, Blu-ray discs, universal serial buses (USBs), memory cards, read-only memories (ROMs), and the like.

[0352] While embodiments have been shown and described above, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure.

Claims

1. An electronic device comprising: sensor; camera; non-volatile memory for storing a plurality of artificial intelligence models trained to recognize objects and for storing information associated with the map; a first processor configured to control the electronic device; and A second processor is configured to identify the object, The first processor is configured to determine an area where the electronic device is located from a plurality of areas included in the map based on sensing data obtained from the sensor, and provide area information about the determined area to the second processor, and The second processor includes a volatile memory and is configured to: loading an artificial intelligence model from among the plurality of artificial intelligence models stored in the non-volatile memory into the volatile memory based on the region information provided by the first processor, and The image obtained through the camera is input to the loaded artificial intelligence model to recognize the object.

2. The electronic device according to claim 1, wherein: Each of the plurality of artificial intelligence models includes a convolutional layer and a fully connected layer trained to recognize an object based on feature information extracted from the convolutional layer; The convolutional layer is a common layer in the multiple artificial intelligence models described above; and The second processor is also configured to load the convolutional layers and the fully connected layers of the artificial intelligence model corresponding to the determined area into the volatile memory.

3. The electronic device according to claim 2, wherein: The plurality of artificial intelligence models include a first model corresponding to a first region of the plurality of regions and a second model corresponding to a second region of the plurality of regions; and The second processor is further configured to: Based on the electronic device being located in the first area, loading the fully connected layer and the convolutional layer corresponding to the first model into the volatile memory, and Based on the electronic device being located in the second area, fully connected layers and convolutional layers corresponding to the second model are loaded into the volatile memory.

4. The electronic device according to claim 1, wherein: The information about the map includes information about the structure of the plurality of areas; and The first processor is further configured to compare the information about the structure with the sensing data obtained from the sensor to determine an area where the electronic device is located from among the plurality of areas.

5. The electronic device according to claim 1, wherein The first processor is further configured to: obtaining information about a structure of a location where the electronic device is located based on sensing data obtained from the sensor, dividing the site into the plurality of areas based on the information obtained about the structure, Information on a map is generated and stored in a storage, the information on the map including information on a structure of each of the divided plurality of areas.

6. The electronic device according to claim 5, wherein: The second processor is further configured to input an image obtained by the camera when the electronic device is located in a specific area among the plurality of areas to at least one of the plurality of stored artificial intelligence models to recognize an object existing in the specific area, and provide information about the recognized object to the first processor; and The first processor is further configured to obtain an artificial intelligence model corresponding to the specific area based on the provided information about the recognized object.

7. The electronic device according to claim 6, wherein: Each of the plurality of stored artificial intelligence models includes a convolutional layer and a fully connected layer trained to recognize a plurality of objects based on feature information extracted from the convolutional layer; and The first processor is further configured to: obtaining a first model including a first portion of a convolutional layer and a fully connected layer trained to recognize the first object based on the first object among the plurality of objects in the first region being recognized as existing based on the information about the objects existing in the first region, and According to the second object among the plurality of objects in the second area being recognized as existing based on the information about the objects existing in the second area, a second model including a second part of a convolutional layer and a fully connected layer trained to recognize the second object is obtained.

8. The electronic device according to claim 1, wherein: The second processor is further configured to input an image obtained by the camera when the electronic device is located in an area among the plurality of areas to at least one artificial intelligence model of the plurality of artificial intelligence models loaded into the volatile memory to recognize an object present at the area, and provide information about the recognized object to the first processor; and The first processor is further configured to update an artificial intelligence model corresponding to the area based on the information about the recognized object provided from the second processor.

9. The electronic device according to claim 8, wherein: The storage stores information about objects present at each of the plurality of areas; and The first processor is further configured to: determining at least one object existing in the area among the plurality of areas based on information stored in a storage about objects existing at each of the plurality of areas, determining an unrecognized object in the area from the determined at least one object based on the information about the recognized object provided from the second processor, and The artificial intelligence model corresponding to the area is updated by removing a portion of the artificial intelligence model corresponding to the area that is trained to recognize the determined unrecognized object.

10. The electronic device according to claim 8, wherein: The storage stores information about objects present at each of the plurality of areas; and The first processor is further configured to: determining at least one object existing in each of the plurality of areas based on information stored in a memory about objects existing in the area, determining whether the recognized object is not included in the determined at least one object existing in the area based on the information about the recognized object provided from the second processor, and Based on the recognition that the recognized object is determined not to be included in at least one of the determined objects present in the area, the trained portion for recognizing the recognized object is added to the artificial intelligence model corresponding to the area to update the artificial intelligence model corresponding to the area.

11. A method for controlling an electronic device using an object identification model, the method comprising: identifying a plurality of regions to be included in the map based on information associated with the map stored in non-volatile memory of the electronic device; determining, based on sensing data obtained from the sensor, an area where the electronic device is located from among the plurality of areas; loading an artificial intelligence model from a plurality of artificial intelligence models stored in a non-volatile memory into a volatile memory based on the determined region; and Objects are recognized by inputting images obtained through the camera into the loaded artificial intelligence model.

12. The control method according to claim 11, wherein: Each of the plurality of artificial intelligence models includes a convolutional layer and a fully connected layer trained to recognize an object based on feature information extracted from the convolutional layer; The convolutional layer is a common layer in the multiple artificial intelligence models described above; The fully connected layer is a layer provided separately to each of the plurality of artificial intelligence models; and Loading into volatile memory involves: Based on the electronic device being located in a first area of the plurality of areas, loading a fully connected layer and a convolutional layer corresponding to a first model of the plurality of artificial intelligence models into a volatile memory, and Based on the electronic device being located in a second area among the multiple areas, a fully connected layer and a convolutional layer corresponding to a second model among the multiple artificial intelligence models are loaded into a volatile memory.

13. The control method according to claim 11, wherein: The information about the map includes information about the structure of each of the plurality of areas; and Determining the area where the electronic device is located includes comparing information about the structure with sensing data obtained from the sensor to determine the area where the electronic device is located from among the plurality of areas.

14. The control method according to claim 11, further comprising: obtaining information about a structure of a location where the electronic device is located based on sensing data obtained from the sensor; dividing the site into the plurality of areas based on the information obtained about the structure; and Information on a map is generated and stored in a storage, the information on the map including information on a structure of each of the divided plurality of areas.

Images