Electronic device and control method therefor

Abstract

A method for controlling an electronic device including an artificial intelligence accelerator according to an embodiment of the present invention comprises the steps of: operating the electronic device in a first mode; receiving an image of a predetermined object through a sensor while operating in the first mode; storing the image and a recognition result of the predetermined object on the basis of a first model; entering a second mode by the electronic device; determining whether the predetermined object is misrecognized, on the basis of a second model; and retraining the first model on the basis of a result of the determination.

Description

Electronic device and method for controlling the same

[0001] The present invention relates to electronic devices and their control methods. For example, the present invention relates to a technology that can eliminate misrecognition targets by robots and facilitate the registration of new recognition targets by applying various artificial intelligence technologies.

[0002] Artificial intelligence (AI) refers to basic intelligence that mimics human intelligence by creating and applying algorithms embedded in a dynamic computing environment. Simply put, AI can be defined as a computer that thinks and behaves like a human.

[0003] Accordingly, various home appliances that apply AI are appearing.

[0004] However, conventional AI-powered robot vacuum cleaners inevitably transmit video data collected in the home environment to servers outside the home. Because video data is highly likely to contain sensitive user privacy information, implementing services utilizing it presents numerous limitations.

[0005] One embodiment of the present invention seeks to protect privacy by eliminating the process of an electronic device with AI applied in the home transmitting information about the home environment to the outside by improving the data processing performance of a robot through on-device learning in which image data collected in the home is not transmitted outside the electronic device.

[0006] One embodiment of the present invention seeks to improve the recognition of a misrecognized object (object, etc.) without transmitting separate data to a server by an electronic device (e.g., a robot vacuum cleaner, a mobile robot, etc.).

[0007] One embodiment of the present invention enables an electronic device to recognize a new object without transmitting separate data to a server.

[0008] A method for controlling an electronic device including an artificial intelligence accelerator according to any one of the embodiments of the present invention comprises the steps of: operating the electronic device in a first mode; receiving an image of an arbitrary object through a sensor while operating in the first mode; storing a recognition result of the arbitrary object and the image based on a first model; entering the electronic device in a second mode; determining whether there is a misrecognition of the arbitrary object based on the second model; and retraining the first model based on the determination result.

[0009] The first mode includes, for example, a state in which the electronic device is running without charging, and the second mode includes, for example, a state in which the electronic device is being charged while power is being supplied.

[0010] The first model, for example, recognizes an arbitrary object within a preset frame time. Meanwhile, the second model, for example, is characterized by increasing the resolution of the image (resolution scaling), increasing the depth of the model (depth scaling), or increasing the number of channels, compared to the first model.

[0011] The above preset frame corresponds to, for example, frame 1.

[0012] The above-described judging step further includes, for example, a step of receiving feedback information related to whether or not there is a misrecognition of the arbitrary object from a mobile device connected in a communication state with the electronic device.

[0013] An electronic device according to one embodiment of the present invention includes a sensor that receives an image of an arbitrary object while operating in a first mode, a memory that stores a recognition result of the arbitrary object and the image based on a first model, and an artificial intelligence accelerator that determines whether there has been a misrecognition of the arbitrary object based on a second model when the electronic device enters a second mode and retrains the first model based on the determination result.

[0014] A method for controlling an electronic device according to an embodiment of the present invention includes the steps of performing a communication connection with a first object, receiving an image of a second object captured through a camera of the first object, displaying the received image of the second object, receiving information about a label of the second object and an action of the first object corresponding to the label from a user, and transmitting the received label and information about the action to the first object.

[0015] When the first object is in a charging state, relearning of the first object is performed based on the image of the second object, the label, and information about the action.

[0016] The method for controlling the electronic device further includes a step of receiving a learning result for an arbitrary image from the first object when the first object is in a charging state.

[0017] The method for controlling the electronic device further includes a step of transmitting feedback on the learning result for the arbitrary image to the first object.

[0018] The electronic device includes a mobile device, the first object includes a robot vacuum cleaner, and the second object includes any object in the home.

[0019] An electronic device according to one embodiment of the present invention includes a transceiver that performs a communication connection with a first object and receives an image of a second object captured through a camera of the first object, a touch display that displays the received image of the second object and receives information about a label of the second object and an action of the first object corresponding to the label from a user, and a controller.

[0020] In particular, the controller is characterized in that it controls the transceiver to transmit information about the received label and the action to the first object.

[0021] It is also within the scope of the present invention for a third party to implement a computer-readable medium (e.g., an application, memory, software, etc.) recording a program for performing any of the above-described methods and various embodiments described in the specification.

[0022] According to one embodiment of the present invention, it is possible to solve the problem of misrecognition of objects that exist only exceptionally within a user's home environment (e.g., a column-shaped screen with a specific pattern is misrecognized as a person).

[0023] According to one of the embodiments of the present invention, privacy-sensitive image data can be improved only through feedback from users, without transmitting such data to an external server or the like.

[0024] According to one embodiment of the present invention, by requesting user reviews of data already organized through a master model review, the burden on users' technical understanding and fatigue from frequent feedback requests can be reduced. Of course, if user reviews are inadequate or the user does not wish to provide reviews, training data classification is possible using only the master model reviews.

[0025] According to one embodiment of the present invention, by securing sufficient computational capacity / computational speed using an artificial intelligence accelerator or the like, the robot learns on its own (on-device) without the help of a network connection or external server, enabling users to quickly experience better functional services. For example, today, the robot asks a chair of a certain shape in the user's home to move out of the way, but after charging it, the robot will no longer ask the same chair of a certain shape in the user's home during the next cleaning.

[0026] In addition to the technical effects described above, it is self-evident that effects that can be inferred by a person skilled in the art from the entire specification should also be taken into consideration.

[0027] FIG. 1 illustrates a case in which an electronic device includes an artificial intelligence accelerator according to one embodiment of the present invention.

[0028] FIG. 2 illustrates a scenario in which different models of AI are applied depending on the mode of an electronic device, according to one embodiment of the present invention.

[0029] FIG. 3 is a drawing for explaining a case in which an electronic device misrecognizes an object according to a prior art.

[0030] FIG. 4 schematically illustrates a process by which an electronic device performs improvement of misrecognition of a recognition target according to one embodiment of the present invention.

[0031] FIG. 5 schematically illustrates a process by which an electronic device adds a new recognition target according to one embodiment of the present invention.

[0032] FIG. 6 is a block diagram illustrating detailed configurations of an electronic device for improving misrecognition according to one embodiment of the present invention.

[0033] FIG. 7 is a flow chart detailing a process by which the electronic device illustrated in FIG. 6 performs error recognition improvement in the first mode.

[0034] FIG. 8 is a flow chart detailing a process by which the electronic device illustrated in FIG. 6 performs error recognition improvement in the second mode.

[0035] FIG. 9 is a flowchart detailing a process by which the electronic device illustrated in FIG. 6 requests user feedback.

[0036] Figure 10 is a flow chart detailing a process by which the electronic device illustrated in Figure 6 performs retraining to improve misrecognition.

[0037] FIG. 11 is a block diagram illustrating detailed configurations of an electronic device for adding a new recognition target according to one embodiment of the present invention.

[0038] Figure 12 is a flow chart detailing the process of collecting data by the electronic device illustrated in Figure 11.

[0039] Figure 13 is a flow chart detailing the process by which the electronic device illustrated in Figure 11 performs learning.

[0040] And, Fig. 14 is a flow chart detailing the process of recognizing a new recognition target after the learning illustrated in Fig. 13 is completed.

[0041] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be assigned the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles.

[0042] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included within the spirit and technical scope of the present invention.

[0043] Terms that include ordinal numbers, such as "first," "second," etc., may be used to describe various components, but these components are not limited by these terms. These terms are used solely to distinguish one component from another. When a component is referred to as being "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components present in between.

[0044] On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0045] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0046] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0047] FIG. 1 illustrates a case in which an electronic device includes an artificial intelligence accelerator according to one embodiment of the present invention.

[0048] As illustrated in FIG. 1, an electronic device (10) according to one embodiment of the present invention may include, for example, an artificial intelligence accelerator.

[0049] AI accelerators are a class of specialized hardware accelerators or computer systems designed to accelerate artificial intelligence and machine learning applications, such as artificial neural networks and machine vision.

[0050] Conventional technology involves robot vacuum cleaners and other devices collecting data, transmitting the collected data to a cloud server, where data classification and learning are performed. The cloud server then transmits the learning results to the robot vacuum cleaner and other devices. In other words, the robot lacks sufficient computing power and functionality for learning. Another problem with conventional technology is that images captured in the home are leaked to external servers.

[0051] On the other hand, the electronic device (10) according to one embodiment of the present invention has the advantage of facilitating improvement of misrecognition targets and registration of new objects by providing sufficient computational power for learning internally.

[0052] For example, as illustrated in Fig. 1, an electronic device (10) including an artificial intelligence accelerator, etc., first collects and classifies data (S100).

[0053] Furthermore, an electronic device (10) according to one embodiment of the present invention performs learning on classified data (S110) and obtains a model output (S120).

[0054] However, another feature of the present invention is that while the electronic device is performing normal operations, it uses a simple learning runtime model, while when the electronic device is in a special state such as charging, it uses a master model that verifies the runtime model. This will be described in more detail below with reference to FIG. 2.

[0055] FIG. 2 illustrates a scenario in which different models of AI are applied depending on the mode of an electronic device, according to one embodiment of the present invention.

[0056] The model illustrated in (a) of FIG. 2 is a first model used by an electronic device (e.g., including an artificial intelligence accelerator) according to one embodiment of the present invention, and may also be named a runtime model.

[0057] The runtime model is applied when the electronic device operates in the first mode (e.g., a state in which a robot vacuum cleaner is running, such as cleaning, without charging).

[0058] Since the video image input to the electronic device must be processed in real time, for example, within 1 frame time, the first model (runtime model) shown in (a) of Fig. 2 is designed to be lightweight in terms of software.

[0059] On the other hand, the model shown in (b) of FIG. 2 is a second model used by an electronic device according to an embodiment of the present invention, and may be named a master model.

[0060] The master model is applied when the electronic device is operating in a second mode (e.g., when the robot vacuum cleaner is charging or entering a charging station).

[0061] To verify that there are no problems with the images and recognition results saved by the robot vacuum cleaner during cleaning time, the master model uses a model that is slower than the runtime model but has good recognition performance.

[0062] In a master model according to one embodiment of the present invention, as a method of increasing performance, for example, a method of increasing the size of an input image (resolution scaling), increasing the depth of the model (depth scaling), or increasing the number of channels (width scaling) is used.

[0063] FIG. 3 is a drawing for explaining a case in which an electronic device misrecognizes an object according to a prior art.

[0064] As shown in (a) of Fig. 3, if an electronic device (e.g., a robot vacuum cleaner) properly recognizes a person and outputs a voice saying, “I am cleaning. Please step aside for a moment,” there is no particular problem.

[0065] On the other hand, as shown in (b) of Fig. 3, if the electronic device misrecognizes a chair as a person and outputs a voice message saying, “We are cleaning. Please step aside for a moment,” then an error has occurred.

[0066] According to conventional technology, there is a problem that an improved artificial intelligence model must be developed separately and updated every time.

[0067] Of course, other conventional technologies require electronic devices to be connected to a server via the Internet. This presents a problem: the electronic device transmits data to the server, and it takes a long time to download models or updates from the server. Furthermore, there's also the problem of electronic devices leaking images taken at home to external servers.

[0068] In contrast, the present invention, designed to address the aforementioned issues and other issues, eliminates the need to transmit data to or receive data from an external server. Therefore, it offers advantages in terms of privacy, security, and speed.

[0069] As an example of implementing this, a fast learning speed can be achieved by using an artificial intelligence accelerator, and user feedback, etc. can also be additionally utilized.

[0070] More specific examples related to this will be described in detail later in FIG. 4, etc.

[0071] FIG. 4 schematically illustrates a process by which an electronic device performs improvement of misrecognition of a recognition target according to one embodiment of the present invention.

[0072] An electronic device (e.g., including an artificial intelligence accelerator, etc.) according to one embodiment of the present invention collects data corresponding to an image captured through a sensor such as a camera (S410).

[0073] Furthermore, the electronic device classifies data (S420) and performs learning (S440) without communicating with the server. Then, the electronic device obtains model output (S450).

[0074] Meanwhile, since errors may occur in the process of the electronic device independently classifying data (S420), it is also within the scope of another right of the present invention to design the device to additionally receive user feedback (S430).

[0075] However, Fig. 4 is a process for improving misrecognition of objects (e.g., people, objects, etc.) by an electronic device, and the process for adding a new object will be described later in Fig. 5.

[0076] FIG. 5 schematically illustrates a process by which an electronic device adds a new recognition target according to one embodiment of the present invention.

[0077] An electronic device (e.g., including an artificial intelligence accelerator, etc.) according to one embodiment of the present invention collects data corresponding to an image captured through a sensor such as a camera (S510).

[0078] Furthermore, the electronic device classifies data (S530) and performs learning (S550) without communicating with the server. Then, the electronic device obtains model output (S560).

[0079] However, unlike the embodiment of FIG. 4 (misrecognition improvement) described above, the embodiment of FIG. 5 (new object registration) is designed to receive user feedback (S520, 540) at each of the data collection (S510) and classification (S530) stages.

[0080] FIG. 6 is a block diagram illustrating detailed configurations of an electronic device for improving misrecognition according to one embodiment of the present invention.

[0081] An electronic device (600) according to one embodiment of the present invention includes a sensor (620), a controller (610) for image analysis and processing, a user feedback interface (640), and a recognition target / recognition result DB (650). However, the controller (610) may be combined with an artificial intelligence accelerator in software or hardware.

[0082] Meanwhile, the artificial intelligence accelerator described in this specification refers to hardware or software capable of accelerating representative operations that constitute an artificial intelligence model. For example, the artificial intelligence accelerator performs row-matrix multiplication, inner product, row-matrix (arithmetic) operations, convolution, pooling, and activation.

[0083] The sensor (620) photographs various objects in the home and transmits the corresponding image data to the controller (610).

[0084] The object recognition module (611) inside the controller (610) recognizes objects, the data classification model (612) classifies objects, and the object learning module (613) performs additional re-learning, etc., by an AI accelerator.

[0085] The recognition target / recognition result DB (650) transmits the object recognition result and classification result received from the controller (610) to the user feedback interface (640).

[0086] In addition, the user feedback interface (640) is designed to improve the misrecognition results of the electronic device (600) through communication with the mobile device (660).

[0087] The mobile device (660) should be designed so that it can be used by a user without having a technical understanding of the present invention.

[0088] Furthermore, for objects that the master model determines to be misrecognized, a double check is performed through communication with a mobile device (660) to determine whether or not the object is actually misrecognized.

[0089] Alternatively, if the master model determines that the object is a normal recognition target, but the probability is low, the normal recognition is double-checked through communication with the mobile device (660) to determine whether it is actually correct.

[0090] However, as previously mentioned, one of the features of the present invention is that the artificial intelligence model varies depending on the mode of the electronic device. This will be described in more detail below with reference to Figures 7 through 10.

[0091] FIG. 7 is a flow chart detailing a process by which the electronic device illustrated in FIG. 6 performs error recognition improvement in the first mode.

[0092] FIG. 7 illustrates a process for collecting data for improving misrecognition of an object in a first mode (e.g., during a cleaning operation) by a robot cleaner according to an embodiment of the present invention.

[0093] First, the robot vacuum cleaner according to one embodiment of the present invention starts cleaning (S710).

[0094] Furthermore, the robot vacuum cleaner receives image data through sensors, etc. (S720).

[0095] It is determined whether a specific object is recognized in the image data received by the runtime model (S730).

[0096] If the above judgment result fails to be recognized, the process returns to step S720.

[0097] On the other hand, if the recognition result above is successful, the image data and recognition result received in step S720 are stored (S740).

[0098] Furthermore, the robot vacuum cleaner determines whether cleaning has been completed (S750).

[0099] If not terminated, return to step S720.

[0100] FIG. 8 is a flow chart detailing a process by which the electronic device illustrated in FIG. 6 performs error recognition improvement in the second mode.

[0101] FIG. 8 illustrates a process for classifying data for improving misrecognition of an object in a second mode (e.g., when a request to enter a charging state or a learning state is received, etc.) by a robot cleaner according to an embodiment of the present invention.

[0102] When a robot cleaner arrives at a charging station and begins charging, for example, the robot cleaner starts a data classification process (S800).

[0103] Furthermore, the robot vacuum cleaner determines whether data to be classified exists (S801).

[0104] As a result of the above judgment (S801), if the data to be classified does not exist, the process is terminated (S809).

[0105] On the other hand, if the classification target data exists as a result of the above judgment (S801), the image data to be classified and the recognition results are loaded (S802). The data used in step S802 may be the results performed in the runtime model.

[0106] The master model mounted on the robot vacuum cleaner according to one embodiment of the present invention determines whether there is a misrecognition (S803).

[0107] If the above judgment result (S803) is determined to be a misrecognition, the robot cleaner designates the image data and recognition result of step S802 as relearning targets (S804).

[0108] And, the robot vacuum cleaner registers itself as a candidate for requesting feedback from a mobile device, etc. (S805).

[0109] On the other hand, if the above judgment result (S803) determines that it is not a misrecognition, the robot cleaner determines again whether the probability of normal recognition is low (S806).

[0110] If the probability of normal recognition is low as a result of the above judgment (S806), the robot cleaner registers itself as a candidate to request feedback from a mobile device, etc. (S807).

[0111] And, the robot vacuum cleaner is finally excluded from the classification target data (S808).

[0112] Meanwhile, the feedback-related processes used in steps S805 and S807 of FIG. 8 will be described in more detail with reference to FIG. 9 below.

[0113] FIG. 9 is a flowchart detailing a process by which the electronic device illustrated in FIG. 6 requests user feedback.

[0114] The mobile device receives a feedback request from a robot vacuum cleaner, etc. (S900).

[0115] The robot vacuum cleaner first determines whether there is data to receive feedback (S910).

[0116] If there is no target to receive feedback as a result of the above judgment (S910), the feedback-related process is terminated (S960).

[0117] On the other hand, if there is a target to receive feedback as a result of the above judgment (S910), the robot cleaner generates question data asking whether the image and recognition result for the object are misrecognized (S920).

[0118] The mobile device that received the question data generated in step S920 outputs it (S930).

[0119] And, if the user response received through the mobile device is a misrecognition, the mobile device transmits the fact of misrecognition to the robot cleaner as a feedback result (S940).

[0120] On the other hand, if the user response received through the mobile device does not correspond to a misrecognition, the robot cleaner is excluded from the data to be received as feedback (S950).

[0121] A robot vacuum cleaner that receives user feedback through mobile devices or other means will proceed with a retraining process using a master model or other similar device. A related embodiment will be described in more detail below in FIG. 10.

[0122] Figure 10 is a flow chart detailing a process by which the electronic device illustrated in Figure 6 performs retraining to improve misrecognition.

[0123] An electronic device (e.g., a robot vacuum cleaner, etc.) according to one embodiment of the present invention starts learning based on a runtime model (S1000).

[0124] However, if the robot cleaner is in a charging state, the robot cleaner is designed to use the data classification results of the master model and user feedback (S1010).

[0125] Therefore, the robot cleaner determines whether there is learning data corresponding to the master model and user feedback (S1020).

[0126] If the above judgment result (S1020) shows that no learning data exists, the learning of the runtime model is terminated (S1040).

[0127] On the other hand, if learning data exists as a result of the above judgment (S1020), retraining of the runtime model is performed using the data classification results and user feedback of the master model (S1030). Then, training of the runtime model is terminated (S1040).

[0128] FIG. 11 is a block diagram illustrating detailed configurations of an electronic device for adding a new recognition target according to one embodiment of the present invention.

[0129] The aforementioned Figure 6 is a block diagram for improving misrecognition of objects (targets) by electronic devices such as robot vacuum cleaners.

[0130] On the other hand, Fig. 11, which will be described below, is a block diagram for an electronic device such as a robot vacuum cleaner to register a new object.

[0131] An electronic device (1100) according to an embodiment of the present invention includes a sensor (1120), a controller (1110) for image analysis and processing, a user feedback interface (1130), a vacuum cleaner driving module (1140), a target learning DB (1150), an event execution module (1160), and a voice module (1170). In particular, unlike FIG. 6, FIG. 11 additionally includes a vacuum cleaner driving module (1140), an event execution module (1160), and a voice module (1170).

[0132] However, the controller (1110) may be combined with an artificial intelligence accelerator in software or hardware.

[0133] The sensor (1120) photographs various objects in the home and transmits the corresponding image data to the controller (1110).

[0134] The data collection module (1111) inside the controller (1110) collects image data corresponding to an object, the object learning module (1112) performs learning on the object, and the object recognition module (1113) completes recognition of which person or object the object corresponds to.

[0135] In particular, we will explain the process by which a robot vacuum cleaner, an example of an electronic device (1100), registers a new object more accurately through feedback with a mobile device (1180).

[0136] A mobile device (1180) transmits a collection request for a new object through a user feedback interface (1130), and the mobile device (1180) receives a video or image of the new object through the user feedback interface (1130).

[0137] The mobile device (1180) inputs a label for a new object (e.g., person / flower pot / TV / dog, etc.) and also registers an action to be performed when the robot cleaner encounters the object.

[0138] Here, for the sake of explanation, let's assume that the new object is a "flowerpot" and that the corresponding action is set to "evasive driving."

[0139] At this time, when a newly registered flower pot is recognized, the robot cleaner triggers evasive driving under the control of the cleaner driving module (1140).

[0140] Here, for the sake of explanation, let's assume that the new object is a "relative" and that the corresponding action is set to "warning message voice output."

[0141] At this time, when a newly registered relative is recognized, the robot vacuum cleaner outputs a voice message, for example, “Please step aside, as we are cleaning” under the control of the event execution module (1160) and the voice module (1170).

[0142] The process of collecting data for adding a new object by an electronic device (e.g., a robot vacuum cleaner) as exemplified in Fig. 11 will be described in detail later with reference to Fig. 12.

[0143] Figure 12 is a flow chart detailing the process of collecting data by the electronic device illustrated in Figure 11.

[0144] As illustrated in FIG. 12, an electronic device (e.g., a robot vacuum cleaner, etc.) according to an embodiment of the present invention starts data collection (S1210).

[0145] The robot vacuum cleaner transmits an image of a new object to the mobile device (S1220).

[0146] It is determined whether there is a request for data collection from a user using a mobile device (S1230).

[0147] If there is no data collection request as a result of the above judgment (S1230), it is determined whether to terminate data collection again (S1270).

[0148] If the user selects to end data collection as a result of the above judgment (S1270), the data collection process is terminated (S1280).

[0149] On the other hand, if the user does not select to end data collection as a result of the above judgment (S1270), the process returns to step S1220.

[0150] Meanwhile, if a data collection request is made based on the above judgment result (S1230), the mobile device receives a label (e.g., name, etc.) and an event for the new object (S1240). In this specification, an event refers to, for example, an action to be taken when a robot vacuum cleaner encounters a new object.

[0151] The robot vacuum cleaner registers images of new objects and label information received in step S1240 as learning targets (S1250).

[0152] Furthermore, the robot vacuum cleaner registers an event (see step S1240) that must be performed when recognizing a new object (S1260).

[0153] Meanwhile, the process of learning after adding a new object as shown in Fig. 12 will be described in more detail in Fig. 13 below.

[0154] Figure 13 is a flow chart detailing the process by which the electronic device illustrated in Figure 11 performs learning.

[0155] When the feedback process described above in FIG. 12 is terminated, an electronic device (e.g., a robot vacuum cleaner, etc.) according to an embodiment of the present invention starts learning a runtime model (S1310).

[0156] First, the robot vacuum cleaner loads learning data from the target learning database (S1320). Next, the robot vacuum cleaner determines whether learning data exists in the database (S1330).

[0157] If the learning data does not exist as a result of the above judgment (S1330), the runtime model learning is terminated (S1350).

[0158] On the other hand, if learning data exists as a result of the above judgment (S1330), runtime model learning is performed and the result is stored in memory, etc. (S1340). Then, runtime model learning is terminated (S1350).

[0159] After the learning illustrated in Fig. 13 is completed, the process of recognizing a new object will be described later in Fig. 14.

[0160] And, Fig. 14 is a flow chart detailing the process of recognizing a new recognition target after the learning illustrated in Fig. 13 is completed. It is assumed that the robot vacuum cleaner is cleaning and therefore operates according to the runtime model (S1410).

[0161] A robot vacuum cleaner according to one embodiment of the present invention receives image data (S1420).

[0162] It is determined whether the image data received at step S1420 corresponds to an object by the runtime model (S1430).

[0163] If the above judgment result (S1430) is not recognized, the process returns to step S1420.

[0164] On the other hand, if the above judgment result (S1430) is recognized, the robot cleaner searches for an event according to the recognition result and performs the searched event (S1440).

[0165] Furthermore, the robot cleaner determines whether the runtime model execution has ended (S1450).

[0166] For example, if a robot vacuum cleaner stops cleaning and enters a charging station, it will shut down.

[0167] On the other hand, if the robot vacuum cleaner is still cleaning, it returns to step S1420.

[0168] The various embodiments of the present invention described above are summarized as follows.

[0169] The present invention relates to a technology for enabling, for example, a robot vacuum cleaner to learn / evolve on its own.

[0170] In particular, a robot vacuum cleaner capable of recognizing objects (objects, people, etc.) using artificial intelligence improves misrecognition of an artificial intelligence classification model (Classification Neural Network) without communicating with the cloud.

[0171] Additionally, it allows for the addition of an already created artificial intelligence classification model (Classification Neural Network) without communication with the cloud.

[0172] In the present specification, the robot vacuum cleaner is described as being used in a typical home, but the present invention is not necessarily limited thereto.

[0173] In the case of robot vacuum cleaners that use AI to recognize objects (such as people) and people, misrecognition can occur due to the nature of AI classification models (Classification Neural Networks). However, no prior technology has been able to collect field-specific problems arising in the field (user's home) and improve them directly in the field without external transmission.

[0174] Even if a user wanted to add a desired recognition target object to an artificial intelligence classification model (Classification Neural Network) that already had a classification target, there was no way to add it directly to a robot vacuum cleaner in the field (user's home) without external transmission.

[0175] And, as mentioned above, transmitting video data externally may raise potential security / privacy issues.

[0176] In order to solve the various problems of the above-mentioned conventional technology, the robot vacuum cleaner is designed to be able to solve problems without the help of an external server by providing sufficient artificial intelligence computing power.

[0177] Additionally, it is advantageous for the service in terms of privacy to have users (e.g., mobile devices) provide feedback on issues that are difficult for the robot vacuum cleaner to determine on its own, rather than having it transmitted externally to a third party (server operator).

[0178] To implement this, in terms of software, for example, a camera sensor input unit that acquires camera images according to various event conditions, an artificial intelligence classification model recognizer that receives image data and drives a runtime artificial intelligence model that recognizes an object, a database that stores the results recognized by the runtime artificial intelligence model and the corresponding image data (in a new target registration scenario, user-specified image-specific data is stored together with a user label), a learning unit that learns data when charging, a data classifier using a master network, a communication unit that receives feedback on the classification results from the user, a learning unit that trains the runtime model on data on which classification (labeling) has been completed, and in a new target registration scenario, runtime model learning can be performed directly in the learning unit without a data classifier.

[0179] To implement this, hardware-wise, for example, camera sensors, storage devices, general computing devices, and artificial intelligence acceleration computing devices can be used.

[0180] Meanwhile, as described above with reference to FIGS. 6 to 10, one embodiment of the present invention is a technology for improving misrecognition of an artificial intelligence model by learning and updating data collected from a robot vacuum cleaner without transmitting the data to an external source.

[0181] First, a real-time processing AI model capable of recognizing objects in real time during cleaning is called a "runtime model." Images captured by the camera sensor during cleaning are processed in real time by the "runtime model" to provide object recognition results. Furthermore, the object images recognized by the "runtime model" are stored in internal storage for later evaluation and improvement.

[0182] Second, while the robot vacuum is charging, the AI ​​model that evaluates the images saved by the above actions and the results of the "runtime model" (whether there is misrecognition) and classifies (labels) the data to train the "runtime model" is called the "master model." The master model has higher performance than the "runtime model" to enable it to evaluate whether there is misrecognition.

[0183] Third, a high amount of computation is required to perform real-time recognition tasks using a 'runtime model' and result evaluation and data classification using a 'master model', and the present invention processes this using an artificial intelligence accelerator installed inside the robot cleaner without transmitting data to the outside (cloud, edge, etc.).

[0184] Fourth, the ‘runtime model’ is trained and updated based on the results evaluated and classified data by the ‘master model’.

[0185] Fifth, at this time, learning the 'runtime model' requires a high amount of computation, and the present invention processes this by using an artificial intelligence accelerator installed inside the vacuum cleaner without transmitting data to the outside (cloud, edge, etc.).

[0186] Sixth, the evaluation and data classification of the 'master model' and the learning of the 'runtime model' are performed while the vacuum cleaner is charging with sufficient hardware resources (operation, memory, etc.) and a stable power supply.

[0187] Seventh, if the power is cut off or the charger is disconnected while the "Master Model" evaluation and data classification and the "Runtime Model" training are not yet complete, the evaluation, data classification, and training will be halted. The next time the charger is connected, the interrupted evaluation, data classification, and training will be resumed.

[0188] Eighth, even if it is not connected to a charger, if the battery is sufficiently charged and no special tasks are required of the vacuum cleaner, it can perform evaluation and data classification of the 'master model' and learning of the 'runtime model'.

[0189] Ninth, before training the 'runtime model' using the results of the evaluation and data classification of the 'master model', you can receive user feedback on the evaluation results and data classification of the 'master model' and whether to use the data to improve the 'runtime model'.

[0190] Tenth, the 'runtime model' that has completed learning will not misrecognize objects or will have a reduced probability of misrecognition based on what it has learned in the next run.

[0191] More specifically, as illustrated in FIG. 6, an electronic device (600) according to an embodiment of the present invention includes a sensor (e.g., 620 illustrated in FIG. 6) that receives an image of an arbitrary object while operating in a first mode, a memory (e.g., 650 illustrated in FIG. 6) that stores a recognition result of the arbitrary object and the image based on a first model, and an artificial intelligence accelerator (e.g., 610 illustrated in FIG. 6) that determines whether there is a misrecognition of the arbitrary object based on a second model when the electronic device (600) enters a second mode and retrains the first model based on the determination result.

[0192] As described above, the first mode includes, for example, a state in which the electronic device (600) is running without being charged. On the other hand, the second mode includes, for example, a state in which the electronic device (600) is being charged while being powered.

[0193] Furthermore, as described above, the first model means that recognition of the arbitrary object is achieved within a preset frame time (e.g., 1 frame). This corresponds to (a) of the previous Fig. 2.

[0194] On the other hand, the second model is characterized by increasing the resolution of the image (resolution scaling), increasing the depth of the model (depth scaling), or increasing the number of channels, compared to the first model. This corresponds to (b) of the previous Fig. 2.

[0195] Furthermore, the electronic device (600) according to one embodiment of the present invention may additionally receive feedback information related to whether or not an arbitrary object has been misrecognized from a mobile device (660) connected in a communication state.

[0196] Meanwhile, as described above with reference to FIGS. 11 to 14, one embodiment of the present invention is a technology for adding a new recognition target to an artificial intelligence model by learning and updating data collected from a robot vacuum cleaner without transmitting the data to an external source.

[0197] First, the user places an object in front of the robot vacuum cleaner and takes a picture of it to acquire a video image.

[0198] Second, the user can access the robot vacuum cleaner operation user interface for the above tasks through a personal display device (smartphone).

[0199] Third, you can check the image of the object being captured by the robot cleaner and take a picture of it through the robot cleaner operation user interface.

[0200] Fourth, after taking a picture through the robot vacuum cleaner operation user interface, the user registers the object by entering the classification (labeling, name) of the object.

[0201] Fifth, the robot vacuum cleaner operation user interface allows you to input the classification (labeling, name) of the object and then set the robot vacuum cleaner's response (e.g., avoidance before collision, specified voice response, etc.).

[0202] Sixth, the robot vacuum cleaner can be driven (moved) through the robot vacuum cleaner operation user interface.

[0203] Seventh, the object images and classifications (labeling, names) captured above are automatically learned into the 'runtime model' with the user-specified classifications (labels) when the robot cleaner enters the charging station.

[0204] Eighth, when the battery is sufficient and there is no separate operation state (such as cleaning), the user can immediately give learning instructions through the robot vacuum cleaner operation user interface.

[0205] Ninth, real-time recognition tasks and learning using a 'runtime model' require a high amount of computation, and the present invention processes this using an artificial intelligence accelerator installed inside the vacuum cleaner without transmitting data to the outside (cloud, edge, etc.).

[0206] Tenth, if the "runtime model" is powered off, disconnected from the charger, or enters a separate operating state (e.g., cleaning) before learning is complete, the ongoing learning process is interrupted. The interrupted evaluation, data classification, and learning process are resumed the next time the charger is connected or when the user instructs the learning process.

[0207] Eleventh, after learning is completed, if the target is recognized by the 'runtime model' from the next cleaning run, the robot cleaner's response specified by the user (e.g., collision avoidance, voice playback, etc.) is performed.

[0208] An electronic device (e.g., corresponding to mobile device 1180 illustrated in FIG. 11) according to one embodiment of the present invention includes a transceiver that performs a communication connection with a first object and receives an image of a second object captured through a camera of the first object, a touch display that displays the received image of the second object and receives information about a label of the second object and an action of the first object corresponding to the label from a user, and a controller, etc.

[0209] In particular, the controller included in the mobile device (1180) controls the transceiver to transmit the information about the received label and the action to the first object.

[0210] The electronic device mentioned here corresponds to, for example, a mobile device (No. 1180 in FIG. 11), the first object corresponds to, for example, a robot vacuum cleaner (No. 1100 in FIG. 11), and the second object includes any object in the home (a person, a flower pot, a dog, etc.).

[0211] When a first object (e.g., a robot vacuum cleaner) is in a charging state, relearning of the first object is performed based on an image of a second object (e.g., an arbitrary object in the home), the label thereof, and information about the action thereof.

[0212] Furthermore, when the first object is in a charging state, the mobile device (1180) receives a learning result for an arbitrary image from the first object.

[0213] Then, the mobile device (1180) transmits feedback on the learning result for the arbitrary image back to the first object.

[0214] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include applications, hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like, and also include media implemented in the form of carrier waves (e.g., transmission via the Internet).

[0215] Although the data processing device and method according to the embodiments of the present invention have been described in specific embodiments, these are merely examples, and the present invention is not limited thereto, and should be construed to have the broadest scope in accordance with the basic idea disclosed in this specification. Those skilled in the art may implement embodiments not specified by combining or replacing the disclosed embodiments, but this also does not exceed the scope of the present invention. In addition, those skilled in the art may easily modify or alter the disclosed embodiments based on this specification, and it is clear that such modifications or alterations also fall within the scope of the present invention.

[0216] Various embodiments for implementing the present invention have been discussed in detail in the previous table of contents, Best Mode for Carrying Out the Invention, so that those skilled in the art can repeatedly implement the invention without any redundant description.

[0217] The present invention is applicable to robot vacuum cleaners, artificial intelligence technology, etc., and thus its industrial applicability is recognized.

Claims

1. A method for controlling an electronic device including an artificial intelligence accelerator, A step in which the electronic device operates in a first mode; A step of receiving an image of an arbitrary object through a sensor while operating in the first mode; A step of storing the recognition result of the arbitrary object and the image based on the first model; A step in which the electronic device enters a second mode; A step of determining whether there is a misrecognition of the arbitrary object based on the second model; and A step of retraining the first model based on the above judgment result. A method for controlling an electronic device including an artificial intelligence accelerator, characterized in that it includes:

2. In paragraph 1, The above first mode is, Including a state in which the electronic device is running without charging, The second mode is, A method for controlling an electronic device including an artificial intelligence accelerator, characterized in that the electronic device is in a charging state in which power is supplied to the electronic device.

3. In paragraph 1, The above first model is, Recognition of the above arbitrary object is performed within the preset frame time, The second model above is, A control method for an electronic device including an artificial intelligence accelerator, characterized in that, in contrast to the first model, the resolution of the image is increased (resolution scaling), the depth of the model is increased (depth scaling), or the number of channels is increased.

4. In paragraph 3, A method for controlling an electronic device including an artificial intelligence accelerator, wherein the above preset frame corresponds to one frame.

5. In paragraph 1, The above judging step is, A step of receiving feedback information related to whether there is a misrecognition of the arbitrary object from a mobile device connected to the electronic device in a communication state. A method for controlling an electronic device including an artificial intelligence accelerator, characterized in that it further includes.

6. In electronic devices, A sensor that receives an image of an arbitrary object while operating in mode 1; A memory for storing the recognition result of the arbitrary object and the image based on the first model; and When the electronic device enters the second mode, an artificial intelligence accelerator that determines whether there is a misrecognition of the arbitrary object based on the second model and retrains the first model based on the determination result. An electronic device characterized by including:

7. In paragraph 6, The above first mode is, Including a state in which the electronic device is running without charging, The second mode is, An electronic device characterized in that it includes a charging state in which power is supplied to the electronic device.

8. In paragraph 6, The above first model is, Recognition of the above arbitrary object is performed within the preset frame time, The second model above is, An electronic device characterized in that, in contrast to the first model, the resolution of the image is increased (resolution scaling), the depth of the model is increased (depth scaling), or the number of channels is increased.

9. In paragraph 8, An electronic device characterized in that the above preset frame corresponds to 1 frame.

10. In paragraph 6, The above artificial intelligence accelerator, An electronic device characterized in that it receives feedback information related to whether or not there is a misrecognition of the arbitrary object from a mobile device connected to the electronic device in a communication state.

11. In a method for controlling an electronic device, A step of receiving an image of a second object captured through a camera of a first object; A step of displaying an image of the second object received above; A step of receiving information about a label of the second object and an action of the first object corresponding to the label from a user; and A step of transmitting the received label and information about the action to the first object. A method for controlling an electronic device, characterized in that it includes:

12. In paragraph 11, If the above first object is in a charging state, A control method for an electronic device, characterized in that relearning of the first object is performed based on information about the image of the second object, the label, and the action.

13. In paragraph 11, If the above first object is in a charging state, A step of receiving learning results for an arbitrary image from the first object A method for controlling an electronic device, characterized in that it further includes:

14. In paragraph 13, A step of transmitting feedback on the learning result for the above arbitrary image to the first object. A method for controlling an electronic device, characterized in that it further includes:

15. In paragraph 11, The above electronic device includes a mobile device, The first object above includes a robot vacuum cleaner, A method for controlling an electronic device, wherein the second object comprises any object in the home.

16. In electronic devices, A transceiver that performs a communication connection with a first object and receives an image of a second object captured through a camera of the first object; A touch display that displays an image of the received second object and receives information about a label of the second object and an action of the first object corresponding to the label from a user; and controller Includes, The above controller, An electronic device characterized in that it controls the transceiver to transmit information about the received label and the action to the first object.

17. In paragraph 16, If the above first object is in a charging state, An electronic device characterized in that relearning of the first object is performed based on the image of the second object, the label, and information about the action.

18. In paragraph 16, If the above first object is in a charging state, The above controller, An electronic device characterized in that it controls the transceiver to receive learning results for any image from the first object.

19. In paragraph 18, The above controller, An electronic device characterized in that it transmits feedback on the learning result for the arbitrary image to the first object and controls the transceiver.

20. In paragraph 16, The above electronic device includes a mobile device, The first object above includes a robot vacuum cleaner, An electronic device characterized in that the second object includes any object in the home.

Images