Electronic device and control method therefor

Abstract

An electronic device and a control method therefor are disclosed. Particularly, the electronic device control method may comprise the steps of: acquiring a first image including an object; inputting the first image into a first neural network model trained using a plurality of second images related to a plurality of predefined types; acquiring first probability information including a first probability that the object corresponds to a first type from among the plurality of types and a second probability that the object corresponds to a second type from among the plurality of types; on the basis of the first image and the difference between the first probability and the second probability being less than a preset first threshold value, using a plurality of third images corresponding to the first type and a plurality of fourth images corresponding to the second type, so as to acquire second probability information corresponding to the type of the object included in the first image; and identifying the type of the object on the basis of the second probability information.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device capable of identifying a type of object included in an image and a method for controlling the same.

[0002] Recently, advancements in technology for more accurately and efficiently recognizing objects in images using neural network models are accelerating. Specifically, when neural network models are implemented on-device, they can be trained on a server based on a large amount of training data and then delivered to the user's electronic device, enabling object recognition in images captured by the electronic device without the need for a server.

[0003] However, if the distribution between the data used for learning and the input data is different after the neural network model is provided to the electronic device, the accuracy of the neural network model may deteriorate, and the so-called dataset drift problem may occur.

[0004] Therefore, in this case, it is necessary to update the neural network model by performing additional training using data with a new distribution. However, performing additional training is difficult with the resources of the electronic device. Furthermore, having the server perform additional training and redistribute the neural network model to the electronic device can consume significant resources and time. Furthermore, if data with a new distribution is collected by the electronic device, there are limitations on collecting it from the server due to privacy concerns.

[0005] The present disclosure is intended to overcome the limitations of the prior art as described above, and an object of the present disclosure is to provide an electronic device and a control method thereof capable of performing accurate object recognition on an image having a new distribution without additional learning of a neural network model, even when the distributions between data used for learning and data actually input are different.

[0006] According to one or more embodiments of the present disclosure for achieving the above-described object, an electronic device includes a memory storing at least one instruction and a processor executing the at least one instruction, wherein the processor obtains a first image including an object, inputs the first image into a first neural network model trained using a plurality of second images related to a plurality of predefined types, obtains first probability information including a first probability that the object among the plurality of types corresponds to the first type and a second probability that the object among the plurality of types corresponds to the second type, and obtains second probability information corresponding to the type of the object included in the first image by using a plurality of third images corresponding to the first type and a plurality of fourth images corresponding to the second type based on the first image, and identifies the type of the object based on the second probability information.

[0007] The plurality of third images and the plurality of fourth images may be identified as the first type and the second type, respectively, based on the first user input, and the plurality of third images and the plurality of fourth images may be identified and stored in the memory before the first image is acquired.

[0008] The processor obtains the second probability information through a second neural network model, and the second neural network model is trained based on a plurality of fifth images that are different from the plurality of second images, and can obtain the second probability information based on a first similarity between the first image and the plurality of third images and a second similarity between the first image and the plurality of fourth images.

[0009] The electronic device further includes a display, and the first type is a type identified through the first neural network model as corresponding to a first probability having a highest probability of corresponding to the object among the plurality of predefined types, and the second type is a type identified through the first neural network model as corresponding to a second probability having a lower probability of corresponding to the object among the plurality of predefined types than the first probability, and the processor can identify the first type as the type of the object if a difference between the first probability and the second probability is greater than or equal to the first threshold value.

[0010] The processor controls the display to display a user interface including information about the first type and information about the second type if the first probability is less than the second threshold value, receives a second user input for selecting the type of the object through the user interface, and identifies the type of the object based on the second user input.

[0011] The processor obtains the second probability information through a second neural network model if the second user input does not correspond to the first type, and identifies the type of the object based on the second probability information, and the second neural network model is trained based on a plurality of fifth images different from the plurality of second images, and can obtain the second probability information based on the first image.

[0012] The second probability information includes a third probability that the object corresponds to the first type and a fourth probability that the object corresponds to the second type, and the processor can identify the first type as the type of the object if the third probability is greater than the fourth probability.

[0013] The processor controls the display to display the user interface if the third probability is less than the fourth probability, receives a third user input for selecting the type of the object through the user interface, and identifies the type of the object based on the third user input.

[0014] The user interface includes a first item corresponding to the first type and a second item corresponding to the second type, and the processor can determine at least one of a size of the first item, a transparency of the first item, a size of the second item, and a transparency of the second item based on the first probability information.

[0015] The memory stores the plurality of third images and the plurality of fourth images, and the processor can update the plurality of third images stored in the memory based on the first image if the first image corresponds to the first type or the second type.

[0016] The processor can update the plurality of third images stored in the memory by replacing the first image with a third image corresponding to the first type or the second type, if the first image corresponds to the first type or the second type.

[0017] The processor may increase the first threshold value by a preset value if the first probability is greater than the second probability, the third probability is less than the fourth probability, and the third user input corresponds to the second type.

[0018] According to one or more embodiments of the present disclosure for achieving the above-described object, a method for controlling an electronic device includes the steps of: obtaining a first image including an object; inputting the first image into a first neural network model learned using a plurality of second images related to a plurality of predefined types; obtaining first probability information including a first probability that the object among the plurality of types corresponds to a first type and a second probability that the object among the plurality of types corresponds to a second type; obtaining second probability information corresponding to the type of the object included in the first image by using a plurality of third images corresponding to the first type and a plurality of fourth images corresponding to the second type based on the first image, wherein a difference between the first probability and the second probability is less than a first threshold value set in advance; and identifying the type of the object based on the second probability information.

[0019] The plurality of third images and the plurality of fourth images may be identified as the first type and the second type, respectively, based on the first user input, and the plurality of third images and the plurality of fourth images may be identified and stored in the memory before the first image is acquired.

[0020] The first type is a type identified through the first neural network model as corresponding to a first probability having the highest probability of corresponding to the object among the plurality of pre-defined types, and the second type is a type identified through the first neural network model as corresponding to a second probability having a lower probability of corresponding to the object among the plurality of pre-defined types than the first probability, and the control method of the electronic device may further include a step of identifying the first type as the type of the object if a difference between the first probability and the second probability is greater than or equal to the first threshold value.

[0021] The above and other aspects and features of the present disclosure will become more apparent by describing one or more embodiments in detail with reference to the accompanying drawings.

[0022] FIG. 1 is a diagram illustrating an electronic device, a first neural network model, and a second neural network model according to one or more embodiments;

[0023] FIG. 2 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments;

[0024] FIG. 3 is a flowchart illustrating a user interface according to one or more embodiments;

[0025] FIG. 4 is a flowchart illustrating a user interface according to one or more embodiments;

[0026] FIG. 5 is a diagram illustrating a user interface according to one or more embodiments;

[0027] FIG. 6 is a diagram illustrating an update of sample data according to one or more embodiments;

[0028] FIG. 7 is a flowchart illustrating an update of a first threshold value according to one or more embodiments;

[0029] FIG. 8 is a block diagram showing the configuration of an electronic device according to one or more embodiments, and

[0030] FIG. 9 is a block diagram detailing the configuration of an electronic device according to one or more embodiments.

[0031] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0032] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0033] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0034] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0035] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0036] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0037] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0038] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0039] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0040] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0041] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0042] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0043] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0044] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0045] FIG. 1 is a diagram illustrating an electronic device (100), a first neural network model, and a second neural network model according to one or more embodiments. FIG. 2 is a flowchart illustrating a control method of an electronic device (100) according to one or more embodiments, and the following description will be made with reference to FIG. 1 and FIG. 2 together.

[0046] The 'electronic device (100)' may be a device capable of identifying the type of object included in an image. For example, the electronic device (100) may be various types of devices, such as a smartphone, a tablet personal computer (PC), a television (TV), a refrigerator, a washing machine, etc., and there is no particular limitation on the type of the electronic device (100).

[0047] The electronic device (100) can identify the type of object included in an image using at least one of a first neural network model and a second neural network model. As illustrated in FIG. 1, both the first neural network model and the second neural network model can be implemented to be included on-device in the electronic device (100). However, even when at least one of the first neural network model and the second neural network model is included in an external device such as a server, and the electronic device (100) is connected to the external device such as a server, various embodiments described below can be similarly applied.

[0048] A "first neural network model" may be a model trained to identify the type of object contained in an image based on a relatively large amount of data. For example, the first neural network model may be trained based on training data including a plurality of images related to a plurality of predefined types and labels for the plurality of images. For example, the first neural network model may include a neural network such as a convolutional neural network (CNN). For example, the first neural network model may be a general object recognition model. The first neural network model may be referred to as an "object recognition model" or a "vision recognition model," and may also be referred to as a "DNN model (deep neural network model)" to emphasize the difference from the second neural network model. For example, the meaning of the first neural network model may be relatively determined in relation to the second neural network model described below.

[0049] A "second neural network model" can be a model that can identify the type of object contained in an image using a relatively small amount of data. For example, a second neural network model also uses a large amount of data during pre-training, but after pre-training, it can distinguish a specific type of image using only a relatively small amount of data (e.g., five images). For example, a second neural network model can be a neural network model referred to as a "few-shot learning model." For example, methods such as meta-learning or prototype networks can be used to train a second neural network model.

[0050] While the first neural network model boasts relatively high accuracy, it suffers from the problem of reduced accuracy (so-called dataset drift) when the distributions of the training data differ from those of the actual input data. Conversely, the second neural network model may exhibit lower accuracy when fed data with similar distributions to the training data. However, it has the advantage of being able to recognize new types of data more quickly when the distributions of the training data and the input data differ.

[0051] Accordingly, an electronic device (100) according to one or more embodiments may utilize a second neural network model together with a first neural network model. Hereinafter, with reference to FIG. 2, various embodiments for identifying a type of an object included in an image using a second neural network model together with a first neural network model will be described.

[0052] As illustrated in FIG. 2, the electronic device (100) may acquire a first image including an object (S210). In one or more embodiments, the "first image" may be an image that is a target of object identification. For example, the first image may be an image input to a first neural network model and a second neural network model, as illustrated in FIG. 1. The term "first image" may be replaced with terms such as "image data" or "input image."

[0053] Hereinafter, when simply referred to as an "object," this may refer to an object included in the first image. The following description will assume that the first image includes a single object, but various embodiments according to the present disclosure may also be applied to cases where the first image includes multiple objects.

[0054] The electronic device (100) inputs a first image into a first neural network model learned using a plurality of second images related to a plurality of predefined types, thereby obtaining first probability information including a first probability for a first type to which an object corresponds among the plurality of types and a second probability for a second type to which an object corresponds among the plurality of types (S220).

[0055] The 'multiple types' may be all types that are the object classification target by the first neural network model, and may be replaced with terms such as multiple 'classes' or 'domains'. As illustrated in Fig. 1, the 'multiple second images' may be images used for training the first neural network model, and may include multiple images for each of the multiple types. For example, the plurality of second images may include multiple images corresponding to the type 'apple', multiple images corresponding to the type 'peach', etc.

[0056] "First probability information" is probability information obtained by the first neural network model, and may include information regarding the probability that an object included in the first image corresponds to each of a plurality of types. For example, since the first probability information includes information regarding which type of object included in the first image corresponds most likely to one of a plurality of types, the first probability information may also be expressed as the object identification result of the first neural network model.

[0057] In particular, the first probability information may include a 'first probability' for the first type to which the object corresponds among the plurality of types and a 'second probability' for the second type to which the object corresponds among the plurality of types.

[0058] Here, the first type may be the type identified as having the highest probability of corresponding objects among the plurality of types through the first neural network model (Top 1 type), and the second type may be the type identified as having the second highest probability of corresponding objects among the plurality of types through the first neural network model (Top 2 type). For convenience of explanation, the following description assumes that the first probability information includes the first probability for the first type, which is the Top 1 type, and the second probability for the second type, which is the Top 2 type.

[0059] However, one or more embodiments are not limited thereto, and for example, the first probability information may include a first probability for a first type having the highest probability of corresponding objects among the plurality of types, a second probability for a second type having the second highest probability of corresponding objects among the plurality of types, and a third probability for a third type having the third highest probability of corresponding objects among the plurality of types.

[0060] If the difference between the first probability and the second probability is less than the preset first threshold value (S230-Y), the electronic device (100) can obtain second probability information indicating the type of the object included in the first image by using a plurality of third images identified as corresponding to the first type and a plurality of fourth images identified as corresponding to the second type according to the first user input through the second neural network model (S240).

[0061] The 'first threshold' may be a preset value for evaluating the reliability of the first neural network model based on the difference between the first probability and the second probability, and may be changed by, for example, a developer or a user. For example, if the first threshold is 0.3, the first probability is 0.8, and the second probability is 0.1, and the difference between the first probability and the second probability is greater than the first threshold, it can be said that the first neural network model clearly classifies the object into a specific type, and it can be said that the reliability of the object identification result of the first neural network model is high.

[0062] If the difference between the first probability and the second probability is less than the first threshold, such as when the first threshold is 0.3, the first probability is 0.5, and the second probability is 0.4, it can be said that the first neural network model cannot classify the object into a specific type, and the reliability of the first neural network model is relatively low.

[0063] Accordingly, if the difference between the first probability and the second probability is less than a preset first threshold, the electronic device (100) can identify the object using the second neural network model, rather than relying solely on the first neural network model. For example, as illustrated in FIG. 1, the electronic device (100) can obtain second probability information indicating the type of the object included in the first image by inputting the first image into the second neural network model and performing inference on the type of the object using sample data.

[0064] If the first probability information includes a first probability for a first type having the highest probability of corresponding an object among the plurality of types, a second probability for a second type having the second highest probability of corresponding an object among the plurality of types, and a third probability for a third type having the third highest probability of corresponding an object among the plurality of types, the processor may determine whether to obtain the second probability information through the second neural network model based on whether a difference between the first probability and the second probability is less than a preset first threshold value, or may determine whether to obtain the second probability information through the second neural network model by comparing the difference between the first probability and the second probability and between the first probability and the third probability, respectively, with preset threshold values.

[0065] The 'second probability information' is probability information obtained by the second neural network model, and may include information on the probability that an object included in the first image corresponds to a specific type. For example, since the second probability information includes information on which type among multiple types the object included in the first image is most likely to correspond to, the second probability information may also be expressed as an object identification result of the second neural network model. In particular, the second probability information may include a 'third probability' that the object corresponds to the first type and a 'fourth probability' that the object corresponds to the second type.

[0066] "Sample data" refers to a set of images identified by a user as corresponding to a specific type. The sample data may be stored in the memory of the electronic device (100) and may be updated according to various embodiments described below with reference to FIG. 6. Images identified by the user as corresponding to a specific type and stored in the memory of the electronic device (100) before the first image to be subjected to object identification is acquired may be used to identify an object included in the first image. Since the sample data includes images identified by the user, it may also be referred to as "user data."

[0067] For example, the second neural network model may use the images identified by the user as corresponding to each of the types included in the first probability information in the sample data to identify the object included in the first image. For example, as illustrated in FIG. 1, before the first image is acquired, a plurality of third images identified as corresponding to the first type and a plurality of fourth images identified as corresponding to the second type may be included in the sample data and stored in the memory of the electronic device (100).

[0068] The number of the plurality of third images and the plurality of fourth images may be preset (e.g., 5), but there is no limitation on the number. In addition, the number of the plurality of third images and the plurality of fourth images to be stored is not limited, but a certain number of images having the highest probability of corresponding to the first type or the second type among them may be used for inference of the second neural network model. When the number of the plurality of third images and the plurality of fourth images is preset, if the number of images less than the specified number is stored in the memory, the electronic device (100) may not use the second neural network model.

[0069] The second neural network model can obtain second probability information based on the first similarity between the first image and the plurality of third images and the second similarity between the first image and the plurality of fourth images.

[0070] For example, the second neural network model can extract and compare feature vectors of the first image and the plurality of third images, respectively, to calculate and / or obtain a 'first similarity' as to whether the first image is similar to the plurality of third images. In addition, the second neural network model can extract and compare feature vectors of the first image and the plurality of fourth images, respectively, to calculate a 'second similarity' as to whether the first image is similar to the plurality of fourth images. In addition, the electronic device (100) can calculate a third probability that the object corresponds to the first type and a fourth probability that the object corresponds to the second type based on the first similarity and the second similarity.

[0071] As illustrated in Fig. 1, the second neural network model can be trained based on a plurality of fifth images that are different from the plurality of second images used to train the first neural network model. For example, if the plurality of second images used to train the first neural network model include a plurality of images corresponding to the type of 'apple' and a plurality of images corresponding to the type of 'peach', the second neural network model can be trained using images corresponding to other types, such as 'airplane' and 'train', rather than using the images corresponding to the types of 'apple' and 'peach'. This is to prevent overfitting and to enable rapid recognition of new types of data.

[0072] After the second neural network model is trained using images corresponding to 'airplane', 'train', etc., when an image corresponding to the type of 'apple' or 'peach' is identified by the user of the electronic device (100), the electronic device (100) can store the identified images as sample data (multiple third images or multiple fourth images), and when an image for 'apple' or 'peach' is input as an identification target, the second probability information indicating the type of object included in the input image can be obtained through the second neural network model (using the multiple third images or multiple fourth images).

[0073] Once the second probability information is acquired, the electronic device (100) can identify the type of object based on the second probability information (S250). For example, if the difference between the first probability and the second probability is less than a preset first threshold, this may indicate that the reliability of the first neural network model is low. Therefore, the electronic device (100) can identify the type of object included in the first image based on the second probability information acquired through the second neural network model, which is a few-shot learning model.

[0074] In one or more embodiments, if the third probability included in the second probability information is greater than or equal to the fourth probability, the electronic device (100) may identify the first type as the type of object. If the third probability included in the second probability information is less than the fourth probability, the electronic device (100) may identify the second type as the type of object.

[0075] In one or more embodiments, the electronic device (100) may identify the type of the object based on the comparison result of the first probability information and the second probability information. For example, if the difference between the first probability and the second probability is less than a preset first threshold, it indicates that the reliability of the first neural network model is evaluated as relatively low, but the type of the object may also be identified by referring to the first probability information together with the second probability information. For example, if the first probability is greater than or equal to the second probability and the third probability is greater than or equal to the fourth probability, the electronic device (100) may identify the first type as the type of the object. If the first probability is less than the second probability and the third probability is less than the fourth probability, the electronic device (100) may identify the second type as the type of the object.

[0076] According to the embodiments described above with reference to FIGS. 1 and 2, when the reliability of the object identification result using the object recognition model (first neural network model) using DNN is not high, the electronic device (100) can obtain a relatively high-accuracy object identification result using images identified according to user input through a few-shot learning model (second neural network model).

[0077] In particular, the electronic device (100) can perform improved and accurate object recognition for images with a new distribution using a few-shot learning model without additional learning for the DNN model even in an on-device environment where additional learning is difficult when the distribution between the data used for learning and the data input is different.

[0078] FIG. 3 is a flowchart illustrating a user interface according to one or more embodiments.

[0079] In the above, the case where the difference between the first probability for the first type to which an object corresponds among the plurality of types and the second probability for the second type to which an object corresponds among the plurality of types is less than a preset first threshold value (S230-Y, S310-N) is described, but in FIG. 3, one or more embodiments related to the case where the difference between the first probability and the second probability is equal to or greater than the first threshold value are described.

[0080] If the difference between the first probability and the second probability is greater than or equal to a first threshold value (S310-Y) and the first probability is greater than or equal to a preset second threshold value (S320-Y), the electronic device (100) can identify the first type as the type of the object (S360).

[0081] Here, the 'second threshold value' may be a preset value for evaluating the reliability of the first neural network model based on the first probability, and may be changed by the developer or user. For example, if the first threshold value and the second threshold value are 0.3 and 0.75, respectively, and the first probability is 0.8 and the second probability is 0.1, and the difference between the first probability and the second probability is greater than or equal to the first threshold value, but the first probability itself is less than the second threshold value, it can be said that the first neural network model clearly classifies the object as the first type. Therefore, since it can be said that the reliability of the first neural network model is relatively very high, the electronic device (100) can confirm that the first type is identified as the type of the object.

[0082] If the difference between the first probability and the second probability is greater than or equal to the first threshold value (S310-Y) and the first probability is less than the second threshold value (S320-N), the electronic device (100) can identify the type of object based on the user input.

[0083] For example, if the difference between the first probability and the second probability is greater than or equal to the first threshold but the first probability is less than the second threshold, such as when the first threshold and the second threshold are 0.3 and 0.75, respectively, and the first probability is 0.6 and the second probability is 0.2, it is difficult to conclude that the first neural network model classifies the object as the first type. For example, since it may not be the case that the reliability of the first neural network model is relatively very high, the electronic device (100) may identify the type of the object based on the user input without confirming the first type as the type of the object.

[0084] For example, if the difference between the first probability and the second probability is greater than or equal to the first threshold value (S310-Y) and the first probability is less than the second threshold value (S320-N), the electronic device (100) may display a user interface including information about the first type and information about the second type (S330).

[0085] For example, the user interface may include identification information for a first type and identification information for a second type. Additionally, the user interface may include a first probability and a second probability. The information for the first type and the information for the second type may be provided in the form of at least one of text and an image.

[0086] For example, the electronic device (100) may display information within the user interface that the first type of name is 'apple' and information that the second type of name is 'peach'. In addition, the electronic device (100) may also display information within the user interface that the probability that the object corresponds to 'apple' is 0.6 and information that the probability that the object corresponds to 'peach' is 0.2. The user interface will be described in more detail with reference to FIG. 5.

[0087] The electronic device (100) may receive a second user input for selecting the type of object through the user interface (S340). For example, if information that the name of the first type is 'apple' and information that the name of the second type is 'peach' are displayed on the user interface, the electronic device (100) may receive a second user input corresponding to one of the following: user feedback that the object included in the first image is an apple, user feedback that the object included in the first image is a peach, and user feedback that the object included in the first image is neither an apple nor a peach.

[0088] If the second user input corresponds to the first type (S350-Y), the electronic device (100) can identify the first type as the type of object (S360). In the example described above, if user feedback is received that the object included in the first image is an apple, the electronic device (100) can identify the type of object as an apple.

[0089] For example, if, as previously mentioned, the first probability information indicates that the first type is the type with the highest probability, and if the user feedback also indicates that the object is the first type, then it can be said that the first probability information output by the first neural network model and the user feedback match, and therefore the electronic device (100) can identify the first type as the type of the object without an additional process.

[0090] If the second user input corresponds to the second type (S350-N), the electronic device (100) can obtain second probability information through the second neural network model (S370) and identify the type of the object based on the second probability information (S380). In the example described above, if user feedback is received that the object included in the first image is a peach, the electronic device (100) does not identify the type of the object as a peach, but as described above with reference to FIGS. 1 and 2, obtains second probability information through the second neural network model and can identify the type of the object based on the second probability information.

[0091] For example, if, as previously mentioned, the first probability information indicates that the first type is the type with the highest probability, and the user feedback indicates that the object is the second type, then it can be said that the first probability information output by the first neural network model does not match the user feedback, and therefore the electronic device (100) can use the second neural network model for an additional verification process. An embodiment of identifying an object using the second probability information obtained through the second neural network model will be described in more detail with reference to FIG. 4. Of course, depending on the embodiment, the second type can be identified as the type of the object as indicated by the user feedback, depending on the user input.

[0092] According to the embodiments described above with reference to FIG. 3, even when the reliability of the object identification result using the first neural network model is relatively high, the electronic device (100) can obtain a more accurate object identification result using a user input or a second neural network model for additional verification.

[0093] FIG. 4 is a flowchart illustrating a user interface according to one or more embodiments.

[0094] In the above, an embodiment of identifying an object type by receiving a second user input through a user interface and using the first probability information and the second user input together when first probability information is acquired through a first neural network model has been described. In the following, an embodiment of identifying an object type by receiving a third user input through a user interface and using the second probability information and the third user input together when second probability information is acquired through a second neural network model will be described.

[0095] First, cases where the second neural network model is used may include cases where the reliability of the object identification result using the first neural network model is not high, and cases where it is difficult to confirm the object identification result even when the first neural network model and user input are used. In addition, when the amount of sample data is a certain amount or more, when the number of sample data updates is a certain number or more, or when the accuracy of the second neural network model is determined to be higher than that of the first neural network model, the second neural network model may be used regardless of the object identification result according to the first neural network model and user input. An embodiment related to comparing the reliability (or accuracy) of the first neural network model and the reliability (or accuracy) of the second neural network model will be described below with reference to FIG. 7.

[0096] As described above, the electronic device (100) can obtain second probability information including information on a third probability that the object corresponds to the first type and a fourth probability that the object corresponds to the second type through the second neural network model (S410), and the second probability information can include a third probability that the object corresponds to the first type and a fourth probability that the object corresponds to the second type.

[0097] If the third probability is greater than the fourth probability (S420-Y), the electronic device (100) can identify the first type as the type of the object (S430). For example, if the first probability information indicates that the first type is the type with the highest probability as previously mentioned, and the second probability information also indicates that the first type is the type with the highest probability, then it can be said that the first probability information output by the first neural network model and the second probability information output by the second neural network model are identical, and therefore the electronic device (100) can identify the first type as the type of the object.

[0098] If the third probability is less than the fourth probability (S420-N), the electronic device (100) can display a user interface including information about the third probability and information about the fourth probability (S440). The electronic device (100) can receive a third user input for selecting the type of object through the user interface (S450). In addition, the electronic device (100) can identify the type of object based on the third user input (S460).

[0099] For example, if, as previously mentioned, the first probability information indicates that the first type is the type with the highest probability, and the second probability information indicates that the second type is the type with the highest probability, then it can be said that the second probability information output by the second neural network model does not match, and therefore the electronic device (100) can receive user input for an additional verification process.

[0100] According to the embodiments described above with reference to FIG. 4, the electronic device (100) can obtain relatively high-accuracy object identification results by using the second neural network model, and can obtain even higher-accuracy object identification results by performing additional verification using user input together with the second neural network model.

[0101] In addition to the embodiments described above, various embodiments of cross-validating object identification results using at least two of the first neural network model, the second neural network model, and user input can be implemented by the present disclosure.

[0102] Although the above has described embodiments in which the input data is an image (i.e., a first image) and the electronic device (100) uses a first neural network model and a second neural network model to identify the type of object included in the image, one or more embodiments according to the present disclosure can be similarly applied to voice recognition.

[0103] For example, the electronic device (100) may acquire a speech signal and input the speech signal into a DNN model trained to identify text corresponding to the speech signal, thereby acquiring a first probability that the speech signal corresponds to a first text and a second probability that the speech signal corresponds to a second text. If the difference between the first probability and the second probability is less than a preset first threshold, the electronic device (100) may acquire second probability information indicating the type of an object included in the speech signal by using a sample speech signal identified as corresponding to the first text and a sample speech signal identified as corresponding to the second text based on a user input through a few-shot learning model. In addition, the electronic device (100) may also identify a text corresponding to the speech signal based on the two probability information.

[0104] FIG. 5 is a diagram illustrating a user interface according to one or more embodiments.

[0105] The user interface may include a first item corresponding to a first type and a second item corresponding to a second type. As illustrated in FIG. 5, the user interface may include a first item in the shape of an apple corresponding to the first type, "apple," and a second item in the shape of a peach corresponding to the second type, "peach."

[0106] The electronic device (100) can determine at least one of the size and transparency of the first item and the second item based on the magnitude of the first probability and the magnitude of the second probability. In addition, the electronic device (100) can display a user interface based on at least one of the determined size and transparency of the first item and the second item.

[0107] For example, if the first probability is 0.7 and the second probability is 0.2, the electronic device (100) may divide the size of the user interface, 600*400 (hereinafter, width*height), in proportion to the sizes of the first and second probabilities, to determine the first item as having a size of 467*311 and the second item as having a size of 133*89. In addition, the electronic device (100) may determine the transparency of the second item as 100-0.2 / 0.7*100=71% to reflect the sizes of the first and second probabilities. At this time, the transparency of the first item and the background of the user interface may be maintained at 0%.

[0108] In addition, the electronic device (100) may display the location, color, etc. of the first item and the second item within the user interface by distinguishing them, may apply a blur effect, etc. to at least one of the first item and the second item, and may also display the first probability and the second probability together with the first item and the second item.

[0109] The electronic device (100) may display the size and transparency of the first item and the second item in the same manner and may not display the first probability and second probability information so that the user can select the type of object without referring to the probability information provided through the first neural network model or the second neural network model.

[0110] According to the embodiments described above with reference to FIG. 5, the electronic device (100) can provide probability information obtained through the first neural network model or the second neural network model in an intuitive manner by visualizing it through a user interface, and can obtain a highly accurate object identification result by using user input received through the user interface.

[0111] FIG. 6 is a diagram illustrating an update of sample data according to one or more embodiments.

[0112] As described above, sample data refers to a set of images identified by the user as corresponding to a specific type, and may be stored in the memory of the electronic device (100). Images corresponding to various types may be stored in the sample data, but for convenience of explanation, a plurality of third images identified by the user as corresponding to the first type and a plurality of fourth images identified as corresponding to the second type will be described below.

[0113] Before the first image that is the target of object identification is acquired, the plurality of third images and the plurality of fourth images can be stored in a memory (e.g., a memory of the electronic device (100) or a memory of an external device), and when the first image is identified as corresponding to the first type or the second type, the plurality of third images or the plurality of fourth images stored in the memory can be updated.

[0114] For example, if the first image is identified as corresponding to the first type, the electronic device (100) may update a plurality of third images stored in the memory based on the first image. For example, if a third user input is received through the user interface as a result of providing a user interface, and if the third user input corresponds to the first type, the electronic device (100) may update a plurality of third images stored in the memory based on the first image.

[0115] Updating the plurality of third images based on the first image may mean replacing one of the plurality of third images with the first image. For example, the memory may store information regarding five probabilities that each of the plurality of third images corresponds to the first type. If the third probability is greater than one of the five probabilities, the electronic device (100) may replace the third image corresponding to one of the five probabilities with the first image and store it in the memory, thereby updating the plurality of third images stored in the memory.

[0116] The memory may store information about weights obtained based on fifth probabilities that each of the plurality of third images corresponds to the first type, and information about the number of days elapsed from the date on which each of the plurality of third images was obtained. Here, the information about the weights may be determined based on at least one of first probability information obtained through the first neural network model, second probability information obtained through the second neural network model, and user input entered through the user interface.

[0117] As in the example of FIG. 6, when the weights of the plurality of third images are 0.45, 0.85, 0.23, 0.57, and 0.12, the electronic device (100) can replace the third image with the smallest weight of 0.12 with the first image. In addition, the electronic device (100) can replace the third image with the oldest elapsed time, that is, the third image with an elapsed time of 30 days, with the first image. If there is a third image with an elapsed time of a preset threshold number of days (e.g., 100 days) or more among the plurality of third images, the third image with an elapsed time of a preset threshold number of days or more can be replaced with the first image regardless of the weight.

[0118] According to the embodiments described above with reference to FIG. 6, the electronic device (100) can further improve the accuracy of object identification of the second neural network model by updating sample data using images in which the type of object is identified according to user input.

[0119] FIG. 7 is a flowchart illustrating an update of a first threshold value according to one or more embodiments.

[0120] As mentioned above, the first threshold is a preset value used to evaluate the reliability of the first neural network model based on the difference between the first probability and the second probability, and can be changed by the developer or user. Furthermore, the first threshold can be changed based on collected data.

[0121] The electronic device (100) can calculate a first intermediate value representing a result of subtracting the accuracy of the first neural network model from the accuracy of the second neural network model based on the accuracy data collected while the first threshold value is A (S710). In addition, the electronic device (100) can calculate a second intermediate value representing a result of subtracting the accuracy of the first neural network model from the accuracy of the second neural network model based on the accuracy data collected while the first threshold value is B (S720).

[0122] Here, 'accuracy data' refers to information indicating which neural network model among the first neural network model and the second neural network model output probability information corresponding to the identification result of the object included in the final input image.

[0123] The 'first median' may represent the result of subtracting the accuracy of the first neural network model from the accuracy of the second neural network model during the first period when the first threshold is set to A, so a larger first median may mean that the second neural network model was more accurate than the first neural network model during the first threshold is set to A. The 'second median' may represent the result of subtracting the accuracy of the first neural network model from the accuracy of the second neural network model during the second period when the first threshold is set to B, so a larger second median may mean that the second neural network model was more accurate than the first neural network model during the first threshold is set to B.

[0124] If the first intermediate value is greater than or equal to the second intermediate value (S730-Y), the electronic device (100) may set A as the first threshold value (S740). For example, if the first intermediate value is greater than the second intermediate value, this may mean that the accuracy of the second neural network model is higher in the first period in which the first threshold value is set to A than in the second period in which the first threshold value is set to B. Accordingly, the electronic device (100) may set A, which is the first threshold value set during the first period, as the first threshold value to identify the object based on the accuracy of the second neural network model, and thereafter identify the object included in the input image.

[0125] FIG. 7 illustrates the case where A is set as the first threshold value when the first intermediate value is equal to the second intermediate value. However, whether A or B is set as the first threshold value when the first intermediate value is equal to the second intermediate value may vary depending on the embodiment.

[0126] If the first intermediate value is less than the second intermediate value (S730-N), the electronic device (100) may set B as the first threshold value (S750). For example, if the first intermediate value is less than the second intermediate value, this may mean that the accuracy of the second neural network model is higher in the second period in which the first threshold value is set to B than in the first period in which the first threshold value is set to A. Accordingly, the electronic device (100) may set B, which is the first threshold value set during the second period, as the first threshold value to identify the object based on the accuracy of the second neural network model, and thereafter identify the object included in the input image.

[0127] In the above, an embodiment in which the first threshold value is changed based on the accuracy data collected over a certain period of time has been described. However, depending on the embodiment, the first threshold value may also be changed based on a single input image. For example, if the object identification result of the first neural network model and the object identification result of the second neural network model conflict with each other, the electronic device (100) may change the first threshold value so that an identification result for the object can be obtained by using the neural network model that outputs an object identification result that matches the user input.

[0128] For example, when the first image described in FIGS. 1 to 6 is acquired, if the first probability acquired through the first neural network model is greater than the second probability (i.e., the object identification result of the first neural network model indicates that the first type is Top 1), the third probability is less than the fourth probability (i.e., the object identification result of the second neural network model indicates that the second type is Top 1), and the third user input corresponds to the second type (i.e., the third user input indicates that the first type is Top 1), the electronic device (100) may increase the first threshold value by a preset value.

[0129] For example, in this case, since the reliability of the second neural network model is higher than that of the first neural network model, the first threshold value may be increased by a preset value so that the reliability of the first neural network model of the electronic device (100) can be judged more strictly. For example, if the first threshold value is 0.3 and the reliability of the second neural network model is identified as higher than that of the first neural network model, the electronic device (100) may upwardly adjust the first threshold value to identify the type of object using the first neural network model when the difference between the first probability and the second probability is 0.31 or more.

[0130] According to the embodiments described above with reference to FIG. 7, the electronic device (100) can obtain more accurate object identification results in the future by comparing the accuracy of the first neural network model with the accuracy of the second neural network model and changing the first threshold value that serves as a basis for calculating the reliability of the first neural network model.

[0131] The method for controlling an electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the method for controlling an electronic device (100) may be stored and provided in a non-transitory computer readable medium.

[0132] For example, in a non-transitory computer-readable recording medium including a program for executing a method for controlling an electronic device (100), the method for controlling an electronic device (100) may include the steps of: obtaining a first image including an object; inputting the first image into a first neural network model trained using a plurality of second images related to a plurality of predefined types, thereby obtaining first probability information including a first probability for a first type to which the object corresponds among the plurality of types and a second probability for a second type to which the object corresponds among the plurality of types; if a difference between the first probability and the second probability is less than a first threshold value set in advance, obtaining second probability information indicating a type of the object included in the first image using a plurality of third images identified as corresponding to the first type and a plurality of fourth images identified as corresponding to the second type according to a first user input through the second neural network model; and identifying the type of the object based on the second probability information.

[0133] In the above, a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100) have been briefly described, but this is only to omit redundant descriptions, and it goes without saying that various embodiments of the electronic device (100) can also be applied to a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100).

[0134] FIG. 8 is a block diagram showing the configuration of an electronic device (100) according to one or more embodiments, and FIG. 9 is a block diagram showing in detail the configuration of an electronic device (100) according to one or more embodiments.

[0135] As illustrated in FIG. 8, an electronic device (100) according to one or more embodiments of the present disclosure includes a memory (110) and a processor (120). In addition, as illustrated in FIG. 9, an electronic device (100) according to one or more embodiments of the present disclosure may further include a communication unit or communication interface (130), an input unit or input interface (140), and an output unit or output interface (150). However, the configurations illustrated in FIGS. 8 and 9 are merely exemplary, and the embodiments are not limited thereto.

[0136] At least one instruction regarding the electronic device (100) may be stored in the memory (110). In addition, an O / S (Operating System) for driving the electronic device (100) may be stored in the memory (110). In addition, various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110). In addition, the memory (110) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0137] For example, various software modules for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110), and the processor (120) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (120), and data reading / recording / modifying / deleting / updating, etc. may be performed by the processor (120).

[0138] In one or more embodiments, the term memory (110) may be used to mean memory (110), ROM, RAM within the processor (120), or a memory card (e.g., micro SD card, memory stick) mounted on the electronic device (100).

[0139] In one or more embodiments, the memory (110) may store a first image, first probability information, second probability information, sample data, etc. In addition, the memory (110) may store data for a first neural network model, data for a second neural network model, a first threshold value, a second threshold value, information for a user interface, etc.

[0140] In addition, various information necessary within the scope of achieving the purpose of one or more embodiments may be stored in the memory (110), and the information stored in the memory (110) may be updated as received from an external device or input by a user.

[0141] The processor (120) controls the overall operation of the electronic device (100). For example, the processor (120) is connected to the configuration of the electronic device (100) including a memory (110), a communication unit (130), an input unit (140), and an output unit (150), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110) as described above.

[0142] The processor (120) may be implemented in various ways. For example, the processor (120) may be implemented as at least one of an application-specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (DSP). The processor (120) may include a central processing unit (CPU), a graphic processing unit (GPU), and a microprocessor unit (MPU).

[0143] In one or more embodiments, the processor (120) may obtain a first image including an object. The processor (120) may input the first image into a first neural network model trained using a plurality of second images related to a plurality of predefined types, thereby obtaining first probability information including a first probability for a first type to which the object corresponds among the plurality of types and a second probability for a second type to which the object corresponds among the plurality of types. If a difference between the first probability and the second probability is less than a preset first threshold, the processor (120) may obtain second probability information indicating a type of the object included in the first image using a plurality of third images identified as corresponding to the first type and a plurality of fourth images identified as corresponding to the second type according to the first user input through the second neural network model. In addition, the processor (120) may identify the type of the object based on the second probability information.

[0144] Various embodiments according to the present disclosure based on the control of the processor (120) have been described above with reference to FIGS. 1 to 7, so a duplicate description will be omitted.

[0145] The communication unit (130) includes a circuit and can communicate with an external device. For example, the processor (120) can receive various data or information from an external device connected via the communication unit (130) and can also transmit various data or information to the external device.

[0146] The communication unit (130) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and an Ultra-Wide Band (UWB) module. Specifically, the WiFi module and the Bluetooth module may each perform communication in the WiFi or Bluetooth manner. When using a WiFi module or a Bluetooth module, various connection information, such as an SSID, may be first transmitted and received, and then communication may be established using this, after which various pieces of information may be transmitted and received.

[0147] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc. And, the NFC module can perform communication in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc. In addition, the UWB module can accurately measure ToA (Time of Arrival), which is the time it takes for a pulse to reach a target, and AoA (Ange of Arrival), which is the pulse arrival angle at the transmitting device, through communication between UWB antennas, and accordingly, precise distance and location recognition is possible within an error range of several tens of centimeters indoors.

[0148] In one or more embodiments, the processor (120) may receive information about a first neural network model and information about a second neural network model from an external device through the communication unit (130). The processor (120) may obtain a first image by receiving a first image from the external device through the communication unit (130). When at least one of the first neural network model and the second neural network model is included in the external device, the electronic device (100) may control the communication unit (130) to transmit data about the first image to the external device, and may receive at least one of the first probability information and the second probability information from the external device through the communication unit (130).

[0149] The input unit (140) includes a circuit, and the processor (120) can receive a user command to control the operation of the electronic device (100) through the input unit (140). For example, the input unit (140) can be configured with components such as a microphone, a camera, and a remote control signal receiving unit. In addition, the input unit (140) can also be implemented in a form included in a display as a touch screen. In particular, the microphone can receive a voice signal and convert the received voice signal into an electrical signal.

[0150] The microphone can acquire a signal for a sound or voice generated from outside the electronic device (100). For example, the microphone can acquire a vibration corresponding to a sound or voice generated from outside the electronic device (100) and convert the acquired vibration into an electrical signal.

[0151] In particular, a microphone according to one or more embodiments may acquire a voice signal for a user's voice generated by the user's speech. The acquired signal may be converted into a digital signal and stored in a memory (110). The microphone may include an analog-to-digital converter (A / D converter) and may operate in conjunction with an A / D converter located external to the microphone.

[0152] In one or more embodiments, the processor (120) may receive user input via a microphone. For example, the processor (120) may receive a user's voice, in the form of a voice signal, via the microphone to select an object type. Furthermore, the processor (120) may identify the type of object corresponding to the user's voice using a learned voice recognition model and natural language understanding model.

[0153] A camera can acquire an image of at least one object. For example, the camera may include an image sensor, which can convert light entering through a lens into an electrical image signal.

[0154] In one or more embodiments, the processor (120) may acquire a first image through a camera, and may also acquire sample data such as a plurality of third images and a plurality of fourth images.

[0155] The output unit (150) includes a circuit, and the processor (120) can output various functions that the electronic device (100) can perform through the output unit (150). In addition, the output unit (150) can include at least one of a display, a speaker, and an indicator.

[0156] The display can output image data under the control of the processor (120). For example, the display can output an image previously stored in the memory (110) under the control of the processor (120). In particular, the display according to one or more embodiments of the present disclosure can display a user interface stored in the memory (110). The display can be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), etc., and in some cases, the display can also be implemented as a flexible display, a transparent display, etc. However, the display is not limited to a specific type.

[0157] The speaker can output audio data under the control of the processor (120).

[0158] The indicator may be lit under the control of the processor (120). For example, the indicator may be lit in various colors under the control of the processor (120). For example, the indicator may be implemented as a light emitting diode (LED), a liquid crystal display panel (LCD), a vacuum fluorescent display (VFD), etc., but is not limited thereto.

[0159] In one or more embodiments, the processor (120) may control a display to display a user interface, convert information about the first type and information about the second type into voice signals, and output the information about the first type and information about the second type through a speaker. For example, the processor (120) may convert information about the first type and information about the second type into voice signals using a learned voice synthesis model.

[0160] An operation (function) related to artificial intelligence according to one or more embodiments is operated through a processor (120) and memory (110) of an electronic device (100).

[0161] The processor (120) may be composed of one or more processors (120). At this time, the one or more processors (120) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processors (120) described above.

[0162] The CPU is a general-purpose processor (120) capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for more efficient execution of complex programs. The CPU is advantageous in a serial processing method, enabling organic linking of previous and subsequent calculation results through sequential calculations. The general-purpose processor (120) is not limited to the examples described above, except in cases where it is specifically referred to as a CPU.

[0163] A GPU is a processor (120) for large-scale operations such as floating-point operations used in graphic processing, and can perform large-scale operations in parallel by integrating a relatively large number of cores. In particular, a GPU may be advantageous compared to a CPU in parallel processing methods such as convolution operations. In addition, a GPU may be used as a co-processor (120) to supplement the functions of a CPU. The processor (120) for large-scale operations is not limited to the examples described above, except in cases where it is specified as the aforementioned GPU.

[0164] An NPU is a processor (120) specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon). Since an NPU is designed specifically according to the required specifications of a company, it has less freedom than a CPU or GPU, but it can more efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor (120) specialized in artificial intelligence operations, an NPU can be implemented in various forms such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), a Vision Processing Unit (VPU), etc. The artificial intelligence processor (120) is not limited to the above-described examples, except in cases where it is specified as an NPU as described above.

[0165] Additionally, one or more processors (120) may be implemented as a System on Chip (SoC). In addition to one or more processors (120), the SoC may further include a memory (110) and a network interface such as a bus for data communication between the processor (120) and the memory (110).

[0166] When a plurality of processors (120) are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the processors (120) among the plurality of processors (120). For example, the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors (120). However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor (120).

[0167] Additionally, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in one processor (120). For example, the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multiple cores included in the processor (120).

[0168] One or more processors (120) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (110). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0169] Here, "creation through learning" may refer to the creation of predefined behavioral rules or artificial intelligence models with desired characteristics by applying a learning algorithm to a large number of learning data sets. This learning may be performed on the device itself, where the artificial intelligence according to the present disclosure is implemented, or through a separate server / system.

[0170] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0171] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0172] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0173] According to one or more embodiments, the methods according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory (110) of a manufacturer's server, an application store's server, or a relay server.

[0174] Each of the components (e.g., modules or programs) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration.

[0175] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0176] The terms "part" or "module" as used herein include a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally formed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0177] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored in the storage medium and operate according to the called instructions, and may include an electronic device (e.g., an electronic device (100)) according to one or more embodiments.

[0178] When the above command is executed by the processor (120), the processor (120) may perform an operation corresponding to the command directly or by using other components under the control of the processor (120). The command may include code generated or executed by a compiler or interpreter.

[0179] Although the embodiments have been described with reference to the drawings, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope defined by the following claims and their equivalents.

Claims

1. In electronic devices, a memory that stores at least one instruction; and a processor for executing at least one instruction; The above processor, Obtain a first image containing an object, Inputting the first image into a first neural network model learned using a plurality of second images related to a plurality of predefined types, Obtain first probability information including a first probability that the object among the plurality of types corresponds to a first type and a second probability that the object among the plurality of types corresponds to a second type, By using a plurality of third images corresponding to the first type and a plurality of fourth images corresponding to the second type based on the first image and the difference between the first probability and the second probability being less than a preset first threshold value, second probability information corresponding to the type of the object included in the first image is obtained, An electronic device that identifies the type of the object based on the second probability information.

2. In paragraph 1, The third plurality of images and the fourth plurality of images are identified as the first type and the second type, respectively, based on the first user input, An electronic device wherein the plurality of third images and the plurality of fourth images are identified before the first image is acquired and stored in the memory.

3. In paragraph 1, The above processor, The second probability information is obtained through the second neural network model, The above second neural network model is, It is learned based on a plurality of fifth images that are different from the plurality of second images, An electronic device that obtains the second probability information based on the first similarity between the first image and the plurality of third images and the second similarity between the first image and the plurality of fourth images.

4. In paragraph 1, The above electronic device, display; including more, The above first type is a type identified through the first neural network model as corresponding to the first probability with the highest probability that the object corresponds among the plurality of types defined above, The second type is a type identified through the first neural network model as corresponding to a second probability that is lower than the first probability among the plurality of types defined above, The above processor, An electronic device that identifies the first type as the type of the object if the difference between the first probability and the second probability is greater than or equal to the first threshold value.

5. In paragraph 4, The above processor, If the first probability is less than the second threshold value, control the display to display a user interface including information about the first type and information about the second type; Receive a second user input for selecting the type of the object through the user interface; An electronic device that identifies the type of the object based on the second user input.

6. In paragraph 5, The above processor, If the second user input does not correspond to the first type, the second probability information is obtained through the second neural network model, Identifying the type of the object based on the second probability information, The above second neural network model is, It is learned based on a plurality of fifth images that are different from the plurality of second images, An electronic device that obtains the second probability information based on the first image.

7. In paragraph 6, The second probability information includes a third probability that the object corresponds to the first type and a fourth probability that the object corresponds to the second type, The above processor, An electronic device that identifies the first type as the type of the object if the third probability is greater than the fourth probability.

8. In paragraph 7, The above processor, If the third probability is less than the fourth probability, control the display to display the user interface; Receives a third user input for selecting the type of the object through the user interface; An electronic device that identifies the type of the object based on the third user input.

9. In paragraph 4, The user interface includes a first item corresponding to the first type and a second item corresponding to the second type, The above processor, An electronic device that determines at least one of the size of the first item, the transparency of the first item, the size of the second item, and the transparency of the second item based on the first probability information.

10. In paragraph 1, The above memory stores the plurality of third images and the plurality of fourth images, The above processor, An electronic device that updates the plurality of third images stored in the memory based on the first image, if the first image corresponds to the first type or the second type.

11. In paragraph 10, The above processor, An electronic device that updates the plurality of third images stored in the memory by replacing a third image corresponding to the first type or the second type with the first image if the first image corresponds to the first type or the second type.

12. In paragraph 8, The above processor, An electronic device that increases the first threshold value by a preset value when the first probability is greater than the second probability, the third probability is less than the fourth probability, and the third user input corresponds to the second type.

13. In a method for controlling an electronic device, A step of obtaining a first image including an object; A step of inputting the first image into a first neural network model learned using a plurality of second images related to a plurality of predefined types; A step of obtaining first probability information including a first probability that the object among the plurality of types corresponds to a first type and a second probability that the object among the plurality of types corresponds to a second type; A step of obtaining second probability information corresponding to the type of the object included in the first image by using a plurality of third images corresponding to the first type and a plurality of fourth images corresponding to the second type based on the first image, wherein the difference between the first probability and the second probability is less than a preset first threshold value; and A method for controlling an electronic device, comprising: a step of identifying the type of the object based on the second probability information; 14. In paragraph 13, The third plurality of images and the fourth plurality of images are identified as the first type and the second type, respectively, based on the first user input, A control method of an electronic device, wherein the plurality of third images and the plurality of fourth images are identified before the first image is acquired and stored in the memory.

15. In paragraph 13, The above first type is a type identified through the first neural network model as corresponding to the first probability with the highest probability that the object corresponds among the plurality of types defined above, The second type is a type identified through the first neural network model as corresponding to a second probability that is lower than the first probability among the plurality of types defined above, The method of controlling the above electronic device is as follows: A control method of an electronic device further comprising: a step of identifying the first type as the type of the object if the difference between the first probability and the second probability is greater than or equal to the first threshold value;

Images