Apparatus for verifying labels assigned to target data and method thereof

Abstract

According to the present disclosure, a method for processing information in an electronic device is disclosed, comprising the steps of: identifying one or more configured models trained based on reference information for verifying a label assigned to data; providing target data and a label assigned to the target data to one or more configured models; and determining whether a label assigned to the target data is appropriate based on evaluation data provided by one or more configured models.

Description

Technology Field

[0001] The present disclosure relates to an apparatus and a method for verifying a label assigned to target data. More specifically, the present disclosure relates to an electronic apparatus and a method for providing target data and a label assigned to the target data to one or more trained models, and determining whether the label assigned to the target data is appropriate based on evaluation data provided by one or more trained models. Background Technology

[0002] With the advancement of electronic technology, there are increasing instances where services are provided by utilizing models trained based on multiple data. For example, artificial intelligence technology can be implemented by training a deep learning model based on multiple data.

[0003] However, if the data used to train the model is inaccurate (e.g., the assigned labels are inaccurate) or the quality of the data is low, the model may not train smoothly, and consequently, the behavior of the trained model may differ from what was intended, which could have an adverse effect on service quality.

[0004] In this regard, prior art documents such as KR102225822B1 or KR102153920B1 may be referenced. The problem to be solved

[0005] The problem that the present embodiment aims to solve is to provide an electronic device and a method thereof that, in order to solve the aforementioned problem, identify one or more configured models trained based on reference information for verifying a label assigned to data, provide target data and a label assigned to the target data to one or more configured models, and determine whether the label assigned to the target data is appropriate based on evaluation data provided by one or more configured models.

[0006] The technical problems to be solved by this embodiment are not limited to those described above, and other technical problems can be inferred from the following embodiments. means of solving the problem

[0007] A method for processing information in an electronic device according to one embodiment includes the steps of: identifying one or more set models trained based on reference information for verifying a label assigned to data; providing target data and a label assigned to said target data to the one or more set models; and determining whether the label assigned to said target data is appropriate based on evaluation data provided by the one or more set models.

[0008] According to one embodiment, the step of verifying one or more set models may include the step of training one or more set models based on the reference information.

[0009] According to one embodiment, the method for processing the information may further include the step of obtaining one or more evaluation guidelines associated with a label assigned to the data; and the step of obtaining the reference information based on the one or more evaluation guidelines.

[0010] According to one embodiment, the step of obtaining one or more evaluation guidelines may include the step of obtaining at least one of the one or more evaluation guidelines based on information contained in a Standard Operating Procedure (SOP).

[0011] According to one embodiment, the step of obtaining one or more evaluation guidelines may include the step of obtaining at least one of the one or more evaluation guidelines based on input information.

[0012] According to one embodiment, the step of acquiring the reference information may include the step of inputting one or more evaluation guidelines into a reference generation model; and the step of generating the reference information based on the output of the reference generation model.

[0013] According to one embodiment, the step of obtaining the reference information may further include the step of requesting verification of the reference information generated based on the output of the reference generation model.

[0014] According to one embodiment, the reference information may include a criterion for determining the appropriateness of one or more of the name information included in the data, image information included in the data, and attribute information included in the data based on a label assigned to the data.

[0015] According to one embodiment, the reference information may include at least one of one or more data samples with appropriate labels and one or more data samples without appropriate labels.

[0016] According to one embodiment, the step of determining whether a label assigned to the target data is appropriate may include determining whether the ratio of data among the evaluation data provided by the one or more set models that evaluates the label assigned to the target data as inappropriate exceeds a set threshold.

[0017] According to one embodiment, the method for processing the information may further include the step of requesting a worker to review the label assigned to the target data when it is determined that the label assigned to the target data is inappropriate.

[0018] According to one embodiment, the worker may include one or more of a worker with higher proficiency than the subject who labeled the target data and a worker with a higher rank than the subject who labeled the target data.

[0019] According to one embodiment, the method for processing the information may further include the step of generating feedback information related to data labeling based on the result of determining whether a label assigned to the data is appropriate for each of the plurality of data.

[0020] According to one embodiment, the electronic device can generate the feedback information based on information of the subject who labeled the plurality of data.

[0021] According to one embodiment, the method for processing the information may further include: a step of identifying value information that matches each of a plurality of candidate data; and a step of determining one or more of the plurality of candidate data as the target data based on the value information.

[0022] According to one embodiment, the step of verifying the value information may include a step of determining the difficulty of the task of assigning a label matching the candidate data for each of the plurality of candidate data.

[0023] An electronic device for processing information according to one embodiment includes a memory for storing instructions and a processor, wherein the processor is connected to the memory and identifies one or more configured models trained based on reference information for verifying a label assigned to data, provides target data and a label assigned to the target data to the one or more configured models, and determines whether the label assigned to the target data is appropriate based on evaluation data provided by the one or more configured models.

[0024] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0025] According to the present disclosure, by determining whether the label assigned to the target data is appropriate, it is possible to prevent a situation in which a model is trained by inaccurate data.

[0026] The effects of the invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description in the claims. Brief explanation of the drawing

[0027] FIG. 1 is a schematic diagram showing a system for processing information according to one embodiment. FIG. 2 is an operation flowchart illustrating a method of processing information in an electronic device according to one embodiment. FIG. 3 is a diagram illustrating, by way of example, the operation of acquiring reference information according to one embodiment. FIG. 4 is a diagram illustrating, in an exemplary manner, an operation of providing target data and a label assigned to the target data to one or more configured models according to one embodiment. FIG. 5 is a diagram illustrating, by way of example, an operation to determine whether a label assigned to target data according to one embodiment is appropriate. FIG. 6 is a diagram illustrating data associated with operations for processing information in an electronic device according to one embodiment. FIG. 7 is a diagram illustrating, by way of example, evaluation request information provided to one or more configured models according to one embodiment. FIG. 8 is an example diagram of the configuration of an electronic device according to one embodiment. Specific details for implementing the invention

[0028] The terms used in the embodiments have been selected to be as widely used as possible, taking into account their functions in the present disclosure; however, these may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section. Therefore, terms used in the present disclosure should be defined not merely by their names, but based on their meanings and the overall content of the present disclosure.

[0029] When a part of a specification is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as “...part” or “...module” as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0030] The expression “at least one of a, b, and c” described throughout the specification may include ‘a alone’, ‘b alone’, ‘c alone’, ‘a and b’, ‘a and c’, ‘b and c’, or ‘a, b, and c all’.

[0031] The "terminal" mentioned below may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all types of handheld-based wireless communication devices such as communication-based terminals like IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), and LTE (Long Term Evolution), smartphones, tablet PCs, etc.

[0032] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0034] In describing the embodiments, technical details that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. This is intended to convey the essence of the present invention more clearly without obscuring it by omitting unnecessary explanations.

[0035] For the same reason, some components in the attached drawings have been exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual dimensions. Identical or corresponding components in each drawing have been assigned the same reference numbers.

[0036] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0037] At this time, it will be understood that each block of the process flow diagrams and combinations of the flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory can also produce a manufactured item containing the means of instruction to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0038] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0039] Embodiments of the present disclosure may be implemented as artificial intelligence (AI) through the processor and memory of an electronic device. The processor may be composed of one or more processors, and the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (digital signal processors), graphics-dedicated processors such as GPUs and VPUs (vision processing units), or AI-dedicated processors such as NPUs. The one or more processors may be controlled to process input data according to predefined operation rules or AI models stored in memory. Alternatively, if the one or more processors are AI-dedicated processors, the AI-dedicated processors may be designed with a hardware structure specialized for processing a specific AI model.

[0040] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using a number of learning data by a learning algorithm. Such learning may be performed on the electronic device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0041] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and can perform neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include, but is not limited to, deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q-networks.

[0042] FIG. 1 is a schematic diagram showing a system for processing information according to one embodiment.

[0043] According to various embodiments, the information processing system (100) includes an electronic device (110). According to one embodiment, the information processing system (100) may further include at least one of one or more configured models (120) and a worker's device (130). According to one embodiment, the information processing system (100) may further include a network that supports the transmission and reception of information between at least some of the electronic device (110), one or more configured models (120), the worker's device (130), and an external device.

[0044] Each of the electronic device (110), one or more configured models (120), and the operator's device (130) may include memory and a processor, and may also include a transceiver depending on the embodiment. Additionally, each of the electronic device (110), one or more configured models (120), and the operator's device (130) represents a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0045] Meanwhile, throughout the embodiments, the electronic device (110), one or more configured models (120), and the operator's device (130) are each referred to as separate devices or servers, but may have a logically separated structure, and at least some of these may be implemented by functions separated from a single device or server. For example, throughout the disclosure, the electronic device (110) and one or more configured models (120) may be implemented by a single device (more specifically, implemented in the form where one or more configured models (120) are stored in the memory of the electronic device (110), and in this case, the process of transmitting and receiving information between the electronic device (110) and one or more configured models (120) can be understood as a process of exchanging data within a single device.

[0046] According to one embodiment, the electronic device (110), one or more configured models (120), and the worker's device (130) may include a number of computer systems or computer software implemented as network servers. For example, at least some of the electronic device (110), one or more configured models (120), and the worker's device (130) may refer to computer systems and computer software that are connected to sub-devices capable of communicating with other network servers via a computer network such as an intranet or the internet, receive requests for work execution, perform work thereon, and provide results. Additionally, at least some of the electronic device (110), one or more configured models (120), and the worker's device (130) may be understood in a broad sense as including a series of applications capable of operating on a network server and various databases built on internal or other connected nodes. For example, at least some of the electronic device (110), one or more configured models (120), and the worker's device (130) may be implemented using a network server program provided in various ways depending on the operating system, such as DOS, Windows, Linux, UNIX, or MacOS.

[0047] Meanwhile, for the sake of convenience of explanation, each operating entity has been referred to as an electronic device (110), one or more set models (120), and a worker's device (130); however, these should be understood as a comprehensive type of device that corresponds to, includes, or can be included in, various types of devices such as computer devices and mobile communication terminals.

[0048] The electronic device (110) is a device that acquires and provides various information. The electronic device (110) can perform various tasks to provide information. Specifically, the electronic device (110) identifies one or more configured models (120) trained based on reference information for verifying a label assigned to data, provides the target data and the label assigned to the target data to one or more configured models (120), and determines whether the label assigned to the target data is appropriate based on evaluation data provided by one or more configured models. According to one embodiment, the electronic device (110) can train one or more configured models (120) based on reference information. According to one embodiment, the electronic device (110) can determine whether the label assigned to the target data is appropriate by determining whether the ratio of data evaluated as inappropriate among the evaluation data provided by one or more configured models (120) exceeds a set threshold. According to one embodiment, if it is determined that a label assigned to the target data is inappropriate, the electronic device (110) may request the operator to further review the label assigned to the target data.

[0049] One or more configured models (120) may include, but are not limited to, models implemented based on artificial intelligence technology. Various embodiments related to specific implementations may fall within the scope of this disclosure, such as the one or more configured models (120) indicating a device in which each model is implemented, or indicating software or a service (e.g., a service provided via the web) in which each model is implemented.

[0050] One or more configured models (120) may not have sufficient performance to perform the data labeling task themselves. In this regard, the expression “not having sufficient performance” may include, but is not limited to, cases where it is impossible for one or more configured models (120) to perform the labeling task. For example, the expression “not having sufficient performance” may include one or more of the following: cases where the accuracy of the labeling result does not meet the level required for service provision (even if it is possible to perform the labeling task), cases where the speed of performing the labeling task does not meet the level required for service provision, and cases where there is insufficient available resources (e.g., tokens that can be consumed to use the model) to perform the overall labeling task. Meanwhile, when a label has already been assigned to the target data, the task of evaluating whether the label is appropriate has a much lower performance hurdle than the labeling task; therefore, in this disclosure, one or more configured models (120) do not directly perform the labeling task but can perform the task of evaluating whether the assigned label is appropriate.

[0051] One or more configured models (120) may receive input from a user or receive information from a device such as an electronic device (110) and perform corresponding actions. For example, one or more configured models (120) may receive reference information and be trained based on the received reference information. Additionally, one or more configured models (120) may receive target data and a label assigned to the target data and provide evaluation data corresponding to the label assigned to the target data.

[0052] Workers may include various entities participating in at least part of the series of processes for providing the service. For example, workers may be developers of scripts related to the service, operators of the service, or individuals involved in data evaluation tasks related to the service.

[0053] The worker's device (130) may be a device operated and managed by the worker. The worker's device (130) may receive input from the worker or receive information from a device such as an electronic device (110) and perform corresponding actions. For example, the worker's device (130) may receive information from the electronic device (110) requesting a review of a label assigned to target data and provide it to the worker (e.g., display via a screen).

[0054] According to one embodiment, the worker using the worker's device (130) may include one or more workers who are more skilled than the subject who labeled the target data (e.g., "one or more labeling workers (401)" of FIG. 4 described later) and workers who are of a higher rank than the subject who labeled the target data. By requesting a review of the label from a person who has a higher skill level or a higher rank than the person who labeled the target data, the credibility of the review result can be increased. However, the scope of the present disclosure is not limited thereto.

[0055] More detailed information regarding the operation of the electronic device (110), one or more set models (120), and the operator's device (130) will be described later in FIGS. 2 to 7, etc.

[0056] The electronic device (110), one or more configured models (120), and the worker's device (130) may include a computer device, a mobile communication terminal, a server, etc. The electronic device (110) may include an input device such as a touch pad, a mouse, or a keyboard for receiving input, or may be connected to an input device, and according to the embodiment, at least one of one or more configured models (120) and the worker's device (130) may be likewise. Additionally, the electronic device (110) may include an output device such as a screen, a speaker, or an interface device for providing information to a user, or may be connected to an output device, and according to the embodiment, at least one of one or more configured models (120) and the worker's device (130) may be likewise. Furthermore, the input and output devices of the electronic device (110), one or more configured models (120), and the worker's device (130) may be integrated or interconnected, and, for example, an interface for receiving input may be displayed on at least one of the electronic device (110), one or more configured models (120), and the worker's device (130).

[0057] Operations related to a series of information provision methods according to various embodiments may be implemented by a single physical device or may be implemented in a manner in which multiple physical devices are organically combined. For example, some of the components included in the information processing system (100) may be implemented by one physical device, and the remaining parts may be implemented by another physical device. For example, one physical device may be implemented as part of an electronic device (110), and another physical device may be implemented as part of one or more configured models (120), part of a worker's device (130), or part of other external devices. In some cases, each component included in the information processing system (100) may be distributed and placed in different physical devices, and the distributed components may be organically combined to be implemented to perform the functions and operations of the information processing system (100). For example, the electronic device (110) of this specification may include at least one sub-device, and some operations described as being performed by the electronic device (110) may be performed by a first sub-device, and other operations may be performed by a second sub-device.

[0058] FIG. 2 is an operation flowchart illustrating a method of processing information in an electronic device according to one embodiment.

[0059] Referring to FIG. 2, an electronic device (110) according to one embodiment checks one or more set models trained based on reference information for verifying a label assigned to data (201). The one or more set models (120) may include a dedicated model created for a service provided by the electronic device (110), or a model not limited to a service provided by the electronic device (110) (e.g., a general-purpose model capable of performing various functions), and various embodiments related to the implementation of one or more set models (120) may fall within the scope of the present disclosure.

[0060] According to one embodiment related to the operation of Drawing No. 201, the electronic device (110) can train one or more set models (120) based on reference information. That is, the entity training one or more set models (120) may be the electronic device (110). However, the scope of the present disclosure is not limited thereto, and various embodiments may fall within the scope of the present disclosure, such as the electronic device (110) using the trained model after the model has been trained by another entity.

[0061] According to one embodiment, the electronic device (110) may obtain one or more evaluation guidelines associated with a label assigned to data and obtain reference information based on one or more evaluation guidelines. Various embodiments may exist regarding the operation of obtaining one or more evaluation guidelines. According to one example, the electronic device (110) may obtain at least one of one or more evaluation guidelines based on information included in a Standard Operating Procedure (SOP). According to one example, the electronic device (110) may obtain at least one of one or more evaluation guidelines based on input information. In this regard, "input information" may be provided (e.g., input via an input device) by a person associated with the service provided by the electronic device (110). For example, "input information" may be provided by a person responsible for the task of setting standards for data labeling. However, the scope of the present disclosure is not limited thereto.

[0062] According to one embodiment relating to the operation of an electronic device (110) acquiring reference information, the electronic device (110) inputs one or more evaluation guidelines into a reference generation model (which may include, but is not limited to, a model implemented based on artificial intelligence technology) and can generate reference information based on the output of the reference generation model. The reference generation model may be understood as a model that has been trained to perform the function of acquiring reference information. The roles performed by the reference generation model may include one or more of the roles of organizing and processing one or more evaluation guidelines and generating reference information in a form that can be used by one or more set models (e.g., reference information having the form of electronic information), but the scope of the present disclosure is not limited thereto.

[0063] According to an embodiment, the electronic device (110) may request verification of reference information generated based on the output of a reference generation model. According to one example, the electronic device (110) may request verification of the generated reference information from a person associated with a service provided by the electronic device (110). As a more specific example, the electronic device (110) may request verification of the generated reference information from the aforementioned "person in charge of setting standards for data labeling." However, the scope of the present disclosure is not limited thereto.

[0064] According to some embodiments, the subject providing the “input information” and the subject requesting verification of the reference information generated by the electronic device (110) may be different subjects. For example, the subject providing the “input information” may be a person with relatively low proficiency among those responsible for setting standards for data labeling, and the subject to whom the electronic device (110) requests verification of the generated reference information may be a person with relatively high proficiency among those responsible for setting standards for data labeling.

[0065] An example relating to the operation of an electronic device (110) acquiring reference information is illustrated in FIG. 3. Referring further to FIG. 3, at least one of one or more evaluation guidelines (320) may be acquired based on information input from a reference setting worker (310). The reference setting worker (310) according to one example may include, but is not limited to, the aforementioned “person responsible for setting standards for data labeling.” Furthermore, according to an embodiment, the operation of acquiring at least one of one or more evaluation guidelines (320) may be performed by the electronic device (110) or by another entity (e.g., by the reference setting worker (310) itself), and the entity performing the operation does not limit the scope of the present disclosure.

[0066] The electronic device (110) can input one or more evaluation guidelines (320) into a reference generation model (330). Based on the output of the reference generation model (330), the electronic device (110) can generate reference information (340).

[0067] According to one embodiment, the electronic device (110) may request verification of the generated reference information (340) (350). For example, the electronic device (110) may request verification of the reference information (340) from the reference setting worker (310) (350). However, the scope of the present disclosure is not limited thereto, and it has already been explained that the entity providing the “input information” and the entity to which the electronic device (110) requests verification of the generated reference information may be different entities.

[0068] Returning to FIG. 2, the reference information according to one embodiment may include a criterion for determining the appropriateness of data based on a label assigned to the data. For example, the reference information may include "a criterion for determining the appropriateness of one or more of name information included in the data, image information included in the data, and attribute information included in the data based on a label assigned to the data."

[0069] According to a more specific example to aid understanding, the reference information includes a first detailed criterion for determining the appropriateness of name information contained in the data based on a label assigned to the data, a second detailed criterion for determining the appropriateness of image information contained in the data based on a label assigned to the data, and a third detailed criterion for determining the appropriateness of attribute information contained in the data based on a label assigned to the data. Based on the set order for each detailed criterion, if the data satisfies the detailed criterion of the previous order, it can be determined whether the data satisfies the detailed criterion of the next order. For example, if the set order includes the first detailed criterion, the second detailed criterion, and the third detailed criterion, if it is determined that the data satisfies the first detailed criterion, the satisfaction of the second detailed criterion is determined; if it is determined that the data satisfies the second detailed criterion, the satisfaction of the third detailed criterion is determined; and if it is determined that the data also satisfies the third detailed criterion, it can be determined that the data conforms to the reference information. (For data that does not satisfy the first detailed criterion, it may be determined immediately that the data does not conform to the reference information without determining whether the second detailed criterion is satisfied, and for data that does not satisfy the second detailed criterion, it may be determined immediately that the data does not conform to the reference information without determining whether the third detailed criterion is satisfied.)

[0070] Meanwhile, when the reference information of the present disclosure includes a plurality of detailed criteria, not all detailed criteria must necessarily be determined based on the label assigned to the data. For example, the reference information may include a detailed criterion for determining whether the name information contained in the data satisfies formal requirements regardless of the label assigned to the data. As a more specific example, the reference information may include a "fourth detailed criterion for determining whether the name information contained in the data is information in a format that conforms to a pre-set rule."

[0071] The detailed criteria described above are examples to aid in understanding the present disclosure, and the scope of the present disclosure is not limited to the examples of the detailed criteria described above. For instance, the criteria information may further include detailed conditions for determining whether a label assigned to data is information in a format that conforms to a preset rule.

[0072] According to one embodiment, reference information may include at least one of one or more data samples to which an appropriate label has been assigned and one or more data samples not to which an inappropriate label has been assigned. This embodiment is not contradictory to the aforementioned "example in which reference information includes a criterion for determining the appropriateness of data based on a label assigned to the data." That is, the reference information of the present disclosure may be implemented by combining both embodiments. For example, when a label A is assigned to target data, one or more configured models (120) may use the reference information as follows: i) identify at least one of a data sample to which a label A has been appropriately assigned and a data sample to which a label A has been inappropriately assigned (i.e., a data sample to which a label A was assigned but was determined to be an inappropriate label); ii) compare one or more of the name information, image information, and attribute information included in the target data with one or more of the name information, image information, and attribute information included in the identified data sample; iii) determine, based on the comparison result, the appropriateness of one or more of the name information, image information, and attribute information included in the target data.

[0073] The electronic device (110) provides target data and a label assigned to the target data to one or more set models (120) (202). The electronic device (110) acquires evaluation data provided by one or more set models (120) (203) and determines whether the label assigned to the target data is appropriate based on the evaluation data (204).

[0074] For understanding the operation of reference numeral 202, which provides target data and a label assigned to the target data to one or more configured models (120), further reference may be made to the example in FIG. 4. Further reference to the example in FIG. 4, the task of assigning a label to the target data may be performed by one or more labeling workers (401), and a labeling tool (402) may be used in the process. Subsequently, the electronic device (110) may provide the target data and a label (403) assigned to the target data to one or more configured models (120) (404).

[0075] According to an embodiment, one or more labeling workers (401) may include a worker using the worker's device (130) or may be included among the workers using the worker's device (130). According to an embodiment, one or more labeling workers (401) may include one or more workers with lower skill levels and lower ranks than the worker using the worker's device (130). However, these embodiments do not limit the scope of the present disclosure.

[0076] Further reference may be made to the example of FIG. 5 to understand the operation of FIG. 203, which obtains evaluation data provided by one or more set models (120), and the operation of FIG. 204, which determines whether a label assigned to target data is appropriate based on the evaluation data. Further reference to FIG. 5, the electronic device (110) may obtain evaluation data based on the output of one or more set models (120) (501). The evaluation data may be the output of one or more set models (120) itself, or it may be data that the electronic device (110) has processed from the output of one or more set models (120) (e.g., removing noise, classifying data according to set rules, etc.).

[0077] Based on the acquired evaluation data, the electronic device (110) can determine whether the label assigned to the target data is appropriate (502). If it is determined that the assigned label is not appropriate (502, no), the electronic device (110) may request a review from a reviewer (503). The reviewer may include a worker using the worker's device (130) or may be included among the workers using the worker's device (130).

[0078] Returning to FIG. 2, according to one embodiment, the electronic device (110) can determine whether the ratio of data in which the label assigned to the target data is evaluated as inappropriate among the evaluation data provided by one or more set models (120) exceeds a set threshold. For example, the threshold may be set to 0, in which case the label assigned to the target data may be determined to be inappropriate if there is at least one piece of data in which the label assigned to the target data is evaluated as inappropriate among the evaluation data provided by one or more set models (120) (i.e., unless one or more set models (120) unanimously determine that the label is appropriate). As another example, the threshold may be set to 0.9999 (a value very close to 1), in which case the label assigned to the target data may be determined to be appropriate if there is at least one piece of data in which the label assigned to the target data is evaluated as appropriate among the evaluation data provided by one or more set models (120) (i.e., unless one or more set models (120) unanimously determine that the label is inappropriate). In another example, the threshold can be set to 0.5, in which case, if the number of evaluation data provided by one or more configured models (120) that evaluates the label assigned to the target data as inappropriate exceeds half of the total (i.e., if the number of data evaluated as inappropriate is a majority of the total), the label assigned to the target data can be determined to be inappropriate. In addition, there may be various other examples related to threshold setting.

[0079] Meanwhile, the operation of determining whether the ratio of data in which the label assigned to the target data is evaluated as inappropriate among the evaluation data provided by one or more configured models (120) exceeds a set threshold can be interpreted as a broad concept that includes several operations in which substantially similar judgments are made. For example, the electronic device (110) may determine whether the ratio of data in which the label assigned to the target data is evaluated as "appropriate" among the evaluation data provided by one or more configured models (120) exceeds a set threshold, and may also determine whether the ratio of data in which the label assigned to the target data is evaluated as inappropriate among the evaluation data provided by one or more configured models (120) is "greater than," "less than," or "less than" the set threshold. Since these operations substantially have almost the same purpose and effect as the operation of “determining whether the ratio of data in which the label assigned to the target data is evaluated as inappropriate among the evaluation data provided by one or more configured models (120) exceeds a set threshold,” they all fall within the scope of an embodiment of the present disclosure, and their specific details can be understood by applying the description related to the embodiment of “determining whether the ratio of data in which the label assigned to the target data is evaluated as inappropriate among the evaluation data provided by one or more configured models (120) exceeds a set threshold.”

[0080] In addition, the electronic device (110) according to one embodiment may determine whether a label assigned to a target data is appropriate by using information on the "number" of data instead of the ratio of data. For example, the electronic device (110) may determine whether the "number" of data that evaluates the label assigned to the target data as inappropriate among the evaluation data provided by one or more set models (120) exceeds a set threshold. In the case of such an embodiment, since it will substantially have almost the same purpose and effect as the operation of "determining whether the ratio of data that evaluates the label assigned to the target data as inappropriate among the evaluation data provided by one or more set models (120) exceeds a set threshold," all fall within the scope of one embodiment of the present disclosure, and the specific details thereof can be understood by applying the description related to the embodiment of "determining whether the ratio of data that evaluates the label assigned to the target data as inappropriate among the evaluation data provided by one or more set models (120) exceeds a set threshold."

[0081] According to one embodiment, if it is determined that a label assigned to the target data is inappropriate, the electronic device (110) may request a worker using the worker's device (130) to review the label assigned to the target data (205) (operation of reference numeral 503 of FIG. 5 described above). The worker using the worker's device (130) may include one or more workers who are more skilled than the person who assigned the label to the target data and workers who are of a higher rank than the person who assigned the label to the target data, but the scope of the present disclosure is not necessarily limited thereto.

[0082] According to one embodiment, based on the result of determining whether a label assigned to a plurality of data is appropriate for each of the data, the electronic device (110) may generate feedback information associated with data labeling (206). The process of determining whether a label assigned to a plurality of data is appropriate for each of the data may include processes of performing the operations of Drawing Nos. 201 to 203 respectively for at least some of the plurality of data. However, the scope of the present disclosure is not limited thereto. For example, for some of the plurality of data, whether a label assigned to the data is appropriate may be determined based on a different process distinct from the operations of Drawing Nos. 201 to 203, and in this case, the electronic device (110) may generate feedback information by combining the results of the determinations made by different processes. That is, just because the data are associated with a single feedback information, the process of determining whether a label assigned to the data is appropriate must not have been the same for all of them.

[0083] According to one embodiment, the electronic device (110) can generate feedback information based on information from the entity that labeled the target data (e.g., “one or more labeling workers (401)” of FIG. 4 described above). According to one example, if it is determined that the data labeled by a specific worker among a plurality of data is particularly inappropriate, the electronic device (110) can generate feedback information regarding the quality of work of the said worker. Specific examples related to this include cases where the data labeled by a specific worker is judged to have an inappropriate label exceeds a set threshold ratio (e.g., 20%), cases where the data labeled by a specific worker is judged to have an inappropriate label exceeds a set threshold number (e.g., 10), and cases where the proportion of data labeled by a specific worker is judged to have an inappropriate label is higher than the proportion of data labeled by another worker is judged to have an inappropriate label by a level exceeding a threshold level (e.g., 1.5 times). These can be understood as examples of "cases where the data labeled by a specific worker is judged to have a particularly inappropriate label."

[0084] According to an embodiment related to the operation of reference numeral 206, the electronic device (110) can generate feedback information regarding the appropriateness of reference information. For example, if the proportion of data among the total data for which the label assigned to the data is judged to be inappropriate exceeds a threshold ratio (e.g., 30%), the electronic device (110) can generate feedback information stating that the reference information is inappropriate or that a review of the appropriateness of the reference information is necessary.

[0085] According to an embodiment, the electronic device (110) may generate feedback information based further on the results of the review after requesting a review of the label assigned to the target data. For example, if the ratio of data in which the judgment was incorrect and the label assigned to the data was appropriate was received according to the operation of reference numeral 205 among the data in which the assigned label was judged to be inappropriate according to the operation of reference numeral 203, the electronic device (110) may generate feedback information stating that the reference information is inappropriate or that a review of the appropriateness of the reference information is required.

[0086] According to an embodiment, the electronic device (110) may determine whether to generate feedback information regarding the appropriateness of reference information by considering information about the subject who assigned the label to the target data. For example, even if the proportion of data among the total data where the label assigned to the data is judged to be inappropriate exceeds a threshold ratio, if the majority (e.g., 70% or more) of them are limited to data worked on by a small number of specific workers, feedback information regarding the quality of work of those workers may be generated, and feedback information regarding the appropriateness of reference information may not be generated.

[0087] In addition, an embodiment that generates feedback information regarding whether the data itself is defective (e.g., whether the data is not related to the service provided by the electronic device (110)) may fall within the scope of the operation of reference numeral 206. Furthermore, an embodiment that generates feedback information regarding the appropriateness of each process (or the appropriateness of the reference information corresponding to that process) based on the ratio of data where the label assigned to each process is judged to be inappropriate (where the electronic device (110) generates feedback information by combining judgment results from different processes) may fall within the scope of the operation of reference numeral 206. Thus, various embodiments may fall within the scope of the operation of reference numeral 206.

[0088] According to one embodiment, the electronic device (110) can identify value information that matches each of a plurality of candidate data and determine one or more of the candidate data as target data based on the value information. For example, the electronic device (110) can determine the target data only for data where the value information indicates a value of a certain level or higher (i.e., provide the data and the label assigned to the data to one or more configured models (120)). By determining the target data by considering the value information in this way, the evaluation work by one or more configured models (120) can be omitted for data with relatively low value, thereby saving resources while allowing evaluation to proceed for important data.

[0089] According to an embodiment relating to the operation of an electronic device (110) verifying value information matching each of a plurality of candidate data, the electronic device (110) can determine the difficulty of the task of assigning a label matching the candidate data for each of the plurality of candidate data. According to the embodiment, the “difficulty of the task” may include the difficulty from the perspective of the entity assigning the label to the data (e.g., “one or more labeling workers (401)” described in FIG. 4) and the difficulty from the perspective of the device (e.g., one or more models (120)), and various related embodiments may fall within the scope of the present disclosure.

[0090] FIG. 6 is a diagram illustrating data associated with operations for processing information in an electronic device according to one embodiment.

[0091] Referring to FIG. 6, data (610) associated with a labeling operation according to one embodiment may include ID information ("id") associated with the labeling operation and ID information ("reference_information_id") associated with reference information.

[0092] The "reference_information_id" included in the data (610) associated with the labeling task may indicate the data (620) associated with the reference information. In one embodiment, the data (620) associated with the reference information may include ID information ("id") associated with the reference information and text information ("evaluation guideline") describing the evaluation guideline.

[0093] As can be understood from the relationship between the data of drawing number 610 and the data of drawing number 620, according to one embodiment of the present disclosure, by including ID information associated with reference information in the labeling work data (610), a device verifying the labeling work data (610) can easily verify what the reference information associated with the data is.

[0094] Data (630) associated with an assigned label according to one embodiment may include ID information ("id") associated with the assigned label.

[0095] Data (640) associated with the determination of whether an assigned label is appropriate according to one embodiment may include ID information ("id") associated with the data (640), ID information ("labelingAttributeId") (641) associated with a labeling attribute, ID information ("labelingTargetAnnotationId") associated with a specific annotation of the labeled data, information ("verdict" and "llm_judgments") (642) indicating the result of the determination of whether the assigned label is appropriate, and time information ("judgedAt") for determining whether the assigned label is appropriate.

[0096] ID information (641) associated with a labeling attribute may indicate data (630) associated with an assigned label. As can be understood from the relationship between the data of reference numeral 630 and the data of reference numeral 640, according to one embodiment of the present disclosure, by including ID information (641) associated with a labeling attribute in the data (640) associated with the determination of whether an assigned label is appropriate, a device verifying the data (640) associated with the determination of whether an assigned label is appropriate can easily determine what the label assigned to the target data was.

[0097] Information (642) indicating the result of a judgment on whether an assigned label is appropriate may include a "verdict" field and a "llm_judgments" field. In this regard, the verdict field is of boolean type and can be understood as a binary field indicating either a value indicating that the label assigned to the target data is appropriate (true) or inappropriate (false). Meanwhile, the llm_judgments field is of text type and can be understood as a field containing detailed information such as the judgment process or the basis for judgment of the entity (typical example being one or more configured models (120)) determining whether the assigned label is appropriate. However, the result of a judgment on whether an assigned label is appropriate in this disclosure does not necessarily have to include both the verdict field and the llm_judgments field, and the information of reference numeral 640 may be implemented in a manner that includes only one of the two, depending on the embodiment.

[0098] In the example of FIG. 6, the "PK" indicator is an abbreviation for Primary Key and indicates that it is unique information associated with the data, and the "FK" indicator is an abbreviation for Foreign Key and indicates that it is information that establishes relationships between data by referencing unique information of other data. However, the PK and FK indicators in FIG. 6 are merely examples to aid understanding and do not limit the scope of the present disclosure.

[0099] FIG. 7 is a diagram illustrating, by way of example, evaluation request information provided to one or more configured models according to one embodiment.

[0100] According to an embodiment related to the operation of reference numeral 202 in FIG. 2, an electronic device (110) may provide evaluation request information, including information on target data and information on a label assigned to the target data, to one or more configured models (120). In this regard, referring to FIG. 7, the information on the target data included in the evaluation request information (700) according to one example may be as shown in reference numeral 710. Referring to reference numeral 710, the information on the target data may include item ID information ("itemid"), item name information ("itemname"), product ID information associated with the item ("productid"), and information on one or more attributes associated with the item ("attributetypeid", "attributetypename", "attributevaluename", "isexposed", etc., which are specified by "catalogitemattributes").

[0101] The information of the label assigned to the target data included in the evaluation request information (700) according to one example may include "categoryid" information as in drawing number 720.

[0102] According to an embodiment, the evaluation request information (700) may further include information for verifying an image related to the target data, such as in Drawing No. 730. According to an embodiment, the information for verifying the image may be provided in the form of a URL, such as "itemimageurl" in Drawing No. 730, or the image information itself may be included in the evaluation request information (700), and various related embodiments may fall within the scope of the present disclosure.

[0103] FIG. 8 is an example diagram of the configuration of an electronic device according to one embodiment.

[0104] Referring to FIG. 8, the electronic device (110) includes a processor (810) and a memory (820), and may further include a transceiver (830) depending on the embodiment. The electronic device (110) can be connected to one or more configured models (120), a worker's device (130), and external devices, etc., through the transceiver (830) and exchange data.

[0105] The processor (810) may include at least one device described through FIGS. 1 to 7 or perform at least one method described through FIGS. 1 to 7. The memory (820) may store information for performing at least one method described through FIGS. 1 to 7. The memory (820) may be a volatile memory or a non-volatile memory.

[0106] The processor (810) can control an electronic device (110) for executing a program and providing information. The code of the program executed by the processor (810) can be stored in memory (820).

[0107] In addition, the electronic device (110) of the embodiment may further include an interface capable of providing information to a user, etc.

[0108] Meanwhile, the present specification and drawings disclose preferred embodiments of the present invention. Although specific terms have been used, they are used merely in a general sense to facilitate the explanation of the technical content of the present invention and to aid in understanding the invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that, in addition to the embodiments disclosed herein, other variations based on the technical concept of the present invention are possible.

[0109] A server or terminal according to the embodiments described above may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, and user interface devices such as a touch panel, a key, a button, etc. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable code or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD (Digital Versatile Disc)). Computer-readable recording media may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium may be readable by a computer, stored in memory, and executed by a processor.

[0110] The present embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the embodiment may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. Similar to how components may be implemented as software programming or software elements, the present embodiment may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, Python, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the present embodiment may employ prior art for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations. The above terms may include the meaning of a series of software processes (routines) in conjunction with processors, etc.

[0111] The aforementioned embodiments are merely examples, and other embodiments may be implemented within the scope of the claims set forth below.

Claims

Claim 1 A method for processing information in an electronic device comprises: obtaining one or more evaluation guidelines associated with a label assigned to data; obtaining reference information for verifying a label assigned to data based on the one or more evaluation guidelines; identifying one or more set models trained based on the reference information; providing target data and a label assigned to the target data to the one or more set models; and determining whether a label assigned to the target data is appropriate based on the evaluation data provided by the one or more set models. Claim 2 A method for processing information according to claim 1, wherein the step of verifying one or more set models includes the step of training one or more set models based on the reference information. Claim 3 delete Claim 4 A method for processing information according to claim 1, wherein the step of obtaining one or more evaluation guidelines includes the step of obtaining at least one of the one or more evaluation guidelines based on information contained in a Standard Operating Procedure (SOP). Claim 5 A method for processing information according to claim 1, wherein the step of obtaining one or more evaluation guidelines includes the step of obtaining at least one of the one or more evaluation guidelines based on input information. Claim 6 A method for processing information according to claim 1, wherein the step of obtaining the reference information comprises: inputting one or more evaluation guidelines into a reference generation model; and generating the reference information based on the output of the reference generation model. Claim 7 A method for processing information, wherein, in claim 6, the step of obtaining the reference information further includes the step of requesting verification of the reference information generated based on the output of the reference generation model. Claim 8 A method for processing information according to claim 1, wherein the reference information includes a criterion for determining the appropriateness of one or more of name information included in the data, image information included in the data, and attribute information included in the data based on a label assigned to the data. Claim 9 A method for processing information according to claim 1, characterized in that the reference information comprises at least one of one or more data samples with appropriate labels and one or more data samples without appropriate labels. Claim 10 A method for processing information according to claim 1, wherein the step of determining whether a label assigned to the target data is appropriate includes the step of determining whether the ratio of data among the evaluation data provided by the one or more set models that evaluates the label assigned to the target data as inappropriate exceeds a set threshold. Claim 11 A method for processing information according to claim 1, further comprising the step of requesting a worker to review the label assigned to the target data when it is determined that the label assigned to the target data is inappropriate. Claim 12 A method for processing information according to claim 11, characterized in that the worker includes one or more of a worker with higher proficiency than the subject who labeled the target data and a worker with a higher rank than the subject who labeled the target data. Claim 13 A method for processing information according to claim 1, further comprising the step of generating feedback information related to data labeling based on the result of determining whether a label assigned to the data is appropriate for each of the multiple data. Claim 14 A method for processing information according to claim 13, wherein the electronic device generates feedback information based on information of the subject who labeled the plurality of data. Claim 15 A method for processing information according to claim 1, further comprising: a step of verifying value information that matches each of a plurality of candidate data; and a step of determining one or more of the plurality of candidate data as the target data based on the value information. Claim 16 A method for processing information according to claim 15, wherein the step of verifying the value information includes the step of determining the difficulty of the task of assigning a label matching the candidate data for each of the plurality of candidate data. Claim 17 A computer-readable, non-transient recording medium having a program for executing the method of paragraph 1 on a computer. Claim 18 An electronic device for processing information, comprising a memory for storing instructions and a processor, wherein the processor is connected to the memory and obtains one or more evaluation guidelines associated with a label assigned to data, obtains reference information for verifying the label assigned to data based on the one or more evaluation guidelines, identifies one or more set models trained based on the reference information, provides target data and a label assigned to the target data to the one or more set models, and determines whether the label assigned to the target data is appropriate based on the evaluation data provided by the one or more set models.

Images