Device and method for semantic segmentation labeling using egg and eye tracking

Abstract

The semantic segmentation labeling device and method using brainwaves and eye tracking according to the embodiment improves the performance of an AI model by collecting EEG (brainwave) and eye tracking data and utilizing the collected data to automatically detect and correct errors in training data. In the existing data labeling process, the likelihood of errors was high due to reduced worker concentration caused by repetitive tasks, and the inspection process also required significant time and cost. To address this, the embodiment analyzes EEG and eye tracking data to monitor the worker's attention state and reactions, and enables the automatic detection of data containing incorrect labeling.

Description

Technology Field

[0001] The technical concept of the present disclosure relates to a semantic segmentation labeling device and method, and more specifically, to a semantic segmentation labeling device and method that generates training data using brainwave and eye-tracking data that monitors incorrectly generated training data. Background Technology

[0002] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.

[0003] Artificial intelligence (AI) is a model that learns patterns and makes predictions based on given training data, and the quality of the training data is a key factor in determining its performance. Since AI models learn patterns through input training data, the more abundant and accurate the training data, the better the model's performance. Conversely, if the training data is insufficient or contains errors, the AI's judgments may become inaccurate. Furthermore, because AI learns the patterns in the training data exactly as they are, if the data contains errors, the AI ​​will learn incorrect patterns. Even small errors, if accumulated, can cause the performance of an AI model to deteriorate rapidly.

[0004] Meanwhile, the data labeling task required to generate training data involves many repetitive operations, such as object identification and data cleaning. Traditionally, data labeling has been performed manually; however, prolonged manual work increases the likelihood of errors due to fatigue and decreased concentration. Such errors can generate incorrect training data and, consequently, degrade the performance of artificial intelligence models.

[0005] Meanwhile, semantic segmentation is a deep learning technique that predicts which object (class) each pixel in an image or video belongs to. Conventional semantic segmentation models must be trained on data manually labeled by humans, but the labeling process is highly labor-intensive. In contrast, utilizing EEG and eye-tracking data enables automatic or semi-automatic labeling. EEG (Electroencephalography) is a technology that measures electrical activity occurring in the brain. Electrodes are attached to the head to record and analyze electrical signals between nerve cells (neurons).

[0006] Figure 1 is a diagram showing the types of brainwaves. Brainwaves are electrical signals generated in the brain, created as nerve cells (neurons) interact. As shown in Figure 1, the types of brainwaves are classified according to frequency bands, and depending on the frequency band, there are delta waves, theta waves, alpha waves, gamma waves, beta waves, etc. Delta waves (0.5-4 Hz) are brainwaves observed in deep sleep or resting states. Theta waves (4-8 Hz) are brainwaves that appear when one is drowsy or in a meditative state, and alpha waves (8-13 Hz) are brainwaves that appear in a stable and relaxed state or in a light meditative state. Beta waves (13-30 Hz) are brainwaves that appear during active cognitive activities such as concentration, alertness, and problem-solving, and gamma waves (30 Hz or higher) are brainwaves that appear in states related to high-level cognitive processing, learning, and memory. Prior art literature

[0007] 1. Korean Patent Registration No. 10-2587288 (Oct. 05, 2023) 2. Korean Patent Registration No. 10-2723713 (Oct. 25, 2024) The problem to be solved

[0008] The semantic segmentation labeling device and method using brainwaves and eye tracking according to the embodiment collects EEG (brainwave) and eye tracking data and utilizes the collected brainwave and eye tracking data to automatically detect and correct errors in training data, thereby improving the performance of an AI model.

[0009] In the existing data labeling process, repetitive tasks led to reduced worker concentration, resulting in a high likelihood of errors. Furthermore, the inspection process required significant time and cost. To address this, the embodiment analyzes EEG and eye-tracking data to monitor the worker's attention state and reactions, and enables the automatic detection of data containing incorrect labels.

[0010] In particular, in the embodiments, if an outlier is detected in the worker's eye movement in a specific image or if the brainwave response changes rapidly, it is determined that the data is highly likely to contain errors.

[0011] In addition, in the embodiment, the semantic segmentation network continuously learns the data detected based on the analysis results of eye movements and brainwave responses to automatically generate more accurate correct answer images.

[0012] In addition, the embodiment improves data quality by iteratively improving the semantic partitioning network.

[0013] In addition, unlike conventional methods, the embodiment enables data quality evaluation and correction using only EEG and eye tracking without using a keyboard or mouse, thereby minimizing the burden of manual work.

[0014] In addition, the semantic segmentation labeling device and method using brainwaves and eye tracking according to the embodiment solves the problem of artificial intelligence performance degradation caused by incorrect training data and enables the automatic construction of a more accurate and reliable dataset.

[0015] However, the problem to be solved according to one embodiment is not limited only to that mentioned above. means of solving the problem

[0016] A semantic segmentation labeling device using brainwaves and eye tracking according to an embodiment comprises: a communication module; a memory storing at least one instruction; and a processor, wherein the at least one instruction is executed by the processor, the labeling device,

[0017] The labeling device collects brainwave (EGG) and eye movement data, analyzes the worker's state based on the collected brainwave and eye movement data, determines the possibility of error in the training data generated according to the analysis of the worker's state, and, for the analysis of the worker's state, detects the worker's state, including concentration, fatigue, and cognitive response, through brainwaves, detects abnormal reactions through eye movement data that records the worker's gaze, and can determine the worker's state of inattention using the brainwaves and abnormal reactions.

[0018] In addition, the labeling device inputs brainwave and eye-tracking data into a machine learning model, and

[0019] The above model analyzes input data and, if an abnormal response is detected where the worker's attention level is lower than a certain level, sets the error probability of the labeling generated at the time the abnormal response was detected to a certain level or higher; if the worker's attention level is higher than a certain level, determines the worker's state as normal and determines that accurate labeling data is generated in the worker's normal state.

[0020] Additionally, the labeling device inputs original RGB images, semantic segmented images, brainwaves, and eye tracking data into a machine learning model to determine the possibility of error in the training data, and the model analyzes all four types of input data, evaluates the state of the worker based on the results of the data analysis, and determines the labeling state of the training data according to the results of the worker state evaluation, the state of the worker includes cognitive response, attention, and gaze pattern, and the labeling state may include accurate, inaccurate, and inattention.

[0021] In addition, the labeling device compares the original RGB image and the semantic segmented image through a machine learning model and further analyzes signal data including brainwave and eye tracking data to detect locations in the semantic segmented image where the probability of an error occurring is above a certain level. The model evaluates the quality of training data generated by an operator based on the original RGB image, the semantic segmentation result, and the brainwave and eye tracking data, and can detect locations in the semantic segmentation result where the probability of an error is above a certain level according to the evaluation result.

[0022] In addition, the labeling device utilizes brainwave and eye-tracking data to analyze errors in training data generated by an operator, generates new correct semantic segmentation images by reflecting the error analysis results, and continuously trains a Semantic Segmentation Network using the generated correct semantic segmentation images.

[0023] The machine learning model can improve the semantic segmentation network by repeatedly training with the generated ground truth semantic segmentation images as training data.

[0024] In addition, the labeling device can evaluate the worker's condition by collecting and analyzing brainwave and eye movement data in real time.

[0025] In addition, the labeling device can evaluate the worker's status by measuring the P300 Event-Related Potential (ERP) that occurs when the worker processes new information.

[0026] In addition, the model can set the error probability to a certain level if objects that are clearly distinguishable in the original RGB image have blurred boundaries or are incorrectly segmented in the semantic segmentation result, and can set the corresponding area as a suspected error location if the location and shape of an object appear differently in the semantic segmentation image and the RGB image. Effects of the invention

[0027] The semantic segmentation labeling device and method using brainwaves and eye tracking according to the embodiment utilize EEG and eye tracking to analyze the worker's attention state in real time, thereby enabling the automatic detection of data with a high probability of error. This allows for the construction of a higher-quality training dataset and improves the accuracy and reliability of the AI ​​model.

[0028] Previously, operators had to manually inspect and modify data, which consumed significant time and cost; however, the embodiment enables the automation of the inspection process by automatically analyzing EEG and eye-tracking data to detect errors. Accordingly,

[0029] It can reduce data labeling and inspection time and maximize work efficiency.

[0030] Furthermore, since there is a high probability that an AI model will experience performance degradation or produce incorrect results if it learns from incorrect training data as is, the embodiments utilize EEG and eye tracking to detect and correct errors in the training data early, thereby ensuring that the AI ​​model learns accurate patterns. This improves the generalization performance of the AI ​​model and prevents performance degradation caused by errors.

[0031] In addition, in the existing method, the burden on workers was significant because data had to be inspected and modified using a keyboard and mouse; however, in the embodiment, data quality evaluation and modification can be performed using only EEG and eye tracking, thereby reducing worker fatigue and minimizing the work burden. This allows for a more intuitive and natural data inspection and labeling process.

[0032] In addition, in the embodiment, since the AI ​​automatically generates ground truth data based on EEG and eye-tracking data, the Semantic Segmentation Network can continuously learn and improve. This helps the AI ​​generate more accurate and precise semantic segmentation results and enables continuous performance improvement through iterative learning.

[0033] While existing data labeling and verification methods required a large workforce, were time-consuming, and costly, the embodiment utilizes EEG and eye tracking to automate the data verification and error correction processes, thereby reducing costs and improving the efficiency of building AI training data.

[0034] Furthermore, the semantic segmentation labeling device and method using EEG and eye tracking according to the embodiment improve the quality of AI training data, maximize the automation and efficiency of the labeling process, and create the effect of preventing performance degradation of the AI ​​model. In addition, since data quality can be evaluated and modified using only EEG and eye tracking without a keyboard or mouse, it reduces the burden on the operator and enables cost reduction.

[0035] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure. Brief explanation of the drawing

[0036] Figure 1 is a diagram showing types of brainwaves. FIG. 2 is a diagram showing a system that restores an image by learning EEG (brainwave) signals in an embodiment. Figure 3 is a diagram showing visual data reconstructing an image seen by a person using EEG (brainwave) signals. FIG. 4 is a diagram illustrating an image labeling (annotation) system based on EEG (brainwave) in an embodiment. FIG. 5 is a block diagram of a semantic segmentation labeling device using brainwaves and eye tracking according to an embodiment. FIG. 6 is a diagram illustrating an EEG (brainwave)-based image labeling (annotation) system and signal analysis using 14 EEG channels in an embodiment. Figure 7 shows eye-tracking and eye-movement analysis A diagram to explain the determination of concentration and areas of interest utilized. FIG. 8 is a conceptual diagram illustrating a semantic segmentation technique according to an embodiment. Figure 9 is a diagram showing the result of successful semantic partitioning. Figure 10 is a diagram showing the result of a failed semantic partitioning. Figure 11 is a diagram illustrating an example of performing class-specific annotation work based on the image source and boundaries. FIG. 12 is a diagram illustrating a method for supplementing semantic segmentation data using EEG (brainwave) and eye-tracking in an embodiment. FIG. 13 is a diagram illustrating the process of detecting defective data (annotation errors) by utilizing EEG (brainwave) and eye movement data in a labeling device according to an embodiment. FIG. 14 is a diagram illustrating a system for detecting defects in AI training data by utilizing EEG (brainwave) and eye movement data in an embodiment. FIG. 15 is a diagram illustrating a system for determining the possibility of defects and error locations in AI training data by utilizing EEG (brainwave) and eye movement data in an embodiment. FIG. 16 is a diagram illustrating a system that utilizes brainwave and eye movement data in an embodiment to detect errors in AI training data and generate semantic segmented images as new correct answers. FIG. 17 is a diagram illustrating a system in an embodiment that continuously trains a semantic segmentation network using EEG (brainwave) and eye movement data and automatically generates new correct semantic images. Specific details for implementing the invention

[0037] Hereinafter, various embodiments of the present disclosure are described in conjunction with the accompanying drawings. As various embodiments of the present disclosure may be subject to various modifications and may have various forms, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the various embodiments of the present disclosure to specific forms, and it should be understood that they include all modifications and / or equivalents and substitutions that fall within the spirit and scope of the various embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals have been used for similar components.

[0038] In various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0039] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0040] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components and may be used to distinguish one component from another.

[0041] When it is mentioned that a component is "connected" or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that a new component may also exist between the component and the other component.

[0042] In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or in a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0043] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0044] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0045] FIG. 2 is a diagram showing a system for restoring an image by learning EEG (brainwave) signals in an embodiment, and FIG. 3 is a diagram showing visual data restored from an image seen by a person using EEG (brainwave) signals. The semantic segmentation labeling device using brainwaves and eye tracking according to the embodiment can restore an image by learning brainwave signals. Referring to FIG. 2, when a user wears an EEG (brainwave) cap, the semantic segmentation labeling device collects the user's brainwave signals and extracts features from the collected brainwave signals. Subsequently, the extracted features are converted into feature vectors. The converted data is mapped to a location in the latent space through an EEG feature mapper. Subsequently, an image decoder restores the original image based on the mapped information.

[0046] In addition, the semantic segmentation labeling unit according to the embodiment performs image encoding and decoding processes. In the embodiment, image encoding (compression) and decoding (restoration) can be performed using a Pretrained VGG11 model. In the embodiment, images can be restored after mapping collected EEG data to a latent space. In the embodiment, input data is converted into the original image using a multi-stage neural network. In the embodiment, the neural network can be designed with a structure including an encoder, a latent space, and a decoder. In addition, in the embodiment, a loss function is used to minimize the difference between the original image and the restored image during the training process. In the embodiment, Mean Squared Error (MSE) Loss is used. Additionally, Distance Loss and Angle Loss can be applied.

[0047] In addition, in the embodiment, an image seen by a person is restored based on EEG signals, and the image is reconstructed by mapping brainwave data to the latent space of a deep learning model.

[0048] This enables the provision of more precise feedback than existing visual BCI (Brain-Computer Interface) systems.

[0049] FIG. 4 is a diagram illustrating an image labeling (annotation) system based on EEG (brainwave) in an embodiment. Referring to FIG. 4, a labeling device (100) according to an embodiment analyzes brainwave signals to automatically detect data outliers and refines the detected outliers. For example, the labeling device (100) utilizes EEG (brainwave) data to automatically detect outliers in training data (images) and improves labeling. In the embodiment, images of a specific category (e.g., pizza) are continuously shown to a user, and images of a different category (outliers) are inserted in between. Subsequently, the labeling device (100) analyzes the user's brainwave response to detect images of a different category (outliers) and uses this to automatically refine the data labeling.

[0050] The image database illustrated in FIG. 4 is a database in which multiple image data are stored. In the embodiment, a specific image category (e.g., pizza, landscape, etc.) is selected from the database and used as training data. In the embodiment, an image sequence is generated and a continuous visual presentation is performed. That is, in the embodiment, images are displayed sequentially and rapidly (Rapid Serial Visual Presentation). For example, while continuously displaying pizza images, an outlier, such as a flower or an animal, can be inserted in the middle. Subsequently, the user's EEG signal (brainwave response) generated by the exposed image is measured. Afterward, the labeling device (100) acquires the EEG signal. In the embodiment, the experiment is conducted while the user wears an EEG cap and views the screen, and the labeling device (100) collects the response (EEG signal) generated in the brain when viewing a specific image (especially an outlier). In the embodiment, the EEG data is transmitted to the brainwave signal classification and labeling process.

[0051] Subsequently, the labeling device (100) performs EEG signal analysis and labeling. In the embodiment, the labeling device (100) analyzes the collected EEG data to detect outliers. Subsequently, it detects images containing outliers, automatically corrects incorrect labels, and distinguishes them into green boxes (correct data) and red boxes (incorrect data). Subsequently, the labeling device (100) removes outliers. In the embodiment, images detected as outliers are filtered or corrected based on the EEG data. In the embodiment, outliers are appropriately classified, and finally, refined training data is generated.

[0052] In addition, the embodiment conducts experiments using blocks and sessions, which are units for rapidly displaying multiple images. A session is a unit of experimental execution that includes multiple blocks. In the embodiment, the image exposure experiment consists of multiple blocks within a single session and can be conducted at regular time intervals.

[0053] Previously, data labeling required manual review by humans, but in the embodiment, incorrect data can be automatically detected and corrected by utilizing EEG signals. Furthermore, the embodiment enhances the reliability of AI training data by automatically detecting and removing outliers. In addition, it prevents data containing errors from being trained on the AI ​​model. Moreover, while manual data verification and error correction were previously required, an automated data labeling system is established using EEG signals. It also enables the AI ​​to create a more sophisticated data cleaning process as it continuously learns.

[0054] FIG. 5 is a block diagram of a semantic segmentation labeling device using brainwaves and eye tracking according to an embodiment. The configuration of the semantic segmentation labeling device (100) using brainwaves and eye tracking shown in FIG. 5 is merely a simplified example. The communication module (110) can be configured regardless of the communication mode, such as wired or wireless, and can be configured with various communication networks such as a Personal Area Network (PAN) or a Wide Area Network (WAN). In addition, the communication module (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication module (110) may be responsible for transmitting and receiving data necessary to perform a technique according to an embodiment of the present disclosure.

[0055] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.

[0056] The memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, the memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). Additionally, the memory (120) stores various types of modules, instruction sets, or models.

[0057] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0058] This processor (130) can train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor (130) can perform calculations for training the neural network, such as processing input data for training, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Additionally, the processor (130) can perform inference for a specific purpose using a model implemented in an artificial neural network manner.

[0059] Figure 6 is a diagram illustrating an EEG (brainwave)-based image labeling (annotation) system and signal analysis using 14 EEG channels in an embodiment.

[0060] In the embodiment, the automatic labeling system can be improved by detecting the P300 signal generated when viewing a specific image (target image). As illustrated in FIG. 6, the embodiment utilizes 14 channels used in an EEG headset. Each channel includes AF3, AF4, F3, F4, F7, F8, FC5, FC6, P7, P8, O1, and O2. In the embodiment, the channels perform different roles depending on their location in the brain. AF3 and AF4 are channels placed in the frontal lobe to collect brainwaves related to attention. F3, F4, FC5, and FC6 are channels placed in the frontal lobe and motor cortex to collect brainwaves related to decision-making and motor responses. Additionally, P7 and P8 are channels placed in the parietal lobe to collect brainwaves related to spatial awareness and visual processing. O1 and O2 are channels placed in the occipital lobe to collect brainwaves related to visual information processing.

[0061] In the embodiment, 14 EEG channels are utilized to measure the user's brainwave response and analyze the response to a specific visual stimulus (image). Subsequently, the processor (130) detects the P300 signal for each EEG channel. Each graph shown in FIG. 6 represents the signal measured in a specific EEG channel (AF3, AF4, F3, F4). The X-axis of each graph in FIG. 6 represents time (ms), and the Y-axis represents the potential difference (μV, microvolt). Additionally, the graphs show the brainwave response when viewing a target image (Target Class) and the brainwave response to a general image (Non-target Class).

[0062] A peak called P300 appears in each graph of FIG. 6. In the embodiment, P300 is a potential associated with cognitive events, a signal that occurs 300ms after a person sees important information. This is related to attention and cognitive response. In the embodiment, it can be seen that the P300 signal appears more strongly when a target image (e.g., pizza) is viewed. In the embodiment, the processor (130) can automatically classify the image that the user is interested in (target image) based on the P300 signal.

[0063] In the embodiment, the processor (130) checks whether a P300 signal appears strongly when a user views a target image, enabling the AI ​​to automatically label it. This enables data labeling that is more intuitive and faster than the existing manual labeling method. Additionally, it allows incorrect labels (outliers) in the AI ​​training data to be corrected based on EEG signals and generates training data with higher reliability by utilizing brainwave responses. Furthermore, while data labeling previously had to be done manually by humans, applying an EEG-based labeling system allows for data refinement in a more objective manner.

[0064] In addition, the embodiment utilizes EEG signals to remove unnecessary data during AI training and to enable the construction of a more accurate dataset. In the embodiment, AI model training using P300 signals can be effectively utilized in visual BCI (Brain-Computer Interface) systems.

[0065] FIG. 7 is a diagram illustrating the determination of concentration and areas of interest using eye-tracking and eye-movement analysis. Referring to FIG. 7, in the embodiment, a processor (130) measures a person's concentration and areas of interest through eye-tracking and eye-movement analysis. This is to analyze the user's visual attention patterns to evaluate which information is receiving more attention. As illustrated in FIG. 7, the processor (130) is observing the user looking at a screen, and an eye-tracking device analyzes the movement of the pupils. In the embodiment, the processor (130) detects the user's pupil movements through a camera or an infrared-based eye-tracking sensor and analyzes which area the user is primarily gazing at. This allows for the detection of focusing more intently on specific parts of the text being read. Subsequently, the processor (130) analyzes the user's eye movement path to automatically track and evaluate the areas of interest.

[0066] Additionally, the processor (130) compares the reading patterns of the reader. In the embodiment, the reading patterns of a proficient reader (a) and an inexperienced reader (b) can be compared. The point shown in FIG. 7 is the area the user gazed at, and the movement of the line is the path the gaze moved. In the case of a proficient reader, the gaze follows a steady flow and shows a stable pattern, reads the text sequentially, and has minimal unnecessary eye movements. On the other hand, the reading pattern of an inexperienced reader shows an irregular pattern in which the gaze moves in various directions, has many movements of looking back at specific parts, and may have low concentration. Through this, the processor (130) can analyze the reader's reading pattern and measure proficiency and comprehension by utilizing eye tracking technology.

[0067] FIG. 8 is a conceptual diagram illustrating a semantic segmentation technique according to an embodiment. Semantic segmentation is a technique that classifies each pixel of an image into a specific object (class) or background. In the embodiment, individual objects or backgrounds in an image are identified by assigning each pixel to a specific class through semantic segmentation. In the embodiment, object classification is performed at the pixel level to clearly distinguish which category objects in the image belong to.

[0068] FIG. 8 illustrates the original image and the semantic segmentation result. The original image is an image of a typical road environment and includes various objects such as trucks, cars, buildings, roads, the sky, and trees. The semantic segmentation result classifies each pixel of the original image into a specific object (class) and represents it by color. For example, roads may appear in purple, the sky in blue, and buildings in gray. In the embodiment, the processor (130) classifies each pixel into a corresponding class (object) through semantic segmentation, thereby allowing for a clearer analysis of the object composition in the image.

[0069] Figure 9 is a diagram showing the result of successful semantic segmentation, Figure 10 is a diagram showing the result of failed semantic segmentation, and Figure 11 is a diagram showing an example of performing class-specific annotation work based on the original image and boundaries. Generally, as illustrated in Figure 11, the task of labeling objects by class based on the original image and boundaries is performed. This is a process of assigning each pixel to a specific class (object) by applying semantic segmentation technology.

[0070] FIG. 12 is a diagram illustrating a method for supplementing semantic segmentation data using EEG (brainwaves) and eye-tracking in an embodiment. Referring to FIG. 12, in an embodiment, a processor (130) records the brainwaves and eye movements of a worker to detect labeling errors and build more accurate training data. In the embodiment, EEG (brainwaves) and eye tracking are additionally recorded during the work process. That is, to prevent or supplement labeling errors in the existing Semantic Segmentation process, the processor (130) additionally utilizes EEG and eye-tracking data.

[0071] In the embodiment, the processor (130) displays the path of the pupils as a line when the operator views an image. Additionally, the point where the user's gaze rests (Fixation Point) can be displayed as a circle. Subsequently, the processor (130) analyzes the eye movement pattern to determine which object (class) the operator focused on more. For example, if the operator carefully observes a specific area in the semantic segmentation result, that part may be important or an area with a high probability of error. The graph visualizing EEG (brainwave) data along the time axis shown in FIG. 12 is the result of measuring visual response by recording voltage changes (μV) in multiple brainwave channels (Fp1, Fp2, etc.). Through this, the processor (130) analyzes how the EEG signal changes when viewing a specific image or object, and can identify the area with high concentration based on the analysis results. In the embodiment, by combining the change in EEG signal and the eye movement pattern, the processor (130) can analyze which part the operator paid more attention to and where the error is likely to occur.

[0072] Through this, the processor (130) can detect and correct parts with a high probability of error based on the worker's gaze and EEG response in the existing Segmentation results. In addition, labeling tasks that previously had to be reviewed manually by a human can be automatically corrected using EEG and eye-tracking data. Furthermore, it prevents the AI ​​model from learning incorrect data (outliers) and allows for the construction of a more accurate and reliable dataset. Additionally, through the embodiment, the possibility of labeling errors occurring when the worker is fatigued or their concentration is reduced can be predicted based on brainwave signals.

[0073] FIG. 13 is a diagram illustrating the process of detecting defective data (annotation errors) by utilizing EEG (brainwave) and eye movement data in a labeling device according to an embodiment.

[0074] Referring to FIG. 13, the processor (130) collects EEG (brainwave) and eye movement data, analyzes the worker's state using the collected data, and determines the possibility of errors in the generated training data based on the analysis results. In the embodiment, the processor (130) measures signals such as concentration, fatigue, and cognitive response in real time through an EEG sensor while the worker performs image labeling. The processor (130) evaluates the worker's cognitive state by detecting patterns of high or low attention concentration when viewing a specific image or object. Additionally, the processor (130) records the fixation points and saccades of the gaze while the worker is looking at the screen. Furthermore, the processor (130) evaluates the worker's attention patterns by analyzing how closely the worker gazed at a specific object during the labeling process and whether they moved on faster than expected. Through this, the processor (130) detects suspicious eye movements. In the embodiment, suspicious eye movements include cases where the worker repeatedly looks at a specific object or exhibits irregular patterns. Subsequently, the processor (130) analyzes the worker's condition using EEG data. In the embodiment, the processor (130) analyzes patterns such as attention, fatigue, and cognitive response from the EEG data. If the results of the brainwave analysis show that the level of concentration is lower than a certain level or the level of fatigue is high, the processor detects the state because the likelihood of errors occurring by the worker increases. In the embodiment, if the processor (130) detects that attention decreases rapidly above a certain level during the work or that abnormal brainwave patterns (P300, changes in Theta Power, etc.) are detected, it determines that the likelihood of errors is higher than a certain level.

[0075] Additionally, the processor (130) can evaluate the level of attention using eye-tracking data. In the embodiment, if the worker looks at a specific area for an excessively short period of time (less than a certain amount) or stares at it for a longer period than necessary during the labeling process, it is determined that the likelihood of the worker's uncertainty increasing increases. This is because, in normal cases, the gaze naturally moves along the object boundaries, but in the case of an error, the gaze movement pattern may be awkward or show a tendency to linger repeatedly in a specific area. In the embodiment, the processor (130) automatically detects locations where labeling errors are expected and classifies specific areas as suspicious data. Subsequently, the processor (130) evaluates the possibility of error by combining EEG and eye-tracking data. In the embodiment, the processor (130) combines EEG and eye-tracking data to compare and analyze the worker's cognitive state and attention patterns. In the embodiment, if specific data is classified as data with a high probability of error when the worker's attention was insufficient during labeling or when an abnormal brainwave response is detected, it is classified as data with a high probability of error. In the embodiment, a Confidence Score is assigned to data with a high probability of error, allowing for the automatic filtering of data that requires further review.

[0076] Additionally, the processor (130) can detect and correct errors by utilizing a machine learning model. In the embodiment, data with a high probability of error is compared with an existing semantic segmentation model to analyze the difference. In the embodiment, the model automatically corrects the quality of the training data based on the worker's state data. Additionally, in the embodiment, a new correct semantic image can be generated to continuously train and improve the AI ​​model.

[0077] In the embodiment, the processor (130) can determine the worker's condition by detecting signals such as concentration, fatigue, and cognitive response when the worker performs image labeling. For example, the processor (130) evaluates the worker's condition by collecting and analyzing EEG (brainwave) and eye movement data in real time. Through this, it can determine the reliability of the labeling work by detecting decreased concentration, accumulated fatigue, and cognitive load. To this end, the EEG (brainwave) sensor measures concentration, fatigue, and cognitive response in real time while the worker performs image labeling. In the embodiment, the processor (130) detects the worker's fatigue in real time by analyzing the change patterns of theta waves, delta waves, and alpha waves, and determines that there is a high possibility of labeling errors if the fatigue level is high. For example, the processor (130) uses beta waves ( It tracks changes in (13~30Hz). Since beta waves increase when concentration is high, the processor (130) determines that the worker is concentrating on labeling when the beta wave value is above a certain level. Also, since beta waves decrease when concentration is low, if the beta wave is below a certain level, it determines that attention is distracted or that the worker is likely to make a mistake.

[0078] Additionally, the processor (130) can evaluate the worker's condition by measuring the Event-Related Potential (ERP) that occurs when the worker processes new information. If the P300 signal appears strongly when viewing a specific image or object, the image is recognized as important information. On the other hand, if the P300 signal is weak or abnormal, the possibility of error increases due to low attention or high fatigue. Based on this, the processor (130) tracks the P300 Event-Related Potential in the brainwave signal, and if the tracked potential is above a certain value, it determines the reliability of the generated training data to be above a certain level. On the other hand, if the tracked potential is below a certain value, it can determine the reliability of the generated training data to be below a certain level.

[0079] In addition, the processor (130) measures the worker's fatigue level using theta waves ( It detects the increase in theta waves (48Hz) and delta waves (δ, 0.54Hz). As the worker's fatigue level increases, theta waves and delta waves increase. When working for a long time, the low-frequency components in the EEG signal become stronger due to fatigue accumulation, and attention decreases rapidly. Accordingly, if the increase in theta waves or delta waves exceeds a certain level, the processor (130) determines that the reliability of the training data generated at that time is below a certain level.

[0080] Additionally, the processor (130) analyzes changes in alpha waves (α, 8~13Hz) and determines that if the alpha waves exceed a certain value, the worker is drowsy or fatigued. In the embodiment, the reliability of the training data generated at the time when the alpha waves exceed a certain value can be determined to be below a certain level.

[0081] Additionally, the processor (130) analyzes the worker's eye movement patterns and gaze time to detect concentration and abnormal reactions. In an embodiment, the processor (130) detects abnormal reactions through eye movement data that records how carefully the worker looked at a specific part of an image.

[0082] Fixation Duration analysis analyzes how long the gaze stays on a specific object or image area. In the embodiment, if the fixation duration is short, the processor (130) determines that the likelihood of a quick glance without paying attention exceeds a certain level.

[0083] If the examination time exceeds a certain level, it is determined that the user is confused or hesitating in that area, and the reliability of the training data generated during that time is set below a certain level.

[0084] Additionally, the processor (130) can determine the reliability of the training data through the gaze pattern. In the embodiment, a normal gaze pattern is a gaze pattern that moves naturally along the boundaries of an object in semantic segmentation. On the other hand, an abnormal gaze pattern includes cases where the gaze is irregularly repeated in a specific part or where the screen is scanned quickly. In the embodiment, if an abnormal gaze pattern is detected, it is determined that the operator's attention is reduced.

[0085] In the embodiment, the processor (130) can detect gaze patterns through the analysis of saccade and fixation patterns. For example, when performing an accurate task, the gaze moves smoothly and stays in important areas for an appropriate amount of time. On the other hand, when attention is low, a pattern appears in which the gaze moves rapidly at a speed exceeding a certain speed (Scanning Effect) or repeatedly gazes at meaningless areas. In the embodiment, if the worker gazes elsewhere without looking at a specific object, the processor (130) determines that the likelihood of the worker making an error is above a certain level. Additionally, if the level of concentration drops sharply in a specific area, it determines that the worker's attention has decreased. In the embodiment, the gaze movement pattern and gaze time data are combined to evaluate whether the worker is performing the labeling task accurately.

[0086] Additionally, the processor (130) can combine EEG signals and eye-tracking data to comprehensively analyze the operator's cognitive state and determine the reliability of the training data based on the analysis results. For example, beta waves increase and P300 activation If a normal eye movement pattern is detected, It is determined that the operator is performing accurate labeling while focused, and the reliability of the training data generated at that time can be set to the highest range. Additionally, the processor (130) decreases beta waves and increases theta waves. If signals indicating rapid eye movement or gaze time shorter than a certain level are detected, it is determined that attention is reduced, and the error probability of the training data generated at that time can be set to a level higher than a certain level.

[0087] Additionally, the processor (130) may determine that the worker is confused if no P300 response is observed and the gaze is repeated in a meaningless area, and may set the reliability of the data generated at that time to a level lower than a certain level. Additionally, the processor (130) may determine that theta waves and delta waves increase If slowing of eye movement is detected, it is determined to be due to accumulated worker fatigue, and the probability of errors in the data generated at that time can be set above a certain level.

[0088] Additionally, the processor (130) performs a worker state-based error detection and action process. In the embodiment, if the worker is focused and the EEG and gaze data show a normal pattern, the reliability of the labeling data is set above a certain level. If the level of concentration decreases and the gaze movement pattern is irregular, the AI ​​model classifies the data as data requiring review. Subsequently, the processor (130) may warn the user to re-check the labeling or enable an automatic correction function. Additionally, if fatigue signals increase in the EEG data and the gaze moves irregularly, the processor (130) may recommend stopping the work or evaluate the data reliability as low.

[0089] Subsequently, the processor (130) combines two signals corresponding to brainwaves and eye-tracking data to detect cases where the worker is inattentive or makes a mistake. In an embodiment, the processor (130) can analyze the worker's state by inputting brainwaves and eye-tracking data into a machine learning model. In an embodiment, if the analysis results show that the worker's level of attention is lower than a certain level or if an abnormal response is detected, the model determines that the labeling has a higher probability of error than a certain level. In an embodiment, if the worker's state is normal, the model determines that accurate data is being generated, and if the worker's state is inattentive or has a high probability of error, it determines that there is a suspected defect.

[0090] Figure 13 illustrates an example in which a model analyzes EEG and gaze data to determine whether the data is normal or potentially defective. Normal data (a) is data where semantic segmentation was performed correctly. Roads, sidewalks, vehicles, the sky, buildings, etc., are correctly classified. Data suspected of being defective (b) is data where semantic segmentation was not performed correctly, or where the boundaries of objects are blurred or contain parts that are misclassified, and where there is a possibility of incorrect labeling.

[0091] In the embodiment, the processor (130) analyzes the worker's condition through EEG and eye-tracking data to automatically detect and correct incorrect data (defective annotations). This enables the automatic detection of labeling errors, thereby allowing for the construction of more reliable AI training data. Additionally, while previously a person had to manually review data errors, this method enables automatic correction using EEG and eye-tracking data.

[0092] In addition, it is possible to detect and prevent data errors caused by worker fatigue, decreased attention, etc., in advance.

[0093] FIG. 14 is a diagram illustrating a system for detecting defects in AI training data by utilizing EEG (brainwave) and eye movement data in an embodiment. Referring to FIG. 14, a processor (130) compares the original RGB image with the semantic segmentation result and further analyzes the signal data to determine the data that may be defective.

[0094] In the embodiment, the processor (130) determines the possibility of error in AI training data by utilizing RGB images, semantic segmentation images, and EEG and eye tracking data. In the embodiment, two signals corresponding to EEG and eye tracking are combined and analyzed with two images including RGB and semantic segmentation images. In the embodiment, the RGB image is the original image, which is the actual captured scene, and the semantic segmentation image is the result of semantic segmentation in which each pixel of the original image is classified into a specific object (Class).

[0095] In the embodiment, EEG (brainwave data) is the result of recording brainwave responses that occur when a user reviews the data. Through this, the processor (130) analyzes concentration, attention, fatigue, etc., to evaluate whether accurate labeling has been performed. In addition, the processor (130) analyzes the user's eye movements through eye tracking data (Eye Movement) to determine which part the user is paying more attention to. Since data containing incorrect labeling is likely to cause the gaze to focus on unexpected places, the processor (130) evaluates the labeling results through the brainwave data and eye tracking data.

[0096] In an embodiment, the processor (130) can evaluate the quality of AI training data by combining four types of data including RGB images, semantic segmented images, EEG, and eye tracking data, and can automatically detect data that may be defective.

[0097] In the embodiment, the processor (130) inputs RGB images, semantic segmentation results, EEG, and eye movement data into a machine learning model and analyzes the input data. In the embodiment, the model evaluates the worker's state (cognitive response, attention, gaze pattern, etc.) to determine whether the data is correctly labeled.

[0098] In the embodiment, the model determines whether the data is normal or has a possibility of being defective based on the results of analyzing four input data. To this end, the model evaluates whether the AI ​​model has properly segmented the data by comparing the original RGB image with the Semantic Segmentation results. For example, if a clear object exists in the RGB image but the boundary is unclear in the Semantic Segmentation result, the model sets the probability of error above a certain level. Additionally, if an object is classified differently from the original image, the probability of error is set above a certain level. Furthermore, if an object is excessively subdivided (over-segmented) through over-segmentation and under-segmentation analysis, the probability of incorrect labeling is set above a certain level. Also,

[0099] If an object is not recognized as a single entity but is divided into multiple parts (under-segmentation), the probability of error is set above a certain level. Additionally, the model evaluates the accuracy of the AI-generated data by comparing RGB images with semantic segmentation results and detects data with a high probability of error. In the embodiment, the model combines EEG, eye tracking, RGB, and Semantic Segmentation data to automatically identify data with a potential for defects. For instance, if concentration is low and fatigue is high, the probability of error is set above a certain level; similarly, if eye tracking data analysis reveals that the worker gazed abnormally at a specific object, the probability of error is set above a certain level.

[0100] In addition, if the AI ​​model misclassifies a specific object based on the comparison results between the RGB image and the semantic segmentation image, the error probability of the AI ​​model's training data is set to a certain level or higher. Furthermore, in the embodiment, the aforementioned RGB image, semantic segmentation image, and data including brainwaves and eye tracking can be combined to evaluate whether the data is normal or if there is a possibility of defects.

[0101] In the embodiments, the model retains normal data as is and modifies or removes data with a high probability of being defective. In the embodiments, the model integrates multiple input data to automatically detect errors in the semantic segmentation results. In the embodiments, the model analyzes EEG and eye-tracking data to detect areas with potential errors in the semantic segmentation results and supports automatic correction or review. In the embodiments, RGB images, semantic segmentation results, EEG, and eye-tracking data are combined to enable the construction of more reliable AI training data.

[0102] FIG. 15 is a diagram illustrating a system for determining the possibility of defects and error locations in AI training data by utilizing EEG (brainwave) and eye movement data in an embodiment. In an embodiment, the processor (130) compares the original (RGB) image with the Semantic Segmentation result and further analyzes the signal data to automatically detect locations where errors are likely to occur. In an embodiment, the processor (130) determines the possibility of errors and suspected locations in AI training data by utilizing the RGB image, the Semantic Segmentation image, and the EEG and eye movement data. To this end, the processor (130) combines and analyzes two signals (EEG + eye movement) and two images (RGB + Semantic). Through this, the processor (130) can determine not only the possibility of defects but also suspected error locations. In an embodiment, the processor (130) analyzes the operator's cognitive response using the EEG and eye movement data and detects specific areas (locations in pixel units) with a high probability of error by comparing the RGB and semantic segmentation data.

[0103] In the embodiment, EEG signal analysis is used to measure the worker's concentration and fatigue levels, thereby identifying tasks with a high probability of error. Additionally, eye-tracking analysis is used to analyze patterns of gazing at specific objects to locate areas with a high probability of error. Furthermore, the embodiment can detect incorrect regions by detecting object segmentation errors in the AI ​​model through a comparison of RGB images and semantic segmentation images. In the embodiment, the machine learning model integrates the three pieces of information mentioned above to automatically detect pixel locations with a high probability of error. In the embodiment, EEG signals enable the identification of locations with a high probability of error based on the worker's cognitive response. Additionally, the model determines that if the P300 signal appears strongly when viewing a specific object or area, the worker has recognized it as important information; conversely, if the P300 signal is weak or abnormal, it determines that the worker is skimming the area or that there is a high probability of error. Through this, the model can set locations where the P300 signal drops sharply above a certain level as suspected error areas. Moreover, the model uses beta waves ( ) decrease and theta wave( If an increase is detected, it means that the worker's concentration has decreased at that location, so the possibility of errors in that area can be set above a certain level.

[0104] In addition, in the embodiment, the model sets the probability of error to a certain level or higher when an object that is clearly distinguishable in the original RGB image has blurred boundaries or is incorrectly segmented in the semantic segmentation result, and sets the corresponding area as a suspected error location if the location and shape of the object appear differently in the semantic segmentation image and the RGB image. Furthermore, the model may mark an area with a high probability of error if a single object is excessively segmented into multiple parts, and may also mark an area with a high probability of error if multiple objects are grouped together.

[0105] In the embodiment, a machine learning model combines the four aforementioned data to evaluate the quality of AI training data and automatically detects specific locations with a high probability of error. In the embodiment, the model analyzes input RGB images, semantic segmentation results, EEG, and eye movement data. Subsequently, the model evaluates the operator's state (cognitive response, attention, gaze patterns, etc.) to determine whether the data has been correctly labeled. Through this, the model can identify specific locations with a high probability of error. In the embodiment, based on the results analyzed by the machine learning model, locations with a high probability of error are marked, and parts of the semantic segmentation image with a high probability of error are highlighted with red boxes. Subsequently, the marked areas can be automatically corrected or further reviewed and corrected. Furthermore, in the embodiment, the model integrates multiple input data to automatically detect locations with a high probability of error in the semantic segmentation results and supports their improvement.

[0106] FIG. 16 is a diagram illustrating a system that utilizes brainwave and eye movement data in an embodiment to detect errors in AI learning data and generate a new correct answer, a semantic segmented image.

[0107] Referring to FIG. 16, the processor (130) detects errors in AI training data by utilizing existing RGB images, semantic segmented images, EEG, and eye-tracking data. In the embodiment, going a step further than the existing error detection method, the machine learning model can automatically generate a new correct semantic segmented image. In the embodiment, parts with a high probability of defects are analyzed and corrected into a new semantic segmented image. To do this, the prediction value of the existing AI model is compared with the parts where errors are detected, and adjustments are made so that the uncertain parts can be retrained. In the embodiment, a corrected image can be generated using a Generative model (e.g., GAN, Autoencoder) or a Fine-tuned Semantic Segmentation model.

[0108] In addition, by applying an automatic ground truth data generation algorithm, the AI ​​model newly predicts accurate object boundaries to correct errors. This complements the results of the existing model and enables the generation of new ground truth semantic segmentation images. The ground truth data generated in the embodiment is further finely adjusted by referring to the original RGB images and eye-tracking data. In the embodiment, the AI ​​model conducts continuous learning using the new ground truth data and is improved to reduce similar errors in the future and generate semantic segmentation results with higher accuracy by learning from the new ground truth data.

[0109] In the embodiment, the machine learning model receives RGB images, original images (actual captured scenes), semantic segmented images, and semantic segmentation results generated by existing AI, and analyzes the input data. Additionally, in the embodiment, EEG (brainwave data) is data that records brainwave responses occurring when a user reviews the data, and through this, attention, cognitive response, fatigue, etc., can be analyzed to evaluate whether there are labeling errors. Eye movement data is data used to analyze the user's eye movements to determine which part they are paying more attention to. In the embodiment, if the operator lingers their gaze on a specific part for a certain period of time or shows a pattern different from what is expected, the model determines that it is an area with a high probability of error above a certain level. In the embodiment, the processor (130) comprehensively analyzes the four types of data mentioned above to identify the location of data with a high probability of defects and generates a new correct semantic image. To this end, the model analyzes the input RGB images, existing semantic segmentation results, EEG, and eye movement data. Subsequently, based on the analysis results, it detects areas with a high probability of error above a certain level, corrects them, and generates a new correct semantic segmented image. In the embodiments, the model can generate new correct answers by correcting errors in existing semantic segmented images. Additionally, in the embodiments, the model automatically generates AI-based corrected semantic segmentation results to improve the quality of the training data.

[0110] FIG. 17 is a diagram illustrating a system that continuously trains a semantic segmentation network using EEG (brainwave) and eye movement data in an embodiment and automatically generates a new correct semantic image. In the embodiment, the semantic segmentation image may be generated based on a semantic segmentation network. In the embodiment, the processor (130) analyzes errors using EEG and eye movement data and generates a new correct image reflecting the results. Subsequently, the processor (130) continuously trains the semantic segmentation network based on the new correct image, and

[0111] We can propose a process to improve data quality by utilizing only EEG and eye movement data without other input devices such as keyboards or mice.

[0112] In the embodiment, the original RGB image is input into a semantic segmentation network to generate semantic images classified by object. The semantic image is the classification result generated through the existing semantic segmentation network.

[0113] EEG (brainwave data) is the result of analyzing attention, cognitive response, and fatigue by recording brainwave responses that occur when a user reviews the data. Additionally, the processor (130) analyzes the user's eye movements through eye-tracking data to determine which part is being focused on more. In the embodiment, if the gaze shows a pattern different from what is expected, it is determined to be an area with a high probability of error.

[0114] The processor (130) comprehensively analyzes data including RGC images, semantic images generated by a semantic segmentation network, EGGs, and eye tracking results to detect errors in the semantic segmentation network and generate new correct semantic segmentation images. In the embodiment, the machine learning model analyzes input RGB images, existing semantic segmentation results, EEGs, and eye movement data. Subsequently, it detects areas with a high probability of error and corrects them to generate new correct semantic segmentation images. In addition, in the embodiment, the processor (130) can detect errors and automatically correct them using only EEG and eye movement data without manual work.

[0115] In addition, the model continuously improves the semantic segmentation network by using new ground truth semantic segmentation images generated by the machine learning network as training data. Through iterative learning, it enhances the accuracy of semantic segmentation and generates new ground truth semantic images. In the embodiment, new ground truths can be generated by correcting errors in existing semantic segmentation images. In the embodiment, the AI ​​model can automatically generate more accurate semantic segmentation results as it is progressively improved. In the embodiment, the ML model utilizes EEG and eye-tracking data to automatically correct areas with a high probability of error, generates new ground truth images, and continuously trains the semantic segmentation network based on this to improve performance.

[0116] The semantic segmentation labeling device and method using brainwaves and eye tracking according to the embodiment utilize EEG and eye tracking to analyze the worker's attention state in real time, thereby enabling the automatic detection of data with a high probability of error. This allows for the construction of a higher-quality training dataset and improves the accuracy and reliability of the AI ​​model.

[0117] Previously, operators had to manually inspect and modify data, which consumed significant time and cost; however, the embodiment enables the automation of the inspection process by automatically analyzing EEG and eye-tracking data to detect errors. Accordingly,

[0118] It can reduce data labeling and inspection time and maximize work efficiency.

[0119] Furthermore, since there is a high probability that an AI model will experience performance degradation or produce incorrect results if it learns from incorrect training data as is, the embodiments utilize EEG and eye tracking to detect and correct errors in the training data early, thereby ensuring that the AI ​​model learns accurate patterns. This improves the generalization performance of the AI ​​model and prevents performance degradation caused by errors.

[0120] In addition, in the existing method, the burden on workers was significant because data had to be inspected and modified using a keyboard and mouse; however, in the embodiment, data quality evaluation and modification can be performed using only EEG and eye tracking, thereby reducing worker fatigue and minimizing the work burden. This allows for a more intuitive and natural data inspection and labeling process.

[0121] In addition, in the embodiment, since the AI ​​automatically generates ground truth data based on EEG and eye-tracking data, the Semantic Segmentation Network can continuously learn and improve. This helps the AI ​​generate more accurate and precise semantic segmentation results and enables continuous performance improvement through iterative learning.

[0122] While existing data labeling and verification methods required a large workforce, were time-consuming, and costly, the embodiment utilizes EEG and eye tracking to automate the data verification and error correction processes, thereby reducing costs and improving the efficiency of building AI training data.

[0123] Furthermore, the semantic segmentation labeling device and method using EEG and eye tracking according to the embodiment improve the quality of AI training data, maximize the automation and efficiency of the labeling process, and create the effect of preventing performance degradation of the AI ​​model. In addition, since data quality can be evaluated and modified using only EEG and eye tracking without a keyboard or mouse, it reduces the burden on the operator and enables cost reduction.

[0124] Meanwhile, the methods according to the various embodiments of the present invention described above can be implemented in the form of an application or software program that can be installed on an existing electronic device.

[0125] In addition, the whole or part of the method may be composed of multiple software function modules and implemented on an operating system (OS). Alternatively, each step may be composed of a single software function module, or each step may be combined to form a single software function module and implemented on an operating system. Therefore, even if all of the embodiments of the present disclosure are not implemented as a single software function module, if multiple software function modules implement each step of the present disclosure and multiple software function modules are implemented on a single operating system, it can be understood that the method of the present disclosure has been implemented.

[0126] In addition, the methods according to the various embodiments of the present invention described above can be implemented solely through software upgrades or hardware upgrades of existing electronic devices. Furthermore, the various embodiments of the present invention described above can also be performed through an embedded server equipped in an electronic device or an external server of the electronic device.

[0127] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented as software comprising instructions stored on a computer-readable recording medium using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0128] Meanwhile, a computer or a similar device may include a device according to the disclosed embodiments, which is capable of calling instructions stored from a storage medium and operating according to the called instructions. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions directly or by using other components under the control of said processor. The instructions may include code generated or executed by a compiler or an interpreter.

[0129] A computer-readable recording medium may be provided in the form of a non-transitory computer-readable recording medium. Here, "non-transitory" simply means that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium. In this context, a non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as registers, caches, or memory. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0130] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 A semantic segmentation labeling device through brainwaves and eye tracking, comprising: a communication module; a memory storing at least one instruction; and a processor, wherein the at least one instruction is executed by the processor, the labeling device collects brainwave (EGG) and eye tracking (Eye Movement) data, analyzes the state of a worker based on the collected brainwave and eye tracking data, and determines the possibility of error in the training data generated according to the results of the worker state analysis, wherein the labeling device detects the worker's state, including concentration, fatigue, and cognitive response when the worker performs image labeling, through brainwaves for the purpose of analyzing the worker's state, detects abnormal reactions through eye tracking data (Eye Movement) that records the worker's gaze, and determines the worker's state of inattention using the brainwaves and abnormal reactions. Claim 2 A labeling device according to claim 1, wherein the labeling device inputs brainwave and eye-tracking data into a machine learning model, and the model analyzes the input data and, if an abnormal response is detected in which the worker's attention state is lower than a certain level, sets the error probability of the labeling generated at the time the abnormal response is detected to a certain level or higher, and if the worker's attention state is higher than a certain level, determines the worker's state as normal and determines that accurate labeling data is generated in the normal state of the worker. Claim 3 In claim 1, the labeling device inputs original RGB images, semantic segmented images, brainwaves, and eye tracking data into a machine learning model to determine the possibility of error in the training data, the model analyzes all four input data, evaluates the state of a worker based on the results of the data analysis, and determines the labeling state of the training data according to the results of the worker state evaluation, the state of the worker includes cognitive response, attention, and gaze pattern, and the labeling state includes accurate, inaccurate, and inattentive. Claim 4 A labeling device according to claim 1, wherein the labeling device compares an original RGB image and a semantic segmented image through a machine learning model, and further analyzes signal data including brainwave and eye tracking data to detect locations in the semantic segmented image where the probability of an error occurring is above a certain level, and the model evaluates the quality of training data generated by an operator based on the original RGB image, the semantic segmentation result, and brainwave and eye tracking data, and detects locations in the semantic segmentation result where the probability of an error is above a certain level according to the evaluation result. Claim 5 In claim 4, the labeling device analyzes errors in training data generated by an operator using brainwave and eye-tracking data, generates new correct semantic segmentation images by reflecting the error analysis results, continuously trains a Semantic Segmentation Network with the generated correct semantic segmentation images, and a machine learning model repeatedly trains the generated correct semantic segmentation images as training data to improve the semantic segmentation network.

Images