Data selection support device, data selection support method, and data selection support program

Abstract

Conventional methods for selecting learning / evaluation data from large amounts of images and point clouds are inefficient because they are manual. Pixel similarity such as hashing and regular downsampling cannot remove approximate overlaps or semantic redundancy, and they are insufficient in identifying small objects and rare events, managing conditions reproducibly, and tracking before and after. The solution automatically generates caption and attribute metadata using a visual language model, and eliminates semantic overlaps that are noteworthy in the domain based on the embedding similarity of images and captions, while narrowing down the results by metadata search and the presence or absence and number of estimated classes. The preprocessing / curation conditions, results, and correspondences are recorded, and the context and selection rationale are visualized in the user interface, making it possible to export across images and point clouds.

Description

[Technical Field]

[0001] The present invention relates to a curation (selection) technology for training data used in machine learning, and in particular to a data selection support device, method, and program that efficiently extracts data that meets specified conditions from visual data such as images and point clouds by combining captions and metadata automatically generated by a visual language model or the like with deduplication (semantic and pixel-wise) based on embedded vectors. [Background technology]

[0002] In recent years, with the advancement of discriminative and generative AI, the importance of "data-centric" development methods that optimize the quality and composition of training data, as well as improving the model itself, has increased. In particular, in fields that handle large amounts of visual data such as images and point clouds, the efficiency and quality of the selection (curation) of training and evaluation datasets directly affects model performance and development costs.

[0003] However, in the past, the typical operation involved manually inspecting large-scale original data sets to determine whether they were necessary or not. This method had the problem that the workload increased significantly as the number of data items increased, and the number of pairs when checking for duplicate data expanded exponentially, making manual brute-force verification virtually impossible.

[0004] The inclusion of duplicate data is known to have multiple adverse effects, including the waste of computing resources (such as GPUs) due to repeated training of the same content, the amplification of bias and reduced generalization performance through over-training of specific patterns, and even overestimation due to data leakage between training and evaluation.

[0005] To address these issues, conventional methods have been used, such as systematic downsampling of video and deduplication based on appearance and hash values. For example, in the field of autonomous driving, data volume is reduced by thinning out consecutive images from on-board sensors, taking into account factors such as driving speed, and then annotation targets are selected.

[0006] On the other hand, deduplication using appearance or hash information as an indicator has shown some effectiveness in removing approximate duplicates, which is difficult to do with regular downsampling. However, because it relies on pixel similarity, it has limitations in that it cannot adequately handle small objects such as pedestrians in the distance, or contextual and semantic information that is difficult to determine from appearance alone.

[0007] In recent years, "semantic deduplication" using image and text embedding vectors and data curation support tools have become well known, such as NVIDIA NeMo Curator, imagededup using hashing and CNN, and Semantic Deduplication. The concept and processing flow of semantic deduplication itself is also explained in various technical documents (Non-Patent Documents 1 to 4). However, even when using a technology like Semantic Dedup, the type of images that are determined to be similar (duplicates) depends on the model, which makes it difficult to apply when defining similarity based on domain requirements.

[0008] On the other hand, there is still much room for technical investigation into the construction of a general-purpose platform that can efficiently and accurately select data from a variety of visual data such as images and point clouds from both the learning and evaluation perspectives by integrating automatically generated captions and metadata (e.g., road environment, sign type, weather, time of day, presence or absence of pedestrians, etc.) using visual language models (VLMs) with semantic deduplication and conditional search based on embedding vectors.

[0009] In particular, with large-scale data such as that used in autonomous driving, there are challenges such as (i) the inefficiency of visually-based sorting, (ii) the limitations of deduplication based on pixel similarity, (iii) the insufficient capture of small objects and rare events, (iv) a design to prevent leaks between learning and evaluation, and (v) reproducible sorting based on set conditions (prompts, search conditions, degree of duplication, etc.). To address these challenges, a system is needed that can handle data consistently as a pipeline while utilizing both notable semantic information and metadata in the domain.

[0010] As described above, improving the curation of visual data (images and point clouds) is important from the perspective of ensuring the reliability of model performance and improving development efficiency, and there is a need to develop selection technology that organically combines semantic information, metadata, and deduplication. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] NVIDIA "NeMo Curator" (OSS for data curation: deduplication workflow using embedded vectors, etc.) URL: https: / / github.com / NVIDIA-NeMo / Curator (Retrieved September 10, 2025) [Non-patent document 2] idealo “imagededup” (image duplicate detection library: deduplication based on hash values ​​and CNN features) URL: https: / / github.com / idealo / imagededup (Retrieved September 10, 2025) [Non-patent document 3] Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, Ari S. Morcos, “SemDeDup: Data-efficient learning at web-scale through semantic deduplication”, Submitted on 16 Mar 2023 (v1), last revised 22 Mar 2023 (this version, v3)URL https: / / arxiv.org / abs / 2303.09540 (Searched on September 10, 2025) [Non-patent document 4] “ruri-v3-310m” (an embedding model for high-precision vectorization of Japanese text) URL: https: / / huggingface.co / cl-nagoya / ruri-v3-310m (accessed September 10, 2025) Summary of the Invention [Problem to be solved by the invention]

[0012] The purpose of this invention is to eliminate the inefficiencies and inconsistencies inherent in the curation process, which involves selecting appropriate data for learning and evaluation with high reproducibility from visual data such as images and point clouds. In particular, in workplaces where annotation can easily become a bottleneck for the entire process, it is necessary to address the fundamental issue of development being hindered due to the inability to quickly extract data to be annotated.

[0013] Traditionally, manual visual selection of original data has been the norm, but when the data scale reaches hundreds of thousands to millions of items, the workload becomes enormous, and in particular, the number of combinations required for duplicate checks increases exponentially, making manual exhaustive selection virtually impossible. The inclusion of duplicate data has multiple adverse effects, including the waste of computing resources, bias amplification and reduced generalization performance due to overfitting, and even damage to the reliability of evaluation due to data leaks between training and evaluation.

[0014] To address this situation, methods such as regular downsampling based on factors such as driving speed and deduplication based on appearance and hash values ​​have been used, but they have limitations, such as not being able to fully remove approximate duplication or semantic redundancy, leaving data that is similar in content. Furthermore, techniques such as Semantic Dedup also rely on models to determine which images are deemed similar (duplicate), making them difficult to apply when defining similarity based on domain requirements. As a result, they tend to be inadequate in detecting small objects in the distance or rare events.

[0015] Furthermore, there is no general-purpose infrastructure in place that organically integrates and handles captions and environmental attributes (road type, weather, time of day, sign content, pedestrian presence, etc.) automatically generated by visual language models (VLMs) with semantic deduplication and conditional search based on embedding vectors in a single pipeline. Furthermore, there is a lack of sufficient reproducibility, such as parameterizing and recording which prompts, search conditions, and degree of duplication were used for preprocessing and selection, and keeping track of the correspondence between data before and after curation.

[0016] In addition, as the judgments required of AI become more sophisticated, the equipment configurations and flows that can consistently handle preprocessing (captioning, attribute metadata assignment, embedding) through condition input, deduplication, and result output across different modalities, such as images (or videos) and point clouds in autonomous driving technology, and images (or videos), point clouds, and joint angle sensors in AI robotics technology, tend to become fragmented, and it is difficult to say that the systematization of user interfaces and result management (pre- and post-curation datasets, condition logs, visualization of selection grounds, etc.) is sufficient. These increase the complexity of systemization and operation, hindering the balance between final data quality and development efficiency.

[0017] Based on the above, the objective of this invention is to realize a curation platform that can efficiently extract the minimum necessary, high-quality data by providing captions and attribute metadata using VLM, semantic deduplication based on image and text embedding, condition management including prompts, search conditions, and redundancy and ensuring their reproducibility, and providing a consistent pipeline spanning images and point clouds. [Means for solving the problem]

[0018] A data selection support device according to a first embodiment of the present invention is a data selection support device that receives an original data set as input and generates curated data from pre-curation data, a first execution unit that generates captions and attribute metadata for images or point clouds using a visual language model, and generates embedding vectors for the images and / or the captions using an embedding model; a first recording unit that records the caption, the attribute metadata, the embedding vector, and preprocessing conditions; a second execution unit that performs filtering and semantic deduplication by referring to the captions, the attribute metadata, and the embedding vectors recorded in the first recording unit and preprocessing conditions based on curation conditions input by a user, the curation conditions including at least metadata search conditions or a degree of deduplication, and identifies curated data; a first output unit that sets the curation conditions and presents data before and after curation; The system is characterized by comprising a second recording unit that records the curation conditions, the results of the filtering and deduplication, and a correspondence relationship log.

[0019] A data selection support device according to a second embodiment of the present invention is a data selection support device according to the first embodiment, characterized in that the first execution unit generates captions by converting images into text and generates attribute metadata required for curation conditions.

[0020] A data selection support device according to a third embodiment of the present invention is a data selection support device according to the first or second embodiment, characterized in that the second execution unit performs semantic duplication elimination processing to determine approximate duplication based on the similarity between the embedding vectors of the image and / or the embedding vectors of the caption, and adjusts the threshold for the determination depending on the degree of duplication elimination.

[0021] A data selection support device according to a fourth embodiment of the present invention is a data selection support device according to either the first or third embodiment, characterized in that the first output unit generates a user interface that displays both pre-curation data and curated data and visualizes the selection basis including at least compatible attribute metadata, neighbor candidates for duplicate elimination, and similarity.

[0022] A data selection support device according to a fifth embodiment of the present invention is the data selection support device according to any one of the first to fourth embodiments, wherein the original data set includes image data and point cloud data; the first execution unit performs preprocessing on the image data and the point cloud data across modalities, including at least one of captioning, attribute metadata assignment, and embedding; The second execution unit is characterized in that it performs curation on either or both of the image data and the point cloud data.

[0023] A data selection support device according to a sixth embodiment of the present invention is the data selection support device according to any one of the first to fifth embodiments, wherein the first recording unit and the second recording unit traceably record the preprocessing conditions and curation conditions, and the correspondence between the preceding and following data; the pre-processing conditions include at least a sampling setting and a prompt used to generate metadata; The curation conditions include at least a metadata search condition or a degree of deduplication.

[0024] A data selection support device according to a seventh embodiment of the present invention is a data selection support device according to any of the first to sixth embodiments, characterized in that the first execution unit generates pre-curation data by pre-processing including downsampling from video.

[0025] A data selection support device according to an eighth embodiment of the present invention is a data selection support device according to any of the first to seventh embodiments, characterized in that the second execution unit narrows down the data by combining the presence or number of class objects estimated by the discriminant model with the attribute metadata.

[0026] A data selection support device according to a ninth embodiment of the present invention is a data selection support device according to any of the first to eighth embodiments, characterized in that the first output unit provides an export function for curated data and is configured to be able to link with an external system as a dataset for annotation or model development.

[0027] A data selection support method according to a tenth embodiment of the present invention is a data selection support method by a computer, (i) taking an original dataset, generating captions and attribute metadata for images or point clouds using a visual language model, and generating embedding vectors for the images and / or the captions using an embedding model; (ii) recording the caption, the attribute metadata, and the embedding vector and preprocessing conditions; (iii) a step of identifying curated data by filtering and semantic deduplication based on curation conditions input by a user, including at least a metadata search condition and a deduplication degree, by referring to the recorded captions, the attribute metadata, the embedding vectors, and preprocessing conditions; (iv) setting the curation conditions and presenting data before and after curation; (v) recording the curation conditions, the results of the filtering and deduplication, and a correspondence log; The present invention is characterized by comprising:

[0028] A data selection support method according to an eleventh embodiment of the present invention is a data selection support method according to the tenth embodiment, characterized in that step (i) includes generating captions by converting images to text and generating attribute metadata required by the curation conditions.

[0029] A data selection support method according to a twelfth embodiment of the present invention is a data selection support method according to the tenth or eleventh embodiment, characterized in that the step (iii) includes performing a semantic deduplication process in which approximate duplication is determined based on the similarity between the embedding vectors of the image and / or the embedding vectors of the caption, and the threshold for the determination is adjusted according to the degree of deduplication.

[0030] A data selection support method according to a thirteenth embodiment of the present invention is a data selection support method according to any of the tenth to twelfth embodiments, characterized in that the step (iv) includes generating a user interface that displays both pre-curation data and curated data and visualizes the selection basis including at least suitable attribute metadata, neighbor candidates for duplicate elimination, and similarity.

[0031] A data selection support method according to a fourteenth embodiment of the present invention is the data selection support method according to any one of the tenth to thirteenth embodiments, wherein the original data set includes image data and point cloud data; The step (i) is characterized by including preprocessing the image data and the point cloud data across modalities, including at least one of captioning, attribute metadata assignment, and embedding.

[0032] A data selection support method according to a fifteenth embodiment of the present invention is a data selection support method according to any one of the tenth to fourteenth embodiments, wherein in the steps (ii) and (v), preprocessing conditions and curation conditions, and correspondences between previous and following data are recorded in a traceable manner; the pre-processing conditions include at least a sampling setting and a prompt used to generate metadata; The curation conditions include at least a metadata search condition or a degree of deduplication.

[0033] A data selection support method according to a 16th embodiment of the present invention is a data selection support method according to any of the 10th to 15th embodiments, characterized in that step (i) includes generating pre-curated data by pre-processing including downsampling from video.

[0034] A data selection support method according to a 17th embodiment of the present invention is a data selection support method according to any of the 10th to 16th embodiments, characterized in that the step (iii) includes narrowing down the data by combining the presence or absence or number of class objects estimated by the discriminant model with the attribute metadata.

[0035] A data selection support method according to an 18th embodiment of the present invention is a data selection support method according to any of the 10th to 17th embodiments, characterized in that step (iv) includes exporting the curated data to an external system as a dataset for annotation or model development.

[0036] A data selection support program according to a 19th embodiment of the present invention is characterized in that it causes a computer to function as the first execution unit, the first recording unit, the second execution unit, the first output unit, and the second recording unit of a data selection support device according to any of the first to ninth embodiments.

[0037] A data selection support program according to a twentieth embodiment of the present invention is characterized in that it causes a computer to execute steps (i) to (v) of the data selection support method according to any of the tenth to nineteenth embodiments. [Effects of the Invention]

[0038] This invention integrates captions and attribute metadata generated by a visual language model with domain-specific semantic deduplication and metadata search based on image / text embeddings, enabling high-speed and high-accuracy pre- and post-curation selection processes. This significantly reduces the workload associated with manual visual selection and pre-annotation preparation time, while also minimizing the waste of computational resources required for training. Furthermore, deduplication reduces data leaks and overfitting between training and evaluation, contributing to improved reliability of model evaluation and generalization performance.

[0039] Furthermore, by managing search conditions (attributes, class presence / absence / number, etc.) and the degree of deduplication as parameters, and systematically recording preprocessing conditions, curation conditions, acceptance / rejection results, and pre- and post-processing correspondences, it is possible to ensure the reproducibility, traceability, and explainability of the processing.The user interface makes it possible to display a comparison of data before and after curation, and visualize the selection basis (matching attributes, neighbor candidates, similarity, etc.), supporting precise data extraction that reduces the oversight of rare events and small objects.

[0040] Furthermore, this invention can be configured as a consistent pipeline across images and point clouds (including video downsampling), and captions and metadata used for deduplication can be flexibly generated using prompts, making it scalable and independent of specific domains, model types, and deployment methods (cloud, on-premise, edge).As a result, it can stably supply high-quality datasets for downstream annotation and model development, and can be smoothly integrated into MLOps and data governance frameworks. [Brief explanation of the drawings]

[0041] [Figure 1] 1 is a diagram showing a schematic configuration of a computer system to which a data selection support device according to an embodiment of the present invention can be applied. [Figure 2] 1 is a block diagram showing a hardware configuration of a data selection support device (computer system) according to the present invention. [Figure 3] 2 shows a series of processing flows executed by the data selection support device according to the present invention. [Figure 4] 1 shows details of processing from pre-processing to curation in the data selection support device of the present invention. [Figure 5] 1 shows a flowchart of pre-processing executed by a data selection support device according to the present invention. [Figure 6] 1 is a flowchart of a curation process according to the present invention. [Figure 7] 1 shows an example of a data structure generated by the preprocessing of the present invention and stored in a recording unit. [Figure 8]10A and 10B are diagrams showing examples of a user interface, illustrating a display screen of a pre-curation dataset and a curation execution screen. [Figure 9] 10A and 10B are diagrams showing specific examples of metadata and captions automatically added by VLM. DETAILED DESCRIPTION OF THE INVENTION

[0042] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings. The individual embodiments of the present invention are not independent and can be appropriately combined with each other for implementation.

[0043] In this specification, "neighborhood candidates" refers to a set of data that are positioned nearby in the embedding space using a specified similarity scale. "Selection basis" refers to the metadata compatibility, similarity, candidate set, etc. that were factors in the decision to accept or reject. "Export" refers to the process of outputting curated data and related metadata / correspondence logs in a specified format to an external device. "Pre-curation data" and "curated data" refer to the data sets finalized at the completion of preprocessing and curation, respectively.

[0044] FIG. 1 is a block diagram showing the hardware configuration of a system including a data selection support device according to one embodiment of the present invention. As shown in FIG. 1, the system S includes terminals 1-1 to 1-N (N is a natural number) used by users and a computer system 2 connected to these terminals 1-1 to 1-N via a communication network CN. An edge configuration in which terminal 1 and computer system 2 are housed in the same housing, or a configuration in which the database device is housed in a separate housing, are also possible. Note that the connecting lines in FIG. 1 indicate logical connections; the actual wiring and equipment configuration are not limited to these. Furthermore, each process of the present invention may be executed by computer system 2 using a program stored on a non-transitory recording medium, or a portion of the program may be executed on terminal 1. A variety of system configurations are possible, including an edge-integrated type in which a processing unit is housed in the same housing as the terminal, a distributed execution type on the cloud, and a three-tier type in which the database is housed in a separate housing. The processing load (preprocessing, curation, and visualization) is allocated according to network bandwidth and latency.

[0045] Terminals 1-1 to 1-N are devices used by users to perform annotation work, etc., and include, for example, multi-function mobile phones (so-called smartphones), tablets, laptops, and desktop computers. These terminals are collectively referred to as terminal 1. Terminal 1 may be an acquisition device having an image sensor (fixed camera, vehicle-mounted camera, etc.) or a point cloud sensor (LiDAR, etc.), or a management terminal / operator terminal that stores existing data. Terminal 1 transmits the original images and point clouds, metadata associated with the data such as time, location, and device settings, and, if necessary, estimation results obtained on the terminal side, to computer system 2 via communication network CN.

[0046] The computer system 2 functions as a data selection support device by executing the data selection support program of the present invention. The computer system 2 is an information processing device including a CPU, a GPU, a main memory, an auxiliary memory, a network interface, a display device, etc., and realizes the functional blocks of the present invention (first execution unit, first recording unit, second execution unit, second recording unit, first output unit, etc.: see Figure 2 for details) through software or software-hardware collaboration. Specifically, the computer system 2 imports original data received from the terminal 1, automatically generates caption / attribute metadata using a visual language model (VLM), calculates image or caption embedding, filters based on metadata conditions, and performs semantic deduplication based on embedding similarity to generate curated data. Furthermore, the computer system 2 records preprocessing conditions, curation conditions, results, and contextual correspondences, and provides operation input, result display, and evidence visualization from the terminal 1 via a user interface. Note that the computer system 2 may be composed of a single computer or multiple computers. Below, an example configured as a single computer is described.

[0047] To improve traceability, the first recording unit 263 and the second recording unit 264 detect log tampering using hash chains, etc., and assign high-precision timestamps and the executor to execution events. Access control and encryption are applied, and the logs can be output in bulk upon audit request.

[0048] Figure 2 is a block diagram showing the hardware configuration of a data selection support device (computer system) according to the present invention. As shown in Figure 2, the computer system 2 includes an input interface 21, a communication module 22, a storage device 23, a memory 24, an output interface 25, and a processor 26. The processor 26 loads a program from the storage device 23 into the memory 24 and executes it, operating as the functional blocks (first execution unit, second execution unit, first recording unit, second recording unit, and first output unit) described below.

[0049] The input interface 21 accepts operational inputs (such as curation conditions and preprocessing conditions) from an administrator or operator and outputs corresponding input signals to the processor 26. The communication module 22 is connected to the communication network CN and exchanges original images, point clouds, metadata, estimation results, and the like with the terminal 1. This communication may be wired or wireless. The storage device 23 stores programs executed by the processor 26 and various data (such as pre- and post-curation data, generated metadata, embedding, and condition logs), and the memory 24 functions as a work area for temporarily storing the programs and data. The output interface 25 can be connected to an external device and can output signals to the external device as needed. The functional blocks realized by the processor 26 are the first execution unit 261, second execution unit 262, first recording unit 263, second recording unit 264, and first output unit 265 shown in the figure. These are logical configurations and may be implemented in a single or distributed manner across multiple cores / threads.

[0050] The program for realizing the functions of the present invention can be distributed via a network, recorded on a storage medium (semiconductor memory, optical disk, magnetic disk, etc.), or provided as a cloud service. The program may be provided in a container, and when read from the medium, a processor executes the method of the present invention.

[0051] The first recording unit 263 stores at least the following as a schema for preprocessing: collection date and time, terminal identifier, data identifier, sampling setting (method, interval, random number seed), image / point cloud normalization condition, visual language model (type, version, prompt), discriminant model (type, version), and embedding model (type, version, number of dimensions). The second recording unit 264 stores the following as a schema for curation: search formula (metadata condition expressed by a logical formula described below), degree of deduplication, number of neighbor candidates, similarity index, threshold, and context correspondence log (original data ID ⇔ result data ID, acceptance / rejection, reason tag, similarity, execution time, executor).

[0052] VLM, discriminative models, and embedding models are interchangeable. The type, training data, and version of each model are recorded as preprocessing conditions, and when changes are made, the differences in the conditions are reflected in the log. This makes it possible to compare and verify results even after updating the model.

[0053] The first execution unit 261 automatically generates captions and attribute metadata (road type, weather, time of day, sign content, presence or absence of pedestrians, etc.) using a visual language model for the original images / point clouds acquired from the terminal 1, and also calculates embedding vectors for the images or captions. If necessary, it also performs preprocessing such as downsampling of the video.

[0054] In this specification, "duplication elimination level" is a dimensionless parameter that indicates the strength of semantic deduplication, and is specified, for example, as a continuous value between 0 and 1 or a step value. The threshold value T is given by a monotonic function such as T = Tmin + (Tmax - Tmin) × deduplication level, and the larger the value, the stricter the deduplication will be. A default value is set in the UI, and the same value is used when re-executing, ensuring the reproducibility of results.

[0055] The second execution unit 262 performs (i) filtering using attribute metadata and the presence / absence / number of estimated classes, and (ii) semantic deduplication based on the similarity of embedding vectors (adjusting the threshold according to the degree of duplication) based on the user's curation conditions (at least metadata search conditions and deduplication level), and identifies the curated data. The first recording unit 263 records the generated captions, attribute metadata, and embedding vectors, as well as preprocessing conditions such as sampling settings and prompts, ensuring reference and reexecutability for subsequent processing. If necessary, it also builds an embedding index, etc., in the database.

[0056] Semantic deduplication uses a high-speed approximate search to find neighboring candidates for each data based on the similarity between embedding vectors (e.g., cosine similarity), identifies candidates with a similarity greater than or equal to a threshold T as a duplicate group, and adopts only representative elements. If necessary, the group is determined by clustering (e.g., connected components), and the representative is selected using a deterministic rule (e.g., ID order) that is not affected by random numbers.

[0057] The second recording unit 264 records the curation conditions, the results of filtering and deduplication, and the correspondence before and after curation (traceability information). The first output unit 265 provides a condition input screen, processing progress, a comparative display of data before and after curation, and visualization of the selection basis (matching attributes, similarity, neighbor candidates, etc.), and outputs the results in a format that can be linked to external devices and systems in response to user operations.

[0058] This computer system 2 can be configured as a single computer or distributed across multiple computing resources, and is scalable according to the application and processing volume. With the above configuration, the computer system 2 in Figure 2 efficiently and reproducibly performs curation processing on general-purpose hardware, from preprocessing (captioning, attribute assignment, and embedding) to condition input, semantic deduplication, and result recording and visualization. Processing is executed in batches or microbatches via a job queue, and computing resources are horizontally scaled according to the load. The GPU pool supports automatic expansion and contraction, and scaling policies can be determined using indicators such as queue length and waiting time.

[0059] For large-scale searches, we generate a neighborhood search index (e.g., HNSW) for the embedding. We record the generation conditions (maximum number of vertices, M,ef) and automatically rebuild it if the model or dimensionality changes. The index can be either memory-resident or disk-resident.

[0060] FIG. 3 shows a series of processing flows executed by the data selection support device (see FIG. 2) according to the present invention. First, in the data preparation process, image data and point cloud data collected from terminals are stored in the original dataset 231 ("images and point clouds"). As preprocessing, captions and attribute metadata are automatically assigned to the original dataset using a visual language model, and the images or captions are embedded into vectors ("preprocessing (caption and metadata assignment and embedded vectorization)"). At this time, sampling settings, the contents of the metadata to be generated, prompts, etc. are entered as preprocessing conditions and recorded for future re-execution.

[0061] The results of the pre-processing are aggregated as pre-curated data 232, which includes (i) sampled images / point clouds, (ii) the metadata, and (iii) embedding vectors based on captions, etc. At this stage, an index for neighborhood search of the embedding vectors may be generated.

[0062] Next, in the curation process, the system accepts user-provided curation criteria (including at least search criteria and deduplication level) and performs (a) metadata-based filtering and (b) semantic deduplication based on embedding similarity. Deduplication involves adjusting a threshold value according to the specified degree of duplication. This extracts the minimum amount of data necessary for learning and evaluation, resulting in a curated dataset 233.

[0063] The generated curated dataset 233 consists of images, point clouds, and corresponding metadata, and is handed over to the subsequent annotation / model development process (right side of the figure). In this flow, upstream data preparation and downstream curation are clearly separated, and preprocessing and curation conditions are recorded, ensuring the reproducibility and traceability of the selection process. Note that preprocessing sampling can be performed using methods such as thinning out video, and caption and metadata assignment and embedding generation are performed in conjunction with the various model inference methods shown in Figure 4 (see Figures 4, 5, and 6 for details).

[0064] Figure 4 shows the details of the process from preprocessing to curation in the data selection support device of the present invention. In the preprocessing step, the user first sets the filtering conditions, such as pixel-by-pixel matching, class objects, class attributes, and image attributes ("Condition Setting"). These are referenced in the inference and deduplication processes described below and are recorded for future use. Next, (i) as a pre-rule application, frames are extracted from the video at a predetermined interval (e.g., 10 seconds) and pixel matching is calculated as needed ("Rule Application"); (ii) the presence and number of class objects are estimated using a discriminative model; (iii) metadata for class attributes and image attributes is generated using image-to-text (VLM) and captions are added; and (iv) the embedding vectors for images and / or text (captions) are calculated using an embedding model ("Model Estimation"). The data is then aggregated into pre-curation data 232.

[0065] Video downsampling can be performed at least one of the following intervals: equal time intervals (e.g., every 10 seconds), equal distance intervals, speed-dependent intervals (coarse intervals in low-speed regions and fine intervals in high-speed regions), or keyframe extraction. Random seed data is recorded as a preprocessing condition for reproducibility.

[0066] In the curation process, filtering is first performed using class information and meta-information based on the curation criteria entered by the user (at least the search criteria and the degree of deduplication). Next, semantic deduplication (e.g., SemDedup) is applied using the similarity between embedding vectors, and redundant data, including near-duplicates, is removed by adjusting the threshold according to the degree of deduplication parameter. The result is the curated dataset233, which is then supplied to downstream processes for learning and evaluation.

[0067] In this embodiment, specific examples of metadata / captions can be generated, such as sign content, road type, weather, time of day, and the presence or absence of pedestrians. The caption text describes the landscape and situation in natural language. These are used as semantic expressions for filtering conditions and deduplication. Furthermore, the deduplication process also assumes the use of OSS (e.g., NeMo Curator) that utilizes embedding vectors, making it possible to reduce semantic redundancy, which is difficult to achieve using conventional downsampling or pixel similarity alone. This allows for the efficient extraction of the minimum necessary, high-quality data.

[0068] Figure 5 shows a flowchart of the pre-processing executed by the data selection support device according to the present invention. First, in step S101, the process starts with acquiring the original data set, and importing images and point clouds collected by a terminal or the like. Next, in step S102, pre-processing conditions, such as at least the sampling rate, the content of the metadata to be generated, and a prompt to be given to the VLM, are accepted as input and recorded for future re-execution. Examples of the prompt contents are as follows: "Please generate captions in Japanese that explain traffic conditions for images taken from the driver's point of view. - There is no need to explain the premise, such as "This image was taken from the driver's point of view." - Describe the location and status of surrounding objects such as cars, motorbikes, bicycles, and pedestrians - Do not describe the behavior of cars, motorcycles, bicycles, or pedestrians - Determine whether there are any signals ahead that drivers should pay attention to. - Include the color of a signal that requires attention **if and only if** - Describe the scenery of the place where you are driving - Include a description of surrounding buildings, if any. - Explaining traffic flow and congestion - Include explanations of traffic signs and road markings, if any. - Include descriptions of weather and time of day

[0069] In step S103, the acquired "all target data" is sequentially listed as the processing target, and the following steps are applied. First, in step S104, the data volume is adjusted by downsampling, such as extracting frames at predetermined intervals from video-derived data. Next, in step S105, metadata is generated by automatically assigning attribute metadata such as road type, weather, time of day, sign content, and pedestrian presence using a visual language model (VLM). In step S106, captions are generated that express the image content in natural language, also using a VLM.

[0070] Finally, in step S107, the embedding vector is generated by calculating the embedding of the image or caption. These outputs (sampled image / point cloud, metadata, embedding) are aggregated as pre-curation data (step S108: processing of all target data completed → end). As a result, the pre-processing flow in Figure 5 conditionally and consistently generates pre-curation data to be input to the subsequent curation stage (metadata search and semantic deduplication based on embedding similarity). The process is repeated for all target data until steps S104 to S107 have been performed.

[0071] Preprocessing can be performed serially in the order of metadata generation → caption generation → embedding generation, or in parallel depending on GPU resources. Each step is designed to be idempotent, and in the event of a failure, the number of retries, skip conditions, and error cause are logged.

[0072] Point cloud data is also preprocessed by voxelization and outlier removal, and attribute metadata such as the presence or absence of roads, shoulders, guardrails, and obstacles, as well as density and ground estimation results are added. Point cloud embedding is calculated using a PointNet system or similar, and is subjected to semantic deduplication in the same way as for images.

[0073] FIG. 6 is a flowchart of the curation process according to the present invention. The process begins in step S201 with the acquisition of a pre-curation dataset, and then reads out the images / point clouds, associated metadata, and embeddings generated and accumulated in the preprocessing of FIG. 5. Next, in step S202, the curation conditions are input by providing the metadata conditions used by the user for search (e.g., the presence or absence of a class, attributes, image attributes, etc.) and a deduplication level, which indicates the deduplication strength described below. These are accepted as "curation conditions (search conditions and deduplication level)." For example, in the case of autonomous driving technology, the attribute metadata required for the curation conditions is attribute information including at least road type, weather, time of day, sign content, region, and the presence or absence of vehicles and pedestrians.

[0074] In step S203, metadata filtering extracts only relevant data using class information and meta-information obtained from the discriminative model and VLM. For example, attributes such as "urban / rural," "highway," "time of day," and "pedestrian presence / absence" can be specified as Yes / No or a value. In step 204, deduplication processing removes redundant data, including near-duplicates, based on the similarity between the image or caption embeddings obtained in preprocessing. Implementation techniques include SemDedup, which uses cosine similarity to determine proximity, and adjusts the threshold according to the input deduplication level. The results obtained through the above process are finalized as a curated dataset and supplied (exported) to subsequent annotation and model development processes.

[0075] 7 shows an example of the data structure generated by the preprocessing of the present invention and stored in the recording unit, where metadata and captions are associated with each data item (Data 1 to Data n) and stored. The metadata is expressed as a set of key-value pairs including scene and object attributes obtained by VLM inference as well as environmental information (traffic signals, signs, urban / mountainous, road type, weather, time of day, presence or absence of pedestrians and vehicles, etc.), and the caption describes the image content in natural language.

[0076] Metadata search expressions are written using the basic unit of <term comparison operator value>, and can be written using AND / OR / NOT, parentheses, ranges (BETWEEN), sets (IN), and boolean expressions (=TRUE / FALSE). Example: is_highway=TRUE AND weather IN {"sunny","cloudy"} AND time_of_day BETWEEN "morning" AND "afternoon".

[0077] In the example of data 1, the metadata includes traffic_lights (e.g., color: "red", rotation: "horizontal") and traffic_signs (e.g., meaning: "no_entry", shape: "diamond", text_in_sign: "No entry"), environmental flags are is_urban: 0, is_mountain_road: 1, is_highway: 1, is_intersection: 0, is_tunnel: 0, is_curve: 0, weather / time are weather: "sunny", time_of_day: "afternoon", and existence flags are has_pedestrian: 0, has_bicycle: 0, has_motorcycle: 0, has_car: 1, has_truck: 0. The corresponding caption describes the road shape, background, weather, traffic conditions, etc. in sentences (for example, "A white car is parked ahead, and a yellow sign is installed behind the car. The word 'Caution' can be seen on the sign. The road curves to the right, with mountains in the background. The weather is clear, with blue skies. Trees grow on both sides of the road, and the scenery is rich in nature. The driving location appears to be a highway. Traffic is relatively calm, and there is no sign of congestion."). These are stored in an ordered table format as shown in the figure.

[0078] This structure allows the curation process to extract desired data by combining high-speed filtering using metadata search criteria (e.g., is_highway=1 and weather="sunny" and has_pedestrian=0) with semantic deduplication based on embedding vectors derived from captions or metadata. The extracted data is finalized as curated data and passed to downstream processes.

[0079] The JSON keys shown in the figure are just examples, and can be added or deleted as desired depending on system requirements. Unknown values ​​are represented as NULL or "unknown," and a version identifier is added to the metadata to maintain schema compatibility. Examples of additional data include vehicle speed, self-position, imaging attitude, and point cloud density index.

[0080] Note that the items and keys in Figure 7 are just examples, and additions, deletions, and type changes (Boolean, categorical, continuous value, array, etc.) can be made according to system requirements. In addition, each record is associated with the preprocessing conditions (prompt, model version, sampling settings, etc.) at the time of generation and a data identifier, ensuring reproducibility and traceability in subsequent processing.

[0081] FIG. 8 is an example of a viewer screen provided by the first output unit 265 of the present invention, showing an operation system for performing curation, inputting conditions, and transferring results while viewing images in a dataset as thumbnails.

[0082] The top of the screen contains the dataset name, tabs (Summary / Viewer / Revision, etc.), a search box, and filter buttons for narrowing down the results. The center displays a grid of thumbnails, file names, and selection checkboxes. The left side displays a class summary (number badges for each class) and attribute dropdowns, allowing users to quickly filter by metadata conditions (e.g., presence or absence of a class and its attributes, image attributes).

[0083] The pull-down menu in the upper right offers commands such as "Output Data," "Curation," "Process Data," and "Copy." Selecting "Curation" activates the second execution unit 262, which performs filtering and semantic deduplication based on the specified conditions (search conditions and deduplication level). After processing, the curated data is confirmed as a new dataset, and can be output (exported) from the same screen for transfer to an external system. Export outputs a manifest (JSON / CSV / Parquet) containing metadata and correspondence logs along with the image / point cloud itself. The output destination is a file, object storage, or via API, and an ID mapping table (original ID → new ID) is also provided to maintain consistency with external annotation systems. The pager, display count switch, and zoom slider in the lower right of the screen ensure easy viewing and operability of large amounts of data. The number and content of selected results are immediately reflected along with updated thumbnails.

[0084] This allows users to visually check the metadata / captions generated in preprocessing, and consistently carry out the process from inputting conditions to checking the results to output, enabling curation operations to be performed with reproducibility and explainability.

[0085] To visualize the selection criteria, the user interface (UI) provides (i) highlighting of metadata items that match the search query, (ii) a list of nearby candidates and similarity scores that were excluded by de-duplication, (iii) a side-by-side display of data before and after curation, and (iv) a jump to the reason tag assigned to each selection / rejection.

[0086] Figure 9 shows a concrete example of captions and attribute metadata generated by the preprocessing according to the present invention. The left side shows an example of a driving scene image, the right side shows metadata (example JSON) automatically assigned to the image, and the bottom shows the caption output by the visual language model (VLM).

[0087] The metadata in this example includes traffic light detection results "traffic_lights" (e.g., color: "red", rotation: "horizontal"), road sign detection results "traffic_signs" (e.g., meaning: "no_entry", shape: "diamond", text_in_sign: "No entry"), as well as environmental attributes "is_urban", "is_mountain_road", "is_highway", "is_intersection", "is_tunnel", "is_curve", weather / time "weather: "sunny", time_of_day: "afternoon", and presence flags "has_pedestrian", "has_bicycle", "has_motorcycle", "has_car", "has_truck", etc. These are standardized as Boolean and categorical values ​​and can be directly used as search criteria during curation (e.g., "is_highway=1 AND weather="sunny" AND has_pedestrian=0").

[0088] A caption might describe the scenery and context in natural language, preserving the semantic content of the image at a high level of granularity. For example, "There is a white car parked ahead, with a yellow sign behind it. The sign reads 'Caution.' The road curves to the right, with mountains in the background. The weather is clear, with blue skies. Trees grow on both sides of the road, creating a natural landscape. The location appears to be a highway. Traffic is relatively calm, with no signs of congestion." However, captions do not necessarily need to be completely accurate. For example, in the example above, the white car ahead could be interpreted as moving rather than "parked," and the location could be an open road with traffic lights. The important thing is not the perfect caption, but rather the presence of a car ahead and the absence of pedestrians, with only other vehicles present. From these, it is important to accurately capture the key features of the image being captioned and generate accurate metadata.

[0089] Metadata and embedding vectors can be generated directly from images or based on captions. Simply extracting feature points within an image can result in distant oncoming vehicles or traffic lights occupying small areas within the image and therefore not being recognized as feature points. In contrast, generating captions that focus on other moving vehicles and the traffic environment allows these features to be extracted into the caption even if they appear small in the image. In this invention, embedding vectors are generated from these captions (and metadata strings, if necessary) and used for semantic deduplication and semantic search based on embedding similarity, effectively reducing approximate overlap and redundancy that are difficult to detect using pixel similarity. Furthermore, when using images extracted through semantic search using metadata and vectors for annotation or various classification and analysis tasks, even if the semantic assignment is incorrect, as in the example above, the presence of natural language captions and itemized metadata makes it easy for a human check to identify inaccuracies. If a human finds an inaccuracy, corrections to the captions or metadata may be accepted, or if the reproducibility of the extraction is more important than the accuracy of the captions, the system may choose not to accept corrections.

[0090] The generated metadata / captions are stored in association with the preprocessing conditions (prompt, model type / version, sampling settings, etc.) in the first recording unit 263, ensuring reproducibility and traceability in subsequent curation processes. This allows attribute filtering and semantic deduplication to be consistently applied from a single image example as shown in Figure 9 to entire large-scale datasets, enabling stable extraction of high-quality data for training and evaluation.

[0091] Embedding can be performed as a linear combination of the image embedding e_img and the caption embedding e_txt, e = αe_img + (1 - α)e_txt (0 ≤ α ≤ 1), with α set according to the target. If one is unavailable, the other is used instead. Captions support multiple languages, and language differences are absorbed by the embedding model. In this embodiment, transportation-related images are used as examples, but this is not limited to this. For example, if a search is intended for people's clothing, terms such as "male," "female," "wearing a coat," and "short sleeves" would be sufficient, while for nature images, terms such as "birds," "mammals," and "insects" would be sufficient. A prompt can be set to generate text that allows searches based on themes of interest in curation. In this embodiment, visible light images are used as examples for ease of understanding, but this is not limited to these. Infrared and ultraviolet images are also possible, and the same can be implemented with visualized (or virtually visualized) data such as point clouds using the detection results of various sensors, such as sound waves, lasers, and other microwaves. [Industrial Applicability]

[0092] The data selection support device, method, and program according to the present invention can efficiently and reproducibly extract high-quality data for learning and evaluation from visual data such as images and point clouds, and can be used in a wide range of industrial fields. For example, in the field of autonomous driving and ADAS, it can reliably extract long-tail cases, including rare events and small objects, from large-scale data obtained from in-vehicle cameras and LiDAR, while removing semantic duplication to prevent data leaks, thereby reducing annotation work and learning computational resources.

[0093] In service robots (cleaning, transportation, security) and smart buildings / factories / warehouses, data for re-learning can be continuously selected from constantly collected video and point cloud data, improving the maintainability of models against changes in the operating environment.Sensor data cleansing and purpose-specific extraction can also be applied as a common platform to smart city / traffic monitoring, infrastructure inspection / construction, manufacturing visual inspection, retail store analysis, etc.

[0094] Furthermore, the present invention systematically records preprocessing conditions, curation conditions, and contextual relationships to ensure explainability and auditability, making it suitable for incorporation into data governance and MLOps platforms, and can be flexibly deployed in any configuration: cloud, on-premise, or edge.

[0095] It can be deployed in any of the following ways: (a) as a multi-tenant cloud SaaS, (b) on-premise and linked to an existing data lake, or (c) integrated into edge devices for on-site pre-processing. Updates can be made to models, rules, and UIs individually. [Explanation of symbols]

[0096] 1: Terminal 1-1~1-N: Terminal 2: Computer Systems 21: Input interface 22: Communication module 23: Storage device 24: Memory 25: Output interface 26: Processor 261: First Executive Division 262: Second Executive Division 263: First Recording Section 264: Second recording section 265: First output section CN: communication line network

Claims

1. A data selection support device that receives an original data set as input and generates curated data from pre-curated data, comprising: a first execution unit that generates captions and attribute metadata for images or point clouds using a visual language model, and generates embedding vectors for the images and / or the captions using an embedding model; a first recording unit that records the caption, the attribute metadata, the embedding vector, and preprocessing conditions; a second execution unit that performs filtering and semantic deduplication based on curation conditions input by a user, the curation conditions including at least a metadata search condition and a deduplication degree, by referring to the captions, the attribute metadata, the embedding vectors, and preprocessing conditions recorded in the first recording unit, to identify curated data; a first output unit that sets the curation conditions and presents data before and after curation; a second recording unit that records the curation conditions, the results of the filtering and deduplication, and a correspondence relationship log; A data selection support device comprising:

2. The data selection support device according to claim 1 , wherein the first execution unit generates a caption by converting an image into text, and generates attribute metadata required for a curation condition.

3. 3. The data selection support device according to claim 1, wherein the second execution unit determines approximate duplication based on the similarity between the embedding vectors of the images and / or the embedding vectors of the captions, and performs semantic duplication elimination processing to adjust a threshold for the determination according to the degree of duplication elimination.

4. 3. The data selection support device according to claim 1, wherein the first output unit generates a user interface that displays both pre-curation data and curated data and visualizes selection grounds including at least suitable attribute metadata, neighbor candidates for deduplication, and similarity.

5. the original dataset includes image data and point cloud data; the first execution unit performs preprocessing on the image data and the point cloud data across modalities, including at least one of captioning, attribute metadata assignment, and embedding; The data selection support device according to claim 1 , wherein the second execution unit performs curation on either or both of the image data and the point cloud data.

6. the first recording unit and the second recording unit record preprocessing conditions and curation conditions, and correspondences between adjacent data, in a traceable manner; The data selection support device according to claim 1 , wherein the preprocessing conditions include at least a sampling setting and a prompt used for generating metadata.

7. The data selection support device according to claim 1 , wherein the first execution unit generates pre-curation data by pre-processing including downsampling from a video.

8. 3. The data selection support device according to claim 1, wherein the second execution unit narrows down the data by combining the presence or absence or number of class objects estimated by a discriminant model with the attribute metadata.

9. The data selection support device according to claim 1 or 2, wherein the first output unit provides a function of exporting the curated data and is configured to be able to link with an external system as a dataset for annotation or model development.

10. A computer-assisted data selection method, comprising: (i) taking an original dataset, generating captions and attribute metadata for images or point clouds using a visual language model, and generating embedding vectors for the images and / or the captions using an embedding model; (ii) recording the caption, the attribute metadata, and the embedding vector and preprocessing conditions; (iii) identifying curated data by filtering and semantic deduplication based on curation conditions input by a user, including at least a metadata search condition and a deduplication degree, with reference to the recorded captions, the attribute metadata, the embedding vectors, and preprocessing conditions; (iv) setting the curation conditions and presenting data before and after curation; (v) recording the curation conditions, the results of the filtering and deduplication, and a correspondence log; A data selection support method including:

11. The data selection support method according to claim 10 , wherein the step (i) includes generating captions by converting images into text and generating attribute metadata required by curation conditions.

12. The data selection support method according to claim 10 or 11, wherein the step (iii) includes performing a semantic deduplication process in which approximate duplication is determined based on the similarity between the embedding vectors of the image and / or the embedding vectors of the caption, and a threshold for the determination is adjusted according to the degree of deduplication.

13. 12. The data selection support method according to claim 10 or 11, wherein step (iv) includes generating a user interface that displays both the pre-curation data and the curated data and visualizes the selection basis, including at least the matched attribute metadata, the neighbor candidates for deduplication, and the similarity.

14. the original dataset includes image data and point cloud data; The data selection support method according to claim 10 or 11, wherein step (i) includes preprocessing the image data and the point cloud data across modalities, the preprocessing including at least one of captioning, attribute metadata assignment, and embedding.

15. In the steps (ii) and (v), the preprocessing conditions and curation conditions, and the correspondence between the preceding and following data are recorded in a traceable manner; The data selection support method according to claim 10 or 11, wherein the preprocessing conditions include at least a sampling setting and a prompt used for generating metadata.

16. The data selection support method according to claim 10 or 11, wherein the step (i) includes generating pre-curated data by preprocessing including downsampling from a video.

17. 12. The data selection support method according to claim 10, wherein the step (iii) includes narrowing down the data by combining the presence or absence or number of class objects estimated by the discriminant model with the attribute metadata.

18. The data selection support method according to claim 10 or 11, wherein step (iv) includes exporting the curated data to an external system as a dataset for annotation or model development.

19. A data selection support program for causing a computer to function as the first execution unit, the first recording unit, the second execution unit, the first output unit, and the second recording unit of the data selection support device according to claim 1 or 2.

20. A data selection support program for causing a computer to execute the steps (i) to (v) of the data selection support method according to claim 10 or 11.

Images