Data merging assistance method, data merging assistance device, and data merging assistance program

Abstract

A data merging assistance device acquires destination information indicating a user's destination relating to data merging in which an additional data set is added to an existing data set owned by a user; retrieving, from the candidate data sets, an additional candidate data set conforming to the destination based on the destination information; a presentation screen generation unit that generates a presentation screen that visualizes a feature in a case where the additional candidate data set is added to the existing data set; and outputting the prompt picture to a display.

Description

Technical Field

[0001] This disclosure relates to techniques for assisting in data merging. Background Technology

[0002] Patent Document 1 discloses a method for identifying reusable project components used to construct new projects. Specifically, Patent Document 1 discloses the following: using Natural Language Processing (NLP), extracting current elements of the new project from a data source; performing meaning parsing of the current elements and existing elements to detect existing elements similar to the current elements; determining a similarity score for the existing elements based on the similarity between the current elements and existing elements; and extracting project components associated with the existing elements based on the similarity score. The extracted project components are reusable project components used to construct new projects.

[0003] However, in the technology of Patent Document 1, the machine unilaterally extracts reusable project components without considering the user's purpose for the project. Therefore, the technology of Patent Document 1 has the problem of failing to achieve data merging that meets the user's purpose.

[0004] Prior art literature

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent No. 7007486 Summary of the Invention

[0007] This disclosure was made in view of such problems and provides a technique for data merging that can achieve the purpose of the user.

[0008] In one aspect of the data merging assistance method disclosed herein, the computer performs the following processing: obtaining purpose information, the purpose information representing the user's purpose related to data merging by adding an additional dataset to an existing dataset owned by the user; based on the purpose information, retrieving additional candidate datasets from candidate datasets that match the purpose; generating a prompt screen that visualizes the features of the merged dataset after adding the additional candidate datasets to the existing dataset; and outputting the prompt screen to a display.

[0009] According to this disclosure, data merging can be achieved in a manner that meets the user's objectives. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example of the structure of a data merging assistance system according to an embodiment of the present disclosure.

[0011] Figure 2This is a flowchart illustrating an example of server processing in this embodiment.

[0012] Figure 3 This is an example of a prompt screen.

[0013] Figure 4 This is a diagram showing details of the sample column and the mapping column. Detailed Implementation

[0014] (The knowledge that forms the basis of this disclosure)

[0015] In recent years, the following operations have been carried out: Transfer learning has been performed on a general base model obtained through self-supervised learning (SSL) on shared foundational data that can be used in a wide variety of businesses, thereby generating a fully learned model suitable for the task the user wants to solve. To obtain the learning data for transfer learning, by extracting additional datasets suitable for the user's desired task from the vast amounts of data accumulated in the enterprise's large-scale database, the development period of the fully learned model can be shortened, and development costs reduced.

[0016] However, it is not easy for machines to extract additional datasets from large-scale databases that meet the user's data merging objectives.

[0017] For example, if an existing dataset contains a large number of "dog" and "cat" data points, and the user wants to "not only identify dogs and cats but also birds," the user might want to extract a bird dataset from a large-scale database as an additional dataset. Conversely, if an existing dataset contains a large number of "dog" and "cat" data points, and the user wants to improve the accuracy of "dog" and "cat" recognition, the user might want to further add "dog" and "cat" datasets from a large-scale database. Thus, the purpose of data merging varies depending on the task the user intends to solve. Therefore, it is not easy for a machine to unilaterally extract additional datasets from a large-scale database that meet the user's data merging objectives.

[0018] When the technology shown in Patent Document 1 is applied to such data merging, the machine extracts additional datasets based on a one-sided benchmark, thus failing to achieve data merging that meets the user's purpose.

[0019] This disclosure was made to solve such a problem.

[0020] (1) In a data merging auxiliary method of one aspect of the present disclosure, the computer performs the following processing: obtaining purpose information, the purpose information representing the user's purpose related to data merging by adding an additional dataset to an existing dataset owned by the user; based on the purpose information, retrieving additional candidate datasets that meet the purpose from candidate datasets; generating a prompt screen that visualizes the features of the merged dataset after adding the additional candidate datasets to the existing dataset; and outputting the prompt screen to a display.

[0021] Based on this structure, additional candidate datasets that match the stated purpose are retrieved from the candidate datasets according to the purpose information. A visual display showing the features of the merged dataset after adding the candidate datasets is output to the screen. Therefore, users can understand the impact of the added candidate datasets on the existing dataset through the visual display and confirm whether the added candidate datasets meet the purpose. Thus, data merging that meets the user's objectives can be achieved.

[0022] (2) In the data merging auxiliary method described in (1) above, the prompt screen may also include: a first distribution representing the distribution of the feature quantities of the existing data constituting the existing dataset in the feature space, and a second distribution representing the distribution of the feature quantities of the merged data constituting the merged dataset in the feature space.

[0023] Based on this structure, the first distribution representing the distribution of feature quantities of the existing dataset and the second distribution representing the distribution of feature quantities of the merged dataset are displayed on the prompt screen, thus enabling users to quantitatively confirm the impact of adding candidate datasets on the existing dataset.

[0024] (3) In the data merging auxiliary method described in (1) or (2) above, the prompt screen may also include: multiple samples of each cluster of the merged dataset obtained by clustering the merged dataset into multiple clusters.

[0025] Based on this structure, the system provides samples from each cluster of the merged dataset that has been clustered, thus enabling users to more specifically confirm the impact of adding additional candidate datasets.

[0026] (4) In the data merging assistance method described in (3) above, it is also possible to further delete the additional candidate data contained in the cluster to which the useless sample belongs from the additional candidate dataset when the instruction of the user to specify a useless sample from the plurality of samples is received. Further, the merged dataset is updated using the deleted additional candidate dataset, and the prompt screen is updated using the updated merged dataset.

[0027] Based on this structure, users can easily remove useless additional candidate data from the additional candidate dataset, and users can identify the changes in the effects caused by the removal.

[0028] (5) In any of the data merging auxiliary methods described in (1) to (4) above, it is also possible to further retrieve the additional candidate dataset from the candidate dataset based on the additional target information obtained, update the merged dataset using the newly retrieved additional candidate dataset, and update the prompt screen using the updated merged dataset.

[0029] According to this configuration, it is possible to interactively re-retrieve additional candidate datasets while considering the additional purposes of users viewing the prompt screen, and to enable users to identify the changes in features after the re-retrieval.

[0030] (6) In any of the data merging auxiliary methods described in (1) to (5) above, the retrieval may also include: retrieving additional candidate datasets that meet the purpose by comparing the feature quantity of the target information with the feature quantity of the candidate data constituting the candidate dataset.

[0031] Based on this structure, additional candidate datasets that meet the objectives are retrieved by comparing the feature values ​​of existing data and the feature values ​​of candidate data, thus enabling accurate retrieval of additional candidate data that meet the objectives.

[0032] (7) The data merging auxiliary method described in any of (1) to (6) above may also include: calculating the consistency of each of the multiple search conditions for each of the multiple candidate data constituting the candidate dataset, and calculating a consistency score for the purpose for each of the multiple candidate data by synthesizing the consistency scores; and retrieving a dataset including candidate data whose consistency scores exceed a threshold as the additional candidate dataset.

[0033] Based on this structure, it is possible to accurately retrieve additional candidate datasets that meet the objectives.

[0034] (8) In any of the data merging auxiliary methods described in (1) to (7) above, the feature quantity may include multiple categories of feature quantities of different types. Further, when the prompt screen receives an instruction to select one category of feature quantity from the multiple categories of feature quantities, it displays the feature of the merged dataset for the one category of feature quantity.

[0035] Based on this structure, users can confirm the impact of adding candidate data by providing prompts for each category feature quantity.

[0036] (9) In the data merging auxiliary method described in (7) above, the existing dataset and the candidate dataset may each include image data, and the multiple category features include at least two of the following: vector features obtained by inputting the image data into the learned model, the metadata of the image data, the brightness distribution of the image data, and annotation information.

[0037] Based on this structure, users can identify additional datasets that meet their purpose by targeting any one of the vector features, metadata, brightness distribution, and annotation information.

[0038] (10) In the data merging auxiliary method described in (9) above, the metadata may also include at least one of the following: the size of the image data, the aspect ratio of the image data, and the date and time of acquisition of the image data.

[0039] Based on this configuration, users can confirm additional datasets that meet their purpose, based on any of the following: size, aspect ratio, or date and time of acquisition.

[0040] (11) In the data merging auxiliary method described in (9) or (10) above, the annotation information may be for each dataset of the existing dataset and the additional candidate dataset, including at least one of the following: the proportion of the category to which the object represented by the annotation assigned to the image data belongs in the dataset, the proportion of the annotated image data in the dataset, and the error rate of the annotation in the dataset.

[0041] Based on this structure, users can confirm the suitability of additional datasets for any of the following purposes: the proportion of classifications, the proportion of annotated image data, and the error rate for annotations.

[0042] (12) In any of the data merging auxiliary methods described in (1) to (11) above, the existing dataset and the additional dataset may also be datasets used for machine learning.

[0043] Based on this structure, users can identify additional candidate datasets in machine learning that align with their objectives.

[0044] (13) In the data merging auxiliary method described in (2) above, the second distribution may also include: multiple distributions corresponding to multiple clusters obtained by clustering the additional candidate data constituting the additional candidate dataset.

[0045] Based on this structure, an indicator can be provided to the user indicating which additional candidate data in the additional candidate dataset is useless.

[0046] (14) In another embodiment of the present disclosure, the data merging auxiliary device is a data merging auxiliary device including a processor, the processor performing the following processing: acquiring purpose information, the purpose information representing the user's purpose related to data merging by adding an additional dataset to an existing dataset owned by the user; based on the purpose information, retrieving additional candidate datasets that meet the purpose from candidate datasets; generating a prompt screen that visualizes the features of the merged dataset after adding the additional candidate dataset to the existing dataset; and outputting the prompt screen to a display.

[0047] Based on this configuration, a data merging auxiliary device can be provided to achieve data merging in a manner that meets the user's objectives.

[0048] (15) In another embodiment of the present disclosure, the data merging aid causes a computer to perform the following processes: obtain purpose information, the purpose information representing the user’s purpose related to data merging by adding an additional dataset to an existing dataset owned by the user; based on the purpose information, retrieve additional candidate datasets from candidate datasets that meet the purpose; generate a prompt screen that visualizes the features of the merged dataset after adding the additional candidate datasets to the existing dataset; and output the prompt screen to a display.

[0049] Based on this structure, a data merging assistant program can be provided to achieve data merging in a manner that meets the user's objectives.

[0050] This disclosure can also be implemented as a data merging auxiliary system that operates through such a data merging auxiliary program. Furthermore, it is of course possible to distribute such a computer program via computer-readable non-transitory recording media such as CD-ROMs or communication networks such as the Internet.

[0051] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, constituent elements in the following embodiments that are not described in the independent claims representing the highest-level concept are described as arbitrary constituent elements. Furthermore, the contents of all embodiments can be combined.

[0052] (Implementation Method)

[0053] Figure 1This is a block diagram illustrating an example of the structure of a data merging assistance system according to an embodiment of the present disclosure. The data merging assistance system includes a server 1 (an example of a data merging assistance device) and a terminal 40. The server 1 and the terminal 40 are communicatively connected to each other via a network NT. An example of a network NT is the Internet. The server 1 is, for example, a cloud server consisting of one or more computers. The terminal 40 is a computer used by a user. An example of a terminal 40 is a desktop computer, a tablet computer, or a portable computer such as a smartphone.

[0054] Server 1 includes a processor 10, a database 20, and a communication unit 30. One example of the processor 10 is a central processing unit (CPU). The processor 10 includes an acquisition unit 11, a feature extraction unit 12, a retrieval unit 13, a screen generation unit 14, and a merging unit 15. The acquisition unit 11 to the merging unit 15 are implemented by the processor executing a data merging auxiliary program. However, this is just one example; the acquisition unit 11 to the merging unit 15 can also be constructed using dedicated hardware circuitry.

[0055] The acquisition unit 11 acquires purpose information, which indicates the user's purpose related to merging data by adding the additional dataset to the user's existing dataset. An example of purpose information is keywords representing the characteristics of the dataset to be added. For example, keywords representing a scene such as "construction site," "highway," or "general road," or keywords representing the category to be added (e.g., dog, cat, bird, etc.) can be used.

[0056] Furthermore, one example of the purpose information is the metadata such as image size that needs to be added to the dataset. Details of the metadata will be described later. Another example of the purpose information is that, starting from the features of the existing dataset, a request message indicating what features are desired can be used as the request message for the additional dataset. Furthermore, if the user wants to include datasets of categories not included (or a few categories) in the existing dataset as additional datasets, a message such as "I want data for general recognition" can be used as the request message. Furthermore, if the user wants to enrich the categories included in the existing dataset as additional datasets, a message such as "I want data specifically for the current categories" can be used as the request message.

[0057] Here, the acquisition unit 11 can also acquire target information through dialogue until it obtains the information required by the retrieval unit 13 to perform the retrieval process. In this case, such as Figure 3As shown, the acquisition unit 11 interacts with the user through the chat bar 200 included in the prompt screen G1 displayed on the display 41 of the terminal 40, and obtains the target information from the messages entered by the user. Alternatively, the acquisition unit 11 may display selection options related to the purpose of data merging on the display 41, and obtain the target information from the items selected by the user. In addition, "until the information required by the retrieval unit 13 to perform the retrieval process" means until the information required to determine one or more retrieval conditions constituting the retrieval formula described later is obtained.

[0058] The existing dataset includes multiple existing data owned by the user. Existing data may be data collected by the user for use in machine learning. In the following description, existing data is described as image data. However, this is just one example; existing data can also be text data or audio data. Existing data may include data related to specific locations such as homes, vehicles, and factories. An example of home data is data representing the actions of people inside a residence, their speech, the operation of electrical equipment, power consumption, and temperature. An example of vehicle data is vehicle operation data. Examples of vehicle operation data include vehicle speed, acceleration, battery capacity, current, voltage, fuel consumption, power consumption, state of charge (SOC), and temperature. An example of factory data includes power consumption, temperature, operation data of various equipment, and data representing the assembly status of components. The existing dataset is associated with the user ID of the user who owns it and is pre-stored in database 20.

[0059] The supplementary dataset includes multiple supplementary data sets. Supplementary data are candidate data ultimately selected from the candidate data stored in database 20 to be added to the existing dataset. In the following description, supplementary data is described as image data. However, this is just an example; supplementary data can also be text data or audio data.

[0060] The feature extraction unit 12 extracts features from the existing dataset and candidate datasets. The features include multiple categories of features of different types.

[0061] Category features include vector features, metadata of image data, brightness distribution, and annotation information. Vector features, for example, consist of multi-dimensional vectors obtained by inputting existing data and candidate data into a pre-learned machine learning model for calculating the vector features. Furthermore, if the existing and candidate data consist of text data, the vector features consist of word vectors. In this case, the vector features can be calculated using word vectors calculated using a pre-learned natural language processing model. If the existing and candidate data consist of audio data, the vector features are calculated using a pre-learned model that transforms the audio data into vectors.

[0062] Meta-information includes the image data's dimensions (image size), aspect ratio, and the date and time the image data was acquired. Image size is represented, for example, by the number of pixels in the width and height of the image data. Aspect ratio is the ratio of the image data's width to its height. The acquisition date and time is the date and time the image data was taken.

[0063] The brightness distribution of image data is data representing the frequency of each brightness level among the multiple brightness levels of the pixels that make up the image data.

[0064] Annotation information includes classification ratio, annotation ratio, and error rate. Classification ratio is the proportion of the classes to which the objects represented by the annotations belong in the existing dataset and each of the additional candidate datasets. Annotation ratio is the proportion of annotated image data in the dataset. Error rate is the error rate for the annotations. Annotations are the notes or comments assigned to image data as training data. Annotations may include, for example, bounding boxes surrounding objects in the image data and labels indicating the class to which the objects enclosed by those bounding boxes belong. For example, in the case of annotated image data of dogs, the annotation consists of a bounding box surrounding the dog and a label indicating that the object enclosed by the bounding box belongs to the class "dog".

[0065] For example, if annotations are assigned to dogs, cats, and birds in the dataset, and their respective proportions are set to 10%, 10%, and 80%, then the classification ratio becomes 1:1:8. For example, if the proportion of annotated image data in the dataset is 30%, then the annotation proportion becomes 30%. For example, if the error rate is 50% when annotated image data is input into the recognizer, then the error rate becomes 50%. Regarding the error rate, a pre-determined value is mapped to the image data.

[0066] Based on the purpose information obtained by the acquisition unit 11, the retrieval unit 13 retrieves additional candidate datasets from the candidate datasets that meet the purpose of data merging. The candidate datasets include multiple candidate data sets. Candidate data are pre-stored data in the database 20 that are intended to become additional data. The candidate data consists of large amounts of data accumulated by the enterprise. For example, candidate data may be data obtained by monitoring residences, cars, and factories using various sensors. Therefore, a large amount of candidate data is stored in the database 20. In the following description, candidate data is described as image data. However, this is just one example; candidate data can also be text data or audio data. Candidate data can also be similar to existing data, such as in-home data, in-vehicle data, and factory data—data related to the site.

[0067] The retrieval unit 13 retrieves additional candidate datasets that meet the purpose by comparing the feature values ​​of the target information with the feature values ​​of the candidate data constituting the candidate dataset.

[0068] In detail, the retrieval unit 13 calculates the consistency degree of each of the multiple conditions for each of the multiple candidate data constituting the candidate dataset, and calculates a consistency score for the purpose of data merging for each of the multiple candidate data based on the calculated consistency degree. In addition, the retrieval unit 13 retrieves the dataset consisting of candidate data whose consistency scores exceed a threshold as an additional candidate dataset.

[0069] Consistency scores can be represented, for example, by the following search query.

[0070] Consistent score = f(s1, s2, ..., si, ..., sn) (1)

[0071] Here, f represents a function that combines si. An example of function f is a function that multiplies si. si represents the consistency of each of the multiple search criteria. i is the index that specifies the search criteria.

[0072] As an example of search criteria, a similarity criterion can be used relative to the keywords contained in the target information. In this case, the higher the similarity between the vector features of the keywords and the vector features of the candidate data, the greater the consistency. Similarity can be, for example, vector distance or cosine similarity.

[0073] Furthermore, as an example of search criteria, similarity or dissimilarity relative to existing datasets can be used. For example, when the target information contains a request message meaning "I want data for general recognition," the greater the dissimilarity of candidate data relative to the vector feature quantities of existing datasets, the greater the consistency. Thus, candidate data including objects belonging to categories other than those included in the existing dataset are retrieved as additional candidate data. Dissimilarity can be, for example, the reciprocal of similarity. For example, when the target information contains a message meaning "I want data specifically for current classifications," the greater the similarity of candidate data relative to the vector feature quantities of existing datasets, the greater the consistency value. In this case, candidate data including objects belonging to the same categories as those included in the existing dataset are retrieved as additional candidate data. When calculating the similarity or dissimilarity between the vector feature quantities of the existing dataset and the vector feature quantities of the candidate data, the search unit 13 can use a representative value of the vector feature quantities of the existing dataset. As a representative value, the average value of the vector feature quantities of the existing data constituting the existing dataset can be used. In addition, when candidate data are assigned labels that indicate classification, the retrieval unit 13 can also use the vector features of the text data contained in the labels to calculate similarity or dissimilarity.

[0074] Furthermore, as an example of a search condition, a filtering condition can be used. For example, if the image size is specified as a filtering condition in the target information, the search unit 13 can calculate the consistency of candidate data that meets the specified filtering condition as "1" and calculate the consistency of candidate data that does not meet the filtering condition as "0".

[0075] When using these three search criteria, the consistency score is represented by the following search expression.

[0076] Consistent score = f(s1×s2×s3) (2)

[0077] Consistency score s1 represents the similarity relative to keywords, consistency score s2 represents the similarity or dissimilarity relative to the existing dataset, and consistency score s3 represents the value of whether the filtering condition is met. For example, when the filtering condition is related to image size (e.g., 640×480 pixels or larger), the consistency score s3 of candidate data that meets the filtering condition is "1", and the consistency score s3 of candidate data that does not meet the filtering condition is "0".

[0078] Here, image size is shown as a filtering condition, but this is just one example; other metadata besides image size can also be used. For example, "aspect ratio greater than a given value" or "the period of the date and time the image data was acquired" could also be used as filtering conditions.

[0079] The screen generation unit 14 generates a prompt screen that visualizes the features of the merged dataset after adding candidate datasets to the existing dataset, and outputs the generated prompt screen to the display 41 of the terminal 40.

[0080] The screen generation unit 14 calculates a first distribution representing the distribution of the feature quantities of the existing data constituting the existing dataset in the feature space, and a second distribution representing the distribution of the feature quantities of the merged data constituting the merged dataset in the feature space, and displays a prompt screen including the first distribution and the second distribution on the display 41.

[0081] The screen generation unit 14 may also, upon receiving an instruction to select one category feature from multiple category feature quantities, calculate a first distribution and a second distribution for one category feature quantity, and display a prompt screen including the calculated first distribution and second distribution on the display 41.

[0082] The screen generation unit 14 can also cluster the merged dataset into multiple clusters, determine samples for each cluster based on the clustering results, and include the determined samples in the prompt screen. As a clustering method, a clustering method using a Gaussian mixture distribution can be used. For each of the multiple clusters, the screen generation unit 14 calculates the centroid of the distribution of feature values ​​in the additional candidate dataset, and determines the additional candidate data closest to the calculated centroid as samples.

[0083] The screen generation unit 14 may also delete the additional candidate data contained in the cluster to which the useless sample belongs from the additional candidate dataset when the acquisition unit 11 receives an instruction from the user to specify a useless sample from multiple samples, update the merged dataset using the additional candidate dataset after the additional candidate data has been deleted, and update the prompt screen displayed on the display 41 using the updated merged dataset.

[0084] The retrieval unit 13 can also, upon receiving additional target information related to data merging from the acquisition unit 11, retrieve additional candidate datasets from the candidate datasets based on the additional target information, update the merged dataset using the re-retrieved additional candidate datasets, and update the prompt screen displayed on the display 41 using the updated merged dataset. The additional target information refers to target information added by the user viewing the prompt screen. For example, a user who confirms from the prompt screen that the proportion of annotated image data in the candidate data included in the additional candidate dataset is low may input a message indicating the need to add annotated images as additional target information. Similarly, a user who confirms from the prompt screen that the proportion of a certain category in the additional candidate dataset is low may input a message indicating the need to add the category with the low proportion as additional target information. Likewise, a user who confirms from the prompt screen that there is insufficient image data of a certain image size or larger in the additional candidate data included in the additional candidate dataset may input a message indicating the need for image data of that image size or larger as additional target information.

[0085] When the user inputs a confirmation instruction via the operation unit 43 indicating that additional candidate data should be added, and the acquisition unit 11 obtains this confirmation instruction via the communication unit 30, the merging unit 15 determines the additional candidate dataset retrieved at a given time point as the additional dataset. Then, the merging unit 15 performs data merging by adding the determined additional dataset to the existing dataset. In this case, the merging unit 15 determines the merged dataset after adding the additional dataset to the existing dataset as the final merged dataset, assigns a user ID to the final merged dataset, and saves it in the database 20.

[0086] Database 20, for example, consists of non-volatile storage devices that store candidate datasets, existing datasets, and the final merged dataset.

[0087] The communication unit 30 consists of a communication circuit that connects the server 1 to the network NT. The communication unit 30 receives destination information and confirmation instructions sent from the terminal 40. Furthermore, the communication unit 30 sends display data for the prompt screen to the terminal 40.

[0088] Terminal 40 includes a display 41, a processor 42, an operation unit 43, and a communication unit 44. The display 41, for example, is a liquid crystal display panel that displays prompts. The processor 42 controls terminal 40. The operation unit 43 consists of input devices such as a keyboard and a touch panel. The operation unit 43 accepts input of target information. The communication unit 44 is a communication circuit that connects terminal 40 to network NT. The communication unit 44 receives display data of prompts sent from server 1, or sends target information received by the operation unit 43 to server 1.

[0089] Figure 2 This is a flowchart illustrating an example of the processing of server 1 in this embodiment. Furthermore, it is assumed that prior to the implementation of this flowchart, the acquisition unit 11 acquired the user ID of the user using terminal 40.

[0090] In step S1, the acquisition unit 11 acquires the target information input by the user in the terminal 40.

[0091] Next, in step S2, the feature extraction unit 12 extracts feature values ​​for all existing data constituting the existing dataset, and extracts feature values ​​for all candidate data constituting the candidate dataset. In this case, the feature extraction unit 12 extracts all category feature values ​​for all existing data, and extracts all category feature values ​​for all candidate data. Here, the existing dataset used is the existing dataset of the user using terminal 40, determined based on the user ID.

[0092] Next, in step S3, the retrieval unit 13 retrieves candidate data from the candidate dataset that matches the purpose indicated by the purpose information based on the purpose information obtained in step S1, and uses this as additional candidate data. In this case, the retrieval unit 13 uses a retrieval formula as shown in equation (1) to calculate a consistency score for each candidate data, and retrieves candidate data whose consistency score exceeds a threshold as additional candidate data. For example, if the purpose information includes a message meaning "wanting an image of a bird" and a message meaning "wanting data for general recognition", the retrieval formula becomes consistency score = f(s1×s2). However, consistency s1 represents the similarity to the keyword "bird", and consistency s2 represents the dissimilarity to the existing dataset.

[0093] Next, in step S4, the screen generation unit 14 clusters the additional candidate data retrieved in step S3 into multiple clusters, and determines the samples of each cluster based on the clustering results.

[0094] Next, in step S5, the screen generation unit 14 uses the communication unit 30 to send display data that causes the multiple samples determined in step S4 to be displayed on the prompt screen to the terminal 40, thereby displaying the multiple samples on the display 41.

[0095] Next, in step S6, the screen generation unit 14 calculates a first distribution representing the distribution of feature quantities of existing data constituting the existing dataset in the feature space, and a second distribution representing the distribution of feature quantities of merged data constituting the merged dataset in the feature space.

[0096] Next, in step S7, the screen generation unit 14 uses the communication unit 30 to send display data that makes the first distribution and the second distribution calculated in step S6 display on the prompt image to the terminal 40, thereby displaying the first distribution and the second distribution on the display 41.

[0097] Next, in step S8, the communication unit 30 sends a message to the terminal 40 to request additional target information or useless samples. As a result, messages such as "Are there any useless samples?" or "Is there a need for additional samples?" are displayed on the display 41.

[0098] Next, in step S9, the acquisition unit 11 determines whether there is input of additional target information. In this case, if additional target information has been input ("Yes" in step S9), the process proceeds to step S10; if no additional target information has been input ("No" in step S9), the process proceeds to step S11.

[0099] Next, in step S10, the retrieval unit 13 uses the additional target information to re-retrieve the additional candidate dataset. For example, if the additional target information includes a message meaning "I want an image of a given size or larger", the retrieval unit 13 can re-retrieve the additional candidate dataset using a search expression that adds the search condition "image size is a given size or larger" to the search expression used in the previous retrieval. In the example above, the search expression is, for example, consistency score = f(s1×s2×s3). In this case, consistency s1 represents the similarity to the keyword "bird", consistency s2 represents the dissimilarity of the candidate data relative to the existing dataset, and consistency s3 represents the value ("1" or "0") of whether the filtering condition "image size is a given size or larger" is met.

[0100] Next, in step S11, the retrieval unit 13 updates the merged dataset with the additional candidate datasets found through the re-search.

[0101] Next, in step S12, the screen generation unit 14 updates the prompt screen using the updated merged dataset. In this case, the screen generation unit 14 simply updates the second distribution displayed on the display 41 using the updated merged dataset. Furthermore, the screen generation unit 14 clusters the updated merged dataset into multiple clusters, determines the samples for each cluster based on the clustering results, and updates the samples of the prompt screen with the determined samples.

[0102] Next, in step S13, the acquisition unit 11 determines whether there is an input of a useless sample. If a useless sample is input ("Yes" in step S13), the process proceeds to step S14; if no useless sample is input ("No" in step S13), the process proceeds to step S15.

[0103] Next, in step S14, the screen generation unit 14 updates the merged dataset by deleting additional candidate datasets belonging to the same cluster as the specified useless samples from the merged dataset.

[0104] Next, in step S15, the screen generation unit 14 updates the prompt screen using the merged dataset updated in step S14. In this case, similar to step S12, the screen generation unit 14 updates the second distribution and sample using the updated merged dataset.

[0105] Next, in step S16, when the merging unit 15 receives a determination instruction from the acquisition unit 11 ("Yes" in step S16), it determines the currently retrieved supplementary candidate dataset as the supplementary dataset, and determines the merged data after adding the determined supplementary dataset to the existing dataset as the final merged dataset (step S17).

[0106] On the other hand, if no confirmation instruction is received by the acquisition unit 11 ("No" in step S16), the process returns to step S8, and the processing after step S8 is executed. By repeating the processing of steps S8 to S16, the user's requirements for data merging are interactively acquired, and a merged dataset suitable for the task that the user wants to solve is generated.

[0107] Figure 3 This diagram illustrates an example of a prompt screen G1. The prompt screen G1 includes a chat bar 200, a sample bar 300, and a mapping bar 400. The chat bar 200 is where the user inputs desired information. The chat bar 200 allows the user to input desired information in a conversational format. In the chat bar 200, the left column represents server-side messages, and the right column represents user-side messages. Server-side messages include messages generated by the acquisition unit 11 to facilitate the input of desired information. User-side messages include user responses to server-side messages.

[0108] In this example, firstly, the acquisition unit 11 displays a message 201 asking the user the purpose of data merging, such as "What kind of data do you need?" In response to this inquiry, the user uses the operation unit 43 to input a message 211 indicating "I want images of the construction site".

[0109] Next, the acquisition unit 11 displays a message 202 such as "Do you need more comprehensive data than the current dataset?" The user responds by inputting "Yes" as a confirmation message 212 using the operation unit 43. Through these dialogues, the acquisition unit 11 acquires purpose information including the keyword "construction site" and a message indicating a desire for general-purpose data. Based on this purpose information, the retrieval unit 13 sets the search query as described below.

[0110] Consistent score = f(s1×s2) (3)

[0111] However, consistency score s1 represents the similarity of candidate data with respect to the keyword "construction site", and consistency score s2 represents the dissimilarity of candidate data with respect to the existing dataset. Next, the retrieval unit 13 uses the retrieval formula to calculate the consistency score of each of the multiple candidate data stored in the database 20, and retrieves a dataset consisting of candidate data whose consistency scores exceed a threshold as an additional candidate dataset.

[0112] Mapping bar 400 includes selection bar 401 and mapping display bar 402. Selection bar 401 allows the user to select one categorical feature from multiple categorical feature quantities. Mapping display bar 402 displays a first distribution and a second distribution related to the categorical feature quantity selected from selection bar 401.

[0113] If a user inputs a selection operation on the selection bar 401 using the operation unit 43, the selection bar 401 displays a drop-down menu that shows a list of multiple category features. The user inputs an operation to select one category feature from the multiple category features that are shown in the list using the operation unit 43. Then, a selection instruction indicating the selected category feature is sent from the terminal 40 to the server 1, and the acquisition unit 11 acquires the selection instruction. The screen generation unit 14 calculates a first distribution and a second distribution for the selected category feature, and uses the communication unit 30 to send display data that displays the first distribution and the second distribution on the prompt screen G1 to the terminal 40. As a result, the mapping display bar 402 displays the first distribution and the second distribution related to the one category feature. In addition, in the initial state, the mapping display bar 402 displays the first distribution and the second distribution related to the default one category feature (e.g., a vector feature).

[0114] Figure 4This is a diagram showing details of sample column 300 and mapping column 400. Mapping display column 402 displays the first and second distributions using a 2D coordinate axis. Here, vector features are selected as a single category feature. Distribution 601 represents the first distribution, and distributions 501, 502, and 503 represent the second distribution. Distribution 601 represents the distribution of vector features related to the existing dataset, and therefore displays a "before" message nearby. Distributions 501–503 represent the distribution of vector features in each cluster of the merged dataset, which is clustered into three clusters. Specifically, distribution 501 represents the distribution of vector features in the cluster to which sample 1 belongs, distribution 502 represents the distribution of vector features in the cluster to which sample 2 belongs, and distribution 503 represents the distribution of vector features in the cluster to which sample 3 belongs. Therefore, the area near distribution 501 is recorded as "sample group 1", the area near distribution 502 is recorded as "sample group 2", and the area near distribution 503 is recorded as "sample group 3", which clearly shows the correspondence between distributions 501 to 503 and the three samples.

[0115] The image generation unit 14 compresses the dimension of the vector feature quantities of the existing data constituting the existing dataset and the merged data constituting the merged dataset into 2 dimensions, and calculates the first distribution and the second distribution. Here, although the vector feature quantities of the existing data and the merged data are compressed into 2 dimensions respectively, they can also be compressed into 3 dimensions. In this case, the mapping display bar 402 can display the first distribution and the second distribution in a display format using 3-dimensional coordinate axes.

[0116] The display format of the first and second distributions shown in the mapping display bar 402 varies depending on the selected categorical feature. For example, if image size is selected as the categorical feature, the image generation unit 14 displays a histogram representing the frequency corresponding to the image size in the existing dataset as the first distribution in the mapping display bar 402, and displays a histogram representing the frequency corresponding to the image size in the merged dataset as the second distribution in the mapping display bar 402. The same applies to aspect ratio and date / time acquisition.

[0117] For example, if the brightness distribution is selected as a single category feature, the image generation unit 14 can display a histogram representing the frequency corresponding to the brightness in the existing dataset as the first distribution, and display a histogram corresponding to the brightness in the merged dataset as the second distribution in the mapping display bar 402.

[0118] For example, if a classification ratio is selected as a single category feature, the screen generation unit 14 can display the value representing the classification ratio in the existing dataset as a first distribution in the mapping display bar 402, and display the value representing the classification ratio in the merged dataset as a second distribution in the mapping display bar 402. An example of displaying the classification ratio is "The classification ratio in the existing dataset is dog: 0.4, cat: 0.1, and the classification ratio in the merged dataset is dog: 0.4, cat: 0.1, crow: 0.2," etc.

[0119] For example, if the annotation ratio is selected as a single category feature, the screen generation unit 14 can display the value representing the annotation ratio in the existing dataset as the first distribution in the mapping display bar 402, and display the annotation ratio in the merged dataset as the second distribution. An example of displaying the annotation ratio is "The proportion of annotated data in the existing dataset is 0.5, and the proportion of annotated data in the merged dataset is 0.4," etc.

[0120] For example, if the error rate is selected as a single category feature, the screen generation unit 14 can display a histogram representing the frequency corresponding to the error rate in the existing dataset as a first distribution in the mapping display bar 402, and display a histogram representing the frequency corresponding to the error rate in the merged dataset as a second distribution in the mapping display bar 402.

[0121] exist Figure 4 In the example, the merged dataset is clustered into 3 clusters, so 3 distributions 501 to 503 are displayed. However, when the merged dataset is clustered into 2 clusters, the 2 distributions are displayed as the second distribution. When the merged dataset is clustered into more than 4 clusters, the more than 4 distributions are displayed as the second distribution.

[0122] Return to reference Figure 3 The user, who is viewing the sample bar 300 and the mapping bar 400, inputs the message "Exclude sample 1 and add nighttime data" 213 using the operation unit 43. As a result, the acquisition unit 11 acquires the target information including the two messages "Exclude sample 1" and "Add nighttime data" as the added target information. The retrieval unit 13 reflects the search conditions corresponding to these two target information in the search formula used in the previous search. The search formula in this case is as follows.

[0123] Consistent score = f(s1×s2×s3×s4) (4)

[0124] In this search query, consistency scores s3 and s4 are added relative to the search query expressed by equation (3). Consistency score s3 represents the similarity of the brightness distribution relative to the given nighttime. Regarding consistency score s4, it represents the value of "0" for additional candidate data belonging to sample 1 and the value of "1" for additional candidate data not belonging to sample 1.

[0125] The retrieval unit 13 uses retrieval formula (4) to calculate the consistency score of the candidate data stored in the database 20, and retrieves a dataset consisting of candidate data whose consistency scores exceed a threshold, as an additional candidate dataset. Then, the screen generation unit 14 uses the retrieved additional dataset to update the merged dataset, and uses the updated merged dataset to update the samples displayed in the sample column 300 and the second distribution displayed in the mapping column 400.

[0126] If, after repeating this process, an additional candidate dataset is finally obtained that the user approves, the user enters a message indicating confirmation into chat box 200. Thus, the additional candidate dataset retrieved at that point is determined to be the additional dataset.

[0127] In this embodiment, based on the objective information, additional candidate datasets that match the objective information are retrieved from the candidate datasets. A visual display showing the features of the merged dataset after adding the candidate datasets to the existing dataset is then output to the display. Therefore, the user can understand the impact of the additional candidate datasets on the existing dataset through the visual display and confirm whether the additional candidate datasets meet the objective. Thus, data merging that meets the user's objective can be achieved.

[0128] The following variations are possible with respect to this disclosure.

[0129] (1) In the above embodiment, the target information is input in the form of a dialog, but this disclosure is not limited to this, and the target information can also be input in the form of a selection. In this case, the acquisition unit 11 can acquire the target information by displaying the selection options related to the target information on the prompt screen G1 in advance and allowing the user to select the selection options. An example of the selection options is a selection bar for selecting whether to use data for general recognition or data specifically for current classification, a selection bar for selecting the image size, etc.

[0130] (2) In Figure 4In the example, the mapping display bar 402 graphically displays the first and second distributions, but this is just one example; the first and second distributions can also be displayed numerically. For instance, regarding the brightness distribution, the mapping display bar 402 could also display the average of the most frequent brightness values ​​of the existing data contained in the existing dataset as the first distribution, and the average of the most frequent brightness values ​​of the merged data contained in the merged dataset as the second distribution.

[0131] (3) The retrieval unit 13 compares the consistency score with the threshold, but this disclosure is not limited thereto. The retrieval unit 13 may also sort the candidate data stored in the database 20 in descending order of consistency score, and retrieve the candidate data ranked higher than the given position as additional candidate data.

[0132] (4) In Figure 1 In this configuration, the data merging auxiliary device is composed of server 1, but it can also be composed of terminal 40. In this case, terminal 40 only needs to include acquisition unit 11 to merging unit 15 and database 20. Furthermore, acquisition unit 11 to merging unit 15 can also be distributed between server 1 and terminal 40.

[0133] Industrial availability

[0134] This disclosure is useful in the field of machine learning technology.

Claims

1. A data merging auxiliary method, The computer performs the following processing: Obtain purpose information, which indicates the user's purpose in relation to the data merging of the additional dataset into the user's existing dataset; Based on the stated objective information, additional candidate datasets that meet the stated objective are retrieved from the candidate datasets; Generate a prompt screen that visualizes the features of the merged dataset after appending the candidate dataset to the existing dataset; The prompt screen is output to the display.

2. The data merging auxiliary method according to claim 1, wherein, The prompt screen includes: a first distribution representing the distribution of the feature quantities of the existing data constituting the existing dataset in the feature space, and a second distribution representing the distribution of the feature quantities of the merged data constituting the merged dataset in the feature space.

3. The data merging auxiliary method according to claim 1 or 2, wherein, The prompt screen includes: multiple samples from each cluster of the merged dataset, obtained by clustering the merged dataset into multiple clusters.

4. The data merging auxiliary method according to claim 3, wherein, Furthermore, upon receiving an instruction from the user specifying a useless sample from the plurality of samples, the additional candidate data contained in the cluster to which the useless sample belongs is deleted from the additional candidate dataset. Furthermore, the merged dataset is updated using the appended candidate dataset after deletion, and the updated merged dataset is used to update the prompt screen.

5. The data merging auxiliary method according to claim 1 or 2, wherein, Furthermore, upon obtaining additional purpose information related to the data merging, the additional candidate dataset is retrieved again from the candidate dataset based on the additional purpose information. The merged dataset is updated using the newly retrieved supplementary candidate dataset, and the updated merged dataset is used to update the prompt screen.

6. The data merging auxiliary method according to claim 1 or 2, wherein, The retrieval includes: retrieving additional candidate datasets that meet the objective by comparing the feature values ​​of the objective information with the feature values ​​of the candidate data constituting the candidate dataset.

7. The data merging auxiliary method according to claim 1 or 2, wherein, The search includes: For each of the multiple candidate data constituting the candidate dataset, the consistency degree of each of the multiple search conditions is calculated, and the consistency degrees are synthesized to calculate a consistency score for the purpose for each of the multiple candidate data; and A dataset consisting of candidate data whose consistency scores exceed the threshold is retrieved as the additional candidate dataset.

8. The data merging auxiliary method according to claim 1 or 2, wherein, The feature quantities include multiple categories of feature quantities of different types. Furthermore, upon receiving an instruction to select one category feature from the plurality of category feature quantities, the prompt screen displays the features of the merged dataset for that one category feature quantity.

9. The data merging auxiliary method according to claim 7, wherein, The existing dataset and the candidate dataset are both composed of image data. The multiple category features include at least two of the following: vector features obtained by inputting the image data into the learned model, the metadata of the image data, the brightness distribution of the image data, and annotation information.

10. The data merging auxiliary method according to claim 9, wherein, The metadata includes at least one of the following: the size of the image data, the aspect ratio of the image data, and the date and time of acquisition of the image data.

11. The data merging auxiliary method according to claim 9, wherein, The annotation information pertains to each of the existing dataset and the additional candidate datasets, and includes at least one of the following: the proportion of the category to which the object represented by the annotation assigned to the image data belongs in the dataset, the proportion of the annotated image data in the dataset, and the error rate of the annotations in the dataset.

12. The data merging auxiliary method according to claim 1 or 2, wherein, The existing dataset and the additional dataset are datasets used for machine learning.

13. The data merging auxiliary method according to claim 2, wherein, The second distribution includes multiple distributions corresponding to multiple clusters obtained by clustering the additional candidate data constituting the additional candidate dataset.

14. A data merging auxiliary device, comprising a processor, The processor performs the following processing: Obtain purpose information, which indicates the user's purpose in relation to the data merging of the additional dataset into the user's existing dataset; Based on the stated objective information, additional candidate datasets that meet the stated objective are retrieved from the candidate datasets; Generate a prompt screen that visualizes the features of the merged dataset after appending the candidate dataset to the existing dataset; The prompt screen is output to the display.

15. A data merging auxiliary program, The computer will perform the following processing: Obtain purpose information, which indicates the user's purpose in relation to the data merging of the additional dataset into the user's existing dataset; Based on the stated objective information, additional candidate datasets that meet the stated objective are retrieved from the candidate datasets; Generate a prompt screen that visualizes the features of the merged dataset after appending the candidate dataset to the existing dataset; The prompt screen is output to the display.

Images