Data integration assistance method, data integration assistance device, and data integration assistance program

Abstract

This data integration assistance device acquires purpose information indicating the purpose of a user with respect to data integration in which an additional dataset is added to an existing dataset owned by the user, searches for an additional candidate dataset that matches the purpose from candidate datasets on the basis of the purpose information, generates a presentation screen image visualizing a feature for a case where the additional candidate dataset is added to the existing dataset, and outputs the presentation screen image to a display.

Description

Data integration support method, data integration support device, and data integration support program

[0001] The present disclosure relates to techniques for supporting data integration.

[0002] Patent Literature 1 discloses a method for identifying reusable project components for building a new project. Specifically, Patent Literature 1 discloses: extracting current requirements for the new project from a data source using natural language processing (NLP); detecting existing requirements similar to the current requirements by performing a semantic analysis of the current requirements and the existing requirements; determining a similarity score for the existing requirements based on the similarity between the current requirements and the existing requirements; retrieving project components related to the existing requirements based on the similarity score; and retrieving the retrieved project components as reusable project components for building the new project.

[0003] However, in the technology of Patent Document 1, reusable project components are extracted unilaterally by a machine, and the user's objectives for the project are not taken into consideration at all. Therefore, the technology of Patent Document 1 has the problem of being unable to realize data integration that meets the user's objectives.

[0004] Patent No. 7007486

[0005] The present disclosure has been made in consideration of such problems, and provides a technology that can realize data integration that meets the user's objectives.

[0006] In one aspect of the present disclosure, a data integration support method includes a computer acquiring purpose information indicating a user's purpose for data integration, which involves adding an additional dataset to an existing dataset owned by the user, searching for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information, generating a presentation screen that visualizes the characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset, and outputting the presentation screen to a display.

[0007] According to the present disclosure, data integration that meets the user's objectives can be realized.

[0008] It is a block diagram showing an example of a configuration of a data integration support system according to an embodiment of the present disclosure.It is a flowchart showing an example of processing by a server according to the embodiment.It is a diagram showing an example of a presentation screen.It is a diagram showing details of a sample column and a map column.

[0009] (Findings underlying the present disclosure) In recent years, a trained model suitable for a task that a user wants to solve has been generated by transfer learning a general-purpose base model obtained by self-supervised learning (SSL) of common base data that can be used in various businesses. If an additional dataset suitable for the task that the user wants to solve is extracted from the massive amount of data accumulated in a large-scale database within a company to obtain training data for transfer learning, the development period for the trained model can be shortened and the development cost can be reduced.

[0010] However, it is not easy for a machine to extract additional data sets from large databases that meet a user's data integration goals.

[0011] For example, if an existing dataset contains many "dog" and "cat" datasets, and the user wishes to "recognize birds in addition to dogs and cats," the user will want to extract a bird dataset from a large-scale database as an additional dataset. On the other hand, if an existing dataset contains many "dog" and "cat" datasets, and the user wishes to improve the recognition accuracy of "dog" and "cat," the user will want to add additional "dog" and "cat" datasets from a large-scale database. In this way, the purpose of data integration varies depending on the task the user wants to solve. Therefore, it is not easy for a machine to unilaterally extract an additional dataset from a large-scale database that meets the user's data integration purpose.

[0012] When the technology described in Patent Document 1 is applied to such data integration, there is a problem in that the machine extracts additional data sets based on unilateral criteria, making it impossible to achieve data integration that meets the user's objectives.

[0013] The present disclosure has been made to solve such problems.

[0014] (1) In one aspect of the present disclosure, a data integration support method includes a computer acquiring purpose information indicating a user's purpose for data integration, which involves adding an additional dataset to an existing dataset owned by the user, searching for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information, generating a presentation screen that visualizes the characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset, and outputting the presentation screen to a display.

[0015] According to this configuration, the candidate datasets are searched for additional candidate datasets that match the purpose indicated by the purpose information based on the purpose information, and a presentation screen that visualizes the characteristics of an integrated dataset obtained by adding the candidate dataset to the existing dataset is output to the display. Therefore, through the presentation screen, the user can understand the impact of the additional candidate dataset on the existing dataset and confirm whether the additional candidate dataset matches the purpose. This makes it possible to realize data integration that matches the user's purpose.

[0016] (2) In the data integration support method described in (1) above, the presentation screen may include a first distribution showing a distribution in a feature space of features of the existing data constituting the existing dataset, and a second distribution showing a distribution in the feature space of features of the integrated data constituting the integrated dataset.

[0017] According to this configuration, a first distribution showing the distribution of the features of the existing dataset and a second distribution showing the distribution of the features of the integrated dataset are displayed on the presentation screen, allowing the user to quantitatively confirm the impact that the candidate dataset to be added will have on the existing dataset.

[0018] (3) In the data integration support method described in (1) or (2) above, the presentation screen may include a plurality of samples for each cluster of the integrated dataset obtained by clustering the integrated dataset into a plurality of clusters.

[0019] According to this configuration, a sample of each cluster of the clustered integrated dataset is presented, allowing the user to more specifically confirm the impact of adding an additional candidate dataset.

[0020] (4) In the data integration support method described in (3) above, when a user instruction specifying an unnecessary sample from the plurality of samples is received, additional candidate data included in a cluster to which the unnecessary sample belongs may be deleted from the additional candidate dataset, and the integrated dataset may be updated using the deleted additional candidate dataset, and the presentation screen may be updated using the updated integrated dataset.

[0021] According to this configuration, it is possible to allow the user to easily delete unnecessary additional candidate data from the additional candidate data set, and also to allow the user to recognize the change in the impact caused by the deletion.

[0022] (5) In the data integration support method described in any of (1) to (4) above, if additional purpose information for the data integration is acquired, the additional candidate dataset may be re-searched from among the candidate datasets based on the additional purpose information, the integrated dataset may be updated using the re-searched additional candidate dataset, and the presentation screen may be updated using the updated integrated dataset.

[0023] According to this configuration, it is possible to interactively re-search for candidate datasets to be added while taking into consideration the purpose of addition of the user viewing the presentation screen, and to allow the user to recognize changes in characteristics after the re-search.

[0024] (6) In the data integration support method described in any one of (1) to (5) above, the search may include searching for additional candidate datasets that meet the purpose by comparing the features of the target information with the features of the candidate data that constitute the candidate dataset.

[0025] According to this configuration, additional candidate data sets that match the purpose are searched for by comparing the feature amounts of the existing data with the feature amounts of the candidate data, so additional candidate data that match the purpose can be accurately searched for.

[0026] (7) In the data integration support method described in any of (1) to (6) above, the method may include calculating a degree of match for each of a plurality of search conditions for each of a plurality of candidate data constituting the candidate dataset, and calculating a match score for the purpose for each of the plurality of candidate data by combining the degrees of match, and searching for a dataset composed of candidate data whose match score exceeds a threshold as the additional candidate dataset.

[0027] This configuration makes it possible to accurately search for additional candidate datasets that meet the purpose.

[0028] (8) In the data integration support method described in any one of (1) to (7) above, the features include a plurality of type-specific features of different types, and further, when the presentation screen receives an instruction to select one type-specific feature from the plurality of type-specific features, it may display the features of the integrated data set for the one type-specific feature.

[0029] According to this configuration, the user can confirm the effect of adding additional candidate data for each type of feature amount via the presentation screen.

[0030] (9) In the data integration support method described in (7) above, the existing dataset and the candidate dataset may each be composed of image data, and the plurality of type-specific features may include at least two of vector features obtained by inputting the image data into a trained model, meta information of the image data, brightness distribution of the image data, and annotation information.

[0031] According to this configuration, the user can check additional data sets that match the purpose for any of the vector feature amount, meta information, brightness distribution, and annotation information.

[0032] (10) In the data integration support method described in (9) above, the meta-information may include at least one of the size of the image data, the aspect ratio of the image data, and the acquisition date and time of the image data.

[0033] This configuration allows the user to check for additional data sets that match the purpose in terms of size, aspect ratio, or acquisition date and time.

[0034] (11) In the data integration support method described in (9) or (10) above, the annotation information may include, for each of the existing dataset and the candidate additional dataset, at least one of the following: the proportion in the dataset of a class to which an object indicated by an annotation attached to the image data belongs; the proportion in the dataset of annotated image data; and the error rate of annotations in the dataset.

[0035] According to this configuration, the user can check additional data sets that meet the purpose in terms of the proportion of classes, the proportion of annotated image data, or the error rate for annotations.

[0036] (12) In the data integration support method described in any one of (1) to (11) above, the existing dataset and the additional dataset may each be a dataset for machine learning.

[0037] This configuration allows the user to check additional candidate datasets that match the user's purpose in machine learning.

[0038] (13) In the data integration support method described in (2) above, the second distribution may include a plurality of distributions corresponding to a plurality of clusters obtained by clustering the additional candidate data that constitutes the additional candidate data set.

[0039] According to this configuration, it is possible to provide the user with an indication of which additional candidate data sets are unnecessary to add.

[0040] (14) In another aspect of the present disclosure, a data integration support device includes a processor, wherein the processor acquires purpose information indicating a user's purpose for data integration, which is to add an additional dataset to an existing dataset owned by the user, searches for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information, generates a presentation screen that visualizes the characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset, and outputs the presentation screen to a display.

[0041] This configuration makes it possible to provide a data integration support device that realizes data integration that meets the user's objectives.

[0042] (15) In another aspect of the present disclosure, a data integration support program causes a computer to execute the following process: acquire purpose information indicating a user's purpose for data integration, which involves adding an additional dataset to an existing dataset owned by the user; search for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information; generate a presentation screen that visualizes the characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset; and output the presentation screen to a display.

[0043] This configuration makes it possible to provide a data integration support program that realizes data integration that meets the user's objectives.

[0044] The present disclosure can also be realized as a data integration support system that operates by such a data integration support program. Needless to say, such a computer program can be distributed on a non-transitory computer-readable recording medium such as a CD-ROM or via a communication network such as the Internet.

[0045] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0046] (Embodiment) Fig. 1 is a block diagram showing an example of the configuration of a data integration support system according to an embodiment of the present disclosure. The data integration support system includes a server 1 (an example of a data integration support device) and a terminal 40. The server 1 and the terminal 40 are connected to each other so that they can communicate with each other via a network NT. An example of the network NT is the Internet. The server 1 is, for example, a cloud server composed of one or more computers. The terminal 40 is a computer used by a user. An example of the terminal 40 is a desktop computer or a portable computer such as a tablet computer or a smartphone.

[0047] The server 1 includes a processor 10, a database 20, and a communication unit 30. An example of the processor 10 is a central processing unit (CPU). The processor 10 includes an acquisition unit 11, a feature extraction unit 12, a search unit 13, a screen generation unit 14, and an integration unit 15. The acquisition units 11 to 15 are realized by the processor executing a data integration support program. However, this is just an example, and the acquisition units 11 to 15 may also be configured with dedicated hardware circuits.

[0048] The acquisition unit 11 acquires purpose information indicating a user's purpose for data integration, which involves adding an additional dataset to an existing dataset owned by the user. An example of the purpose information is a keyword indicating the characteristics of the dataset to be added. Examples of the keyword include a keyword indicating a scene such as "construction site," "highway," or "general road," or a keyword indicating a class to be added (e.g., dog, cat, bird, etc.).

[0049] An example of the purpose information is meta-information such as the image size to be added as a dataset. Details of the meta-information will be described later. Another example of the purpose information can be a request message indicating the characteristics of the dataset desired as an additional dataset, taking into account the characteristics of the existing dataset. If the user requests an additional dataset that includes classes not included in the existing dataset (or a small number of classes), a message such as "I want data for general recognition" can be used as the request message. If the user requests an additional dataset that includes more data on classes included in the existing dataset, a message such as "I want data specialized for the current classes" can be used as the request message.

[0050] Here, the acquisition unit 11 may acquire the target information interactively until the information necessary for the search unit 13 to execute the search process is obtained. In this case, the acquisition unit 11 may interact with the user through a chat field 200 included in the presentation screen G1 displayed on the display 41 of the terminal 40, as shown in FIG. 3, and acquire the target information from a message entered by the user. Alternatively, the acquisition unit 11 may display options regarding the purpose of data integration on the display 41 and acquire the target information from an item selected by the user. Note that "until the information necessary for the search unit 13 to execute the search process is obtained" refers to until the information necessary to determine one or more search conditions constituting a search formula, which will be described later, is obtained.

[0051] The existing data set includes multiple existing data owned by the user. The existing data is, for example, data collected by the user as learning data for machine learning. In the following description, the existing data is described as image data. However, this is just an example, and the existing data may be text data or audio data. The existing data is, for example, data related to a site, such as in-home data, in-vehicle data, and factory data. An example of in-home data is data indicating a person's movements, speech content, the operating status of electrical appliances, power consumption, and temperature inside a residence. An example of in-vehicle data is automobile operation data. An example of automobile operation data is the automobile's speed, acceleration, battery capacity, current, voltage, fuel efficiency, power consumption, state of charge (SOC), and temperature. An example of factory data is data indicating power consumption, temperature, operation data of various facilities, and the assembly status of parts. The existing data set is stored in advance in the database 20 in association with the user ID of the user who owns the existing data set.

[0052] The additional data set includes a plurality of additional data. The additional data is candidate data that is finally selected from the candidate data stored in the database 20 as data to be added to the existing data set. In the following description, the additional data is described as being image data. However, this is just an example, and the additional data may be text data or audio data.

[0053] The feature extraction unit 12 extracts feature amounts from the existing dataset and the candidate dataset. The feature amounts include a plurality of type-specific feature amounts of different types.

[0054] The type-specific features include vector features, meta information of image data, brightness distribution, and annotation information. Vector features are composed of multidimensional vectors obtained by inputting existing data and candidate data into a trained model that has been machine-learned in advance to calculate vector features, for example. If the existing data and candidate data are composed of text data, the vector features are composed of word vectors. In this case, word vectors calculated using a trained natural language processing model can be used as vector features. If the existing data and candidate data are composed of audio data, the vector features are calculated using a trained model that converts audio data into vectors.

[0055] The meta information includes the size of the image data (image size), the aspect ratio of the image data, and the date and time the image data was acquired. The image size is expressed, for example, using the number of pixels in the width and height of the image data. The aspect ratio is the ratio between the width and height of the image data. The acquisition date and time is the date and time the image data was captured.

[0056] The luminance distribution of image data is data that represents the frequency of each of a plurality of luminance values ​​of each pixel data that constitutes the image data.

[0057] The annotation information includes a class ratio, an annotation ratio, and an error rate. The class ratio is the ratio in each of the existing dataset and the candidate additional dataset of a class to which an object indicated by an annotation attached to image data belongs. The annotation ratio is the ratio in the dataset of annotated image data. The error rate is the error rate for the annotation. The annotation is annotation information attached to the image data as training data. The annotation includes, for example, a bounding box surrounding an object included in the image data and a tag indicating the class to which the object surrounded by the bounding box belongs. For example, in the case of image data to which an annotation is attached to a dog, the annotation is composed of a bounding box surrounding the dog and a tag indicating that the class to which the object surrounded by the bounding box belongs is dog.

[0058] For example, if annotated dogs, cats, and birds are added to a dataset at a ratio of 10%, 10%, and 80%, respectively, the class ratio is 1:1:8. For example, if the ratio of image data to which annotations are added in a dataset is 30%, the annotation ratio is 30%. For example, if the error rate is 50% when annotated image data is input to a recognizer, the error rate is 50%. A pre-measured value of the error rate is associated with the image data.

[0059] The search unit 13 searches the candidate datasets for an additional candidate dataset that matches the purpose of data integration based on the purpose information acquired by the acquisition unit 11. The candidate dataset includes multiple candidate data. The candidate data is data that is stored in advance in the database 20 and is a candidate for additional data. The candidate data is composed of big data accumulated by companies. For example, the candidate data is data obtained by monitoring homes, automobiles, and factories using various sensors. Therefore, a large amount of candidate data is stored in the database 20. In the following description, the candidate data will be described as image data. However, this is just an example, and the candidate data may be text data or audio data. Like existing data, the candidate data may also be data related to the field, such as in-home data, in-vehicle data, and factory data.

[0060] The search unit 13 searches for an additional candidate dataset that matches the purpose by comparing the feature amount of the target information with the feature amount of the candidate data that constitutes the candidate dataset.

[0061] Specifically, the search unit 13 calculates the degree of match for each of a plurality of conditions for each of a plurality of candidate data constituting the candidate data set, and calculates a match score for the purpose of data integration for each of the plurality of candidate data from the calculated degree of match. Furthermore, the search unit 13 searches for a data set composed of candidate data whose match score exceeds a threshold as an additional candidate data set.

[0062] The match score is expressed, for example, by the following search formula:

[0063] Match score = f (s1, s2, ..., si, ..., sn) (1) Here, f represents some function that combines si. An example of the function f is a function that multiplies si. si represents the degree of match for each of multiple search conditions. i is an index that specifies the search condition.

[0064] An example of a search condition is the similarity condition for keywords contained in the target information. In this case, the higher the similarity between the vector feature of the keyword and the vector feature of the candidate data, the higher the degree of match. The similarity can be, for example, vector distance or cosine similarity.

[0065] Furthermore, as an example of a search condition, a condition of similarity or dissimilarity with an existing dataset can be employed. For example, if the target information contains a request message stating "general recognition data is desired," the greater the dissimilarity of the vector features of candidate data to the existing dataset, the greater the degree of match. As a result, candidate data including objects belonging to classes other than those included in the existing dataset are searched for as additional candidate data. For example, the inverse of similarity can be used as the dissimilarity. For example, if the target information contains a message stating "data specialized for the current class is desired," the greater the similarity of candidate data to the existing dataset, the greater the degree of match. In this case, candidate data including objects belonging to the same class as the class included in the existing dataset are searched for as additional candidate data. When calculating the similarity or dissimilarity between the vector features of an existing dataset and those of candidate data, the search unit 13 may use a representative value of the vector features of the existing dataset. The representative value can be the average value of the vector features of the existing data constituting the existing dataset. Note that if a tag indicating a class is attached to the candidate data, the search unit 13 may calculate the similarity or dissimilarity using the vector features of text data included in the tag.

[0066] For example, if an image size is specified as a filtering condition in the target information, the search unit 13 may calculate the degree of match of candidate data that satisfies the specified filtering condition as "1" and the degree of match of candidate data that does not satisfy the filtering condition as "0."

[0067] When these three search conditions are used, the match score is expressed by the following search formula:

[0068] Match score = f (s1 × s2 × s3) (2) The match score s1 indicates the similarity to the keyword, the match score s2 indicates the similarity or dissimilarity to an existing dataset, and the match score s3 indicates a value indicating whether the filtering condition is satisfied. For example, if the filtering condition is a condition related to image size (e.g., 640 × 480 pixels or more), the match score s3 of candidate data that satisfies this filtering condition is "1," and the match score s3 of candidate data that does not satisfy this filtering condition is "0."

[0069] Here, image size is used as a filtering condition, but this is just an example, and meta information other than image size may also be used. For example, "aspect ratio is equal to or greater than a predetermined value" or "period of time when image data was acquired" may be used as filtering conditions.

[0070] The screen generation unit 14 generates a presentation screen that visualizes the characteristics of the integrated dataset in which the additional candidate dataset has been added to the existing dataset, and outputs the generated presentation screen to the display 41 of the terminal 40.

[0071] The screen generation unit 14 calculates a first distribution that indicates the distribution in feature space of the features of the existing data that make up the existing dataset, and a second distribution that indicates the distribution in feature space of the features of the integrated data that make up the integrated dataset, and displays a presentation screen including the first distribution and the second distribution on the display 41.

[0072] When the screen generation unit 14 receives an instruction to select one type-specific feature from multiple type-specific features, it may calculate a first distribution and a second distribution for the one type-specific feature, and display a presentation screen including the calculated first distribution and second distribution on the display 41.

[0073] The screen generator 14 may cluster the integrated data set into multiple clusters, determine samples for each of the multiple clusters based on the clustering results, and include the determined samples on the presentation screen. A clustering method using a Gaussian mixture distribution may be employed as the clustering method. The screen generator 14 may calculate the center of gravity of the distribution of the feature quantities of the additional candidate data set for each of the multiple clusters, and determine the additional candidate data closest to the calculated center of gravity as the sample.

[0074] When the acquisition unit 11 acquires a user instruction specifying an unnecessary sample from multiple samples, the screen generation unit 14 may delete the additional candidate data included in the cluster to which the unnecessary sample belongs from the additional candidate dataset, update the integrated dataset using the additional candidate dataset from which the additional candidate data has been deleted, and update the presentation screen displayed on the display 41 using the updated integrated dataset.

[0075] When the acquisition unit 11 acquires additional objective information regarding data integration, the search unit 13 may re-search for additional candidate datasets from among the candidate datasets based on the additional objective information, update the integrated dataset using the re-searched additional candidate dataset, and update the presentation screen displayed on the display 41 using the updated integrated dataset. The additional objective information is objective information additionally input by a user viewing the presentation screen. For example, if a user confirms from the presentation screen that the proportion of annotated image data in the candidate data included in the additional candidate dataset is low, the user inputs a message requesting the addition of more annotated images as additional objective information. For example, if a user confirms from the presentation screen that the proportion of a certain class in the additional candidate dataset is low, the user inputs a message requesting the addition of the class with the low proportion as additional objective information. For example, if a user confirms from the presentation screen that the additional candidate data included in the additional candidate dataset lacks image data of a certain image size or larger, the user inputs a message requesting image data of a size larger than that image size as additional objective information.

[0076] When a confirmation instruction indicating that the additional candidate data should be confirmed as additional data is input by the user using the operation unit 43 and the acquisition unit 11 acquires the confirmation instruction using the communication unit 30, the integrating unit 15 confirms the currently searched additional candidate data set as the additional data set. Then, the integrating unit 15 performs data integration by adding the confirmed additional data set to the existing data set. In this case, the integrating unit 15 confirms the integrated data set in which the additional data set is added to the existing data set as the final integrated data set, assigns a user ID to the final integrated data set, and stores it in the database 20.

[0077] The database 20 is configured, for example, by a non-volatile storage device, and stores the candidate dataset, the existing dataset, and the final integrated dataset.

[0078] The communication unit 30 is configured with a communication circuit that connects the server 1 to the network NT. The communication unit 30 receives the objective information, confirmation instructions, and the like transmitted from the terminal 40. The communication unit 30 also transmits display data for the presentation screen to the terminal 40.

[0079] The terminal 40 includes a display 41, a processor 42, an operation unit 43, and a communication unit 44. The display 41 is composed of, for example, a liquid crystal display panel and displays a presentation screen. The processor 42 controls the terminal 40. The operation unit 43 is composed of input devices such as a keyboard and a touch panel. The operation unit 43 accepts input of target information. The communication unit 44 is a communication circuit that connects the terminal 40 to the network NT. The communication unit 40 receives display data for the presentation screen sent from the server 1 and transmits target information accepted by the operation unit 43 to the server 1.

[0080] 2 is a flowchart showing an example of the processing of the server 1 in this embodiment. It is assumed that the acquisition unit 11 has acquired the user ID of the user who uses the terminal 40 before this flowchart is executed.

[0081] In step S1 , the acquisition unit 11 acquires the target information input by the user at the terminal 40 .

[0082] Next, in step S2, the feature extraction unit 12 extracts features for all existing data constituting the existing dataset, and also extracts features for all candidate data constituting the candidate dataset. In this case, the feature extraction unit 12 extracts all type-specific features for all existing data, and also extracts all type-specific features for all candidate data. Here, the existing dataset used is the existing dataset of the user who uses the terminal 40, and is identified by the user ID.

[0083] Next, in step S3, the search unit 13 searches the candidate datasets for candidate data that matches the purpose indicated by the purpose information, as additional candidate data, based on the purpose information acquired in step S1. In this case, the search unit 13 calculates a match score for each candidate data set using a search formula such as that shown in equation (1), and searches for candidate data whose match score exceeds a threshold as additional candidate data. For example, if the purpose information includes a message requesting images of birds and a message requesting data for general-purpose recognition, the search formula is match score = f(s1 x s2). Here, the degree of match s1 indicates the degree of similarity to the bird keyword, and the degree of match s2 indicates the degree of dissimilarity to the existing dataset.

[0084] Next, in step S4, the screen generation unit 14 clusters the additional candidate data retrieved in step S3 into a plurality of clusters, and determines samples for each of the plurality of clusters from the clustering results.

[0085] Next, in step S5, the screen generation unit 14 displays the multiple samples on the display 41 by transmitting display data for displaying the multiple samples determined in step S4 on the presentation screen to the terminal 40 using the communication unit 30.

[0086] Next, in step S6, the screen generation unit 14 calculates a first distribution indicating the distribution of features in the feature space of the existing data that constitutes the existing dataset, and a second distribution indicating the distribution of features in the feature space of the integrated data that constitutes the integrated dataset.

[0087] Next, in step S7, the screen generation unit 14 displays the first distribution and the second distribution calculated in step S6 on the display 41 by transmitting display data to the terminal 40 using the communication unit 30 to display the first distribution and the second distribution in the presentation image.

[0088] Next, in step S8, a message requesting additional target information or unnecessary samples is sent to the terminal 40 using the communication unit 30. As a result, a message such as "Are there any unnecessary samples?" or "Do you have any additional requests?" is displayed on the display 41.

[0089] Next, in step S9, the acquisition unit 11 determines whether or not additional target information has been input. In this case, if additional target information has been input (YES in step S9), the process proceeds to step S10, and if additional target information has not been input (NO in step S9), the process proceeds to step S11.

[0090] Next, in step S10, the search unit 13 re-searches for additional candidate datasets using the additional target information. For example, if the additional target information includes a message indicating that images of a predetermined size or larger are desired, the search unit 13 may re-search for additional candidate datasets using a search formula in which the search condition "image size is a predetermined size or larger" is added to the search formula used in the previous search. In the above example, the search formula may be, for example, a match score = f(s1 x s2 x s3). In this case, the match score s1 indicates the similarity to the bird keyword, the match score s2 indicates the dissimilarity of the candidate data to the existing dataset, and the match score s3 indicates a value ("1" or "0") indicating whether the filtering condition "image size is a predetermined size or larger" is satisfied.

[0091] Next, in step S11, the search unit 13 updates the integrated dataset with the additional candidate datasets that have been re-searched.

[0092] Next, in step S12, the screen generator 14 updates the presentation screen using the updated integrated data set. In this case, the screen generator 14 may use the updated integrated data set to update the second distribution displayed on the display 41. Furthermore, the screen generator 14 may cluster the updated integrated data set into multiple clusters, determine samples for each of the multiple clusters from the clustering results, and update the samples on the presentation screen with the determined samples.

[0093] Next, in step S13, the acquisition unit 11 determines whether or not an unnecessary sample has been input. If an unnecessary sample has been input (YES in step S13), the process proceeds to step S14. If an unnecessary sample has not been input (NO in step S13), the process proceeds to step S15.

[0094] Next, in step S14, the screen generator 14 updates the integrated dataset by deleting from the integrated dataset any additional candidate datasets that belong to the same cluster as the designated unnecessary sample.

[0095] Next, in step S15, the screen generator 14 updates the presentation screen using the integrated data set updated in step S 14. In this case, the screen generator 14 may update the second distribution and the samples using the updated integrated data set, similar to step S12.

[0096] Next, in step S16, if a confirmation instruction is acquired by the acquisition unit 11 (YES in step S16), the integration unit 15 confirms the currently searched additional candidate dataset as an additional dataset, and confirms the integrated data obtained by adding the confirmed additional dataset to the existing dataset as the final integrated dataset (step S17).

[0097] On the other hand, if the acquisition unit 11 does not acquire a confirmation instruction (NO in step S16), the process returns to step S8, and the processes from step S8 onward are executed. By repeating the processes of steps S8 to S16, an integrated data set suitable for the task that the user wants to solve is generated while interactively incorporating the user's requests for data integration.

[0098] FIG. 3 is a diagram showing an example of the presentation screen G1. The presentation screen G1 includes a chat field 200, a sample field 300, and a map field 400. The chat field 200 is a field where the user inputs target information. The chat field 200 prompts the user to input target information in an interactive format. In the chat field 200, the left column indicates a server-side message, and the right column indicates a user-side message. The server-side message includes a message generated by the acquisition unit 11 to prompt the user to input target information. The user-side message includes a user's reply message to the server-side message.

[0099] In this example, the acquisition unit 11 first displays a message 201 asking the user about the purpose of data integration, such as "What kind of data do you need?" In response to this question, the user uses the operation unit 43 to input a message 211 indicating that "I want images of the construction site."

[0100] Next, the acquisition unit 11 displays a message 202 asking, "Do you need more extensive data than the current data set?" In response to this, the user inputs a message 212 indicating agreement, "Yes," using the operation unit 43. Through these interactions, the acquisition unit 11 acquires target information including the keyword "construction site" and a message indicating that general-purpose data is desired. By acquiring this target information, the search unit 13 sets a search formula as follows:

[0101] Match score = f ( s1 × s2 ) (3) where the degree of match s1 indicates the similarity of the candidate data to the keyword “construction site”, and the degree of match s2 indicates the dissimilarity of the candidate data to the existing dataset. Next, the search unit 13 calculates the match score for each of the multiple candidate data stored in the database 20 using this search formula, and searches for a dataset consisting of candidate data whose match score exceeds a threshold as an additional candidate dataset.

[0102] The map field 400 includes a selection field 401 and a map display field 402. The selection field 401 allows the user to select one type-specific feature from a plurality of type-specific feature values. The map display field 402 displays a first distribution and a second distribution for the one type-specific feature value selected from the selection field 401.

[0103] When the user selects the selection field 401 using the operation unit 43, the selection field 401 displays a pull-down menu displaying a list of multiple type-specific feature quantities. The user uses the operation unit 43 to select one type-specific feature quantity from the displayed list of multiple type-specific feature quantities. A selection instruction indicating the selected type-specific feature quantity is then transmitted from the terminal 40 to the server 1, and the acquisition unit 11 acquires the selection instruction. The screen generation unit 14 calculates a first distribution and a second distribution for the selected type-specific feature quantity and transmits display data for displaying the first distribution and the second distribution on the presentation screen G1 to the terminal 40 using the communication unit 30. As a result, the map display field 402 displays the first distribution and the second distribution for the single type-specific feature quantity. In its initial state, the map display field 402 displays the first distribution and the second distribution for the default type-specific feature quantity (e.g., a vector feature quantity).

[0104] FIG. 4 shows details of the sample column 300 and the map column 400. The map display column 402 displays the first and second distributions in a display format using two-dimensional coordinate axes. Here, vector features are selected as the type-specific feature 1. Distribution 601 indicates the first distribution, and distributions 501, 502, and 503 indicate the second distribution. Since distribution 601 indicates the distribution of vector features related to an existing dataset, the message "before" is displayed nearby. Distributions 501 to 503 indicate the distribution of vector features for each cluster of an integrated dataset that has been clustered into three clusters. In detail, distribution 501 indicates the distribution of vector features in the cluster to which sample 1 belongs, distribution 502 indicates the distribution of vector features in the cluster to which sample 2 belongs, and distribution 503 indicates the distribution of vector features in the cluster to which sample 3 belongs. Therefore, "sample group 1" is written near distribution 501, "sample group 2" is written near distribution 502, and "sample group 3" is written near distribution 503, clearly indicating the correspondence between distributions 501 to 503 and the three samples.

[0105] The screen generator 14 may calculate the first distribution and the second distribution by compressing the number of dimensions of the vector feature quantities of the existing data constituting the existing dataset and the integrated data constituting the integrated dataset to two dimensions. Here, the vector feature quantities of the existing data and the integrated data are compressed to two dimensions, but they may also be compressed to three dimensions. In this case, the map display field 402 may display the first distribution and the second distribution in a display format using three-dimensional coordinate axes.

[0106] The display format of the first distribution and the second distribution displayed in the map display field 402 differs depending on the selected type-specific feature. For example, if image size is selected as the type-specific feature, the screen generation unit 14 may display a histogram showing the frequency according to image size in the existing dataset as the first distribution in the map display field 402, and may also display a histogram showing the frequency according to image size in the integrated dataset as the second distribution in the map display field 402. The same applies to the aspect ratio and the acquisition date and time.

[0107] For example, if a luminance distribution is selected as one type-specific feature, the screen generation unit 14 displays a histogram showing the frequency according to luminance in the existing data set as a first distribution, and also displays a histogram according to luminance in the integrated data set as a second distribution in the map display field 402.

[0108] For example, when class proportions are selected as one type-specific feature, the screen generation unit 14 displays numerical values ​​indicating the class proportions in the existing datasets as a first distribution in the map display field 402, and also displays numerical values ​​indicating the class proportions in the integrated dataset as a second distribution in the map display field 402. An example of how class proportions are displayed is, "The class proportions in the existing datasets are dog: 0.4, cat: 0.1, and the class proportions in the integrated dataset are dog: 0.4, cat: 0.1, crow: 0.2."

[0109] For example, if the annotation ratio is selected as one type-specific feature, the screen generation unit 14 may display a numerical value indicating the annotation ratio in the existing dataset as a first distribution in the map display field 402, and may display the annotation ratio in the integrated dataset as a second distribution. An example of how the annotation ratios are displayed may be, "The ratio of annotated data in the existing dataset is 0.5, and the ratio of annotated data in the integrated dataset is 0.4."

[0110] For example, if an error rate is selected as one type-specific feature, the screen generation unit 14 displays a histogram showing the frequency according to the error rate in the existing dataset as a first distribution in the map display field 402, and also displays a histogram showing the frequency according to the error rate in the integrated dataset as a second distribution in the map display field 402.

[0111] In the example of Figure 4, the integrated data set is clustered into three clusters, so three distributions 501 to 503 are displayed; however, if the integrated data set is clustered into two clusters, the two distributions will be displayed as the second distribution, and if the integrated data set is clustered into four or more clusters, four or more distributions will be displayed as the second distribution.

[0112] Returning to FIG. 3 , a user who viewed the sample column 300 and the map column 400 inputs a message 213, "Exclude sample 1 and add night data," using the operation unit 43. As a result, the acquisition unit 11 acquires target information including two messages, "Exclude sample 1" and "Add night data," as additional target information. The search unit 13 reflects search conditions corresponding to these two target information in the search formula used in the previous search. The search formula in this case is as follows:

[0113] Match score = f (s1 × s2 × s3 × s4) (4) In this search formula, match scores s3 and s4 are added to the search formula shown in formula (3). Match score s3 indicates the similarity to a predetermined nighttime luminance distribution, and match score s4 indicates a value of "0" for additional candidate data belonging to sample 1 and a value of "1" for additional candidate data not belonging to sample 1.

[0114] The search unit 13 calculates the match score of the candidate data stored in the database 20 using the search formula (4), and searches for a dataset consisting of candidate data whose match score exceeds a threshold as an additional candidate dataset. The screen generation unit 14 then updates the integrated dataset using the searched additional dataset, and updates the sample displayed in the sample column 300 and the second distribution displayed in the map column 400 using the updated integrated dataset.

[0115] By repeating this process, when the user finally finds a candidate additional dataset that satisfies the user, the user inputs a confirmation message in the chat field 200. This determines the candidate additional dataset that has been searched for at this point as the additional dataset.

[0116] As described above, according to this embodiment, candidate datasets to be added that match the purpose indicated by the purpose information are searched for from among the candidate datasets based on the purpose information, and a presentation screen that visualizes the characteristics of an integrated dataset obtained by adding the candidate dataset to an existing dataset is output to the display. Therefore, through the presentation screen, the user can understand the impact that the candidate dataset will have on the existing dataset and confirm whether the candidate dataset to be added matches the purpose. This makes it possible to realize data integration that matches the user's purpose.

[0117] The present disclosure can employ the following modifications.

[0118] (1) In the above embodiment, the target information is input in an interactive format. However, the present disclosure is not limited to this. The target information may be input in a selection format. In this case, the acquisition unit 11 may acquire the target information by displaying options related to the target information on the presentation screen G1 in advance and allowing the user to select an option. Examples of options include a selection field that allows the user to select whether they want data for general recognition or data specific to the current class, a selection field that allows the user to select an image size, etc.

[0119] (2) In Fig. 4, the map display field 402 graphically displays the first distribution and the second distribution, but this is just an example, and the first distribution and the second distribution may be displayed numerically. For example, with respect to the luminance distribution, the map display field 402 may display the average value of the most frequent luminance values ​​of the existing data included in the existing data set as the first distribution, and may display the average value of the most frequent luminance values ​​of the integrated data included in the integrated data set as the second distribution.

[0120] (3) Although the search unit 13 compares the matching score with a threshold, the present disclosure is not limited to this. The search unit 13 may rank the candidate data stored in the database 20 in descending order of matching score, and search for candidate data ranked higher than a predetermined rank as additional candidate data.

[0121] (4) In Fig. 1, the data integration support device is configured as a server 1, but it may also be configured as a terminal 40. In this case, the acquisition unit 11 to the integration unit 15 and the database 20 may be provided in the terminal 40. Furthermore, the acquisition unit 11 to the integration unit 15 may be distributed between the server 1 and the terminal 40.

[0122] The present disclosure is useful in the field of machine learning.

Claims

1. A data integration support method, comprising: a computer acquiring purpose information indicating a user's purpose for data integration, which is to add an additional dataset to an existing dataset owned by the user; searching for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information; generating a presentation screen that visualizes the characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset; and outputting the presentation screen on a display.

2. A data integration support method as described in claim 1, wherein the presentation screen includes a first distribution showing a distribution in a feature space of features of existing data constituting the existing dataset, and a second distribution showing a distribution in the feature space of features of integrated data constituting the integrated dataset.

3. The data integration support method according to claim 1 or 2, wherein the presentation screen includes a plurality of samples for each cluster of the integrated data set obtained by clustering the integrated data set into a plurality of clusters.

4. The data integration support method according to claim 3, further comprising the steps of: when receiving an instruction from the user to specify an unnecessary sample from the plurality of samples, deleting from the additional candidate data set the additional candidate data included in a cluster to which the unnecessary sample belongs; updating the integrated data set using the deleted additional candidate data set; and updating the presentation screen using the updated integrated data set.

5. A data integration support method as claimed in claim 1 or 2, further comprising the steps of: when additional purpose information for the data integration is obtained, re-searching the additional candidate dataset from among the candidate datasets based on the additional purpose information; updating the integrated dataset using the re-searched additional candidate dataset; and updating the presentation screen using the updated integrated dataset.

6. A data integration support method according to claim 1 or 2, wherein the search includes searching for an additional candidate dataset that matches the purpose by comparing features of the target information with features of candidate data that constitute the candidate dataset.

7. A data integration support method as claimed in claim 1 or 2, wherein the search includes: calculating a degree of match for each of a plurality of search conditions for each of a plurality of candidate data constituting the candidate dataset, and calculating a match score for the purpose for each of the plurality of candidate data by combining the degrees of match; and searching for a dataset composed of candidate data whose match score exceeds a threshold as the additional candidate dataset.

8. A data integration support method as described in claim 1 or 2, wherein the features include a plurality of type-specific features of different types, and further, when the presentation screen receives an instruction to select one type-specific feature from the plurality of type-specific features, it displays the features of the integrated data set for the one type-specific feature.

9. The data integration support method according to claim 7, wherein the existing dataset and the candidate dataset are each composed of image data, and the multiple type-specific features include at least two of vector features obtained by inputting the image data into a trained model, meta information of the image data, a brightness distribution of the image data, and annotation information.

10. The data integration support method according to claim 9, wherein the meta-information includes at least one of the size of the image data, the aspect ratio of the image data, and the acquisition date and time of the image data.

11. The data integration support method described in claim 9, wherein the annotation information includes, for each of the existing dataset and the candidate additional dataset, at least one of the following: the proportion in the dataset of a class to which an object indicated by an annotation assigned to the image data belongs, the proportion in the dataset of annotated image data, and an annotation error rate in the dataset.

12. The data integration support method according to claim 1 or 2, wherein the existing dataset and the additional dataset are each a dataset for machine learning.

13. The data integration support method according to claim 2, wherein the second distribution includes a plurality of distributions corresponding to a plurality of clusters obtained by clustering the additional candidate data constituting the additional candidate data set.

14. A data integration support device including a processor, wherein the processor acquires purpose information indicating a user's purpose for data integration, which is to add an additional dataset to an existing dataset owned by the user, searches for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information, generates a presentation screen that visualizes characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset, and outputs the presentation screen to a display.

15. A data integration support program that causes a computer to execute the following processes: acquire purpose information indicating a user's purpose for data integration, which is to add an additional dataset to an existing dataset owned by the user; search for an additional candidate dataset that matches the purpose from among candidate datasets based on the purpose information; generate a presentation screen that visualizes the characteristics of an integrated dataset in which the additional candidate dataset has been added to the existing dataset; and output the presentation screen to a display.

Images