Data creating apparatus, data creating method, program, and recording medium

Abstract

To create teacher data by selecting various kinds of image data according to the intention of a user from a huge amount of image data according to the purpose and application of machine learning.SOLUTION: In the data creating apparatus, the data creating method, the program, and the recording medium of the present invention, the first condition for selecting the first selected image data from the plurality of image data based on the incidental information is set, the first selected image data in which the incidental information conforming to the first condition is recorded is selected from the plurality of image data, and the second condition for selecting the second selected image data from the non-selected image data not conforming to the first condition among the plurality of image data based on the incidental information is set. Depending on whether or not the second condition is adopted, it is determined whether teacher data is created based on the first sorted image data or teacher data is created based on the first sorted image data and the second sorted image data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] One embodiment of the present invention relates to a data creation device, a data creation method, a program, and a recording medium for creating training data for machine learning in an artificial intelligence. [Background technology]

[0002] When using training data to train an AI to perform machine learning, annotation work is important to select appropriate training data according to the purpose and use of the machine learning (the purpose and use of the AI). However, selecting image data for creating appropriate training data from a huge amount of image data according to the purpose and use of the machine learning, and creating training data based on the selected image data, requires considerable effort and processing time, resulting in a surge in the cost of creating training data.

[0003] In response to this, in recent years, it has been proposed to automatically select image data from a huge amount of image data and create training data based on the selected image data (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-150381 [Patent Document 2] Japanese Patent Application Publication No. 2019-114243 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Documents 1 and 2, image data for creating training data is automatically selected from a huge amount of image data, which has the problem that the image data selected may not be in line with the user's intentions.

[0006] In response to this, a user can set selection conditions themselves to select image data that meets the user's intentions from a vast amount of image data. However, in this case, only image data that meets the selection conditions set by the user is selected, and image data that meets selection conditions not set by the user is not selected, which creates the problem that it is difficult to select a wide variety of image data from a vast amount of image data according to the purpose and use of machine learning.

[0007] Therefore, an object of one embodiment of the present invention is to provide a data creation device, a data creation method, a program, and a recording medium that can create training data by selecting a wide variety of image data that is in line with the user's intentions from a vast amount of image data, depending on the purpose and application of machine learning. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the present invention provides a data creation device that creates training data for machine learning from multiple image data on which additional information is recorded, the data creation device comprising a processor, the processor performing the following steps: a setting process that sets a first condition for selecting first selected image data from the multiple image data based on the additional information; a selection process that selects first selected image data from the multiple image data on which additional information that meets the first condition is recorded; a proposal process that proposes second conditions for selecting second selected image data from non-selected image data that does not meet the first condition based on the additional information among the multiple image data; and a creation process that creates training data based on the first selected image data if the user does not adopt the second condition, and creates training data based on the first selected image data and the second selected image data if the user adopts the second condition.

[0009] Here, when the user adopts the second condition, it is preferable that the processor executes a second selection process to select second selection image data from the non-selected image data, which has recorded therein supplementary information that meets the second condition.

[0010] Furthermore, it is preferable that the processor executes machine learning based on an adoption result as to whether or not the user adopted the second condition, and the suggestion process suggests the second condition based on the machine learning of the adoption result.

[0011] Furthermore, the processor preferably executes a notification process for notifying information relating to the second condition.

[0012] Preferably, the first condition and the second condition include an item related to the supplementary information and a content related to the item.

[0013] Furthermore, it is preferable that the first condition and the second condition have the same items but different contents.

[0014] Furthermore, the item is preferably information regarding whether or not the image data can be used as training data.

[0015] Preferably, the permission information includes at least one of user information regarding the use of the image data, restriction information regarding restrictions on the purpose of use of the image data, and copyright holder information of the image data.

[0016] Furthermore, it is preferable that the content of the first condition is to select image data based on the feasibility information, and the content of the second condition is to select image data for which the feasibility information is not recorded, or image data for which the feasibility information stating that there are no restrictions on the use of the image data is recorded.

[0017] Furthermore, the items are preferably items relating to the type of subject appearing in the image based on the image data.

[0018] Preferably, the first condition is a condition related to a subject appearing in an image based on image data, and the suggestion process is a process of suggesting the second condition based on characteristics of the subject of the first condition.

[0019] Furthermore, the proposing process is preferably a process of proposing a second condition that is a higher-level concept obtained by abstracting the first condition.

[0020] The present invention also provides a data creation method for creating training data for machine learning from multiple image data on which additional information is recorded, the data creation method including: a setting step for setting a first condition for selecting first selected image data from the multiple image data based on the additional information; a selection step for selecting first selected image data from the multiple image data on which additional information that meets the first condition is recorded; a proposal step for proposing second conditions for selecting second selected image data from non-selected image data that does not meet the first condition based on the additional information among the multiple image data; and a creation step for creating training data based on the first selected image data if the user does not adopt the second condition, and creating training data based on the first selected image data and the second selected image data if the user adopts the second condition.

[0021] The present invention also provides a program for causing a computer to execute each process of any of the above data creation devices.

[0022] The present invention also provides a computer-readable recording medium having recorded thereon a program for causing a computer to execute each process of any of the above-described data creation devices. [Effects of the Invention]

[0023] According to the present invention, it is possible to provide a data creation device, a data creation method, a program, and a recording medium that can create training data by selecting a wide variety of image data that is in line with the user's intentions from a huge amount of image data, depending on the purpose and application of machine learning. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a block diagram illustrating a configuration of a data processing system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram illustrating an internal configuration of the data creation device shown in FIG. 1 according to an embodiment. [Figure 3] FIG. 2 is a conceptual diagram illustrating an internal structure of image data according to an embodiment. [Figure 4] 10 is a conceptual diagram illustrating an embodiment of a selection process for selecting, from a plurality of image data, first selection image data in which incidental information that meets a first condition is recorded. [Figure 5] FIG. 10 is a conceptual diagram illustrating an embodiment of a second selection process for selecting, from non-selected image data, second selection image data in which incidental information that meets a second condition is recorded. [Figure 6] FIG. 2 is a conceptual diagram illustrating a configuration of supplementary information according to an embodiment. [Figure 7] FIG. 2 is a conceptual diagram illustrating a configuration of shooting condition information according to an embodiment. [Figure 8] FIG. 2 is a conceptual diagram illustrating a configuration of subject information according to an embodiment. [Figure 9] FIG. 2 is a conceptual diagram illustrating a configuration of image quality information according to an embodiment. [Figure 10] FIG. 10 is a conceptual diagram illustrating a configuration of availability information according to an embodiment. [Figure 11] FIG. 2 is a conceptual diagram illustrating a configuration of history information according to an embodiment. [Figure 12] 2 is a flowchart illustrating an embodiment of the operation of the data processing system shown in FIG. 1. [Figure 13] FIG. 10 is a conceptual diagram illustrating an embodiment of an input screen for a user to input selection conditions. [Figure 14] FIG. 10 is a conceptual diagram illustrating an embodiment of a presentation screen that proposes a second condition. DETAILED DESCRIPTION OF THE INVENTION

[0025] A data creation device, a data creation method, a program, and a recording medium according to one embodiment of the present invention will be described in detail below, based on a preferred embodiment shown in the accompanying drawings. However, the embodiment described below is merely an example given to facilitate understanding of the present invention, and the present invention is not limited thereto. In other words, the present invention may be modified or improved from the embodiment described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0026] Furthermore, in this specification, the concept of "device" includes not only a single device that performs a specific function, but also multiple devices that exist independently of each other but cooperate to perform a specific function. Furthermore, in this specification, the term "person" means an entity that performs a specific action, and this concept includes individuals, groups, corporations, and organizations, as well as computers and devices that constitute artificial intelligence.

[0027] Figure 1 is a block diagram showing the configuration of a data processing system according to one embodiment of the present invention. The data processing system 10 shown in Figure 1 includes a data creation device 12, a machine learning device 14, and multiple user terminal devices 16 (16a, 16b, ...). The data creation device 12, the machine learning device 14, and each of the multiple user terminal devices 16 are bidirectionally connected via a network 18 such as the Internet or a mobile data communication line, and are capable of sending and receiving various data to and from each other.

[0028] The data creation device 12 and the machine learning device 14 may be configured as separate devices as in this embodiment, or may be integrated into a single device. The data processing system 10 may include multiple user terminal devices 16 as in this embodiment, but it is not essential that the system include multiple user terminal devices 16; it is sufficient that the system includes at least one user terminal device 16.

[0029] The data creation device 12 performs annotation work to create training data for artificial intelligence to perform machine learning from multiple image data to which additional information has been recorded (added), and is composed of a computer such as a PC (Personal Computer), workstation, or server, and is equipped with an input device, a display, memory (storage device), communication device, control device, etc.

[0030] Artificial intelligence (AI) is the realization of intelligent functions such as inference, prediction, and judgment using hardware and software resources. AI is realized by any algorithm, such as an expert system, case-based reasoning (CBR), Bayesian network, or subsumption architecture. Machine learning is a technology that learns patterns and judgment criteria from data and uses them to predict and judge unknown things, as well as analytical techniques related to AI.

[0031] Fig. 2 is a block diagram of an embodiment showing the internal configuration of the data creation device shown in Fig. 1. As shown in Fig. 2, the data creation device 12 includes an acquisition processing unit 20, an image memory 22, a setting processing unit 24, a selection processing unit 26, a proposal processing unit 28, a notification processing unit 30, a second selection processing unit 32, and a creation processing unit 34.

[0032] Image data is input to an acquisition processing unit 20, which is connected to an image memory 22. A first condition is input to a setting processing unit 24, which is connected to a selection processing unit 26. A selection processing unit 26 and a second selection processing unit 32 are each connected to the image memory 22, and a creation processing unit 34 is connected to the selection processing unit 26 and the second selection processing unit 32. Training data is output from the creation processing unit 34. The adoption result of the second condition is input to the second selection processing unit 32 and the proposal processing unit 28, and the second condition is output from the proposal processing unit 28. A notification processing unit 30 is connected to the proposal processing unit 28, and a notification is output from the notification processing unit 30.

[0033] The acquisition processing unit 20 executes an acquisition process for acquiring a plurality of image data from at least one of a plurality of image data supply sources.

[0034] The source of the image data is not particularly limited, but for example, the acquisition processing unit 20 can acquire image data selected (specified) by the user on the user terminal device 16, image data posted on a website where images can be made public or shared, such as an SNS (Social Networking Service), image data stored in online storage or an image server, etc.

[0035] As shown in Fig. 3, each of the plurality of image data has recorded thereto additional information. The additional information includes various tag information (label information). For example, the additional information may be recorded as header information of the image data, or the additional information may be prepared as additional information data separate from the image data, and the image data and the additional information data corresponding to this image data may be recorded in association with each other. A detailed description of the additional information will be given later.

[0036] The image memory 22 stores a plurality of image data.

[0037] The image memory 22 may acquire a plurality of image data acquired by the acquisition processing unit 20, or a plurality of image data may be stored in the image memory 22 in advance. The image memory 22 is not particularly limited, but may be, for example, a variety of recording media such as a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a RAM (Random Access Memory), an SD card (Secure Digital card), or a USB memory (Universal Serial Bus memory), or may be an online storage or an image server.

[0038] The setting processing unit 24 executes a setting process for setting a first condition regarding the supplementary information.

[0039] The first condition is a selection condition for selecting (searching) first selected image data based on the supplementary information from among a plurality of image data stored in the image memory 22. The first condition will be described in detail later.

[0040] The method for setting the first condition is not particularly limited, but for example, the setting processing unit 24 can set a selection condition input by a user as the first condition. For example, when creating an artificial intelligence for the purpose (purpose of use) of estimating whether or not a subject in an image is an "apple," the user inputs the selection condition "apple" as the first condition. In this case, the setting processing unit 24 sets the selection condition "apple" input by the user as the first condition. The first condition may be one selection condition, or may be an AND condition or an OR condition of two or more selection conditions.

[0041] Alternatively, the setting processing unit 24 can prepare a table that stores, for each machine learning purpose and application, an application associated with a first condition corresponding to that application, and use this table to set the first condition associated with the machine learning purpose and application entered by the user. In this case, the user may manually enter the machine learning purpose and application as the selection condition, or may use a pull-down menu or the like to select a desired application from a list of machine learning purposes and applications stored in the table.

[0042] As shown in Figure 4, the selection processing unit 26 performs a selection process to select image data (first selected image data) from multiple image data, which has recorded therein supplementary information that meets the first condition set by the setting processing unit 24.

[0043] The method for selecting the first selected image data is not particularly limited, but for example, the selection processing unit 26 can compare the first condition with the incidental information recorded in each of the plurality of image data, and thereby select, from the plurality of image data, the first selected image data in which incidental information that matches the first condition is recorded. For example, if the first condition is "apple," the selection processing unit 26 selects the first selected image data in which incidental information corresponding to "apple" is recorded.

[0044] The incidental information that meets the first condition may include incidental information that completely matches the first condition, as well as incidental information that encompasses the first condition. For example, if the first condition is "apple," then in addition to incidental information corresponding to "apple," it may also include incidental information corresponding to "red apple," etc.

[0045] The proposal processing unit 28 executes a proposal process to propose a second condition regarding the ancillary information from among the multiple image data, non-selected image data that does not meet the first condition, i.e., non-selected image data that was not selected as the first selected image data.

[0046] The non-selected image data is image data other than the image data selected as the first selected image data among the plurality of image data, and includes one or more image data. The second condition is a selection condition different from the first condition, and is a selection condition for selecting second selection image data different from the first selection image data from non-selected image data based on incidental information. The second condition is a selection condition that is automatically set by the proposal processing unit 28 and proposed to the user regardless of instructions from the user. The second condition will be described in detail later.

[0047] The method for proposing the second condition is not particularly limited, but for example, the proposal processing unit 28 can prepare a table that stores, for each first condition, an association between the first condition and the second condition corresponding to this first condition, and use this table to propose the second condition associated with the first condition. For example, if the first condition is "apple," adding image data of peaches, which have a similar appearance to apples, can improve the accuracy of the estimation results by the artificial intelligence. Therefore, if "apple" and "peaches" are associated in the table, the setting processing unit 24 proposes "peaches" as the second condition.

[0048] The timing of proposing the second condition is not particularly limited, but may be, for example, between the proposal process that proposes the first condition and the selection process that selects the first selected image data, or between the selection process and the creation process that creates the training data described below.

[0049] Alternatively, the proposal processing unit 28 may propose a second condition estimated from the first condition using artificial intelligence for performing the proposal processing. As with the first condition, the second condition may be one selection condition, or an AND or OR condition of two or more selection conditions.

[0050] The notification processing unit 30 executes a notification process for notifying information relating to the second condition proposed by the proposal processing unit .

[0051] The information regarding the second condition is not particularly limited, but examples include the reason for proposing the second condition, the number of times or adoption rate that the same second condition has been adopted in the past, and the accuracy of the content (estimated result) of the second condition proposed by the artificial intelligence. For example, if the first condition is "apple" and the proposal processing unit 28 proposes "peaches" as the second condition, the notification processing unit 30 notifies the user of the reason for the proposal, such as "adding image data of peaches, which have a similar appearance to apples, will improve the accuracy of the estimation results by the artificial intelligence." By notifying the user of the reason for the proposal in this way, the user can know the reason for the proposal of the second condition, and can easily decide whether or not to adopt the second condition based on the reason for the proposal.

[0052] The notification method is not particularly limited, but for example, a text message may be displayed on the user terminal device 16, or the text message may be read aloud using voice synthesis, or both may be performed.

[0053] When the user adopts the second condition in response to the proposal of the second condition by the proposal processing unit 28, the second selection processing unit 32 executes a second selection process to select image data (second selected image data) having recorded therein incidental information that conforms to the second condition from among the non-selected image data, as shown in Fig. 5. In other words, when the user does not adopt the second condition, the second selection processing unit 32 does not execute the second selection process and does not select the second selected image data.

[0054] The second selection processing unit 32 can select the second selected image data from the non-selected image data in the same way as the selection processing unit 26 selects the first selected image data from a plurality of image data.

[0055] If the user does not adopt the second condition in response to the proposal of the second condition by the proposal processing unit 28, the creation processing unit 34 executes a creation process to create teacher data based on the first selected image data, since the second selected image data is not selected. On the other hand, if the user adopts the second condition, the creation processing unit 34 executes a creation process to create teacher data based on the first selected image data and the second selected image data, since the second selected image data is selected.

[0056] The creation processing unit 34 may use the first selected image data or the second selected image data itself as training data, or may create training data by performing various image processing on at least one of the first selected image data and the second selected image data.

[0057] In this embodiment, the acquisition processing unit 20, the setting processing unit 24, the selection processing unit 26, the proposal processing unit 28, the notification processing unit 30, the second selection processing unit 32 and the creation processing unit 34 are configured by a processor and a program executed by this processor.

[0058] The machine learning device 14 creates a machine-learned inference model by having the artificial intelligence perform machine learning using multiple pieces of training data created by the data creation device 12.

[0059] The inference model constructed by machine learning can be any mathematical model, such as a neural network, convolutional neural network, recurrent neural network, attention, transformer, generative adversarial network, deep learning neural network, Boltzmann machine, matrix factorization, factorization machine, m-way factorization machine, field-aware factorization machine, field-aware neural factorization machine, support vector machine, Bayesian network, decision tree, or random forest.

[0060] The user terminal device 16 causes the data creation device 12, the machine learning device 14, etc. to perform various processes in response to instructions input by the user. In this embodiment, in response to instructions input by the user, the user terminal device 16 causes the data creation device 12 to create training data according to the purpose and use of the machine learning, causes the machine learning device 14 to perform machine learning using the training data with an artificial intelligence to create a trained estimation model, and causes the trained estimation model to perform estimation according to the purpose and use of the machine learning.

[0061] The user terminal device 16 is configured by a computer such as a desktop PC, a notebook PC, a tablet PC, or a smartphone, and includes an input device, a display, a memory (storage device), a communication device, a control device, and the like.

[0062] Next, the supplementary information will be described.

[0063] The incidental information includes various tag information (label information) used to select first selected image data that meets a first condition from among a plurality of image data, and to select second selected image data that meets a second condition from among non-selected image data. The incidental information is not particularly limited, but includes at least one of shooting condition information, subject information, image quality information, acceptability information, history information, and usage information as tag information, as shown in FIG.

[0064] The shooting condition information is information about the shooting conditions of an image based on image data, and includes at least one of shooting device information, shooting environment information, and image processing information as tag information in Exif (Exchangeable Image File Format) format, as shown in Figure 7.

[0065] The photographic device information is information relating to the photographic device (camera), and includes information such as the manufacturer of the photographic device, the model name of the photographic device, and the type of light source the photographic device has. Shooting environment information is information about the shooting environment of the image, and includes information such as the shooting date and time, the season at the time of shooting, the shooting location, the name of the place where the image was taken, the exposure conditions at the time of shooting (f-number, ISO sensitivity, shutter speed, etc.), the weather at the time of shooting, and the illuminance (amount of sunlight) at the time of shooting. Image processing information is information about the image processing that the photographing device performs on the image, and includes information such as the name of the image processing, the characteristics of the image processing, the model of device that can perform the image processing, and the area within the image where the processing was performed.

[0066] The subject information is information about the subject appearing in the image based on the image data, and as shown in FIG. 8, includes at least one of identification information, position information, size information, etc. of the subject in the image.

[0067] The identification information is information about the type (classification), state, characteristics (color, shape, pattern, etc.) of the subject in the image. For example, information such as the type of subject being an "apple," its state being ripe, and its characteristics being red and round corresponds to the identification information. Position information is information relating to the position of a subject in an image, and includes, for example, information on a predetermined position of a rectangular area when the subject in the image is surrounded by a bounding box (for example, the coordinate position of one vertex in the rectangular area). The size information is information relating to the size of the area occupied by the subject in the image, and includes, for example, information on the coordinate positions of the two apex angles on the diagonal of the rectangular area.

[0068] The image quality information is information relating to the image quality of a subject captured in an image based on image data, and includes at least one of resolution information, brightness information, and noise information of the subject, as shown in FIG.

[0069] The perceived resolution information is information about the perceived resolution of a subject in an image, and includes, for example, information about the degree of blur and shake of the subject, the resolution of the subject, etc. The degree of blur and shake of the subject may be expressed in terms of the number of pixels, evaluated in stages such as a rank or grade of 1 to 5, evaluated in terms of a score, or may be the result of a sensory evaluation evaluated in stages on a scale based on human sensibility. Brightness information is information relating to the brightness (luminance value) of a subject in an image, and includes, for example, information such as the luminance value of each RGB (red, green, blue) color at each pixel in a rectangular area surrounding the subject. The noise information is information relating to noise of the subject in the image, and includes, for example, information on the S / N value (signal-to-noise ratio) within a rectangular area surrounding the subject.

[0070] The subject information and image quality information are assigned to each subject in the image. That is, if an image contains multiple subjects, the subject information and image quality information corresponding to each subject are assigned to each subject.

[0071] The permission information is information regarding the use of image data as training data, and includes at least one of user information, restriction information, copyright holder information, and the like, as shown in FIG.

[0072] User information is information about the user of the image data, and includes, for example, information that restricts the use of the image data to a specific user, such as "only available to Mr. A" or "only available to Company B," and information that there are no restrictions on the use of the image data, such as "anyone can use it." User information includes at least one of information about users who are authorized to use the image data and information about users who are not authorized to use it.

[0073] The restriction information is information regarding restrictions on the purpose of use of image data, and includes, for example, information restricting the purpose of use of image data, such as "commercial use is restricted," and information indicating that there are no restrictions on the purpose of use of image data, such as "can be used for any purpose."

[0074] The copyright holder information is information relating to the copyright holder of the image data, and includes, for example, information identifying the copyright holder of the image data, such as "The copyright holder is Company B," and information indicating that there is no copyright holder of the image data, such as "No copyright holder." Note that the copyright holder information is not limited to the copyright holder of the image data, but may also be information relating to the creator of the image data, such as ID (Identification), nickname, etc.

[0075] The permission information, i.e., the user information, the restriction information, and the copyright holder information, may each further include period information regarding the period during which the image data can be used. That is, the user information, the restriction information, and the copyright holder information may each include information regarding the restrictions on the period during which the image data can be used, such as the expiration date during which the image data can be used, the period during which the image data can be used free of charge or for a fee, etc.

[0076] It is desirable to ensure security of the availability information by encrypting or hashing it to prevent unauthorized tampering.

[0077] The history information is information about the learning history of past machine learning using image data, and includes at least one of count information, user information, correct tag information, incorrect tag information, adoption information, and accuracy information, as shown in Figure 11.

[0078] The number of times information is information about the number of times image data has been used to create training data in past machine learning. User information is information about users who used image data to create training data in past machine learning. The correct tag information and incorrect tag information are information indicating whether the training data created based on image data was used as correct data or incorrect data in past machine learning. The adoption information is information on whether or not training data created based on image data was adopted as incorrect answer data in past machine learning. The accuracy information is information about the accuracy of the estimation results obtained by artificial intelligence that has undergone machine learning using training data created based on image data in past machine learning.

[0079] The usage information is information about the learning purpose of machine learning (learning purpose of artificial intelligence), and more specifically, it is information that indicates what kind of artificial intelligence machine learning the training data created based on image data can be used for. Therefore, by referring to the usage information, it is possible to identify what kind of artificial intelligence machine learning the image data can be used to create training data for.

[0080] Of the incidental information, the shooting condition information, subject information, and image quality information can be assigned to the image data by automatically generating tag information using the photographing device that captured the image. All incidental information, i.e., shooting condition information, subject information, image quality information, availability information, history information, and usage information, may be assigned to the image data by the user manually inputting tag information into the user terminal device 16. Alternatively, tag information may be automatically estimated from the image data using artificial intelligence for assigning tag information, and the estimated tag information may be assigned to the image data.

[0081] Next, the first and second conditions will be explained.

[0082] Any selection conditions can be used as the first and second conditions depending on the purpose and application of the machine learning. When creating an artificial intelligence to estimate whether a subject in an image is an "apple," for example, "apple" can be set as the first condition, and for example, "peaches" can be suggested as the second condition, which is different from the first condition.

[0083] The first and second conditions may include an item related to the incidental information and a content related to the item. In other words, the first and second conditions may be AND conditions of two selection conditions, namely, the item and the content.

[0084] An item represents a category of a higher concept that encompasses multiple tag information of the same type, and content represents, for each category, the individual elements of the lower concept that belong to that category. For example, if the item is "fruit," its content is "apple," "peach," "tangerine," etc. If the item is "automobile," its content is "passenger car," "bus," "truck," etc. If the item is "seaweed," its content is "kelp," "wakame," "mozuku," etc.

[0085] As described above, the first condition and the second condition are defined by the item and the content, and as the first condition and the second condition, for example, conditions having the same item but different content can be used. The items may also include items related to the type of subject, as described above. The items may also include items related to the characteristics of the subject, the position and size of the subject in the image, etc. Furthermore, the items may include at least one of items related to image data acceptability information, items related to shooting conditions, items related to image quality, and items related to image data history information.

[0086] To explain with some specific examples, if the item and content of the first condition are set to "user information" and "only usable by Company B," the item and content of the second condition can be set to "user information" that is the same as the first condition and "anyone can use," which is different from the first condition. Similarly, if the item and content of the first condition are set to "fruit" and "apple," the item and content of the second condition can be set to "fruit" and "peaches." If the item and content of the first condition are set to an AND condition of "fruit" and "apple" with "weather" and "sunny," the item and content of the second condition can be set to an AND condition of "fruit" and "peaches" with "weather" and "cloudy." If the item and content of the first condition are set to "trees" and "trees," the item and content of the second condition can be set to "trees" and "forest." If the item and content of the first condition are set to "car" and "passenger car," the item and content of the second condition can be set to "car" and "bus."

[0087] In this way, by proposing the second condition, the selection of the second selected image data used to create the training data can be facilitated and the amount of training data can be increased, thereby improving the accuracy of the estimation results produced by artificial intelligence.

[0088] Among the above examples, examples such as "fruit," "tree," and "car" propose as second conditions selection conditions that are highly similar to the first conditions. In this way, by proposing second conditions that are highly similar to the first conditions, for example, training data that will be correct data can be created based on the first selected image data, training data that will be incorrect data can be created based on the second selected image data, and machine learning can be performed on the artificial intelligence using these training data, which makes it possible to correctly distinguish between similar objects, thereby improving the accuracy of the estimation results by the artificial intelligence.

[0089] The first and second conditions may be selection conditions with different items and the same content, or selection conditions with different items and content.

[0090] Also, the item may be propriety information, as in the example of "user information" described above. Furthermore, when the content of the first condition is to select image data based on propriety information, the content of the second condition may be to select image data for which propriety information is not recorded, or image data for which propriety information indicating that there are no restrictions on the use of the image data is recorded.

[0091] For example, if the first condition is set for the item "User Information" as "Available only to Company B," then the second condition can be proposed for the item "User Information." Similarly, if the first condition is set for the item "Restricted Information" as "Restrict commercial use," then the second condition can be proposed for the item "Restricted Information" as "Available for any purpose." Furthermore, if the first condition is set for the item "Copyright Holder Information" as "Copyright holder is Company B," then the second condition can be proposed for the item "Copyright Holder Information" as "No copyright holder."

[0092] In this way, by proposing a second condition for selecting image data for which no feasibility information is recorded, or image data for which feasibility information is recorded stating that there are no restrictions on the use of the image data, it is possible to promote the selection of second selected image data that is not restricted by feasibility information in addition to the first selected image data, and increase the number of selected image data used to create training data, thereby improving the accuracy of the estimation results produced by artificial intelligence.

[0093] The item may also be an item relating to the type of subject appearing in the image based on the image data.

[0094] For example, if a condition of "apple" is set for the item "fruit" as the first condition, a condition of "strawberry" can be proposed for the item "fruit" as the second condition. That is, the type of subject that is the item of the first condition and the second condition is "fruit," and its contents are "apple" and "strawberry." In this case, for example, the first selected image data is used to create training data for correct data, and the second selected image data is used to create training data for incorrect data.

[0095] Furthermore, as described above, if the item is an item related to the type of subject in the image and the content includes characteristics of the subject, the proposal processing unit 28 may propose characteristics different from the characteristics of the subject in the first condition as the content of the second condition.

[0096] For example, if the first condition is set as "apples of variety B produced in prefecture A" for the item "fruit," then the second condition can be set as "apples of variety D produced in prefecture C." In other words, the type of subject, which is the item of the first and second conditions, is "fruit," the content is "apple," and the characteristics of the subject are "place of origin" and "variety." This makes it possible to prevent bias in data due to the characteristics of the subject when selecting selected image data for creating training data, and also to increase the number of selected image data.

[0097] In addition, when the first condition is a condition related to a subject appearing in an image based on image data, the proposal processing unit 28 may perform a proposal process to propose a second condition based on the characteristics of the subject of the first condition, such as color, shape, pattern, etc.

[0098] For example, if the first condition is "tangerine," the second condition can be "an oval, orange object" based on the characteristics of "tangerine." That is, the subject of the first condition is "tangerine," and its characteristics are "oval" and "orange." In this case, the second sorted image data includes image data of "tangerines" that does not have tag information for "tangerines" recorded, and image data that is not a "tangerine" but has characteristics similar to a "tangerine," such as an orange ball. This makes it possible to create training data that serves as correct answer data based on image data of "tangerines" that does not have tag information for "tangerines," and to create training data that serves as incorrect answer data based on image data that is not a "tangerine" but has characteristics similar to a "tangerine." In this case, for example, a human would look at an image based on image data that has tag information for an "oval, orange object" that resembles a "tangerine" and determine that it is a "tangerine" (correct answer data) or not a "tangerine" (incorrect answer data).

[0099] Furthermore, the proposal processing unit 28 may perform a proposal process in which a second condition, which is a higher-level concept obtained by abstracting the first condition, is proposed.

[0100] For example, if "kelp" is set as the first condition, the proposal processing unit 28 can propose "seaweed," which is a higher-level concept of "kelp," as the second condition.

[0101] In this case, image data with tag information for "kelp" recorded is selected as the first selected image data selected based on the first condition, "kelp," but image data with tag information for "wakame" or "mozuku" recorded is not selected.

[0102] In response to this, by proposing the second condition "seaweed," which is a higher concept than the first condition "kelp," differences in language, food culture, etc. can be compensated for. To explain in more detail, people in countries where "seaweed" is eaten often use the words "kelp," "wakame," and "mozuku" differently, while people in countries where "seaweed" is not eaten often use the words "kelp," "wakame," and "mozuku" collectively as "seaweed." Taking this into consideration, by setting the second condition to "seaweed," it is possible to select image data that has tag information such as "seaweed" and "wakame" or "seaweed" and "mozuku" recorded from multiple image data, even if the tag information for "kelp" is not recorded. This makes it possible to select a larger number of selected image data related to "kelp."

[0103] As another example, if "shijimi" (freshwater clam), "asari" (clam), "hamaguri" (clam), etc. are set as the first condition, the proposal processing unit 28 can propose "shellfish" (shellfish), which is a superordinate concept of these, as the second condition. The same applies to other examples.

[0104] Next, the operation of the data processing system 10 will be described with reference to the flowchart shown in FIG.

[0105] First, the acquisition processing unit 20 executes an acquisition process (acquisition step) to acquire a plurality of image data from at least one of a plurality of image data supply sources (step S1). The image data acquired by the acquisition processing unit 20 is stored in the image memory 22.

[0106] On the other hand, the user inputs selection conditions for selecting image data according to the purpose and application of machine learning, for example, to the user terminal device 16. The selection conditions input by the user are transmitted from the user terminal device 16 to the data creation device 12.

[0107] In response to this, the setting processing unit 24 executes a setting process (setting step) for setting a first condition related to the additional information (step S2).

[0108] Next, the selection processing unit 26 executes a selection process (selection step) to select first selected image data from the multiple image data stored in the image memory 22, the first selected image data having accompanying information recorded thereon that meets the first condition set by the setting processing unit 24 (step S3).

[0109] Next, the proposal processing unit 28 executes a proposal process (proposal step) to propose a second condition related to the incidental information (step S4). Also, the notification processing unit 30 executes a notification process (notification step) to notify information related to the second condition proposed by the proposal processing unit 28 (step S5).

[0110] As a result, if the user does not adopt the second conditions proposed by the proposal processing unit 28 (No in step S6), the second selection processing (second selection step) by the second selection processing unit 32 is not executed. That is, the second selected image data is not selected. In this case, the creation processing unit 34 executes a creation process (creation step) for creating teacher data based on the first selected image data (step S7).

[0111] On the other hand, if the user adopts the second condition (Yes in step S6), the second selection processing unit 32 executes a second selection process (second selection step) to select second selection image data having recorded thereon supplementary information that meets the second condition from among the non-selected image data (step S8). In this case, the creation processing unit 34 executes a creation process (creation step) to create teacher data based on the first selected image data and the second selected image data (step S9). This teacher data is transmitted from the data creation device 12 to the machine learning device 14.

[0112] If the user does not adopt the second condition, the proposal processing unit 28 may repeatedly execute the proposal process (proposal step) to propose the second condition in response to an instruction from the user.

[0113] Next, in the machine learning device 14, the artificial intelligence is trained by machine learning using the training data transmitted from the data creation device 12, and a machine-learned inference model is created (step S10).

[0114] Next, the user inputs image data of the estimation target to cause this artificial intelligence to perform estimation corresponding to the intended use, into the user terminal device 16. An instruction to input the image data of the estimation target is transmitted from the user terminal device 16 to the machine learning device 14.

[0115] In response to an instruction for image data to be estimated input by a user, the image data to be estimated sent from the user terminal device 16 is input to the artificial intelligence in the machine learning device 14, and the artificial intelligence uses the trained estimation model to perform estimation on the image data to be estimated according to the purpose and use of the machine learning. The estimation result by the artificial intelligence is sent from the machine learning device 14 to the user terminal device 16.

[0116] Next, in the user terminal device 16, various processes are performed using the results of the estimation by the artificial intelligence transmitted from the machine learning device 14.

[0117] As a concrete example of the above-mentioned series of steps, we will explain the case of creating artificial intelligence for the purpose of estimating whether or not the subject in an image is a mandarin orange, in other words, the case of having the artificial intelligence learn mandarin oranges by machine learning.

[0118] As described above, the acquisition processing unit 20 executes an acquisition process (acquisition step) for acquiring multiple pieces of image data. Meanwhile, the user inputs, into the user terminal device 16, selection conditions for selecting image data to be used for machine learning the "tangerine" into the artificial intelligence.

[0119] In this case, as shown in Fig. 13, an input screen for the user to input selection conditions is displayed on the display of user terminal device 16. In the example shown in Fig. 13, the input screen for selection conditions displays a message at the top saying "Please input image data selection conditions," and below this message, input fields for inputting the type of subject, whether or not commercial use is permitted, user information, etc. are displayed in sequence.

[0120] For example, the user inputs selection conditions on a selection condition input screen, as shown in Figure 13, for selecting image data whose subject type is "tangerine" and which is available for commercial use and can only be used by Company B.

[0121] In response to this, the setting processing unit 24 executes a setting process (setting step) to set the first condition, and the selection processing unit 26 executes a selection process (selection step) to select, from among multiple image data, first selection image data having recorded thereon supplementary information that meets the first condition.

[0122] Next, the proposal processing unit 28 executes a proposal process (proposal step) for proposing the second condition, and the notification processing unit 30 executes a notification process (notification step) for notifying information related to the second condition.

[0123] In this case, as shown in Fig. 14, a proposal screen proposing second conditions different from the first conditions is displayed on the display of the user terminal device 16. In the example shown in Fig. 14, the proposal screen for the second conditions sequentially displays the selection conditions proposed as the second conditions, the reason for proposing the second conditions, and an input field for whether or not to adopt the second conditions.

[0124] In the example shown in Figure 14, the second condition displays a selection condition for selecting image data in which the type of subject is also "tangerine," the subject "tangerine" is located in the center of the image, and the image is "copyright-free." The reason for suggesting the second condition displays a message saying, "You can increase the amount of training data used in machine learning." Furthermore, in the input field for whether or not to adopt the second condition, a message saying, "Do you want to adopt this selection condition?" is displayed, with "Yes" and "No" buttons displayed below.

[0125] 14, the proposal processing unit 28 may propose, as the second condition, a selection condition for selecting image data in which the type of subject is a "persimmon," the subject "persimmon" is located outside the center of the image, and the image data has "no copyright holder." In this case, as the reason for proposing the second condition, for example, a message such as "You will be able to correctly distinguish between similar objects" is displayed.

[0126] In the input field for whether or not to adopt the second condition, the user presses the "Yes" button if the proposal is to be adopted, or presses the "No" button if the proposal is not to be adopted.

[0127] As a result, if the user presses the "No" button and does not adopt the second condition, the creation processing unit 34 creates teacher data based on the first selected image data. On the other hand, if the user presses the "Yes" button to adopt the second condition, the second selection processing unit 32 selects second selected image data from the non-selected image data, which has additional information recorded that meets the second condition, and the creation processing unit 34 creates teacher data based on the first selected image data and the second selected image data.

[0128] In this example, the first selected image data and the second selected image data, which have tag information for "mandarin oranges" recorded as supplementary information, are used to create teacher data that will serve as correct answer data for the AI ​​to learn "mandarin oranges" by machine learning. On the other hand, the second selected image data, which has tag information for "persimmons" recorded as supplementary information, is used to create teacher data that will serve as incorrect answer data for the AI ​​to learn by machine learning that "persimmons" are not "mandarin oranges."

[0129] The subsequent operations are as described above.

[0130] As a result, the data creation device 12 can select a wide variety of image data that meets the user's intentions from a huge amount of image data, depending on the purpose and application of machine learning. Then, appropriate training data can be automatically created in a short time based on the wide variety of image data selected from a huge amount of image data, which can significantly reduce the cost of creating training data and significantly improve the accuracy of the estimation results obtained by artificial intelligence.

[0131] Note that proposal processing unit 28 may cause an artificial intelligence for performing proposal processing to perform machine learning based on whether or not the user adopted the second condition, i.e., based on the adoption result of the second condition, and propose the second condition based on the machine learning of the adoption result of the second condition. In this case, the first condition to be estimated is input to the artificial intelligence, and the artificial intelligence estimates the second condition from the first condition using a trained estimation model.

[0132] When proposing second conditions, it is considered that second conditions that the user has adopted in the past are more likely to be adopted by the user than second conditions that the user has not adopted in the past. Therefore, the proposal processing unit 28 preferentially proposes second conditions that the user has adopted in the past over second conditions that the user has not adopted in the past. Furthermore, the proposal processing unit 28 may preferentially propose second conditions that the user has adopted more frequently in the past over second conditions that the user has adopted less frequently in the past. Furthermore, it is not necessary to propose second conditions that the user has not adopted in the past.

[0133] Based on machine learning of the results of adopting the second condition, for example, by repeatedly suggesting the second condition that has been adopted most frequently based on the number of times the user has adopted the second condition in the past, the likelihood that the user will adopt the second condition can be gradually increased.

[0134] In this case, the users may be the same or different users, and the user may be one user or multiple users. For example, the proposal processing unit 28 can store a history of information regarding whether or not the user has adopted the second condition and information regarding the number of times the user has adopted the second condition in association with a first condition corresponding to this second condition, and acquire the history of information regarding whether or not the user has adopted the second condition and information regarding the number of times the user has adopted the second condition, which is stored in association with the first condition.

[0135] Furthermore, the proposal processing unit 28 may cause the artificial intelligence for performing the proposal processing to perform machine learning based on the accuracy of the estimation result by the artificial intelligence, and propose the second condition based on the machine learning of the estimation result.

[0136] When performing machine learning on an artificial intelligence, if the accuracy of the estimation results by the first artificial intelligence when a first user has previously adopted a second condition is higher than the accuracy of the estimation results by the second artificial intelligence when a second user has not previously adopted the same second condition, it is considered that the accuracy of the estimation results by the artificial intelligence can be improved when this second condition is adopted rather than when this second condition is not adopted.

[0137] Therefore, the proposal processing unit 28 proposes the second conditions for the first artificial intelligence that the first user previously adopted if the accuracy of the estimation result by the first artificial intelligence when the first user adopts the second conditions is higher than the accuracy of the estimation result by the second artificial intelligence when the second user does not adopt these second conditions. In other words, the proposal processing unit 28 preferentially proposes second conditions that the user previously adopted and that improved the accuracy of the estimation result by the artificial intelligence, over second conditions that the user previously adopted and that reduced the accuracy of the estimation result by the artificial intelligence. Furthermore, it is not necessary to propose second conditions that the user previously adopted and that reduced the accuracy of the estimation result by the artificial intelligence.

[0138] Based on the history of accuracy of the inference results made by the artificial intelligence, by repeatedly proposing second conditions that the user has adopted in the past and which have improved the accuracy of the inference results made by the artificial intelligence, the accuracy of the inference results made by the artificial intelligence can be gradually improved.

[0139] In this case, the first user and the second user may be the same user or different users, and the first user and the second user may be one user or multiple users. The proposal processing unit 28 may, for example, store the history of the accuracy of the estimation results by the artificial intelligence in association with a second condition for the artificial intelligence, and acquire the history of the accuracy of the estimation results by the artificial intelligence associated with the second condition.

[0140] In the device of the present invention, the hardware configuration of the processing units that perform various processes such as the acquisition processing unit 20, setting processing unit 24, selection processing unit 26, proposal processing unit 28, notification processing unit 30, second selection processing unit 32 and creation processing unit 34 may be dedicated hardware or various processors or computers that execute programs.

[0141] Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute software (programs) and function as various processing units, programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacture, such as FPGAs (Field Programmable Gate Arrays), and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically for performing specific processes.

[0142] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Also, multiple processing units may be configured with one of the various processors, or two or more of the multiple processing units may be combined into one processor.

[0143] For example, as typified by server and client computers, one processor is configured by combining one or more CPUs and software, and this processor functions as multiple processing units. Another form is the use of a processor that realizes the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by system-on-chip (SoC).

[0144] Furthermore, the hardware configuration of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0145] The method of the present invention can be implemented by, for example, a program that causes a computer to execute each step. Also, a computer-readable recording medium on which this program is recorded can be provided. [Explanation of symbols]

[0146] 10 Data Processing System 12 Data creation device 14 Machine Learning Device 16 User terminal equipment 18 Network 20 Acquisition processing unit 22 Image Memory 24 Setting processing section 26 Sorting processing section 28 Proposal Processing Unit 30 Notification processing section 32 Second sorting processing section 34 Creation processing section

Claims

1. A data creation device that creates training data for machine learning from a plurality of image data to which supplementary information is recorded, a processor; The processor: a setting process for setting a first condition for selecting first selection image data from the plurality of image data based on the supplementary information; a selection process for selecting, from the plurality of image data, the first selection image data in which incidental information that meets the first condition is recorded; a setting process for setting a second condition for selecting second selection image data from non-selection image data that does not conform to the first condition based on the supplementary information among the plurality of image data; A data creation device that executes a process of determining whether to create the teacher data based on the first selected image data or based on the first selected image data and the second selected image data, depending on whether the second condition is adopted or not.

2. The data creation device described in claim 1, wherein the processor, when the user adopts the second condition, executes a second selection process to select the second selected image data from the non-selected image data, which has recorded thereon additional information that meets the second condition.

3. the processor executes machine learning based on an adoption result of whether or not the user adopted the second condition; The data creation device according to claim 1 , wherein the setting process proposes the second condition based on the machine learning of the adoption results.

4. The data creation device according to claim 1 , wherein the processor executes a notification process for notifying information relating to the second condition.

5. 3. The data creation device according to claim 1, wherein the first condition and the second condition include an item related to the additional information and a content related to the item.

6. The data creation device according to claim 5 , wherein the first condition and the second condition have the same item but different contents.

7. The data creation device according to claim 6 , wherein the item is information on whether or not the image data can be used as the training data.

8. 8. The data creating device according to claim 7, wherein the permission information includes at least one of user information regarding use of the image data, restriction information regarding restrictions on the purpose of use of the image data, and copyright information of the image data.

9. the first condition is a condition for selecting image data based on the availability information, The data creation device according to claim 7, wherein the second condition is to select image data for which the availability information is not recorded or image data for which the availability information is recorded indicating that there are no restrictions on the use of the image data.

10. 7. The data creation device according to claim 6, wherein the item is an item relating to the type of subject appearing in an image based on the image data.

11. the first condition is a condition related to a subject appearing in an image based on image data, 3. The data creation device according to claim 1, wherein the setting process is a process of setting the second condition based on a feature of the subject of the first condition.

12. 3. The data creation device according to claim 1, wherein the setting process is a process of setting the second condition, which is a superordinate concept obtained by abstracting the first condition.

13. A data creation method for creating training data for machine learning from a plurality of image data to which supplementary information is recorded, comprising the steps of: a setting step in which a processor sets a first condition for selecting first selection image data from the plurality of image data based on the supplementary information; a selection step in which a processor selects, from the plurality of image data, the first selection image data in which incidental information that meets the first condition is recorded; a setting step in which a processor sets a second condition for selecting second selection image data from non-selection image data that does not meet the first condition based on the supplementary information, among the plurality of image data; A data creation method including a step in which a processor decides whether to create the teacher data based on the first selected image data or based on the first selected image data and the second selected image data, depending on whether the second condition is adopted or not.

14. 3. A program for causing a computer to execute each process of the data creation device according to claim 1 or 2.

15. 3. A computer-readable recording medium having recorded thereon a program for causing a computer to execute each process of the data creation device according to claim 1 or 2.

Images