Information processing apparatus, information processing method, and storage medium

Abstract

An information processing apparatus, an information processing method, and a storage medium are disclosed. A candidate data determination unit acquires an estimation result of a score that indicates a likelihood of a label being added as an annotation to target data. A label candidate output unit receives a designation of a candidate label from a user. The candidate data determination unit determines candidate data from among a plurality of labeled data included in a feature space, the candidate data indicating a plurality of labeled data distributed in each of quadrants in which the feature space is divided, wherein the label is added as an annotation to the labeled data, and wherein the feature space is defined using the score that indicates the likelihood of the label being added as an annotation as an axis. The candidate data determination unit determines the candidate data for each of the plurality of quadrants based on the labeled data included in each of the plurality of quadrants.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a storage medium. BACKGROUND

[0002] As a program for creating training data to be used for machine learning, an annotation tool for adding correct answer information (correct answer label) to be learned is used. Some annotation tools have a functional group (i.e., a functional group for assisting a user) for reducing the workload of a user, for example, to create training data.

[0003] For example, in order to create a machine learning model with higher performance, it can be desirable to add more accurate correct answer information to training data and use a larger amount of training data. Therefore, some annotation tools have a function for assisting a user to more efficiently add more accurate correct answer information to target data for training purposes. For example, Japanese Patent Application Publication No. 2019-114018 discusses an example of a function that assists a user in determining which correct answer information to add to target data by classifying data to which correct answer information has been added based on the correct answer information and prompting the classified data as reference information. SUMMARY

[0004] As the number of data to which correct answer information has been added increases, the number of candidate data to be used to prompt reference information also increases. In this case, it can become very difficult for a user to examine all the data. In addition, even if some data is extracted at random for prompting reference information, the reference information prompted based on the extraction result can not always be used as a reference when a user determines which correct answer information to add to target data.

[0005] Embodiments of the present disclosure assist a user in a more desirable pattern of labels to be added as annotations.

[0006] According to embodiments of the present disclosure, an information processing apparatus includes an acquisition unit configured to acquire an estimation result of a score indicating a likelihood of a label being added as an annotation to target data; a candidate label reception unit configured to receive a designation of a candidate label, the candidate label being a candidate for the label to be added as an annotation to the target data; a determination unit configured to determine, from a plurality of labeled data included in a feature space, candidate data indicating a plurality of labeled data distributed in each quadrant in which the feature space is divided, wherein the label is added as an annotation to the labeled data, and wherein the feature space is defined using the score indicating the likelihood of the label being added as an annotation as an axis; and a candidate data output unit configured to output the candidate data determined for each quadrant to a predetermined output destination.

[0007] Other features of embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to one embodiment.

[0009] Figure 2 is a block diagram showing an example of a functional configuration of an information processing apparatus according to one embodiment.

[0010] Figure 3 is a table showing an example of a data structure for managing label data according to one embodiment.

[0011] Figure 4 is a table showing an example of a data structure for managing score estimation results according to one embodiment.

[0012] Figure 5 is a flowchart showing an example of processing to be executed by an information processing apparatus according to one embodiment.

[0013] Figure 6 shows an example of an operation screen for receiving designation of a label candidate according to one embodiment.

[0014] Figure 7 shows an example of an output screen for prompting information based on candidate data according to one embodiment.

[0015] Figure 8 is a flowchart showing an example of processing to be executed by an information processing apparatus according to one embodiment.

[0016] Figures 9A-9F are diagrams each showing an example of processing for dividing a region of point cloud data into a plurality of quadrants according to one embodiment.

[0017] Figure 10 is a flowchart showing an example of processing to be executed by an information processing apparatus according to one embodiment.

[0018] Figure 11 is a block diagram showing another example of a functional configuration of an information processing apparatus according to one embodiment.

[0019] Figure 12 is a flowchart showing another example of processing to be executed by an information processing apparatus according to one embodiment.

[0020] Figure 13 is a flowchart showing another example of processing to be executed by an information processing apparatus according to one embodiment.

[0021] Figure 14 FIG. 8 is a block diagram illustrating another example of a functional configuration of an information processing apparatus according to an embodiment.

[0022] Figure 15 FIG. 9 is a flowchart illustrating another example of a process to be executed by an information processing apparatus according to an embodiment.

[0023] Figure 16 FIG. 10 illustrates an example of a dialog box for prompting notification information according to an embodiment. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings. The same reference numbers are used throughout the drawings and the specification to refer to the same or like parts. The embodiments are not limiting of the present disclosure.

[0025] SUMMARY OF THE ANNOUNCEMENT

[0026] Supervised learning is an example of a technique for training a machine learning model based on so-called machine learning, that is, for building a machine learning model. In supervised learning, a data set including training data in which data to be input to a learning model is associated with a correct answer label to be predicted based on the data is used to build a machine learning model. In the case of building a machine learning model, for example, if such a data set does not exist or is incomplete, the data set is built by an annotation operation of adding a correct answer label as an annotation to the data after collecting the data to be input. In some cases, an annotation tool including a function for assisting a user in performing an operation of adding a correct answer label to data is used, so that the user can more easily perform the annotation operation.

[0027] The annotation tool prompts a user with data (hereinafter also referred to as target data) on an image, a document, or the like to be annotated, and receives a specification of a correct answer label to be added to the target data as an annotation from the user. Then, the annotation tool adds the correct answer label specified by the user to the target data, thereby generating training data to be included in a data set.

[0028] Among various types of annotation tools, there is a tool that uses a machine learning model (hereinafter also referred to as a "trained model") built based on preliminary machine learning to efficiently perform a labeling operation to add a correct answer label to target data as described above. As a special example, a tool using a trained model causes the trained model to analyze target data to extract a label candidate to be added to the target data as an annotation, and prompts the user with the extracted label candidate. This enables the user to select a candidate to be added to the target data as a correct answer label from the label candidate prompted by the annotation tool.

[0029] To create higher-performance machine learning models, it may be desirable to add more accurate correct answer information to the training data and use a larger amount of training data. Therefore, some annotation tools have user-assistive features that allow users to more effectively add more accurate correct answer information to the target data used for training purposes.

[0030] As an example of such user assistance functionality provided by the aforementioned annotation tools, a function is proposed that categorizes data with added correct answer information based on the correct answer information and uses this as reference information to prompt the categorized data.

[0031] On the other hand, as the amount of data with correct answer information increases, the number of candidate data for use as hints also increases. In this situation, it can become very difficult for users to check all the data. Furthermore, even if some data is randomly extracted for hints, the conditions used for extraction may not always be appropriate. Therefore, when a user determines which correct answer information to add to the target data, the hints based on the extraction results may not always be used as a reference.

[0032] Therefore, this disclosure proposes a technique related to an annotation tool that can assist users in selecting tags to be added to target data as annotations in a more ideal manner.

[0033] In the following text, the existing trained machine learning model is also referred to as the pre-trained model, and the candidate labels to be selected as the correct answer information to be added to the data to be annotated are also referred to as candidate labels. The data to be annotated is also referred to as the target data, and the data already annotated with correct answer information is also referred to as labeled data.

[0034] Reference Figure 1 Examples of hardware configurations of an information processing apparatus 100 according to exemplary embodiments of the present disclosure are described. For example... Figure 1 As shown, the information processing apparatus 100 according to this exemplary embodiment includes a central processing unit (CPU) 111, a read-only memory (ROM) 112, and a random access memory (RAM) 113. The information processing apparatus 100 also includes an auxiliary storage device 114, an output device 115, an input device 116, and a communication interface (I / F) 117. The CPU 111, ROM 112, RAM 113, auxiliary storage device 114, output device 115, input device 116, and communication I / F 117 are interconnected via a bus 118.

[0035] The CPU 111 is a central processing unit that controls various operations of the information processing apparatus 100. For example, the CPU 111 can control operations of the entire information processing apparatus 100. The ROM 112 stores control programs, a boot program, and the like that can be executed by the CPU 111. The RAM 113 is a main memory of the CPU 111 and functions as a work area or a temporary storage area for loading various programs.

[0036] The auxiliary storage device 114 stores various data and various programs. The auxiliary storage device 114 is realized by a storage device capable of temporarily or persistently storing various data, such as a nonvolatile memory typified by a hard disk drive (HDD) and a solid state drive (SSD).

[0037] The output device 115 is a device that outputs various information and is used to present various information to a user. In the present exemplary embodiment, the output device 115 is realized by a display device such as a display. The output device 115 displays various display information, thereby presenting information to the user. In another example, the output device 115 can be realized by a sound output device that outputs a sound such as a voice or an electronic sound. In this case, the output device 115 outputs a sound such as a voice or an electronic sound, thereby presenting information to the user. The device used as the output device 115 can be appropriately changed depending on a medium used to present information to the user.

[0038] The input device 116 is used to receive various instructions from a user. In the present exemplary embodiment, the input device 116 includes input devices such as a mouse, a keyboard, and a touch panel. In another example, the input device 116 can include a sound collecting device, for example, a microphone, to collect a voice spoken by a user. In this case, various analysis processing such as acoustic analysis and natural language processing is performed on the collected voice, thereby it is possible to recognize the content of the voice instruction as an instruction from the user. The device used as the input device 116 can be appropriately changed depending on a method of recognizing an instruction from a user. A plurality of types of devices can be used as the input device 116.

[0039] The communication I / F 117 is used for communication with an external apparatus via a network. The device used as the communication I / F 117 can be appropriately changed depending on a type of a communication path or a communication method to be applied.

[0040] The CPU 111 loads a program stored in the ROM 112 or the auxiliary storage device 114 into the RAM 113 and executes the program, thereby realizing the functional configuration shown in Figure 2 、 Figure 11 and Figure 14 . The functions shown in Figure 5 、 Figure 8 、 Figure 10 are realized by executing the program.Figure 12 , Figure 13 and Figure 15 the processes illustrated in the flowcharts of

[0041] As a storage medium for providing a program, for example, a floppy disk, a hard disk, an optical disk, and a magneto-optical disk can be used. Other examples of the storage medium can include a compact disc (CD)-ROM, a recordable CD (R), a magnetic tape, a non-volatile memory card, a ROM, and a digital versatile disc (DVD).

[0042] The program can be directly executed by the computer, or can be executed under the management of a basic software such as an operating system (OS) running on the computer.

[0043] The program read from the storage medium can be processed by a function expansion board installed on the computer, a function expansion unit connected to the computer, or the like.

[0044] A first example embodiment of the present disclosure will be described below. In order to distinguish the information processing apparatus according to the first example embodiment from the information processing apparatuses according to other example embodiments, hereinafter, for the convenience of explanation, the information processing apparatus according to the first example embodiment will be referred to as an information processing apparatus 200.

[0045] In the present example embodiment and the other example embodiments described below, for the convenience of explanation of the technical features of the present disclosure, the description can be made by focusing on a case where image data is used as data to which correct answer information (a label) is added as an annotation. However, the image data is merely an example and is not intended to limit the type of data to which correct answer information is added by the information processing apparatus to which the technology according to the present disclosure is applied. In other words, the type of data to which correct answer information is added can be appropriately changed within a range that does not depart from the basic idea of the technology according to the present disclosure.

[0046] (Functional Configuration)

[0047] An example of the functional configuration of the information processing apparatus 200 according to the present example embodiment will be described with reference to Figure 2 The information processing apparatus 200 includes a pre-trained model reading unit 201, a labeled data management unit 202, a score estimation unit 203, an estimation result management unit 204, a candidate data determination unit 206, and a candidate data extraction unit 207. The information processing apparatus 200 further includes a label candidate input unit 205, a candidate data output unit 208, and a candidate data input unit 209.

[0048] The pre-trained model reading unit 201 reads a learning model (pre-trained model) constructed by previously training a model. The unit that reads the pre-trained model is not particularly limited as long as it can be referred to by the pre-trained model reading unit 201. The pre-trained model reading unit 201 can load the read pre-trained model into a memory of the information processing apparatus 200.

[0049] The marker data management unit 202 manages marker data (for example, marker image data). For example, the marker data management unit 202 can store a plurality of pieces of marker data in a manner that each piece of marker data can be read separately. The marker data management unit 202 can manage the marker data using a database or the like.

[0050] For example, Figure 3 An example of a data structure for managing marker data is shown. In Figure 3 In the example shown, an identification (ID) is assigned to each piece of marker data, and information about a label added to the marker data and information about a path to the marker data are associated with the ID. As a specific example, "Label A" is added as a label to marker data to which "1" is assigned as an ID, and the marker data is stored at a location indicated by "AAA\BBB\CCC".

[0051] In Figure 3 In the example shown, the marker data is managed as a file. However, the method for managing the marker data is not particularly limited as long as data can be managed. As a specific example, the marker data management unit 202 can manage the marker data as Base64 format data. In this case, the marker data management unit 202 can convert the file of the marker data into Base64 format data as needed.

[0052] The score estimation unit 203 estimates, based on the read pre-trained model, correct answer information (label) that can be added to target data to be annotated and a score that represents a likelihood that the target data is data indicated by the correct answer information. Specifically, the score estimation unit 203 can use the pre-trained model read by the pre-trained model reading unit 201 to estimate a label that indicates the target data, and estimate a score that represents a likelihood that the target data is data indicated by the label. For example, the score is expressed in the form of a probability that the target data is indicated by the label (that is, a probability that the target data is data indicated by the label).

[0053] The estimation result management unit 204 manages correct answer information (label) that can be added to target data and a score for the correct answer information estimated by the score estimation unit 203.

[0054] For example, Figure 4An example of a data structure for managing estimation results, which are estimation results for each target data regarding labels that can be added to the target data and scores of the labels, is shown. In Figure 4 In the example shown, information corresponding to the score estimation result of each label is associated with the ID assigned to each piece of mark data. In Figure 4 In the example shown, the score estimation result for each of "Label A", "Label B", "Label C", and "Label D" is associated with the ID. As a specific example, mark data to which "1" is assigned as the ID indicates that the score of "Label A" is "0.7", the score of "Label B" is "0.4", the score of "Label C" is "0.2", and the score of "Label D" is "0.1".

[0055] The label candidate input unit 205 receives a designation of a candidate (hereinafter also referred to as a candidate label) of a label to be added as an annotation to target data from a user who performs an annotation operation. As a specific example, in a case where the user wants to know which label to add to target data, the label candidate input unit 205 functions as an interface for inputting a label candidate to the information processing apparatus 200.

[0056] The candidate data determination unit 206 determines mark data for prompting reference information to a user who performs an annotation operation as candidate data. Specifically, the candidate data determination unit 206 determines candidate data on the basis of one or more candidate labels input via the label candidate input unit 205 and the information managed by the estimation result management unit 204. The processing performed by the candidate data determination unit 206 will be described in detail below.

[0057] The candidate data extraction unit 207 reads mark data matching a desired condition from the mark data management unit 202. For example, the candidate data extraction unit 207 can extract mark data determined as candidate data by the candidate data determination unit 206 from the data managed by the mark data management unit 202.

[0058] The candidate data output unit 208 outputs information corresponding to the candidate data (mark data) extracted by the candidate data extraction unit 207 to a predetermined output destination. Thus, the candidate data output unit 208 can prompt the user with information corresponding to the candidate data as reference information, on the basis of which the user can determine which correct answer information to add to target data.

[0059] The candidate data input unit 209 receives a designation of candidate data from the user, the candidate data being closer in meaning to the target data among the candidate data for which information is prompted to the user. As a specific example, the candidate data input unit 209 can receive a designation of candidate data closer to the target data from among the candidate data prompted to the user by the candidate data output unit 208 from the user.

[0060] (Process)

[0061] Next, an example of the process to be executed by the information processing apparatus 200 according to the present exemplary embodiment will be described below. The description will be made with reference to Figure 5 An example of a series of processes for prompting reference information to add a label to target data will be described.

[0062] In step S501, the label candidate input unit 205 receives a designation of a candidate of a label to be added as an annotation to the target data from the user. For example, Figure 6 An example of an operation screen (user interface) on which a designation of a candidate label is received from the user is shown.

[0063] The operation screen 600 includes an image display region 610, a label candidate prompt region 620, a determination button 601, and a sample image display button 602. The image display region 610 is a region in which an image 611 corresponding to image data (target data) to be annotated is displayed. The label candidate prompt region 620 is a region in which one or more candidate labels are prompted as candidates of a label to be added as an annotation to the target data, and a designation of at least part of the one or more candidate labels is received from the user. The determination button 601 is a button that receives an instruction from the user for determining the candidate label designated in the label candidate prompt region 620 as a label to be added to the target data. The sample image display button 602 is a button that receives an instruction from the user for displaying a sample image.

[0064] In Figure 6 In the example shown, a target region 612 to which a label is to be added as an annotation is designated with respect to the image 611 displayed in the image display region 610. For example, the target region 612 is set in accordance with a designation by the user using the pointer 630. For example, in order to operate the pointer 630, an input device 116 such as a pointing device connected to the information processing apparatus 200 is used. As a specific example, a region having four corners corresponding to four points designated on the image 611 by the pointer 630 can be set as the target region 612. This operation method is merely an example, and the operation method for setting the target region 612 is not particularly limited as long as at least a part of the image 611 can be designated.

[0065] In Figure 6In the example shown, candidate buttons 621 to 624 corresponding to the candidate tags of "Tag A" to "Tag D", respectively, are prompted in the tag candidate prompt region 620. The candidate buttons 621 to 624 are buttons for receiving, from the user, a designation of the candidate tag associated with the corresponding candidate button, respectively.

[0066] Figure 6 The example shown is merely an example, and the configuration for prompting information about receiving a designation of a candidate tag is not necessarily limited to Figure 6 the example shown. For example, the number of buttons prompted in the tag candidate prompt region 620 is not limited to four. For example, in other embodiments, the number of buttons can be changed according to the number of candidate tags. The candidate tags that can be selected can be changed according to the object to be annotated. As a specific example, the candidate tags that can be selected can be determined according to an estimation result based on a pre-trained model for a partial image included in the designated target region 612. In this case, a predetermined number (e.g., four) of tags with higher scores among a series of tags obtained as the estimation result can be determined as the candidate tags.

[0067] After the target region 612 is designated and any one of the candidate buttons 621 to 624 is pressed, the tag candidate input unit 205 acquires the tag corresponding to the pressed candidate button as the candidate tag when the sample image display button 602 is pressed. The pressing of each of the candidate buttons 621 to 624 and the pressing of the sample image display button 602 are implemented by, for example, using the pointing device 630 or the like as an operation pointer.

[0068] Referring back to Figure 5 In step S502, the candidate data determination unit 206 sets a feature space using scores indicating the likelihood that the target data is the data indicated by the candidate tags as an axis, based on the information managed by the estimation result management unit 204 and the candidate tags acquired in step S501. The process for setting the feature space will be described in detail below with reference to Figure 8

[0069] In step S503, the candidate data determination unit 206 determines the candidate data (e.g., candidate images) for which information was prompted to the user by using the feature space set in step S502. The process for determining the candidate data for which information was prompted to the user will be described in detail below with reference to Figure 10

[0070] In step S504, the candidate data extraction unit 207 extracts the candidate data from the data managed by the labeled data management unit 202 according to the determination result in step S503.

[0071] ​​In step S505, the candidate data output unit 208 outputs information corresponding to the candidate data extracted in step S504 to a predetermined output destination, thereby prompting the user with the information. As a specific example, the candidate data output unit 208 can cause a display device such as a display to display an image corresponding to the image data extracted as the candidate data, thereby prompting the user with the image. For example, Figure 7 An example of an output screen (user interface) for prompting the user with information corresponding to the candidate data is shown. In particular, Figure 7 An example of a screen for prompting the user with an image corresponding to the image data extracted as the candidate data is shown.

[0072] The output screen 700 includes a candidate image display area 710, an end button 701, and a re-search button 702. The candidate image display area 710 is an area in which an image corresponding to the image data extracted as the candidate data (hereinafter also referred to as "candidate image") is displayed. In the example shown in Figure 7 In the example shown, in the candidate image display area 710, candidate images 711 to 714 are displayed as images corresponding to the image data extracted as the candidate data. In the vicinity of each of the candidate images 711 to 714, information indicating a tag added to each of the candidate images 711 to 714 is displayed. The end button 701 is a button for receiving an instruction from the user regarding termination of display of the candidate images. The re-search button 702 is a button for receiving an instruction from the user regarding re-search of the candidate images.

[0073] Figure 7 The example shown is merely an example, and the configuration for prompting the information corresponding to the candidate data is not necessarily limited to Figure 7 the example shown. For example, the number of candidate images displayed in the candidate image display area 710 is not limited to four. Four or more images can be displayed, or four or fewer images can be displayed. If it is difficult to display a series of candidate images to be displayed in the candidate image display area 710, the candidate image display area 710 can be configured to be scrollable by a scroll operation using a scroll bar or the like, thereby extending the display area of the candidate images.

[0074] Referring back to Figure 5 In step S506, the candidate data determination unit 206 determines whether or not an instruction to terminate the processing for extracting the candidate data is indicated. For example, if the end button 701 is pressed on the output screen 700 shown in Figure 7

[0075] ​In step S506, if the candidate data determination unit 206 determines that an instruction to terminate the processing for extracting candidate data is not indicated (NO in step S506), the processing proceeds to step S507. In step S507, the candidate data input unit 209 receives a designation of at least one of the candidate data for which information was prompted to the user in step S505 from the user. Further, the candidate data determination unit 206 acquires the candidate data whose designation was received from the user by the candidate data input unit 209, from the candidate data input unit 209.

[0076] As a specific example, the candidate data input unit 209 receives a designation of any one of the candidate images 711 to 714 displayed in the candidate image display region 710 on the output screen 700 illustrated in FIG. 8. Figure 7 Then, when the re-search button 702 is pressed in a state where any one of the candidate images 711 to 714 is designated, the candidate data input unit 209 acquires the candidate data (image data) corresponding to the designated candidate image, and outputs the acquired candidate data to the candidate data determination unit 206. Then, the candidate data determination unit 206 performs the processing of step S502 and the subsequent steps based on the candidate data acquisition result.

[0077] On the other hand, in step S506, if the candidate data determination unit 206 determines that an instruction to terminate the processing for extracting candidate data is indicated (YES in step S506), the processing illustrated in FIG. 7 is terminated. Figure 5

[0078] Next, an example of the processing regarding the setting of the feature space illustrated in the processing of step S502 in FIG. 6 will be described with reference to Figure 8 Figure 5

[0079] In step S801, the candidate data determination unit 206 sets a feature space using each of the candidate labels acquired in step S501 as an axis.

[0080] In step S802, the candidate data determination unit 206 plots the marker data in the feature space set in step S801 based on the information managed by the estimation result management unit 204 (information corresponding to the estimation result obtained by the score estimation unit 203). Thus, a point cluster data corresponding to the marker data is defined on the feature space.

[0081] In step S803, the candidate data determination unit 206 determines whether or not the candidate data acquired in the processing of step S507 illustrated in FIG. 8 (for example, image data corresponding to the designated candidate image) is input. Figure 5

[0082] ​​​​If the candidate data determination unit 206 determines that no candidate data is input in step S803 (NO in step S803), the processing proceeds to step S804. In step S804, the candidate data determination unit 206 acquires the maximum value and the minimum value of the scores of the respective candidate labels of each axis defining the feature space as the range of the point group data in the feature space defined in step S802.

[0083] In step S805, the candidate data determination unit 206 calculates the intermediate value between the maximum value and the minimum value acquired in step S804 for each axis (i.e., each candidate label) with respect to the setting of the feature space.

[0084] In step S806, the candidate data determination unit 206 divides at least the region in which the point group data exists in the feature space into a plurality of quadrants on the basis of the maximum value, the minimum value, and the intermediate value acquired or calculated for each axis with respect to the setting of the feature space. Thus, a plurality of quadrants are set in at least the region in which the point group data exists in the feature space.

[0085] For example, Figures 9A-9F is a diagram illustrating an example of the processing of setting a plurality of quadrants by dividing the region in which the point group data exists in the feature space. In Figures 9A-9F In the example illustrated in FIG. 9, for the convenience of explanation of the technical features according to the present exemplary embodiment, the feature space 900 is set as a two-dimensional space using the score of the label A as the horizontal axis 901 and using the score of the label B as the vertical axis 902 for the convenience of explanation.

[0086] Specifically, Figure 9A The state in which the feature space 900 set in the processing of step S801 is plotted with the points 903 corresponding to the respective labeled data in the processing of step S802 is schematically illustrated.

[0087] Figure 9B The region 910 in which the point group data exists is schematically illustrated. The region 910 is defined on the basis of the maximum value and the minimum value acquired for each axis in the processing of step S804.

[0088] Figure 9C The state in which the region 910 in which the point group data exists in the feature space is divided into a plurality of quadrants 911 to 914 on the basis of the calculation result of the intermediate value in the processing of step S805 is schematically illustrated. Specifically, in Figure 9C In the example illustrated in FIG. 9, the region 910 is divided into the plurality of quadrants 911 to 914 by a straight line passing through the intermediate value of the score of the label A and being perpendicular to the horizontal axis 901 and a straight line passing through the intermediate value of the score of the label B and being perpendicular to the vertical axis 902.

[0089] Although Figures 9A-9FThe example shown shows a case where two candidate labels are used for the sake of simplifying the explanation, and three or more candidate labels can be used to set the feature space. In this case, the feature space set in step S801 is a three-dimensional or more feature space.

[0090] Referring back to Figure 8 In step S803, if the candidate data determination unit 206 determines that the candidate data is input (YES in step S803), the processing proceeds to step S807. In step S807, the candidate data determination unit 206 identifies a point corresponding to the candidate data acquired in the processing in step S507 (the candidate data specified by the user) among the point cluster data set in the feature space. Then, the candidate data determination unit 206 acquires the coordinates of the identified point 903 in the feature space.

[0091] In step S808, the candidate data determination unit 206 sets a new region by halving the lengths of the edges of the region combining the four quadrants set in step S806 with the coordinates acquired in step S807 as the center.

[0092] Next, in step S806, the candidate data determination unit 206 divides the new region set in step S808 into a plurality of new quadrants based on a process similar to the process of step S806 described above.

[0093] Referring back to Figures 9A-9F , Figure 9D A state in which the point 915 corresponding to the candidate data acquired in step S507 is identified in the processing of step S807 is schematically shown. Figure 9E A state in which a new region 920 obtained by halving the lengths of the edges of the region combining the four quadrants 911 to 914 (i.e., the region 910) with the point 915 as the center is set is schematically shown. Thus, the new region 920 is set using the point 915 corresponding to the candidate data specified by the user as a base point.

[0094] Figure 9F A state in which the new region 920 set in the feature space is divided into a plurality of new quadrants 921 to 924 in the processing of step S806 after the processing of steps S807 and S808 is schematically shown.

[0095] Figures 9A-9F The examples show a case where the region set based on the maximum value and the minimum value on each axis is divided into a plurality of quadrants based on the intermediate value between the maximum value and the minimum value. However, these examples are not intended to limit the method of setting the quadrants. As a specific example, clustering of the point cluster data set in step S802 can be performed, and each cluster set by the clustering can be set as a quadrant.

[0096] Next, the processing of the step S503 will be described. Figure 10 Figure 5 An example of the processing of the candidate data indicated by the processing of the step S503 will be described.

[0097] In the step S1001, the candidate data determination unit 206 calculates the position (coordinates) of the center points of the respective quadrants set in the processing of the step S806. Figure 8

[0098] In the step S1002, the candidate data determination unit 206 calculates the distance in the feature space between the position of the center point calculated for each quadrant in the step S1001 and the position of each point included in the point group data set in the processing of the step S802. Figure 8

[0099] In the step S1003, the candidate data determination unit 206 identifies the point closest to the center point of each quadrant on the basis of the calculation results obtained in the step S1002 (i.e., the distance between each point included in the point group data and the center point of each quadrant). Further, the candidate data determination unit 206 determines the mark data corresponding to the identified point for each quadrant as the candidate data representing the quadrant. Then, the candidate data extraction unit 207 extracts the mark data determined as the candidate data by the candidate data determination unit 206 from the mark data managed by the mark data management unit 202.

[0100] Through the above-described control processing, the user sequentially specifies (e.g., selects) the more likely candidate data from the candidate data prompted as the reference information. Therefore, the range of the candidate data in the feature space to be prompted as the reference information is sequentially limited. Therefore, even in the case where there is no significant difference in reliability among a plurality of different tags, the user can sequentially specify the more likely candidate data from the prompted candidate data, so that it is possible to effectively narrow down the semantic range of the candidate data for which the information is prompted.

[0101] This configuration makes it possible for the user to select the tag desired to be added to the target data through a simple operation without the need to check a large number of candidate data (mark data). In other words, with the technology according to the present exemplary embodiment, it is expected that even in the case where the tags indicating different subjects do not significantly differ in reliability, so that the user is expected to carefully perform the tag operation, it is possible to obtain the beneficial effect of assisting the user to perform the marking operation in a more desirable manner.

[0102] ​​​A second exemplary embodiment of the present disclosure will be described below. In order to distinguish the information processing apparatus according to the second exemplary embodiment from the information processing apparatuses according to the other exemplary embodiments, the information processing apparatus according to the second exemplary embodiment will be referred to as an information processing apparatus 1100 below for ease of description.

[0103] In the above-described first exemplary embodiment, the set feature space is divided into a plurality of quadrants, and one candidate data is extracted from each quadrant. In the second exemplary embodiment, the number of candidate data to be extracted from each quadrant is changed based on a score representing the likelihood that the target data to be annotated is data indicated by each candidate label. Therefore, the differences between the features of the technology according to the second exemplary embodiment and the technology according to the first exemplary embodiment will be mainly described below, and the substantially same parts as the first exemplary embodiment will not be described in detail.

[0104] (Functional Configuration)

[0105] An example of the functional configuration of the information processing apparatus 1100 according to the present exemplary embodiment will be described with reference to Figure 11 The information processing apparatus 1100 according to the present exemplary embodiment is different from the information processing apparatus 200 illustrated in Figure 2 in that the information processing apparatus 1100 includes a target data input unit 1101.

[0106] The target data input unit 1101 receives input of target data to be annotated. The information processing apparatus 1100 according to the present exemplary embodiment estimates a label that can be added to the target data received by the target data input unit 1101 and a score of the label based on a pre-trained model, and uses the estimation result for processing for determining candidate data. The processing and the processing to be performed by the information processing apparatus 1100 according to the present exemplary embodiment will be described in detail below.

[0107] (Processing)

[0108] Next, an example of the processing to be performed by the information processing apparatus 1100 according to the present exemplary embodiment will be described in detail below.

[0109] First, an example of a series of processing procedures for prompting reference information to add a label to target data will be described with reference to Figure 12 The example illustrated in Figure 12 is different from the example illustrated in Figure 5 in that the processing of steps S1201 and S1202 is added, and a part of the processing of step S503 is changed.

[0110] In step S1201, the target data input unit 1101 receives the target data to be annotated. To receive the target data input, for example, a reference can be used. Figure 6 The described operation screen (user interface) is 600.

[0111] As a specific example, when the sample image display button 602 is pressed after the target area 612 is specified in the image display area 610, the image data of the portion of the image displayed in the image data display area 610 that corresponds to the target area 612 can be obtained as the target data.

[0112] In step S1202, the scoring estimation unit 203 estimates the labels that can be added to the target data obtained in step S1201 and the score indicating the probability that the target data is the data indicated by the label.

[0113] Subsequently, the processes in steps S501 and S502 are performed, and then in step S503, the process for determining candidate data is performed.

[0114] Reference Figure 13 describe Figure 12 The example shown illustrates the processing of step S503. Figure 13 The example shown is the same as Figure 10 The difference in the example shown is that the processing of steps S1301 and S1302 is replaced by the processing of step S1003.

[0115] In step S1301, the candidate data determination unit 206 plots the target data in the feature space set in step S502 based on the score of each label estimated for the target data in step S1202.

[0116] In step S1302, the candidate data determination unit 206 calculates the distance between the position of the target data plotted in the feature space in step S1301 and the center point of each quadrant set in step S1001. Then, the candidate data determination unit 206 controls the number of target data to be determined from each quadrant, thereby determining a larger number of candidate data from quadrants that are closer to the position of the target data.

[0117] As a specific example, the candidate data determination unit 206 can determine three candidate data closest to the center point of the quadrant closest to the position of the target data from the quadrant closest to the position of the target data. In this case, the candidate data determination unit 206 can determine two candidate data closest to the center point of the quadrant sharing one side with the quadrant closest to the position of the target data from the quadrant sharing one side with the quadrant closest to the position of the target data. In addition, the candidate data determination unit 206 can determine one candidate data closest to the center point of the quadrant sharing one point with the quadrant closest to the position of the target data from the quadrant sharing one point with the quadrant closest to the position of the target data. Then, the process of steps S504 to S507 is executed as shown in the example. Figure 5 Figure 12

[0118] In the above example, the number of candidate data to be extracted from each quadrant is set to a fixed number according to the proximity to the position of the target data. However, the operation of the information processing apparatus according to the present example embodiment is not necessarily limited to this manner.

[0119] As a specific example, the number of candidate data to be extracted from each quadrant can be changed based on the user's designation according to the proximity to the position of the target data. In another example, the number of candidate data to be extracted from each quadrant can be changed according to the distance between the position of the target data and the center point of each quadrant. Although the center point is used as the representative point of each quadrant in the above example embodiment, any point other than the center point can be used.

[0120] A third example embodiment of the present disclosure will be described below. In order to distinguish the information processing apparatus according to the third example embodiment from the information processing apparatuses according to the other example embodiments, the information processing apparatus according to the third example embodiment will be referred to as an information processing apparatus 1400 for convenience of explanation below.

[0121] In the above second example embodiment, the number of candidate data to be extracted from each quadrant is changed based on the score indicating the likelihood that the target data to be annotated is the data indicated by each candidate label. In the present example embodiment, if the distance between the target data selected by the user and the candidate data in the feature space increases by more than a predetermined multiple, predetermined information is provided to the user based on the score indicating the likelihood that the target data is the data indicated by each candidate label. Therefore, the difference between the features of the technology according to the third example embodiment and the features of the technology according to the second example embodiment will be mainly described below, and the substantially same parts as the second example embodiment will not be described in detail.

[0122] (Functional Configuration)

[0123] Reference will be made to Figure 14 ​​An example of a functional configuration of the information processing apparatus 1400 according to the present exemplary embodiment will be described. The information processing apparatus 1400 according to the present exemplary embodiment is different from the information processing apparatus 1100 illustrated in FIG. 12 in that the information processing apparatus 1400 includes a notification unit 1401. Figure 11 The notification unit 1401 causes a predetermined output unit to provide notification information in accordance with a predetermined condition, thereby presenting the notification information to the user. As a specific example, if the distance between the target data and the candidate data selected by the user in the feature space increases by more than a predetermined multiple, the notification unit 1401 can present the user with notification information indicating a warning.

[0124] The notification unit 1401 causes a predetermined output unit to provide notification information in accordance with a predetermined condition, thereby presenting the notification information to the user. As a specific example, if the distance between the target data and the candidate data selected by the user in the feature space increases by more than a predetermined multiple, the notification unit 1401 can present the user with notification information indicating a warning.

[0125] (Process)

[0126] Next, an example of the process to be executed by the information processing apparatus 1400 according to the present exemplary embodiment will be described below with reference to Figure 15 , particularly by focusing on a series of processes of presenting reference information for adding a label to the target data. Figure 15 The example illustrated in FIG. 15 is different from the example illustrated in FIG. 13 in that the process of steps S1501 to S1504 is added as a process after step S507. Figure 12 In step S1501, the candidate data determination unit 206 calculates the distance between the position of the target data plotted in step S1301 and the position of the candidate data specified in step S507 in the feature space set in step S502.

[0127] In step S1502, the candidate data determination unit 206 determines whether the distance calculated in step S1501 is increased by more than a predetermined multiple in the case where the process of step S507 is continuously executed multiple times. As a specific example, the candidate data determination unit 206 can determine whether the distance calculated in step S1501 is increased by more than three times.

[0128] In step S1502, if the candidate data determination unit 206 determines that the distance calculated in step S1501 is not increased by more than a predetermined multiple (NO in step S1502), the process returns to step S503. In this case, the process of step S503 and the subsequent steps are executed again.

[0129]

[0130] ​On the other hand, in step S1502, if the candidate data determination unit 206 determines that the distance calculated in step S1501 has increased by a predetermined multiple or more (YES in step S1502), the process proceeds to step S1503. In step S1503, the notification unit 1401 causes the predetermined output unit to provide notification information (for example, notification information indicating a warning), thereby prompting the notification information to the user.

[0131] For example, Figure 16 An example of a dialog box (user interface) that prompts predetermined notification information to the user as display information is shown. The dialog box 1600 displays notification information indicating a warning (for example, character information), thereby prompting the notification information to the user. The dialog box 1600 includes an OK button 1601, a redo button 1602, and a close button 1603.

[0132] The OK button 1601 is a button for receiving an instruction from the user to continue the processing that is currently being executed. The redo button 1602 is a button for interrupting the processing that is currently being executed and receiving an instruction from the user to redo the processing for determining the candidate data (to execute the processing again in accordance with the setting of the feature space). The close button 1603 is a button for receiving an instruction from the user to close the dialog box 1600. In the present exemplary embodiment, when the close button 1603 is pressed, similar processing to that performed when the OK button 1601 is pressed is performed.

[0133] In step S1504, the candidate data determination unit 206 determines whether the redo processing is selected. As a specific example, if it is detected that the redo button 1602 is pressed on the dialog box 1600 shown in FIG. 16, the candidate data determination unit 206 can determine that the redo processing is selected. On the other hand, if it is detected that the OK button 1601 or the close button 1603 is pressed on the dialog box 1600, the candidate data determination unit 206 can determine that the redo processing is not selected. Figure 16

[0134] In step S1504, if the candidate data determination unit 206 determines that the redo processing is selected (YES in step S1504), the process returns to step S502. In this case, the feature space setting processing exemplified by the processing of step S502 is executed again.

[0135] On the other hand, in step S1504, if the candidate data determination unit 206 determines that the redo processing is not selected (NO in step S1504), the process returns to step S503. In this case, the processing for determining the candidate data exemplified by the processing of step S503 is executed again.

[0136] ​The above-described present exemplary embodiment illustrates an example in which notification information (e.g., a warning) is provided when the distance between the target data and the selected candidate data increases by more than a predetermined multiple. However, the processing to be performed by the information processing apparatus according to the present exemplary embodiment is not necessarily limited to this.

[0137] As a specific example, notification information can be provided depending on whether the target data and the selected candidate data are separated from each other by more than a predetermined distance. In another example, the distance between the target data and the candidate data can be accumulated each time the candidate data is selected, and notification information can be provided when the accumulated value of the distance exceeds a threshold value. The threshold value for determining the number of times, the distance, the accumulated value of the distance, and the like can be a fixed value, or can be changed based on an instruction from the user.

[0138] In the example of the above-described processing, the determination in step S1504 provides the user with two options, i.e., an option to continue the processing and an option to redo the processing from the setting of the feature space. However, the options that the user can select in the determination are not limited to this example.

[0139] As a specific example, an option to restart the processing from the state in which the previous candidate data was selected can be provided. In this case, processing for storing the state of the feature space at the time when the previous candidate data was selected can be added. In this case, the option to restart the processing from the state in which the previous candidate data was selected can be provided in step S1504. Figure 16 An interface (e.g., a button) for receiving an instruction from the user to restart the processing from the state in which the previous candidate data was selected can be provided on the dialog box 1600 illustrated.

[0140] Embodiments of the present disclosure can also be realized by a process in which a program for realizing one or more functions according to the above-described exemplary embodiments is provided to a system or an apparatus via a network or a recording medium, and a processor in the computer of the one or more systems or apparatuses reads and executes the program. Embodiments can also be realized by a circuit (e.g., an application specific integrated circuit (ASIC)) for realizing one or more functions according to the above-described exemplary embodiments.

[0141] The above-described embodiments are merely examples, and can be modified or changed in various ways without departing from the technical scope according to the present disclosure.

[0142] For example, the functions of the information processing apparatus according to the present exemplary embodiments can be realized by a plurality of apparatuses in a cooperative manner. As a specific example, some of the components of the information processing apparatus illustrated in Figure 2 , Figure 11 and Figure 14 may be provided in an external device different from the information processing apparatus. Furthermore, the functions of the information processing apparatus according to the present exemplary embodiments can be realized by a combination of hardware and software. Figure 2 , Figure 11 and Figure 14The load on the processing performed by at least some of the components of the illustrated information processing apparatus is distributed to a plurality of apparatuses. Although the exemplary embodiments are described by focusing on a case in which image data is mainly used as data to which a label is added as an annotation, the type of data is not necessarily limited to the above-described image data. Furthermore, the method of presenting the information related to the prompt and the method of receiving the designation of the target data from the user can be appropriately changed according to the type of data to which a label is added as an annotation.

[0143] According to the embodiments of the present disclosure, it is possible to help a user to select a label to be added as an annotation in a more desirable manner.

[0144] Other Embodiments

[0145] Embodiments of the present disclosure can also be implemented by a method of providing, through a network or various storage media, software (program) that performs the functions of the above-described embodiments to a system or apparatus, which reads and executes the program by a computer or a central processing unit (CPU), a micro processing unit (MPU).

[0146] While exemplary embodiments have been described herein, it should be understood that the present application is not limited to the disclosed exemplary embodiments. The scope of the claims should be construed to include all possible embodiments and equivalents thereof.

Claims

1. An information processing apparatus, the information processing apparatus comprising: The acquisition unit is configured to acquire an estimate of the score representing the probability of adding a label as an annotation to the target data, wherein multiple candidate labels and target data are input into a training model, and the training model outputs a score based on the input multiple candidate labels and target data; A candidate tag receiving unit is configured to receive a designation of candidate tags, which are candidates for tags to be added as annotations to the target data; A determining unit is configured to determine candidate data from a plurality of labeled data included in a feature space, the candidate data representing a plurality of labeled data distributed in quadrants in which the feature space is divided, wherein labels are added to the labeled data as annotations, and wherein the feature space is defined using scores representing the probability of adding labels as annotations as axes, wherein the feature space has a plurality of axes, each corresponding to the magnitude of the label score of the candidate labels; and The candidate data output unit is configured to output candidate data determined for each quadrant to a predetermined output destination.

2. The information processing apparatus according to claim 1, wherein, The determining unit divides the region where the labeled data is distributed into quadrants based on the median value between the maximum and minimum scores of each labeled data item for each of the multiple candidate labels.

3. The information processing device according to claim 1, in, The determining unit divides the new region into multiple new quadrants and determines new candidate data for each of the multiple new quadrants based on the labeled data included in each quadrant. The new region is defined using specified candidate data from multiple candidate data output to the output destination as a base point. The candidate data output unit outputs new candidate data determined for each of the multiple new quadrants to the output destination.

4. The information processing apparatus according to claim 3, wherein, The following processes are performed sequentially: dividing the new region into the plurality of new quadrants by the determining unit, wherein the new region is defined using a specified candidate data from a plurality of candidate data output to the output destination as a base point; determining new candidate data for each of the plurality of new quadrants based on the labeled data included in each of the plurality of new quadrants; and outputting the new candidate data determined for each of the plurality of new quadrants to the output destination by the candidate data output unit.

5. The information processing apparatus according to claim 3, wherein, The new region is defined such that its size is limited compared to the region previously divided into multiple quadrants.

6. The information processing apparatus according to claim 3, further comprising: The notification unit is configured to notify a predetermined output destination of notification information when the distance to the target data in the feature space calculated for each sequentially specified candidate data is increased by a predetermined multiple relative to the distance calculated for previously specified candidate data.

7. The information processing apparatus according to claim 1, wherein, The determining unit controls the number of candidate data determined for each of the multiple quadrants based on the distance between the specified target data and each of the multiple quadrants in the feature space.

8. The information processing apparatus according to claim 7, wherein, The determining unit is controlled such that for each quadrant that is closer to the specified target data in the feature space, the number of candidate data determined for each quadrant is greater.

9. An information processing method executed by an information processing device, the information processing method comprising: Obtain an estimated score, which represents the probability of adding a label as an annotation to the target data, wherein multiple candidate labels and target data are input into a training model, and the training model outputs a score based on the input multiple candidate labels and target data; Receive the designation of candidate tags, which are candidates for tags to be added as annotations to the target data; Candidate data is determined from multiple labeled data included in a feature space, the candidate data representing multiple labeled data distributed in various quadrants of the feature space, wherein labels are added to the labeled data as annotations, and wherein the feature space is defined using scores representing the probability of adding labels as annotations as axes, wherein the feature space has multiple axes, each corresponding to the magnitude of the label score of the candidate label; and The candidate data determined for each quadrant will be output to the predetermined output destination.

10. A non-transitory computer-readable storage medium storing a computer-executable program for causing a computer to execute the method of claim 9.

Images