Information processing device, information processing method, and program

Abstract

This information processing device is provided with: a correct label specification means that uses a machine learning model to specify a correct label to be assigned to each of a plurality of sets of image data; a correct label modification means that modifies the specified correct label of image data having an inappropriate correct label among the plurality of sets of image data to which the correct label has been specified to be assigned by the correct label specification means; and a weighting means that assigns a weight to each of the plurality of sets of image data. The weighting means sets the weight assigned to the image data to which the correct label modified by the correct label modification means is assigned, to be greater than the weight assigned to each set of image data to which the correct label specified by the correct label specification means is assigned. Thus, the information processing device supports a user's decision-making regarding a correct label to be assigned to each of the plurality of sets of image data.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] There are known techniques for identifying data that causes difficulty in image recognition. For example, Patent Literature 1 discloses a technique for identifying a part of an image that causes an incorrect inference, while modifying a part of the image represented by image data in which an incorrect label has been inferred, so as to maximize the score of the inferred label.

[0003] Japanese Patent Application Publication No. 2020-197875

[0004] The technology described in Patent Document 1 has the problem that it does not generate machine learning data by placing emphasis on data that is difficult to recognize, and therefore does not contribute much to improving recognition accuracy.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technology that contributes to improving recognition accuracy by generating machine learning data with an emphasis on data that is difficult to recognize.

[0006] An information processing device according to an exemplary aspect of the present disclosure includes a correct label identification means that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction means that corrects the correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification means is inappropriate; and a weight assignment means that assigns a weight to each of the plurality of image data, wherein the weight assignment means sets a weight to be assigned to image data to which the correct label corrected by the correct label correction means has been assigned that is greater than the weight to be assigned to image data to which the correct label identified by the correct label identification means has been assigned.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes a correct label identification process that uses a machine learning model to identify a correct label to be assigned to each of a plurality of image data; a correct label correction process that corrects the correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification process is inappropriate; and a weight assignment process that assigns a weight to each of the plurality of image data, wherein the weight assignment process sets a weight to be assigned to image data to which the correct label corrected by the correct label correction process has been assigned that is greater than the weight to be assigned to image data to which the correct label identified by the correct label identification process has been assigned.

[0008] A program according to an exemplary aspect of the present disclosure causes a computer to execute a correct label identification process that uses a machine learning model to identify a correct label to be assigned to each of a plurality of image data; a correct label correction process that corrects the correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification process is inappropriate; and a weight assignment process that assigns a weight to each of the plurality of image data, wherein the weight assignment process sets a weight to be assigned to image data to which the correct label corrected by the correct label correction process has been assigned that is greater than the weight to be assigned to image data to which the correct label identified by the correct label identification process has been assigned.

[0009] According to one exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technology can be provided that contributes to improving recognition accuracy by generating machine learning data with an emphasis on data that is difficult to recognize.

[0010] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flow diagram showing the flow of an information processing method according to the present disclosure. FIG. 3 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 4 is a flow diagram showing the flow of an information processing method according to the present disclosure. FIG. 5 is a flow diagram showing the flow of an information processing method according to the present disclosure. FIG. 6 is a diagram showing an example of how correct labels and weights are expressed according to the present disclosure. FIG. 7 is a diagram showing an example of how correct labels and weights are expressed according to the present disclosure. FIG. 8 is a diagram showing an example of a method of assigning weights according to the degree of deviation according to the present disclosure. FIG. 9 is a block diagram showing the configuration of a computer that functions as an information processing device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Overview of Information Processing Device 1) An overview of the information processing device 1 according to this exemplary embodiment will be described. As an example, the information processing device 1 is a device that identifies correct labels to be assigned to image data using a machine learning model, corrects inappropriate labels from the identified correct labels, and assigns a greater weight to image data to which the corrected correct labels have been assigned than to other image data. The image data to be processed by the information processing device 1 may be, for example, image data representing a still image or image data representing a moving image.

[0014] (Configuration of information processing device 1) The configuration of the information processing device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes a correct label identification unit 11, a correct label correction unit 12, and a weighting unit 13.

[0015] (Correct Label Identification Unit 11) The correct label identification unit 11 identifies a correct label to be assigned to each of a plurality of image data using a machine learning model MM. The machine learning model MM may be, for example, a model that has been machine-learned by referring to existing image data. Furthermore, the image data may be, for example, received via a communication unit, read from a storage device, or acquired via an input interface (such as a USB).

[0016] (Correct Label Correction Unit 12) The correct label correction unit 12 corrects the correct label to be assigned to image data for which the correct label identified by the correct label identification unit 11 is inappropriate, among a plurality of image data.

[0017] Whether a correct label is appropriate or inappropriate may be determined, for example, by (i) comparing it with the estimation result of another model (another model with higher estimation accuracy), (ii) comparing it with a correct label previously assigned to the image data, or (iii) visually inspecting it at any time by a user. In this case, as an example, the correct label correction unit 12 outputs the image data input to the machine learning model MM and the correct label identified by the correct label identification unit 11 to an output device such as a display device. If the correct label is incorrect, the user inputs the correct label using an input device. The correct label correction unit 12 determines whether the correct label identified by the correct label identification unit 11 is appropriate or inappropriate based on the information input via the input device.

[0018] (Weighting Unit 13) The weighting unit 13 assigns a weight to each of the plurality of image data. Here, the weighting unit 13 sets a weight to be assigned to the image data to which the correct label corrected by the correct label correction unit 12 has been assigned, to be larger than the weight to be assigned to the image data to which the correct label identified by the correct label identification unit 11 has been assigned.

[0019] For example, if the correct label identified by the correct label identification unit 11 using the machine learning model MM is inappropriate, the image data to which the correct label is assigned is considered to be more difficult to recognize by the machine learning model MM. The weighting unit 13 may, for example, assign a larger weight to such image data that is more difficult to recognize than to image data to which an appropriate correct label is assigned.

[0020] Also, for example, the weighting unit 13 may set a predetermined value as a weight for image data for which the correct answer label is appropriate, and set a value greater than the predetermined value as a weight for image data for which the correct answer label is inappropriate.

[0021] Furthermore, the image data to which the correct label and weight have been assigned after processing by the weighting unit 13 may be output, for example, to the outside of the information processing device 1. As a specific example of a method for outputting the image data to the outside of the information processing device 1, the image data may be stored in a storage device external to the information processing device 1, may be transmitted to the outside of the information processing device 1 via a communication unit, or may be output to an output device (display, printer, speaker, etc.) external to the information processing device 1.

[0022] Furthermore, the image data that has been processed by the weighting unit 13 and assigned a correct label and a weight may be used, for example, as training data for training another model, or may be used for retraining a model.

[0023] (Specific examples of correct labels) The correct labels identified by the correct label identification unit 11 may be, for example, at least one of a label indicating an object class, information about the position of an object, a label indicating an action class, and information about the time period during which the action is performed.

[0024] The label indicating the object class is, for example, a label indicating what object is included as a subject in the image represented by the image data. The label indicating the object class may be, for example, a name indicating the object, such as a shovel, a dump truck, a doctor, or a patient.

[0025] The information about the object position indicates, for example, the position of an object included as a subject in the image represented by the image data. Note that the information about the object position may be represented by a rectangle (quadrilateral) in the image, as will be described later in the description of Figures 7 and 9.

[0026] The label indicating the action class is, for example, a label indicating the action performed by an object included as a subject in a video represented by image data. The label indicating the action class may indicate the content of the action, such as, for example, "I'm currently leveling the ground."

[0027] The information regarding the time period during which an action is performed indicates, for example, the time period from which frame to which frame an action is performed by an object included as a subject in the video represented by the image data, as will be described later in the explanation of Figure 8.

[0028] (Effects of Information Processing Device 1) As described above, the information processing device 1 includes a correct label identification means that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model, a correct label correction means that corrects the correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification means is inappropriate, and a weight assignment means that assigns a weight to each of the plurality of image data, wherein the weight assignment means is configured to set a weight to be assigned to image data to which the correct label corrected by the correct label correction means is assigned greater than a weight to be assigned to image data to which the correct label identified by the correct label identification means is assigned. Therefore, the information processing device 1 has the effect of contributing to improved recognition accuracy by increasing the weight of data to which an inappropriate correct label, which can be said to be data that is difficult to recognize, is assigned.

[0029] (Flow of Information Processing Method S1) The flow of information processing method S1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of information processing method S1. As shown in Fig. 2, information processing method S1 includes a correct label identification process (step) S11, a correct label correction process (step) S12, and a weighting process (step) S13.

[0030] (Step S11) In step S11, the correct label identification unit 11 identifies a correct label to be assigned to each of the plurality of image data using the machine learning model MM.

[0031] (Step S12) In step S12, the correct label correction unit 12 corrects the correct label to be assigned to the image data for which the correct label identified by the correct label identification unit 11 is inappropriate, among the plurality of image data.

[0032] (Step S13) In step S13, the weighting unit 13 assigns a weight to each of the plurality of image data. Here, the weighting unit 13 sets the weight to be assigned to the image data to which the correct label corrected by the correct label correction unit 12 has been assigned to be greater than the weight to be assigned to the image data to which the correct label identified by the correct label identification unit 11 has been assigned.

[0033] (Effects of Information Processing Method S1) As described above, information processing method S1 includes a correct label identification process that uses a machine learning model to identify a correct label to be assigned to each of a plurality of image data pieces, a correct label correction process that corrects the correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification process is inappropriate, and a weighting process that assigns a weight to each of the plurality of image data pieces, wherein the weighting process is configured to set a weight to be assigned to image data to which the correct label corrected by the correct label correction process is assigned greater than a weight to be assigned to image data to which the correct label identified by the correct label identification process is assigned. Therefore, information processing method S1 has the effect of contributing to improved recognition accuracy by increasing the weight of data to which an inappropriate correct label, which can be said to be data that is difficult to recognize, is assigned.

[0034] (Modification of Information Processing Device 1) A modification of the information processing device 1 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example configuration of the correct label correction unit 12 in the modification of the information processing device 1. As shown in Fig. 3, the correct label correction unit 12 may include, for example, a presentation unit 121 and a correct label change unit 122.

[0035] (Presentation Unit 121) The presentation unit 121 may, for example, present all or part of the plurality of image data to the user U together with the correct labels identified by the correct label identification unit 11. The presentation unit 121 may, for example, present all or part of the plurality of image data to the user U together with the correct labels identified by the correct label identification unit 11 via a display device. Specific examples of such a display device include a display and a touch panel.

[0036] (Correct label modification unit 122) For example, based on the operation of user U, the correct label modification unit 122 may modify the correct label to be assigned to image data specified by user U among the image data presented by the presentation unit 121 to the correct label specified by user U.

[0037] For example, the user U may specify image data presented by the presentation unit 121 based on the operation of an input device. Specific examples of such input devices include a keyboard, a mouse, a touch panel, etc. Furthermore, the user U may specify, for example, image data presented by the presentation unit 121 for which the correct label identified by the correct label identification unit 121 is determined to be inappropriate, and instruct the correct label changing unit 122 to assign a correct label that the user U determines to be appropriate to the image data.

[0038] (Specifying a change in the correct label for each similar image data group) The presentation unit 121 may, for example, classify all or part of the multiple image data so that similar image data are included in the same image data group and present the classified data to the user U. Here, the correct label modification unit 122 may, for example, change the correct label assigned to the image data belonging to each image data group to a correct label specified by the user U.

[0039] For example, the presenting unit 121 may extract features from multiple image data and classify the image data such that image data in which the distance between the features is less than a predetermined threshold is included in the same image data group. Furthermore, for example, the presenting unit 121 may perform a clustering process on multiple image data and classify the image data such that image data included in the clusters generated by the clustering process are included in the same image data group.

[0040] (Assigning correct labels to multiple image data with a single operation) For example, when the correct label modification unit 122 changes the correct label to be assigned to first image data specified by user U to the correct label specified by user U, it may also change the correct label to be assigned to second image data among the multiple image data that is similar to the first image data to the same correct label as the correct label assigned to the first image data.

[0041] (Presenting Image Data with Reference to Accuracy of Correct Label) The correct label identification unit 11 may, for example, calculate an accuracy that indicates the accuracy of the correct label identified using the machine learning model MM. Here, the presentation unit 121 may select image data to be presented to the user U with reference to the accuracy calculated by the correct label identification unit 11.

[0042] The accuracy of the correct label represents, for example, the degree of accuracy of the correct label. The method for calculating the accuracy of the correct label may be determined, for example, based on an existing judgment result by the user. Furthermore, for example, the presentation unit 121 may present to the user U only image data whose accuracy calculated by the correct label identification unit 11 is below a predetermined threshold.

[0043] (Effects of the Variation of Information Processing Device 1) As described above, in the variation of the information processing device 1, the correct label correction means is configured to include: a presentation means that presents all or part of the multiple image data to the user together with the correct labels identified by the correct label identification means; and a correct label change means that, based on the user's operation, changes the correct label to be assigned to image data specified by the user among the image data presented by the presentation means to the correct label specified by the user. Therefore, according to the variation of the information processing device 1, in addition to the effects achieved by the information processing device 1, an effect is obtained in that the correct label assigned to the image data can be changed at the discretion of the user.

[0044] Furthermore, in a modified example of the information processing device 1, the presenting means classifies all or part of the multiple image data so that similar image data are included in the same image data group and presents the classified data to the user, and the correct label changing means changes the correct label assigned to the image data belonging to each image data group to a correct label designated by the user. Therefore, according to the modified example of the information processing device 1, in addition to the effects achieved by the information processing device 1, the effect that the user can collectively designate a change in the correct label for each similar image data can be obtained.

[0045] Furthermore, in a modified example of the information processing device 1, the correct label changing means is configured to, when the correct label to be assigned to first image data designated by the user is changed to the correct label designated by the user, change the correct label to be assigned to second image data among the plurality of image data that is similar to the first image data to the same correct label as the correct label assigned to the first image data. Therefore, according to the modified example of the information processing device 1, in addition to the effect achieved by the information processing device 1, an effect is obtained in that, when the user changes the correct label assigned to image data, the correct labels to be assigned to image data similar to the image data can be similarly changed.

[0046] Furthermore, in a modified example of the information processing device 1, the correct label identification means further calculates an accuracy indicating the accuracy of the correct label identified using the machine learning model, and the presentation means selects image data to be presented to the user by referring to the accuracy calculated by the correct label identification means. Therefore, according to the modified example of the information processing device 1, in addition to the effects achieved by the information processing device 1, an effect is obtained in that the image data to be presented to the user can be limited by referring to the accuracy of the correct label.

[0047] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0048] (Overview of Information Processing Device 1A) The information processing device 1A is, as an example, a device for efficiently generating training data for machine learning. For example, when training an image recognition model by machine learning with reference to multiple image data, it is thought that the recognition accuracy of the image recognition model after training will be improved if the machine learning is performed with a greater emphasis on image data that is difficult for existing machine learning models to recognize.

[0049] The information processing device 1A is a device that, for example, uses a machine learning model to identify correct labels to be assigned to image data processed using the information processing device 1, corrects inappropriate labels among the identified correct labels, i.e., labels that are difficult to recognize, and then assigns a greater weight to image data to which the corrected correct labels have been assigned than to other image data.

[0050] Specific examples of fields in which the information processing device 1A can be utilized include labor-intensive industries such as the construction industry, and medical and healthcare fields.

[0051] (Configuration of information processing device 1A) The configuration of information processing device 1A will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of information processing device 1A. In addition to a correct label identification unit 11, a correct label correction unit 12, and a weighting unit 13 that are included in information processing device 1, information processing device 1A also includes a learning unit 14 and a recognition unit 15.

[0052] (Learning unit 14) The learning unit 14 may, for example, perform machine learning on the image recognition model VM by referring to a plurality of image data weighted by the weighting unit 13. For example, the learning unit 14 may perform machine learning on the image recognition model VM by multiplying the weights assigned to the image data by the weighting unit 13 by a loss function calculated during learning.

[0053] (Recognition unit 15) The recognition unit 15 may recognize a specific object included as a subject in the image data or a specific action of an object included as a subject in the image data, for example, by using the image recognition model VM machine-learned by the learning unit 14. The result of the recognition unit 15 recognizing the object or action included in the image data may be output to the outside of the information processing device 1A, for example.

[0054] (Assigning Weights According to Positional Deviation of Objects) The machine learning model MM may be, for example, an object recognition model. Here, the correct label may include, for example, the position of a specific object included as a subject in the image data. In addition, the weighting unit 13 may set a weight to be assigned to the image data according to, for example, the difference between the position identified by the correct label identifying unit 11 and the position corrected by the correct label correcting unit 12.

[0055] For example, the weighting unit 13 may set a larger weight to be assigned to the image data as the difference between the position identified by the correct label identification unit 11 and the position corrected by the correct label correction unit 12 increases. Furthermore, for example, if an image includes multiple objects, the weighting unit 13 may set a weight based on a statistical value (addition value, multiplication value, weighted addition value, etc.) of the differences between the positions of the multiple objects.

[0056] (Assigning Weights According to Differences in Time Periods During which Actions are Performed) The machine learning model MM may be, for example, a behavior recognition model. Here, the correct label may include, for example, a time period during which an object included as a subject in the image data performs a specific action. Then, the weighting unit 13 may set a weight to be assigned to the image data according to, for example, the difference between the time period identified by the correct label identification unit 11 and the time period corrected by the correct label correction unit 12.

[0057] For example, the weighting unit 13 may set a larger weight to be assigned to the image data as the difference between the time period identified by the correct label identification unit 11 and the time period corrected by the correct label correction unit 12 increases.

[0058] (Flow of information processing method S1A executed by information processing device 1A) The flow of information processing method S1A executed by information processing device 1A will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of information processing method S1A. In addition to the correct label identification process (step) S11, correct label correction process (step) S12, and weighting process (step) S13 included in information processing method S1, information processing method S1A includes a learning process (step) S14 and a recognition process (step) S15.

[0059] (Step S14) In step S14, the learning unit 14 may, for example, refer to a plurality of image data weighted by the weighting unit 13 and perform machine learning of the image recognition model VM.

[0060] (Step S15) In step S15, the recognition unit 15 may recognize a specific object included as a subject in the image data or a specific action of an object included as a subject in the image data, for example, by using the image recognition model VM machine-learned by the learning unit 14.

[0061] (Flow of Information Processing Method S1B Executed by Information Processing Device 1A) The flow of information processing method S1B executed by information processing device 1A will be described with reference to FIG. 6. FIG. 6 is a flowchart showing the flow of information processing method S1B. Information processing method S1B includes the correct label identification process (step) S11, the correct label correction process (step) S12, the weighting process (step) S13, the learning process (step) S14, and the recognition process (step) S15 included in information processing method S1A, but differs from information processing method S1A in that there is a process of returning to the correct label identification process (step) S11 after the learning process (step) S14. Note that it is assumed that the processes of steps S11 to S14 included in information processing method S1A have been performed at least once before the processes in each of the following steps.

[0062] (Step S11 Following Step S14) In step S11 following step S14, the correct label identification unit 11 may, for example, replace the machine learning model MM with the image recognition model VM that has been machine learned by the learning unit 14. Here, the correct label identification unit 11 may, for example, identify the correct label to be assigned to each of the multiple image data using the machine learning model MM that has been replaced with the image recognition model VM.

[0063] (Step S12) In step S12, the correct label correction unit 12 may correct the correct label to be assigned to image data for which the correct label identified by the correct label identification unit 11 is inappropriate, for example, among the multiple image data.

[0064] (Step S13) In step S13, the weighting unit 13 may, for example, assign a weight to each of the plurality of image data.

[0065] (Step S14) In step S14, the learning unit 14 may refer to the plurality of image data weighted by the weighting unit 13 and train the image recognition model VM by machine learning.

[0066] (Repetition of Steps S11 to S14) The information processing device 1A may repeatedly execute a cycle including the processes in the above-described steps S11 to S14 in the information processing method S1B, for example.

[0067] (Exemplary Expression of Correct Labels and Weights for Objects) Fig. 7 is a diagram showing exemplary expressions of correct labels and weights for objects included as subjects in image data. In the example of Fig. 7, correct labels and weights assigned to objects included as subjects in image data are expressed as annotation classes. Note that the exemplary annotation classes in Fig. 7 include labels indicating specific object classes and information regarding the positions of the objects.

[0068] In the example of the annotation class in Figure 7, for an object with an object serial number "id" of 1, the image data name "image_name" is "image001.png", the position "bbox" of the object is [x1, y1, x2, y2] (a rectangular range between x1 and x2 on the x coordinate and between y1 and y2 on the y coordinate in image001.png), the code "category_id" representing the category of the object is 1 (a code representing a person), and the weight "weight" is 1.

[0069] In the example of the annotation class in Figure 7, for the object with the object serial number "id" of 2, the image data name "image_name" is "image001.png", the position "bbox" of the object is [x3, y3, x4, y4] (a rectangular range between x3 and x4 on the x coordinate and between y3 and y4 on the y coordinate in image001.png), the code "category_id" representing the category of the object is 1 (a code representing a person), and the weight "weight" is 1.

[0070] (Exemplary Expression of Correct Labels and Weights Related to Actions) Fig. 8 is a diagram showing an exemplary expression of correct labels and weights related to actions of an object included as a subject in image data. In the example of Fig. 8, the correct labels and weights assigned to actions of an object included as a subject in image data representing a moving image are expressed as annotation classes. Note that the exemplary annotation classes in Fig. 8 include a label indicating a specific action class and information related to the time period during which the action is performed.

[0071] In the example of the annotation class in Figure 8, for an object with an object serial number "id" of 1, the image data name "image_name" is "image002.mp4", the time period "duration" during which the action is performed is [f1, f2] (the range between the f1th frame and the f2th frame in image002.mp4), the code "action_id" indicating what the action is is 1 (a code indicating "at work"), and the weight "weight" is 1.

[0072] (Example of a method for assigning weights according to the degree of deviation of an object) FIG. 9 is a diagram showing an example of a method for assigning weights according to the degree of deviation of the position of an object included as a subject in image data. As shown in the example of FIG. 9 , assuming that: Solid line rectangle: appropriate object position Dotted line rectangle: example 1 of object position identified by the correct label identification unit 11 Dashed line rectangle: example 2 of object position identified by the correct label identification unit 11, the deviation amount between the appropriate object position and the above-mentioned object position example 1 may be calculated, for example, by the formula shown in FIG. 9 . In the example of FIG. 9 , if the deviation amount between the appropriate object position and the above-mentioned object position example 2 is calculated in the same way, the deviation amount between the appropriate object position and the above-mentioned object position example 2 will be larger than the deviation amount between the appropriate object position and the above-mentioned object position example 1.

[0073] In addition, the weighting unit 13 may set the threshold value for the amount of deviation in advance in several stages, for example, if 0≦amount of deviation<0.5, weight=2, if 0.5≦amount of deviation<0.8, weight=3, otherwise weight=4, and set the weight to change depending on the amount of deviation.

[0074] (Effects of Information Processing Device 1A) As described above, information processing device 1A employs a configuration further including a learning means for performing machine learning on an image recognition model by referring to the plurality of image data weighted by the weighting means. Therefore, in addition to the effects achieved by information processing device 1, information processing device 1A can achieve the effect of further improving the recognition accuracy of the image recognition model by performing machine learning on an image recognition model that places more emphasis on image data that is difficult to recognize by existing machine learning models.

[0075] Furthermore, the information processing device 1A is configured to further include a recognition unit that uses an image recognition model machine-learned by the learning unit to recognize a specific object included as a subject in the image data or a specific action of an object included as a subject in the image data. Therefore, in addition to the effects of the information processing device 1, the information processing device 1A can achieve the effect of being able to recognize an object or action of an object included in the image data with higher accuracy.

[0076] Furthermore, in the information processing device 1A, the machine learning model is an object recognition model, the correct label includes the position of a specific object included as a subject in the image data, and the weighting means sets the weight to be assigned to the image data according to the difference between the position identified by the correct label identifying means and the position corrected by the correct label correcting means. Therefore, in addition to the effects achieved by the information processing device 1, the information processing device 1A has the effect of being able to set the weight to be assigned to the image data according to the deviation in the position where the object is recognized.

[0077] Furthermore, in the information processing device 1A, the machine learning model is a behavior recognition model, the correct label includes a time period in which an object included as a subject in the image data performs a specific action, and the weighting means sets a weight to be assigned to the image data according to the difference between the time period identified by the correct label identifying means and the time period corrected by the correct label correcting means. Therefore, in addition to the effects achieved by the information processing device 1, the information processing device 1A has the effect of being able to set a weight to be assigned to the image data according to the difference in the time period in which the action was recognized.

[0078] The information processing device 1A is configured to repeatedly execute a cycle including: (1) replacing the machine learning model with an image recognition model trained by a learning means; (2) identifying a correct label by a correct label identification means using the machine learning model replaced with the image recognition model to identify a correct label to be assigned to each of a plurality of image data; (3) correct label correction means correcting a correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification means is inappropriate; (4) weighting means assigning a weight to each of the plurality of image data; and (5) learning means training the image recognition model by machine learning, with reference to the plurality of image data weighted by the weighting means. Thus, the information processing device 1A can achieve the effect of assigning a higher weight to image data to which an inappropriate correct label is repeatedly assigned.

[0079] [Example of Software Implementation] Some or all of the functions of the information processing devices 1 and 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0080] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0081] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0082] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0083] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0084] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0085] [Appendix 1] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0086] (Supplementary Note 1) An information processing apparatus comprising: a correct label identification means that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction means that corrects the correct label to be assigned to image data of the plurality of image data for which the correct label identified by the correct label identification means is inappropriate; and a weight assignment means that assigns a weight to each of the plurality of image data, wherein the weight assignment means sets a weight to be assigned to image data to which the correct label corrected by the correct label correction means has been assigned that is greater than a weight to be assigned to image data to which the correct label identified by the correct label identification means has been assigned.

[0087] (Supplementary Note 2) The information processing device according to Supplementary Note 1, further comprising: a learning unit that performs machine learning on an image recognition model by referring to the plurality of image data weighted by the weighting unit.

[0088] (Supplementary Note 3) The information processing device according to Supplementary Note 2, further comprising a recognition means for recognizing a specific object included as a subject in the image data, or a specific action of an object included as a subject in the image data, using the image recognition model machine-learned by the learning means.

[0089] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the correct label correction means comprises: a presentation means for presenting to a user all or part of the plurality of image data together with the correct label identified by the correct label identification means; and a correct label change means for changing, based on the user's operation, the correct label to be assigned to image data specified by the user among the image data presented by the presentation means, to the correct label specified by the user.

[0090] (Appendix 5) The information processing device described in Appendix 4 is characterized in that the presentation means classifies all or part of the multiple image data so that similar image data are included in the same image data group and presents them to the user, and the correct label change means changes the correct label to be assigned to image data belonging to each image data group to a correct label specified by the user.

[0091] (Appendix 6) The information processing device described in Appendix 5, characterized in that when the correct label changing means changes the correct label to be assigned to first image data specified by a user to the correct label specified by the user, it changes the correct label to be assigned to second image data among the plurality of image data that is similar to the first image data to the same correct label as the correct label assigned to the first image data.

[0092] (Supplementary Note 7) The information processing device according to Supplementary Note 4, wherein the correct label identification means further calculates an accuracy indicating the accuracy of the correct label identified using the machine learning model, and the presentation means selects image data to be presented to a user by referring to the accuracy calculated by the correct label identification means.

[0093] (Appendix 8) The information processing device described in Appendix 3, characterized in that: the machine learning model is an object recognition model; the correct label includes the position of a specific object included as a subject in the image data; and the weighting means sets the weight to be assigned to the image data according to the difference between the position identified by the correct label identification means and the position corrected by the correct label correction means.

[0094] (Supplementary Note 9) The information processing device described in Supplementary Note 3, characterized in that the machine learning model is a behavior recognition model, the correct label includes a time period during which an object included as a subject in the image data performs a specific action, and the weighting means sets a weight to be assigned to the image data according to a difference between the time period identified by the correct label identifying means and the time period corrected by the correct label correcting means.

[0095] (Supplementary Note 10) The information processing device according to any one of Supplementary Notes 2, 3, 8, and 9, characterized in that it repeatedly executes a cycle including: (1) a process of replacing the machine learning model with the image recognition model machine-learned by the learning means; (2) a process by the correct label identification means of identifying a correct label to be assigned to each of the plurality of image data using the machine learning model replaced by the image recognition model; (3) a process by the correct label correction means of correcting a correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification means is inappropriate; (4) a process by the weighting means of assigning a weight to each of the plurality of image data; and (5) a process by the learning means of machine learning an image recognition model by referring to the plurality of image data to which weights have been assigned by the weighting means.

[0096] (Supplementary Note 11) An information processing method including: a correct label identification process that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction process that corrects the correct label to be assigned to image data of the plurality of image data for which the correct label identified by the correct label identification process is inappropriate; and a weight assignment process that assigns a weight to each of the plurality of image data, wherein the weight assignment process sets a weight to be assigned to image data to which the correct label corrected by the correct label correction process has been assigned greater than a weight to be assigned to image data to which the correct label identified by the correct label identification process has been assigned.

[0097] (Supplementary Note 12) The information processing method according to Supplementary Note 11, further comprising a learning process for machine learning an image recognition model by referring to the plurality of image data to which the weighting process has been applied.

[0098] (Supplementary Note 13) The information processing method according to Supplementary Note 12, further comprising a recognition process for recognizing a specific object included as a subject in the image data, or a specific action of an object included as a subject in the image data, using the image recognition model machine-learned by the learning process.

[0099] (Supplementary Note 14) The information processing method described in any one of Supplementary Notes 11 to 13, characterized in that the correct label correction process includes: a presentation process that presents all or part of the plurality of image data to a user together with the correct labels identified by the correct label identification process; and a correct label change process that, based on a user operation, changes the correct label to be assigned to image data specified by the user among the image data presented by the presentation process to the correct label specified by the user.

[0100] (Appendix 15) The information processing method described in Appendix 14, characterized in that the presentation process classifies all or part of the multiple image data so that similar image data are included in the same image data group and presents them to the user, and the correct label change process changes the correct label to be assigned to image data belonging to each image data group to a correct label specified by the user.

[0101] (Appendix 16) The information processing method described in Appendix 15, characterized in that, when the correct label change process changes the correct label to be assigned to first image data specified by a user to the correct label specified by the user, the correct label to be assigned to second image data among the plurality of image data that is similar to the first image data is changed to the same correct label as the correct label to be assigned to the first image data.

[0102] (Supplementary Note 17) The information processing method according to Supplementary Note 14, wherein the correct label identification process further calculates an accuracy indicating the accuracy of the correct label identified using the machine learning model, and the presentation process selects image data to be presented to a user by referring to the accuracy calculated by the correct label identification process.

[0103] (Supplementary Note 18) The information processing method described in Supplementary Note 13, characterized in that: the machine learning model is an object recognition model; the correct label includes the position of a specific object included as a subject in the image data; and the weighting process sets a weight to be assigned to the image data according to the difference between the position identified by the correct label identification process and the position corrected by the correct label correction process.

[0104] (Supplementary Note 19) The information processing method described in Supplementary Note 13, characterized in that: the machine learning model is a behavior recognition model; the correct label includes a time period during which an object included as a subject in the image data performs a specific action; and the weighting process sets a weight to be assigned to the image data according to a difference between the time period identified by the correct label identification process and the time period corrected by the correct label correction process.

[0105] (Supplementary Note 20) The information processing method according to any one of Supplementary Notes 12, 13, 18, and 19, characterized in that the method repeatedly executes a cycle including: (1) a process of replacing the machine learning model with the image recognition model trained by machine learning through the learning process; (2) a process of the correct label identification process of identifying a correct label to be assigned to each of the plurality of image data using the machine learning model replaced with the image recognition model; (3) a process of the correct label correction process of correcting a correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification process is inappropriate; (4) a process of the weighting process of assigning a weight to each of the plurality of image data; and (5) a process of the learning process of machine learning an image recognition model by referring to the plurality of image data to which weights have been assigned by the weighting process.

[0106] (Supplementary Note 20) A program for causing a computer to operate as the information processing device according to any one of Supplementary Notes 1 to 10, the program causing the computer to function as each of the means.

[0107] [Appendix 2] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0108] (Supplementary Note 1) An information processing device comprising at least one processor, wherein the at least one processor executes: a correct label identification process that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction process that corrects a correct label to be assigned to image data of the plurality of image data for which a correct label identified by the at least one processor in the correct label identification process is inappropriate; and a weight assignment process that assigns a weight to each of the plurality of image data, wherein the weight assignment process sets a weight to be assigned to image data to which the correct label corrected by the at least one processor in the correct label correction process is assigned greater than a weight to be assigned to image data to which the correct label identified by the at least one processor in the correct label identification process is assigned.

[0109] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0110] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the at least one processor further executes a learning process to machine-train an image recognition model by referring to the plurality of image data weighted by the weighting process.

[0111] (Supplementary Note 3) The information processing device described in Supplementary Note 2, characterized in that the at least one processor further performs a recognition process to recognize a specific object included as a subject in the image data or a specific action of an object included as a subject in the image data using the image recognition model machine-learned by the learning process.

[0112] (Supplementary Note 4) The information processing device described in any one of Supplementary Notes 1 to 3, characterized in that in the correct label correction process, the at least one processor executes: a presentation process that presents all or part of the plurality of image data to a user together with the correct label identified by the at least one processor in the correct label identification process; and a correct label change process that, based on a user operation, changes the correct label to be assigned to image data specified by the user among the image data presented by the at least one processor in the presentation process to the correct label specified by the user.

[0113] (Appendix 5) The information processing device described in Appendix 4, characterized in that in the presentation process, the at least one processor classifies all or part of the multiple image data so that similar image data are included in the same image data group and presents them to the user, and in the correct label change process, the at least one processor changes the correct label to be assigned to image data belonging to each image data group to a correct label specified by the user.

[0114] (Appendix 6) The information processing device described in Appendix 5, characterized in that, in the correct label change process, when the at least one processor changes the correct label to be assigned to first image data specified by a user to the correct label specified by the user, it changes the correct label to be assigned to second image data among the multiple image data that is similar to the first image data to the same correct label as the correct label assigned to the first image data.

[0115] (Supplementary Note 7) In the correct label identification process, the at least one processor further calculates an accuracy indicating the accuracy of the correct label identified using the machine learning model, and in the presentation process, the at least one processor selects image data to be presented to a user by referring to the accuracy calculated by the at least one processor in the correct label identification process. The information processing device described in Supplementary Note 4 is characterized in that

[0116] (Appendix 8) The information processing device described in Appendix 3, characterized in that: the machine learning model is an object recognition model; the correct label includes the position of a specific object included as a subject in the image data; and in the weighting process, the at least one processor sets the weight to be assigned to the image data according to the difference between the position identified by the at least one processor in the correct label identification process and the position corrected by the at least one processor in the correct label correction process.

[0117] (Supplementary Note 9) The information processing device described in Supplementary Note 3, characterized in that the machine learning model is a behavior recognition model, the correct label includes a time period during which an object included as a subject in the image data performs a specific action, and in the weighting process, the at least one processor sets a weight to be assigned to the image data according to a difference between the time period identified by the at least one processor in the correct label identification process and the time period corrected by the at least one processor in the correct label correction process.

[0118] (Supplementary Note 10) The at least one processor repeatedly executes a cycle including: (1) a process of replacing the machine learning model with the image recognition model trained by the at least one processor in the learning process; (2) a process of the at least one processor identifying a correct label to be assigned to each of the plurality of image data using the machine learning model replaced with the image recognition model in the correct label identification process; (3) a process of the at least one processor correcting a correct label to be assigned to image data for which the correct label identified by the at least one processor in the correct label identification process is inappropriate among the plurality of image data; (4) a process of the at least one processor assigning a weight to each of the plurality of image data; and (5) a process of the at least one processor training an image recognition model by machine learning, by referring to the plurality of image data to which weights are assigned by the at least one processor in the weight assignment process. 10. The information processing device according to any one of claims 2, 3, 8, and 9.

[0119] 1, 1A Information processing device 11, 121 Correct label identification unit 12 Correct label correction unit 13 Assignment unit 14 Learning unit 15 Recognition unit 121 Presentation unit 122 Correct label change unit C1 Processor C2 Memory

Claims

1. An information processing device comprising: a correct label identification means for identifying a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction means for correcting the correct label to be assigned to image data for which the correct label identified by the correct label identification means is inappropriate among the plurality of image data; and a weight assignment means for assigning a weight to each of the plurality of image data, wherein the weight assignment means sets a weight to be assigned to image data to which the correct label corrected by the correct label correction means has been assigned greater than a weight to be assigned to image data to which the correct label identified by the correct label identification means has been assigned.

2. The information processing device according to claim 1, further comprising a learning means for machine learning an image recognition model by referring to the plurality of image data weighted by the weighting means.

3. The information processing device according to claim 2, further comprising a recognition means for recognizing a specific object included as a subject in the image data, or a specific action of an object included as a subject in the image data, using the image recognition model machine-learned by the learning means.

4. The information processing device of claim 1, wherein the correct label correction means comprises: a presentation means for presenting to a user all or part of the plurality of image data together with a correct label identified by the correct label identification means; and a correct label change means for changing, based on the user's operation, a correct label to be assigned to image data specified by the user among the image data presented by the presentation means, to the correct label specified by the user.

5. The information processing device of claim 4, wherein the presentation means classifies all or part of the multiple image data so that similar image data are included in the same image data group and presents the classified data to the user, and the correct label changing means changes the correct label to be assigned to image data belonging to each image data group to a correct label specified by the user.

6. The information processing device of claim 5, wherein the correct label changing means, when changing the correct label to be assigned to first image data specified by a user to the correct label specified by the user, changes the correct label to be assigned to second image data among the plurality of image data that is similar to the first image data to the same correct label as the correct label assigned to the first image data.

7. The information processing device according to claim 4, wherein the correct label identification means further calculates an accuracy indicating the accuracy of the correct label identified using the machine learning model, and the presentation means selects image data to be presented to a user by referring to the accuracy calculated by the correct label identification means.

8. The information processing device of claim 3, wherein the machine learning model is an object recognition model, the correct label includes a position of a specific object included as a subject in the image data, and the weighting means sets a weight to be assigned to the image data in accordance with a difference between the position identified by the correct label identifying means and the position corrected by the correct label correcting means.

9. The information processing device of claim 3, wherein the machine learning model is a behavior recognition model, the correct label includes a time period during which an object included as a subject in the image data performs a specific action, and the weighting means sets a weight to be assigned to the image data in accordance with a difference between the time period identified by the correct label identification means and the time period corrected by the correct label correction means.

10. An information processing device as described in claim 2, which repeatedly executes a cycle including: (1) a process of replacing the machine learning model with the image recognition model trained by the learning means; (2) a process by the correct label identification means of identifying a correct label to be assigned to each of the plurality of image data using the machine learning model replaced by the image recognition model; (3) a process by the correct label correction means of correcting a correct label to be assigned to image data among the plurality of image data for which the correct label identified by the correct label identification means is inappropriate; (4) a process by the weighting means of assigning a weight to each of the plurality of image data; and (5) a process by the learning means of machine learning an image recognition model by referring to the plurality of image data to which weights have been assigned by the weighting means.

11. An information processing method comprising: a correct label identification process that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction process that corrects the correct label to be assigned to image data among the plurality of image data, for which the correct label identified by the correct label identification process is inappropriate; and a weight assignment process that assigns a weight to each of the plurality of image data, wherein the weight assignment process sets a weight to be assigned to image data to which the correct label corrected by the correct label correction process has been assigned greater than a weight to be assigned to image data to which the correct label identified by the correct label identification process has been assigned.

12. A program that causes a computer to execute: a correct label identification process that identifies a correct label to be assigned to each of a plurality of image data using a machine learning model; a correct label correction process that corrects the correct label to be assigned to image data among the plurality of image data, for image data for which the correct label identified by the correct label identification process is inappropriate; and a weight assignment process that assigns a weight to each of the plurality of image data, wherein the weight assignment process sets a weight to be assigned to image data to which the correct label corrected by the correct label correction process has been assigned greater than a weight to be assigned to image data to which the correct label identified by the correct label identification process has been assigned.

Images