Information processing device, information processing method, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program capable of easily improving recognition accuracy of a recognition model. The present invention involves: evaluating a recognition rate of a recognition model for verification data composed of a label and an image generated on the basis of a plurality of conditions; specifying a condition of a low recognition rate lower than a predetermined recognition rate in the recognition model; generating low recognition rate data, composed of a label and an image, of the recognition model on the basis of the specified low recognition rate condition; and repeatedly training the recognition model by using the generated low recognition rate data as training data until the low recognition rate condition is resolved. The present invention can be applied to a technique for improving the recognition rate of a recognition model.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable easy improvement of the recognition rate of a recognition model.

[0002] Technology that uses machine learning to generate recognition models that recognize the type and state of people and objects based on image information is becoming widespread.

[0003] Recognition models are generated by machine learning using training data, but they may not be able to learn enough for certain scenes, resulting in insufficient recognition rates.

[0004] For this reason, training is often required to improve the recognition rate for images of specific scenes with low recognition rates.

[0005] Therefore, a technology has been proposed to improve the recognition rate by extracting images similar to images of specific scenes with recognition rates lower than a predetermined value from the verification images used to verify the recognition rate of a recognition model and using them for learning (see Patent Document 1).

[0006] In addition, a technology has been proposed that collects images actually taken in a multinational environment, extracts images from the collected images that are similar to images of specific scenes with low recognition rates, and uses these images for learning to improve the recognition rate (see Patent Document 2).

[0007] JP 2019-109924 A JP 2019-021201 A

[0008] However, in the techniques disclosed in Patent Documents 1 and 2, the extracted images must be manually labeled, and the greater the number of extracted images, the greater the labor and burden involved in labeling.

[0009] Furthermore, in the technologies disclosed in Patent Documents 1 and 2, the images of scenes with low recognition rates used for learning are images generated in advance as verification images or images extracted from images aggregated in advance, so there is a limit to the number of images that can be used for learning. As a result, it is not possible to secure the number of images for learning that is actually required, and there is a possibility that the recognition rate will not be sufficiently improved.

[0010] The present disclosure has been made in consideration of such circumstances, and in particular, easily improves the recognition accuracy of a recognition model by generating images of scenes whose recognition rate in the recognition model is lower than a predetermined value, along with labels.

[0011] An information processing device and program according to one aspect of the present disclosure are an information processing device including an evaluation unit that evaluates the recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions and identifies conditions for a low recognition rate that is lower than a predetermined recognition rate for the recognition model, and a low recognition rate data generation model that generates low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for a low recognition rate, and a program that causes a computer to function as the evaluation unit and the low recognition rate data generation model.

[0012] An information processing method according to one aspect of the present disclosure is an information processing method including the steps of: evaluating the recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions; identifying conditions for a low recognition rate that is lower than a predetermined recognition rate for the recognition model; and generating low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for a low recognition rate.

[0013] In one aspect of the present disclosure, the recognition rate of a recognition model for verification data consisting of images and labels generated based on multiple conditions is evaluated, conditions with a low recognition rate lower than a predetermined recognition rate for the recognition model are identified, and low recognition rate data for the recognition model consisting of the images and the labels is generated based on the identified conditions with a low recognition rate.

[0014] FIG. 1 is a diagram illustrating an overview of the present disclosure. FIG. 1 is a diagram illustrating a configuration example of a first embodiment of a recognition model improvement system of the present disclosure. FIG. 2 is a diagram illustrating functions realized by the recognition model improvement system. FIG. 3 is a diagram illustrating a verification data generation model and a low recognition rate data generation model. FIG. 4 is a diagram illustrating a model of the present disclosure to which Unidiffuser is applied. FIG. 5 is a diagram illustrating a method for evaluating a recognition model by a recognition model evaluation unit. FIG. 6 is a diagram illustrating examples of images and labels generated when the condition for a low recognition rate is a label. FIG. 7 is a diagram illustrating examples of images and labels generated when the condition for a low recognition rate is an image. FIG. 8 is a diagram illustrating examples of images and labels generated when the condition for a low recognition rate is text. FIG. 9 is a flowchart illustrating recognition model improvement processing by the recognition model improvement system of FIG. 3. FIG. 10 is a diagram illustrating a configuration example of a second embodiment of a recognition model improvement system of the present disclosure. FIG. 11 is a diagram illustrating a configuration example of a third embodiment of the recognition model improvement system of the present disclosure. FIG. 12 is a diagram illustrating an application example of the present disclosure. FIG. 13 is a diagram illustrating an example of the configuration of a general-purpose computer.

[0015] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0016] Hereinafter, embodiments for carrying out the present technology will be described. The description will be made in the following order: 1. Overview of the present disclosure 2. First embodiment 3. Second embodiment 4. Third embodiment 5. Application example 6. Example of execution by software

[0017] <<1. Overview of the Present Disclosure>> The present disclosure particularly aims to easily improve the recognition rate of a recognition model by generating, together with labels, images of scenes for which the recognition rate in the recognition model is lower than a predetermined value.

[0018] First, an overview of the present disclosure will be described. Technology for generating a recognition model by machine learning, such as identifying people or objects in an image or recognizing their states, based on image information, is widely used.

[0019] Recognition models are generated by machine learning using training data, but because there are limitations on the types of images that can be used as training data, they may not be able to learn sufficiently for images under certain conditions (scenes), resulting in a recognition rate (the success rate of recognition in the recognition process using the recognition model) lower than a specified value and preventing proper recognition.

[0020] For this reason, it is often necessary to perform learning (relearning) to improve the recognition accuracy for images under specific conditions (scenes) where the recognition rate is low.

[0021] Therefore, a technology has been proposed that improves recognition accuracy by extracting images similar to images of specific scenes with a recognition rate lower than a predetermined value, labeling them as correct answers, and allowing them to learn.

[0022] However, in this case, the correct labels must be manually assigned to the extracted images, and the greater the number of extracted images, the greater the effort and burden involved in labeling.

[0023] Furthermore, images of scenes with low recognition rates are images extracted from pre-generated images or pre-aggregated images, and are not newly created images. Therefore, the number of images actually required to improve the recognition rate cannot be secured, and the recognition rate may not be sufficiently improved.

[0024] For this reason, in the present disclosure, by applying existing image generation technology that generates images by specifying text or images, an image generation unit is configured that generates images by associating them with labels based on conditions, labels, etc.

[0025] Furthermore, by using the image generation unit of the present disclosure described above, the process of steps S1 to S3 shown in FIG. 1 is repeated to improve the recognition rate of the recognition model.

[0026] 1, first, in step S1, the image generation unit generates images of various conditions (scenes) by associating them with labels. Then, using the images generated by the image generation unit, a recognition process is executed using a recognition model, and the recognition results are compared with the corresponding labels to obtain a recognition rate, and conditions (scenes) with a lower recognition rate than a predetermined rate are identified.

[0027] Here, for example, if a user uses a recognition model in advance to identify conditions (scenes) with a recognition rate lower than a predetermined recognition rate, an image of the pre-identified conditions (scenes) may be generated.

[0028] Furthermore, even if a recognition model is used in advance by a user, there may be conditions (scenes) where the recognition rate is lower than a predetermined recognition rate that the user is unaware of. For this reason, in the initial process, images of a wide variety of conditions (scenes), not just the conditions (scenes) specified in advance, may be generated together with the corresponding labels.

[0029] Next, in step S2, the image generating unit generates a large number of images and labels that satisfy conditions (scenes) lower than the specified recognition rate.

[0030] Then, in the process of step S3, the recognition model is trained using images and labels that are in conditions (scenes) that are lower than the specified recognition rate.

[0031] After the process of step S3, the process returns to step S1, and the processes of steps S1 to S3 are repeated until a condition (scene) with a lower recognition rate than the predetermined recognition rate is not specified (does not exist).

[0032] Furthermore, in the second or subsequent processing of step S1, among images of conditions (scenes) for which the recognition rate is known to be lower than a predetermined value in the initial processing, conditions (scenes) for which the recognition rate is also lower than a predetermined value even in the recognition model after learning may be identified.

[0033] By performing the above process, it is possible to identify conditions (scenes) with low recognition rates, generate images and labels for the identified conditions (scenes) with low recognition rates, and repeat the learning process using the generated images and labels for the conditions (scenes) with low recognition rates until the recognition rate becomes higher than a predetermined value, thereby improving the recognition rate of the recognition model.

[0034] In this case, the image generation unit of the present disclosure generates images and labels for conditions (scenes) where the recognition rate is lower than a predetermined value, thereby reducing the effort and cost of manually assigning labels to each image.

[0035] Furthermore, since images and labels for various conditions (scenes) can be generated while reducing the effort and cost, it becomes possible to easily generate a large amount of training data. As a result, it becomes possible to easily and appropriately identify conditions (scenes) for which the recognition rate is lower than a predetermined value based on the recognition results of the recognition model when images and labels for various conditions (scenes) are used.

[0036] Furthermore, since it is possible to generate a large number of images and labels for conditions (scenes) where the identified recognition rate is lower than a predetermined value, it becomes possible to easily prepare a sufficient number of images and labels required to raise the recognition rate of the recognition model above a predetermined value, and as a result, it becomes possible to improve the recognition rate of the recognition model with high accuracy.

[0037] <<2. First Embodiment>> Next, with reference to FIG. 2, a configuration example of a recognition model improvement system to which the technology of the present disclosure is applied will be described.

[0038] 2 is composed of smartphones 31-1 to 31-n, a cloud server 32, and a network 33. The smartphones 31-1 to 31-n and the cloud server 32 are configured to be able to communicate with each other via the network 33, which is represented by a public line, the Internet, or the like, and are configured to be able to exchange, for example, data and programs with each other.

[0039] In addition, when there is no particular need to distinguish between the smartphones 31-1 to 31-n in FIG. 2, they will be simply referred to as smartphones 31, and the same will be used for other components.

[0040] The smartphone 31 is carried by the user and has a so-called recognition function that identifies people or objects that are subjects in images captured by a camera (not shown) or images supplied from other smartphones 31 via the network 33, or recognizes their type or status.

[0041] The recognition function for people or objects that are the subject of an image here refers to a function for identifying individuals or their status based on images of people or facial images that are the subject of an image, or a function for identifying the type or status of an object that is the subject of an image.

[0042] More specifically, the smartphone 31 includes a recognition model improvement request unit 51 and a recognition model 52 .

[0043] The recognition model 52 is configured by machine learning and includes a DNN (Deep Neural Network) and has the above-mentioned recognition functions, such as identifying people or objects in an image based on the image, or recognizing their type or state.

[0044] When using the recognition model 52, if a user reports that the recognition rate for an image under a specific condition (scene) feels low to the user, or if the recognition rate calculated from the actual recognition result is lower than a predetermined value, the recognition model improvement request unit 51 requests an improvement from the cloud server 32 via the network 33.

[0045] When the cloud server 32 receives a request from the smartphone 31 to improve the recognition model 52, it identifies the conditions (scenes) in which the recognition rate in the recognition model 52 is low, and re-trains the recognition model to improve the recognition rate under the identified conditions, thereby updating the recognition model 52 of the smartphone 31 for which the improvement request was made.

[0046] More specifically, the cloud server 32 includes a recognition model improvement processing unit 71 and a recognition model update unit 72 .

[0047] When the recognition model improvement processing unit 71 receives a request to improve the recognition model 52 from the smartphone 31, it identifies conditions (scenes) that result in a low recognition rate in the recognition model 52. More specifically, the recognition model improvement processing unit 71 generates images and labels for various conditions (scenes).

[0048] In addition, the recognition model improvement processing unit 71 is equipped with a recognition model similar to the recognition model 52 built into the smartphone 31, and uses the generated images and labels of various conditions (scenes) to calculate the recognition rate of the recognition model 52, and identifies conditions (scenes) with a low recognition rate from the recognition rate results.

[0049] Then, the recognition model improvement processing unit 71 generates images and labels for the identified conditions (scenes) with a low recognition rate, and re-trains the recognition model 52 using the generated images and labels for the identified conditions (scenes) with a low recognition rate.

[0050] In addition, the recognition model improvement processing unit 71 uses images and labels of the identified conditions (scenes) with a low recognition rate to obtain recognition results using the re-trained recognition model, and determines from the recognition results whether the decline in the recognition rate of the recognition model has improved.

[0051] Furthermore, if it is determined from the recognition result that the decline in the recognition rate of the recognition model has not been improved, the same process is repeated until the recognition rate exceeds a predetermined value.

[0052] Then, if it is determined from the recognition results that the decline in the recognition rate of the recognition model has been improved, the recognition model update unit 72 updates the recognition model 52 of the smartphone 31 for which an improvement request was made via the network 33 with a recognition model in which the decline in the recognition rate has been improved.

[0053] <Functions Realized by the Recognition Model Improvement System in Fig. 2> Next, functions realized by the recognition model improvement system in Fig. 2 will be described with reference to Fig. 3. Note that, in particular, functions realized by the cloud server 32 will be described here.

[0054] As described above, the cloud server 32 includes the recognition model improvement processing unit 71 and the recognition model update unit 72 .

[0055] Of these, the recognition model improvement processing unit 71 comprises a condition generating unit 91 , a verification data generating model 92 , a recognition model 93 , a recognition model evaluating unit 94 , a low recognition rate data generating model 95 , and a learning unit 96 .

[0056] When the condition generating unit 91 receives a request for improving the recognition rate of the recognition model 52 from the smartphone 31, the condition generating unit 91 generates conditions for generating verification data in the verification data generating model 92 at the subsequent stage.

[0057] If the improvement request supplied from the recognition model improvement request unit 51 is, for example, a request to improve a decline in recognition rate under specific conditions, the condition generation unit 91 supplies the specific conditions included in the improvement request to the verification data generation model 92.

[0058] Furthermore, if the improvement request supplied from the recognition model improvement request unit 51 is a request to improve the overall decline in recognition rate, the condition generation unit 91 generates various conditions to improve the overall decline in recognition rate and supplies them to the verification data generation model 92.

[0059] The verification data generation model 92 generates pairs of images serving as verification data and corresponding labels serving as correct answers based on the conditions supplied from the condition generation unit 91, and supplies these to the recognition model 93. The verification data generation model 92 will be described in detail later.

[0060] The recognition model 93 is, for example, a deep neural network (DNN) generated based on machine learning performed by the learning unit 96 , and is ultimately updated as the recognition model 52 of the smartphone 31 .

[0061] The recognition model 93 recognizes the image based on a pair of the image serving as verification data supplied by the verification data generation model 92 and its correct label, and outputs the recognition result together with the correct label to the recognition model evaluation unit 94.

[0062] It should be noted that, before an improvement request is received, the recognition model 93 is assumed to be completely identical to the recognition model 52 of the smartphone 31. However, when an improvement request is received, the recognition model 93 is improved, and when it is confirmed that the recognition rate has been sufficiently improved, the recognition model 93 is updated to become the recognition model 52. Therefore, the recognition model 93 is not identical to the recognition model 52 until it is updated upon receiving an improvement request.

[0063] The recognition model evaluation unit 94 evaluates the recognition model 93 by comparing the recognition results of the recognition model 93 with the labels using pairs of images and labels as verification data. The recognition model evaluation unit 94 identifies conditions of low recognition rates from the evaluation results of the recognition model 93 where the recognition rate is lower than a predetermined threshold, and outputs the conditions to the low recognition rate data generation model 95.

[0064] The low recognition rate data generation model 95 generates pairs of images that will become low recognition rate data and their correct labels based on the low recognition rate conditions supplied from the recognition model evaluation unit 94, and supplies these to the learning unit 96.

[0065] The low recognition rate data generation model 95 is basically the same as the verification data generation model 92, but differs in that when generating image and label data, the image and label are generated as low recognition rate data based on low recognition rate conditions identified based on the evaluation results of the recognition model evaluation unit 94.

[0066] The low recognition rate data generation model 95 will be described in detail later together with the verification data generation model 92.

[0067] The learning unit 96 uses pairs of images that are low recognition rate data and their correct labels supplied by the low recognition rate data generation model 95 to train (re-train) the recognition model 93, thereby improving the decline in recognition rate through learning.

[0068] After the recognition model 93 has been trained by the training unit 96, the recognition model 93 again executes the recognition process using the verification data, and the recognition model evaluation unit 94 identifies conditions of low recognition rate from the recognition results. If conditions of low recognition rate exist and are identified from the recognition results of the recognition model 93, it means that the improvement of the recognition model 93 is not sufficient, so the identified conditions of low recognition rate are output to the low recognition rate data generation model 95, which generates low recognition rate data, and the training unit 96 re-trains the recognition model 93 again based on the low recognition rate data.

[0069] That is, the same process is repeated until the recognition model evaluation unit 94 no longer extracts conditions that result in a low recognition rate from the recognition results of the recognition model 93, that is, until the deterioration of the recognition rate is improved.

[0070] When no conditions for a low recognition rate are extracted from the recognition results of the recognition model 93, it is considered that the recognition rate of the recognition model 93 has been sufficiently improved. Therefore, the recognition model update unit 72 updates the recognition model 52 of the smartphone 31 using the recognition model 93 with an improved recognition rate.

[0071] <Verification Data Generation Model and Low Recognition Rate Data Generation Model> Next, the verification data generation model 92 and the low recognition rate data generation model 95 will be described in detail.

[0072] The verification data generation model 92 and the low recognition rate data generation model 95 both generate labels corresponding to images based on conditions. In Fig. 4, the verification data generation model 92 and the low recognition rate data generation model 95 are denoted as generation models 92, 95.

[0073] As shown in FIG. 4, the verification data generation model 92 and the low recognition rate data generation model 95 generate pairs of images and corresponding labels based on conditions specified by at least one of various labels consisting of key points, face bounding boxes, and semantic segmentation (seg), as well as text.

[0074] In FIG. 4, only key points KP are specified as conditions, and an example is shown in which the verification data generation model 92 and the low recognition rate data generation model 95 generate an image CP and also generate key points (key points) CKP, a face bounding box (face bbox) CFBB, and a semantic segmentation (seg) CS as corresponding labels.

[0075] 4, only key points KP are specified as conditions, but at least one of other text and labels may also be specified. When multiple labels and texts are specified as conditions, multiple conditions are combined. Furthermore, the verification data generation model 92 and the low recognition rate data generation model 95 may generate an image and at least one other label.

[0076] <Method of Generating the Verification Data Generation Model and the Low Recognition Rate Data Generation Model> Next, a method of generating the verification data generation model 92 and the low recognition rate data generation model 95 will be described with reference to FIG.

[0077] The verification data generation model 92 and the low recognition rate data generation model 95 may utilize, for example, an application of a technology called UniDiffuser. UniDiffuser generates images, text, or images and text according to input conditions, as well as generates text from images and images from text.

[0078] For more information about UniDiffuser, see "One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale" by Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu (https: / / arxiv.org / pdf / 2303.06555.pdf).

[0079] The verification data generation model 92 and the low recognition rate data generation model 95 of the present disclosure may be configured to simultaneously generate labels in addition to images and text by, for example, extending UniDiffuser. The term "label" here refers to multiple types of labels, such as bounding boxes (faces, human bodies, etc.), keypoints (human body models), and semantic segmentation.

[0080] More specifically, as shown in FIG. 5, different noise addition intensities (t ｘ , t ｙ , t ｃ ) and a noise-added feature set (x ｔｘ , y ｔｙ , c ｔｃ ) to determine the noise (E θ x , E θ y , E θ c ) is learned.

[0081] The verification data generation model 92 and the low recognition rate data generation model 95 of the present disclosure are formed based on this model M11.

[0082] That is, by learning labels like model M11, it is possible to generate pairs of images that conform to the input conditions and labels that are the correct answer by setting an appropriate noise addition strength according to the input conditions and recursively repeating the noise estimation process.

[0083] Note that model M11 in Figure 5 is configured by adding an additional training partial model M2 (dashed frame in the figure) to the trained partial model M1 (solid frame in the figure) used to construct the original UniDiffuser, in order to adapt it to the labels of the present disclosure.

[0084] That is, the original UniDiffuser is configured to support both text and images by being based on the trained partial model M1.

[0085] In contrast, the verification data generation model 92 and the low recognition rate data generation model 95 of the present disclosure are formed based on a model M11 formed by adding an additional learning partial model M2 corresponding to the label to the learned partial model M1.

[0086] Therefore, when forming model M11, a trained partial model M1 trained on an existing image and text pair is used, and an additional trained partial model M2 corresponding to the label is added, thereby reducing the cost of training.

[0087] However, model M11 may be formed by training that can generate image and label pairs, similar to the case of forming the trained partial model M1 used to train the original UniDiffuser.

[0088] In this specification, an example will be described in which the verification data generation model 92 and the low recognition rate data generation model 95 are models that are an extension of UniDiffuser, so that labels can be generated simultaneously in addition to images and text. However, the verification data generation model 92 and the low recognition rate data generation model 95 are not limited to models that extend UniDiffuser, and may be other models that have similar functions, as long as they have similar functions as models that extend UniDiffuser.

[0089] <Examples of Conditions for a Low Recognition Rate Identified by the Recognition Model Evaluation Unit> Next, specific examples of evaluation by the recognition model evaluation unit 94 will be described with reference to FIG.

[0090] For example, consider a case where the condition generating unit 91 generates conditions consisting of texts "sunny day" and "bad weather day."

[0091] In this case, the verification data generation model 92 generates pairs of images and labels that satisfy the respective conditions of the text "sunny day" and "bad weather day" as verification data, and supplies these to the recognition model 93. Here, the verification data generation model 92 generates multiple pairs of images and labels that satisfy the respective conditions of the text "sunny day" and "bad weather day" as verification data, and supplies these to the recognition model 93.

[0092] The recognition model 93 performs a predetermined recognition process based on images consisting of text that satisfy the conditions of "sunny day" and "bad weather day" as verification data supplied from the verification data generation model 92, and supplies the recognition results together with the corresponding labels to the recognition model evaluation unit 94.

[0093] When the recognition model evaluation unit 94 obtains the recognition result of the recognition model 93 and the correct label, it calculates the recognition rate (recognition success rate) for each condition, identifies conditions lower than a predetermined threshold as conditions with low recognition rates, and outputs the identified conditions with low recognition rates to the low recognition rate data generation model 95.

[0094] FIG. 6 shows an example in which the recognition success rate for images under the condition of a "fine day" is 97%, and the recognition success rate for images under the condition of a "bad weather day" is 65%.

[0095] For example, if the threshold for the recognition success rate is set to 90%, the recognition model evaluation unit 94 identifies the condition "bad weather day" as a condition with a low recognition rate and outputs it to the low recognition rate data generation model 95.

[0096] In addition to the conditions shown in FIG. 6, the recognition model evaluation unit 94 may also set a low recognition rate condition for a specific label when the recognition success rate for that specific label is lower than a threshold value.

[0097] For example, even in an image of a "bad weather day," if there is a key point that allows the subject to be recognized as striking a specific pose, the recognition model evaluation unit 94 may set the label of the key point indicating a "bad weather day" and a specific pose as a condition for a low recognition rate.

[0098] In addition, the recognition model evaluation unit 94 may extract at least one of the labels, text, and images (the images themselves that had a low recognition rate in the evaluation results of the recognition model 93) that are associated with images with a recognition success rate lower than a predetermined value as conditions for a low recognition rate, and output them to the low recognition rate data generation model 95.

[0099] <Example of Generating Low Recognition Rate Data in Low Recognition Rate Data Generation Model> Next, an example of generating low recognition rate data in the low recognition rate data generation model 95 will be described with reference to FIGS.

[0100] (Part 1) For example, consider a case where the condition for a low recognition rate is a text "bad weather day" and a label L1 consisting of a key point that is a specific pose, as shown in FIG.

[0101] The label L1 in FIG. 7 is a key point indicating a pose in which one person is standing side by side on the left side of the figure and another person who is shorter than the first person is standing side by side on the right side.

[0102] The low recognition rate data generation model 95 generates low recognition rate data, such as that shown on the right, based on the conditions for the low recognition rate, and outputs it to the learning section 96 as learning data.

[0103] 7, the low-recognition-rate data generation model 95 generates images P1 to P3 of a bad weather day, in which two people are standing side by side, with the person on the right being shorter than the person on the left, in accordance with a low-recognition-rate condition consisting of text such as "bad weather day" and a label L1 consisting of a key point that is a specific pose. In this case, the low-recognition-rate data generation model 95 also outputs label L1 as a label common to all of images P1 to P3. While FIG. 7 shows an example in which three images are generated, in reality, many more images are generated as required for learning.

[0104] The learning unit 96 uses the low recognition rate data, which combines the images P1 to P3 and the label L1, as learning data to retrain the recognition model 93. This enables the recognition model 93 to improve the recognition success rate for images with conditions similar to those with a low recognition rate, which are composed of text such as "bad weather day" and the label L1 consisting of key points indicating a specific pose.

[0105] (Part 2) In the above, an example has been described in which the condition for a low recognition rate is composed of the text "bad weather day" and the label L1 consisting of a key point that is a specific pose, but the condition for a low recognition rate may also be an image itself whose recognition rate by the recognition model 93 is lower than a threshold. Here, consider the case where the condition for a low recognition rate is image P11, as shown in Figure 8.

[0106] Image P11 in FIG. 8 is an image taken on a rainy day (bad weather day) in which a man is present on the left and a woman is present on the right, standing side by side.

[0107] The low recognition rate data generation model 95 generates low recognition rate data, for example, as shown on the right, based on the image P11 that satisfies the conditions for this low recognition rate, and outputs it to the learning unit 96 as learning data.

[0108] 8, the low-recognition-rate data generation model 95 generates images P21 and P22 of a rainy or snowy day (bad weather day) in which a man is present on the left and a woman is standing next to each other on the right, in accordance with the low-recognition-rate condition of image P11. In this case, the low-recognition-rate data generation model 95 also outputs key points KP21 and KP22 as labels for images P21 and P22.

[0109] The learning unit 96 retrains the recognition model 93 using training data that combines the images P21 and P22 with the key points KP21 and KP22 that serve as labels. This enables the recognition model 93 to improve the recognition success rate for images that are similar to the low recognition rate conditions, such as the image P11, in which a man is present on the left and a woman is standing next to each other on the right, and the weather is bad.

[0110] (Part 3) In the above, an example has been described in which an image and one type of label are generated based on the condition of a low recognition rate.

[0111] However, based on the condition of low recognition rate, multiple corresponding labels may be generated along with the image.

[0112] For example, as shown in Figure 9, if the condition for a low recognition rate is the text "an image showing one person," the low recognition rate data generation model 95 may generate low recognition rate data consisting of multiple images, such as those shown on the right, and multiple labels for each of the images based on the text "an image showing one person," which is the condition for a low recognition rate, and output this data to the learning unit 96 as learning data.

[0113] 9, the low-recognition-rate data generation model 95 generates low-recognition-rate data consisting of images P31 to P33 showing one person in accordance with a low-recognition-rate condition consisting of text such as "an image showing one person." In this case, the low-recognition-rate data generation model 95 also generates and outputs key points KP31 to KP33 and human body bounding boxes (boxes indicating the area where a human body is present) BB31 to BB33 as labels for the images P31 to P33 as low-recognition-rate data.

[0114] The learning unit 96 re-learns the recognition model 93 using learning data that combines the images P31 to P33, the key points KP31 to KP33 that serve as labels, and the human body bounding boxes BB31 to BB33.

[0115] The recognition model 93 not only learns images P31 to P33 generated from "images showing one person," which is a condition with a low recognition rate, but also learns labels consisting of the corresponding key points KP31 to KP33 and human body bounding boxes BB31 to BB33, thereby making it possible to further improve the recognition success rate for "images showing one person," which is a condition with a low recognition rate.

[0116] That is, for example, if the recognition model 93 is a model that estimates key points from an image, as a preprocessing step for estimating the key points, for example, a human body bounding box may be estimated from within the image, and the key points may be estimated as the posture of a person within the estimated human body bounding box.

[0117] In such cases, keypoints can be estimated with a higher recognition rate by learning the correct positions of the human body bounding boxes required for keypoint estimation as labels.

[0118] (Part 4) In the above, we have explained an example in which an image is generated based on an RGB image under the condition of a low recognition rate, but images in various data formats may be generated depending on the processing content of the recognition process of the recognition model and the data format of the image to be handled.

[0119] For example, as shown in Figure 10, if the condition for a low recognition rate is text such as "an image showing two people in a dark environment," the low recognition rate data generation model 95 may generate low recognition rate data based on the text, "an image showing two people in a dark environment," which is the condition for a low recognition rate, along with images in multiple data formats such as those shown on the right, and labels consisting of key points for each image, and output this as learning data to the learning unit 96.

[0120] 10, the low-recognition-rate data generation model 95 generates, as low-recognition-rate data, images of two people in a dark environment, such as image P41 made of RAW data and image P42 made of black-and-white image data, in accordance with a low-recognition-rate condition consisting of text such as "an image showing two people in a dark environment." In this case, the low-recognition-rate data generation model 95 also generates and outputs key points KP41 and KP42 as labels for images P41 and P42 as low-recognition-rate data.

[0121] The learning unit 96 retrains the recognition model 93 using learning data that combines an image P41 consisting of RAW data, an image P42 consisting of a black and white image, and key points KP41 and KP42 that serve as corresponding labels.

[0122] Recognition model 93 is trained based on images P41 and P42 of different data formats that are generated from an "image of two people in a dark environment," which is a condition with a low recognition rate. As a result, in the example of Figure 10, it is possible to improve the recognition success rate of "an image of two people in a dark environment," at least for RAW data and black and white image data.

[0123] Furthermore, when, for example, pseudo-labeling keypoints from an image based on an image taken in a dark environment, sufficient accuracy may not be achieved due to the dark environment, and if the image is learned by associating it with the pseudo-labels, there is a risk that the recognition success rate may not be adequately improved.

[0124] However, as shown in Figure 10, the low recognition rate data generation model 95 generates low recognition rate data by associating it with images of different data formats and assigning key points, and then uses the generated low recognition rate data as training data to train the recognition model 93, thereby making it possible to improve the recognition success rate by estimating labels such as key points for images in dark environments in the recognition model 93.

[0125] <Recognition Model Improvement Processing> Next, the recognition model improvement processing performed by the recognition model improvement system 11 of FIG. 3 will be described with reference to the flowchart of FIG.

[0126] In step S31, the recognition model improvement request unit 51 determines whether or not there is a user request to improve the recognition model 52. For example, if the user operates an operation panel (not shown) or the like to request improvement of the recognition model 52, the recognition model improvement request unit 51 determines that there is a request to improve the recognition model 52 based on an operation signal corresponding to the operation content of the operation panel, and the process proceeds to step S32.

[0127] In addition, if information that can be considered to be a condition for a low recognition rate is input based on the user's experience when using recognition model 52 or the results of statistical processing of the recognition success rate of the recognition results of recognition model 52, the input of the information that can be considered to be a condition for a low recognition rate will be accepted along with the request for improvement.

[0128] In step S32, the recognition model improvement request unit 51 transmits a request for improving the recognition model 52 to the cloud server 32 via the network 33. At this time, if there is information that can be considered to be the above-mentioned conditions for a low recognition rate, the recognition model improvement request unit 51 transmits the information that can be considered to be the conditions for a low recognition rate to the cloud server 32 together with the improvement request.

[0129] In step S51, the condition generating unit 91 determines whether or not a request for improving the recognition model 52 has been transmitted from the smartphone 31 via the network 33. If a request for improving the recognition model 52 has been transmitted from the smartphone 31 in step S51, the process proceeds to step S52.

[0130] In step S52, the condition generating unit 91 generates various conditions required for generating verification data to be generated in the verification data generation model 92, and supplies them to the verification data generation model 92. At this time, if information that can be considered as a condition for a low recognition rate has been transmitted together with the improvement request, the condition for a low recognition rate transmitted together with the improvement request may be output to the verification data generation model 92.

[0131] In step S53 , the verification data generation model 92 generates a pair of an image and a label to serve as verification data based on the conditions supplied from the condition generation unit 91 , and supplies the pair to the recognition model 93 .

[0132] In step S54, the recognition model 93 executes recognition processing based on the pair of image and label serving as verification data, and outputs the recognition result and the corresponding label to the recognition model evaluation unit 94.

[0133] In step S55, the recognition model evaluation unit 94 calculates the recognition success rate based on the recognition result of the recognition model 93 and the corresponding label, evaluates the recognition model 93, and generates an evaluation result consisting of the recognition success rate.

[0134] In step S56, the recognition model evaluation unit 94 determines, based on the evaluation results, whether or not there is a recognition result in which the recognition success rate is lower than a predetermined recognition rate.

[0135] In step S56, if there is a recognition result in which the recognition success rate is lower than a predetermined recognition rate, the process proceeds to step S57.

[0136] In step S57, the recognition model evaluation unit 94 identifies the conditions for a low recognition rate based on the recognition result that is lower than a predetermined recognition rate, and generates a low recognition rate data generation model 95.

[0137] In step S58, the low recognition rate data generation model 95 generates low recognition rate data consisting of images and labels based on the conditions for low recognition rate, and outputs the data to the learning unit 96 as learning data.

[0138] In step S59, the learning unit 96 uses the images and labels that make up the low recognition rate data as learning data to train the recognition model 93, and the process returns to step S54.

[0139] That is, in step S56, the processing of steps S54 to S59 is repeated until there are no recognition results with a recognition success rate lower than the predetermined recognition rate, low recognition rate data is generated, and re-learning of the recognition model 93 is repeated.

[0140] Then, in step S56, if there are no more recognition results with a recognition success rate lower than the predetermined recognition rate, the process proceeds to step S60.

[0141] In step S60, the recognition model update unit 72 supplies the recognition model 93, which has been re-trained and no longer contains recognition results with low recognition success rates, to the smartphone 31 via the network 33 for updating the recognition model 52 of the smartphone 31.

[0142] In step S33 , the recognition model 52 of the smartphone 31 is updated with the recognition model 93 supplied from the cloud server 32 .

[0143] If there is no request for improvement of the recognition model in step S31, steps S32 and S33 are skipped and the process proceeds to step S34.

[0144] Also, in step S51, if there is no request for improvement of the recognition model, the processes of steps S52 to S60 are skipped and the process proceeds to step S61.

[0145] In steps S34 and S61, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to steps S31 and S51, respectively, and the subsequent processes are repeated.

[0146] Then, in steps S34 and S61, if an instruction to end the process is given, the process ends.

[0147] In step S56, if there is no recognition result with a recognition success rate lower than a predetermined recognition rate in the initial processing, even if there is a request to improve the recognition model, no improvement is necessary, so that the smartphone 31 may be notified that improvement is not necessary, and the processing of steps S60 and S33 may be skipped. Also, the unimproved recognition model 93 may be supplied to the smartphone 31 as is, and the recognition model 52 may be updated with the supplied (unimproved) recognition model 93 while remaining in an unimproved state.

[0148] Through the above processing, for example, if a user feels that the recognition success rate is low while using the recognition model 52, or if the recognition success rate is actually low, the user can provide an improvement request to the cloud server 32, and images and labels will be generated under various conditions as verification data.

[0149] Next, a recognition process is performed using the generated verification data with a recognition model 93 (corresponding to the current recognition model 52), the recognition success rate is calculated from the recognition results, low recognition rate conditions where the recognition success rate is lower than a predetermined threshold are identified, and low recognition rate data consisting of images and labels for the identified low recognition rate conditions is generated.

[0150] Then, by repeating the process of re-learning the recognition model 93 using low recognition rate data as training data until the recognition success rate becomes higher than a predetermined value, it becomes possible to improve the recognition success rate of the recognition model 93 and realize improvements to the recognition model 52.

[0151] In this case, when generating the verification data and low recognition rate data, multiple images and labels are associated and generated in pairs, which makes it possible to reduce the effort and cost of manually assigning labels to each image.

[0152] In addition, while reducing the effort and cost, it is possible to generate a large number of images and labels under various conditions (scenes) as verification data, making it possible to easily and appropriately identify conditions with low recognition rates based on the recognition results of the recognition model when images and labels under various conditions are used.

[0153] Furthermore, since a large amount of low recognition rate data consisting of images and labels under low recognition rate conditions can be generated as training data, it becomes possible to train the recognition model 93 using a larger number of images and labels under low recognition rate conditions, thereby making it possible to appropriately improve the recognition success rate for low recognition rate conditions.

[0154] As a result, it is possible to easily determine the conditions for low recognition rate, efficiently generate low recognition rate data based on the determined conditions for low recognition rate, and then train the recognition model 93, thereby making it possible to easily and appropriately improve the recognition success rate of images corresponding to the conditions for low recognition rate.

[0155] In the above, an example has been described in which the smartphone 31 constituting the recognition model improvement system 11 requests the cloud server 32 to improve the recognition rate of the recognition model 52 installed in the smartphone 31, the cloud server 32 identifies the conditions for a low recognition rate of the recognition model 93 corresponding to the recognition model 52, low recognition rate data is generated based on the identified conditions for a low recognition rate, and is used as training data to train the recognition model 93, thereby updating the recognition model 52 of the smartphone 31.

[0156] However, the smartphone 31 may be provided with a configuration corresponding to the recognition model improvement processing unit 71 in the cloud server 32, so that the smartphone 31 alone can improve the recognition rate of the recognition model 52. That is, the smartphone 31 may independently identify conditions for a low recognition rate of the recognition model 52, and based on the identified conditions for a low recognition rate, generate low recognition rate data and repeat the process of training the recognition model 52, thereby improving the recognition rate of the recognition model 52.

[0157] <<3. Second embodiment>> In the above, an example has been described in which a request for improvement is received, verification data is generated, the verification data is used to identify conditions for a low recognition rate of a recognition model, low recognition rate data is generated based on the conditions for the low recognition rate, and the low recognition rate data is used as training data to train a recognition model, thereby improving the recognition success rate.

[0158] However, in cases where verification data is generated periodically based on a large number of conditions required for evaluating the recognition model 93, it is also possible to evaluate the recognition model 93 with the updated verification data each time the verification data is updated, identify conditions for a low recognition rate, generate low recognition rate data, and use this as training data to train and improve the recognition model 93, thereby updating the recognition model 52 of the smartphone 31, without waiting for an improvement request from the smartphone 31.

[0159] Figure 12 shows an example configuration of a recognition model improvement system 11' in which, using a configuration other than the cloud server 32 (not shown), verification data is periodically updated based on a large number of conditions required for evaluating the recognition model 93, the recognition model 93 is evaluated using the updated verification data, conditions for low recognition rate are identified, low recognition rate data is generated, and this is used as training data to train and improve the recognition model 93, thereby updating the recognition model 52 of the smartphone 31.

[0160] In the recognition model improvement system 11' of FIG. 12, components having the same functions as those in the recognition model improvement system 11 of FIG. 3 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0161] That is, the configuration of the recognition model improvement system 11′ in FIG. 12 differs from that of the recognition model improvement system 11 in FIG. 3 in that, for example, the smartphone 31 is designated as a smartphone 31′, and in the cloud server 32, the recognition model improvement processing unit 71 is designated as a recognition model improvement processing unit 71′.

[0162] The smartphone 31' has a configuration in which the recognition model improvement request unit 51 in the smartphone 31 is omitted.

[0163] Furthermore, the recognition model improvement processor 71 ′ is configured such that a verification data acquisition unit 131 is provided instead of the condition generation unit 91 and the verification data generation model 92 in the recognition model improvement processor 71 .

[0164] The verification data acquisition unit 131 acquires the verification data 121 that is updated by a component (not shown) other than the cloud server 32 , and supplies the data to the recognition model 93 .

[0165] The verification data 121 is an image and a label generated based on, for example, conditions obtained from information declared by users of other smartphones 31′ that indicates a lower recognition success rate than users of other smartphones 31′, or conditions of a low recognition rate specified by other recognition model improvement processing units 71′, and is updated when new conditions of a low recognition rate are obtained from other smartphones 31′ or other recognition model improvement processing units 71′. By evaluating the recognition model 93 using this verification data 121, it becomes possible to narrow down and specify the conditions of a low recognition rate of the recognition model 93.

[0166] With this configuration, each time the verification data 121 is updated based on the numerous conditions required for evaluating the recognition model 93, the verification data acquisition unit 131 acquires the verification data 121 and supplies it to the recognition model 93, thereby evaluating the recognition model 93, identifying conditions for a low recognition rate, generating low recognition rate data, and training the recognition model 93 using the low recognition rate data as training data, thereby making it possible to efficiently improve the recognition success rate of the recognition model 93.

[0167] The recognition model improvement process by the recognition model improvement system 11′ in FIG. 12 is the same as the process described with reference to the flowchart in FIG. 11, except that the process of generating verification data each time an improvement request is sent is changed to a process of obtaining verification data 121 each time it is updated, and therefore its description will be omitted.

[0168] <<4. Third embodiment>> In the above, we have described an example in which the recognition model evaluation unit 94 calculates the recognition success rate by comparing the recognition result by the recognition model 93 with the label, and identifies conditions for a low recognition rate depending on the recognition success rate.

[0169] However, it is also possible to identify conditions for low recognition rate using, for example, large language models based on the recognition results from the recognition model 93 and information on the labels, and the low recognition rate data generation model 95 generates low recognition rate data consisting of images and labels based on the identified conditions for low recognition rate.

[0170] FIG. 13 shows an example of the configuration of a recognition model improvement system 11″ that identifies conditions for low recognition rates from information on the recognition results by the recognition model 93 and labels, for example, using large language models.

[0171] In the recognition model improvement system 11'' in FIG. 13, components having the same functions as those in the recognition model improvement system 11 in FIG. 3 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0172] That is, the recognition model improvement system 11'' in FIG. 13 differs from the recognition model improvement system 11 in FIG. 3 in that the recognition model improvement processing unit 71 in the cloud server 32 is replaced with a recognition model improvement processing unit 71''.

[0173] Moreover, the recognition model improvement processing unit 71 ″ is configured such that, instead of the recognition model evaluation unit 94 in the recognition model improvement processing unit 71 , a recognition model evaluation LLM (Large Language Models) 151 is provided.

[0174] The recognition model evaluation LLM 151 is configured using a large-scale language model, and generates information that identifies the conditions for low recognition rate from the recognition results by the recognition model 93 using verification data and information on the correct label, and outputs this information to the low recognition rate data generation model 95.

[0175] In this case, the recognition model evaluation LLM 151 may generate information that specifies the conditions for a low recognition rate, and may also generate information that specifies the labels that the low recognition rate data generation model 95 should generate together with the image as low recognition rate data to be used as training data according to the conditions for a low recognition rate.

[0176] The recognition model improvement process by the recognition model improvement system 11'' in Figure 13 is the same as the process described with reference to the flowchart in Figure 11, except that the process of evaluating the recognition model 93 and identifying conditions for a low recognition rate is performed by the recognition model evaluation LLM 151 instead of the recognition model evaluation unit 94, so a description of this process will be omitted.

[0177] <<5. Application Example>> (Use in Contrastive Learning) There is a learning method called contrastive learning, and it is possible to apply this contrastive learning to the learning of the recognition model 93. In general contrastive learning, the same image subjected to different data augmentation is used as a positive pair, and the other data is used as a negative pair.

[0178] As a result, through contrastive learning, it becomes possible to easily identify images that are similar to the data-augmented image as a positive pair as originating from the same image, and to easily identify images that are similar to the data-augmented image as a negative pair as originating from a different image.

[0179] Therefore, by using contrastive learning to train models that can generate images and labels depending on conditions, such as the verification data generation model 92 and low recognition rate data generation model 95 of the present disclosure, it is possible to perform learning that is more specialized for downstream tasks.

[0180] For example, consider a case where contrastive learning is applied to learning that enables the recognition model 93 to estimate key points (human body poses). In this case, for example, the low recognition rate data generation model 95 generates a plurality of conditions consisting of various correct labels under low recognition rate conditions and images according to the labels, as shown in FIG.

[0181] Figure 14 shows an example in which the low recognition rate data generation model 95 generates images P51 to P53 and images P61 to P63 corresponding to key points KP51 and KP61 as the body pose of a single person, as a low recognition rate condition.

[0182] In this case, the learning unit 96 performs contrastive learning by treating images with the same correct label as a positive pair and other images as a negative pair, which is expected to enable learning specialized for human body pose estimation depending on the human body pose.

[0183] In other words, when images P51 to P53 and images P61 to P63 are treated as a group of images as Positive Pairs, Positive Pairs and Negative Pairs are set according to the human body pose, and learning specialized for estimation according to the human body pose is expected.

[0184] More specifically, in the case of FIG. 14, images P51 to P53 generated from the same key point KP51 and images P61 to P63 generated from the same key point KP61 are learned as positive pairs.

[0185] Furthermore, images P51 to P53 generated from the same key point KP51 and images P61 to P63 generated from the same key point KP61 are learned as a mutually negative pair.

[0186] As a result, under the condition of a single human body pose, images P51 to P53 and images P61 to P63 would simply be learned as a group of images. However, by contrasting and learning the images generated for each key point KP51, KP61, which are labels consisting of different human body poses, in association with the labels, it is expected that learning specialized for human body pose estimation will become possible.

[0187] <<6. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0188] 15 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0189] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0190] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0191] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0192] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0193] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0194] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0195] 15 realizes the functions of the recognition model improvement processing unit 71 in FIG. 3, the recognition model improvement processing unit 71' in FIG. 12, and the recognition model improvement processing unit 71'' in FIG.

[0196] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0197] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0198] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0199] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0200] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0201] The present disclosure may also be configured as follows. <1> An information processing device comprising: an evaluation unit that evaluates a recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions, and identifies conditions for a low recognition rate lower than a predetermined recognition rate for the recognition model; and a low recognition rate data generation model that generates low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for the low recognition rate. <2> The low recognition rate data generation model defines a different noise addition intensity for each of the images, text, and labels, and generates pairs of the images and the labels that comply with the conditions by using an estimation model that estimates the noise added to each of the images, text, and labels from image features, text features, and label features to which noise has been added, generated according to the noise addition intensity, setting the noise addition intensity in accordance with the conditions and recursively repeating a process of estimating the noise. <3> The information processing device according to <2>, wherein the estimation model is configured by adding an additional estimation model to a basic estimation model that defines a different noise addition intensity for each of the image and the text and estimates the noise added to each of the image and the text from a group of image features and a group of text features to which the noise has been added and generated in accordance with the noise addition intensity. <4> The information processing device according to <3>, wherein the low recognition rate data generation model is a model obtained by extending UniDiffuser, and the UniDiffuser generates pairs of the image and the text that comply with the conditions by using the basic estimation model to set the noise addition intensity according to the conditions and recursively repeating a process of estimating the noise.<5> The information processing device according to <1>, wherein the condition for the low recognition rate is at least one of an image with a low recognition rate in the recognition model, text expressing a condition for generating an image with a low recognition rate, and a label that is a condition for generating an image with a low recognition rate. <6> The information processing device according to <1>, wherein the low recognition rate data generation model generates the image and a plurality of corresponding labels based on the condition for the low recognition rate. <7> The information processing device according to <1>, further including a learning unit that trains the recognition model using the low recognition rate data as training data. <8> The evaluation unit identifies the condition for the low recognition rate in the recognition model trained by the learning unit, and the low recognition rate data generation model regenerates the low recognition rate data for the recognition model consisting of the image and the label based on the condition for the low recognition rate in the recognition model trained by the learning unit that is identified by the evaluation unit, and the learning unit trains the recognition model using the low recognition rate data regenerated by the low recognition rate data generation model as training data. <9> The information processing device according to <8>, wherein a series of processes in which the evaluation unit identifies the conditions for the low recognition rate, the low recognition rate data generation model regenerates the low recognition rate data, and the learning unit trains the recognition model are repeated until the evaluation unit is no longer able to identify the conditions for the low recognition rate. <10> The information processing device according to <7>, wherein the learning unit trains the recognition model by contrastive learning using the low recognition rate data as training data. <11> The information processing device according to <1>, further including: a condition generation unit that generates the plurality of conditions; and a verification data generation model that generates the verification data consisting of pairs of the image and the label based on the plurality of conditions generated by the condition generation unit.<12> The information processing device according to <11>, wherein the verification data generation model defines a different noise addition intensity for each of the image, the text, and the label, and uses an estimation model that estimates the noise added to each of the image, the text, and the label from a group of image features, a group of text features, and a group of label features to which noise has been added, generated according to the noise addition intensity, to set the noise addition intensity according to the condition and recursively repeat a process of estimating the noise, thereby generating pairs of the image and the label that comply with the condition. <13> The information processing device according to <1>, wherein the verification data is composed of pairs of the image and the label generated under the low recognition rate condition in another information processing device that includes the evaluation unit and the low recognition rate data generation model. <14> The information processing device according to any of <1> to <13>, wherein the evaluation unit is composed of a large-scale language model. <15> The information processing device according to any one of <1> to <14>, wherein the label includes at least one of a bounding box (face, human body, etc.), keypoints (human body model), and semantic segmentation corresponding to the image. <16> An information processing method comprising the steps of: evaluating a recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions, identifying conditions for a low recognition rate lower than a predetermined recognition rate for the recognition model, and generating low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for a low recognition rate. <17> A program causing a computer to function as a low recognition rate data generation model that includes: an evaluation unit that evaluates the recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions, and identifying conditions for a low recognition rate lower than a predetermined recognition rate for the recognition model; and generating low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for a low recognition rate.

[0202] 11, 11', 11'' Recognition model improvement system, 31, 31' Smartphone, 32 Cloud server, 51 Recognition model improvement request unit, 52 Recognition model, 71, 71' Recognition model improvement processing unit, 72 Recognition model update unit, 91 Condition generation unit, 92 Verification data generation model, 93 Recognition model, 94 Recognition model evaluation unit, 95 Low recognition rate data generation model, 96 Learning unit, 121 Verification data, 131 Verification data acquisition unit, 151 LLM (Large Language Models) for recognition model evaluation

Claims

1. An information processing device comprising: an evaluation unit that evaluates the recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions, and identifies conditions for a low recognition rate that is lower than a predetermined recognition rate in the recognition model; and a low recognition rate data generation model that generates low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for the low recognition rate.

2. The information processing device of claim 1, wherein the low recognition rate data generation model defines a different noise addition intensity for each of the image, text, and label, and generates pairs of the image and label that comply with the condition by recursively repeating the process of setting the noise addition intensity in accordance with the condition and estimating the noise using an estimation model that estimates the noise added to each of the image, text, and label from a group of image features, a group of text features, and a group of label features to which noise generated in accordance with the noise addition intensity has been added.

3. The information processing device according to claim 2, wherein the estimation model is configured by adding an additional estimation model to a basic estimation model that defines a different noise addition intensity for each of the image and the text, and estimates the noise added to each of the image and the text from a group of image features and a group of text features to which the noise has been added and which have been generated in accordance with the noise addition intensity, and that defines a different noise addition intensity for each of the image and the label, and estimates the noise added to each of the image and the label from a group of image features and a group of label features to which the noise has been added and which have been generated in accordance with the noise addition intensity.

4. The information processing device of claim 3, wherein the low recognition rate data generation model is an extended model of UniDiffuser, and the UniDiffuser uses the basic estimation model to set the noise addition intensity according to the conditions, and recursively repeats the process of estimating the noise, thereby generating pairs of the image and the text that comply with the conditions.

5. The information processing device according to claim 1, wherein the conditions for a low recognition rate are at least one of an image with a low recognition rate in the recognition model, text expressing the conditions for generating an image with a low recognition rate, and a label that is a condition for generating an image with a low recognition rate.

6. The information processing device according to claim 1, wherein the low recognition rate data generation model generates the image and a plurality of corresponding labels based on the low recognition rate condition.

7. The information processing apparatus according to claim 1, further comprising a learning unit for learning the recognition model using the low recognition rate data as learning data.

8. The information processing device described in claim 7, wherein the evaluation unit identifies conditions for the low recognition rate in the recognition model learned by the learning unit, the low recognition rate data generation model re-generates the low recognition rate data of the recognition model consisting of the image and the label based on the conditions for the low recognition rate in the recognition model learned by the learning unit identified by the evaluation unit, and the learning unit trains the recognition model using the low recognition rate data re-generated by the low recognition rate data generation model as training data.

9. The information processing device of claim 8, wherein the series of processes in which the evaluation unit identifies the conditions for the low recognition rate, the low recognition rate data generation model regenerates the low recognition rate data, and the learning unit learns the recognition model are repeated until the evaluation unit is no longer able to identify the conditions for the low recognition rate.

10. The information processing device according to claim 7, wherein the learning section trains the recognition model by contrastive learning using the low recognition rate data as training data.

11. The information processing device according to claim 1, further comprising: a condition generation unit that generates the plurality of conditions; and a verification data generation model that generates the verification data consisting of pairs of the image and the label based on the plurality of conditions generated by the condition generation unit.

12. The information processing device according to claim 11, wherein the verification data generation model defines a different noise addition intensity for each of the images, text, and labels, and generates pairs of the images and labels that comply with the conditions by recursively repeating the process of setting the noise addition intensity in accordance with the conditions and estimating the noise using an estimation model that estimates the noise added to each of the images, text, and labels from a group of image features, a group of text features, and a group of label features to which noise generated in accordance with the noise addition intensity has been added.

13. The information processing device according to claim 1, wherein the verification data is composed of pairs of the image and the label generated based on the conditions set as the low recognition rate in another information processing device equipped with the evaluation unit and the low recognition rate data generation model.

14. The information processing device according to claim 1, wherein the evaluation unit is composed of a large-scale language model.

15. The information processing device according to claim 1, wherein the labels include at least one of a bounding box (face, human body, etc.), keypoints (human body model), and semantic segmentation corresponding to the image.

16. An information processing method comprising the steps of: evaluating the recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions; identifying conditions for a low recognition rate lower than a predetermined recognition rate for the recognition model; and generating low recognition rate data for the recognition model consisting of the images and the labels based on the identified conditions for a low recognition rate.

17. A program that causes a computer to function as a low recognition rate data generation model, comprising an evaluation unit that evaluates the recognition rate of a recognition model for verification data consisting of images and labels generated based on a plurality of conditions, and identifies conditions for a low recognition rate that is lower than a predetermined recognition rate for said recognition model, and generates low recognition rate data for said recognition model consisting of said images and said labels based on the identified conditions for a low recognition rate.

Images