Information processing system and information processing program

Abstract

This invention provides an information processing system and program that distinguish and estimate features of a person in an image that should be processed and features that do not need to be processed. [Solution] The terminal 10 constituting the information processing system is generated based on each pair of data that associates an unprocessed image with a processed image. An estimation unit 12 is provided that estimates processing information for a new image and outputs the estimated processing information using an estimation model 15 that performs machine learning to output processing information for an unprocessed image, based on training data that associates the type of processing performed on the unprocessed image, processing location information, and processing information including the unprocessed location information with the unprocessed image in order to obtain the processed image.

Description

Technical Field

[0001] The present disclosure relates to an information processing system and an information processing program.

Background Art

[0002] Patent Document 1 discloses an editing system including an editing device for editing a video file used for broadcasting, the editing system including: a face image storage server that acquires a face image of a performer included in the video file and records the face image in association with time code information of the performance video of each performer; a performance video detection unit that compares the face image recorded in the face image storage server with a face image to be searched for included in a video file of a specific program and detects a performance video in the specific program; and a similar face image detection device that, based on the performance video detected by the performance video detection unit, detects other performance videos in which the person of the face image to be searched for appears in the specific program by similar face image search and notifies the editing device of the time code information of the detected performance video in association with the performer information that is the search target. The editing device edits the video file of the specific program using the time code information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Using known extraction techniques, feature parts of a person may be extracted from an image and the extracted feature parts may be processed. Among the extracted feature parts, there may be parts that one does not want to process, but in conventional extraction techniques, information indicating whether processing is necessary for the extracted feature parts is not included, so feature parts that one does not want to process also become targets for processing like other feature parts.

[0005] The purpose of this disclosure is to provide an information processing system and information processing program that can distinguish and estimate feature parts of a person contained in an image that should be processed and feature parts that do not need to be processed. [Means for solving the problem]

[0006] The information processing system according to the first embodiment includes a processor, which generates data based on each pair of data that associates a first image of a person before processing with a second image in which the feature portion of the person contained in the first image has been processed, and the data that associates the first image with the type of processing performed on the first image to obtain the second image, processing position information representing the position of the feature portion in the processed first image, and unprocessed position information representing the position of the feature portion in the unprocessed first image. Using an estimation model that has been machine-learned to output the type of processing, the processing position information, and the unprocessed position information for the first image, the processor estimates the type of processing, the processing position information, and the unprocessed position information from a new image which is the first image that does not contain a person and has not been used in the machine learning of the estimation model, and outputs the type of processing, the processing position information, and the unprocessed position information for the new image.

[0007] In the information processing system according to the second embodiment, the processor outputs the type of processing, processing location information, and unprocessed location information for the new image together with the new image before processing.

[0008] The information processing system according to the third embodiment, in the information processing system according to the second embodiment, has the processor refrain from performing image processing, represented by the estimated type of processing, on the positions representing the feature portions of a person in the new image represented by the estimated raw position information, while performing image processing, represented by the estimated type of processing, on the positions representing the feature portions of a person in the new image represented by the estimated processing position information, and further outputting the processed new image.

[0009] The information processing system according to the fourth embodiment is an information processing system according to the first embodiment in which the estimation model is a model prepared for each person in which machine learning has been performed using only the multiple training data including the first image which contains the same person, and each estimation model is associated with identification information that identifies the person included in the first image which constitutes the training data used for machine learning, and the processor estimates the type of processing, the processing location information, and the unprocessed location information from the new image using the estimation model to which the identification information of the person included in the new image is associated.

[0010] The information processing system according to the fifth embodiment, in the information processing system according to the fourth embodiment, outputs a mark representing the position of the feature portion of a person in the new image represented by the estimated processed position information, and the position of the feature portion of a person in the new image represented by the estimated unprocessed position information, superimposed on at least one of the new image before processing and the new image after processing.

[0011] The information processing system according to the sixth embodiment, in the information processing system according to the fifth embodiment, outputs a first mark, which is a mark representing the position of a person's feature portion in the new image represented by the estimated raw position information, and a second mark, which is a mark representing the position of a person's feature portion in the new image represented by the estimated processed position information, in different display formats.

[0012] The information processing system according to the seventh embodiment, in the information processing system according to the sixth embodiment, when the processor receives a predetermined operation on the first mark from the user, changes the first mark on which the predetermined operation has been performed to the second mark, changes the unprocessed position information associated with the mark that has been changed from the first mark to the second mark to the processed position information, and when the processor receives a predetermined operation on the second mark from the user, changes the second mark on which the predetermined operation has been performed to the first mark, changes the processed position information associated with the mark that has been changed from the second mark to the first mark to the unprocessed position information.

[0013] In the information processing system according to the eighth aspect, in the information processing system according to the seventh aspect, when the processor receives another new image that contains the same person as the person in the new image on which the predetermined operation has been performed, it transfers the setting status of the first mark and the second mark in the new image on which the predetermined operation has been performed to the other new image.

[0014] The information processing system according to the ninth aspect is an information processing system according to any one of the fifth to seventh aspects, wherein the processor outputs a new image in which a string representing the position of a characteristic part of a person in the new image, associated with the mark, is superimposed along with the mark.

[0015] The information processing system according to the tenth embodiment, in the information processing system according to the fifth embodiment, the processor identifies the attributes of the characteristic parts of a person in the new image from the new image, and outputs the marks, whose display form is changed for each attribute, superimposed on at least one of the new image before processing and the new image after processing.

[0016] The information processing system according to the 11th embodiment, in the information processing system according to the 10th embodiment, outputs the mark corresponding to the attribute with the highest number of occurrences among the attributes identified from the new image, superimposed on at least one of the new image before processing and the new image after processing.

[0017] In the information processing system according to the 12th embodiment, in the information processing system according to the 11th embodiment, when the processor receives an output instruction from the user to output a mark corresponding to a different attribute from the mark being output, the processor further superimposes and outputs a mark corresponding to the attribute that is the next most numerous among the attributes identified from the new image, after the attribute represented by the mark being output, onto at least one of the new image before processing and the new image after processing.

[0018] In the information processing system according to the 13th embodiment, if the new image contains multiple people, the processor estimates the type of processing, the processing location information, and the unprocessed location information for each person from the new image using the estimation model to which the identification information of each person is associated.

[0019] In the information processing system according to the 14th embodiment, if there is no estimation model associated with the identification information of a person in the information processing system according to the 13th embodiment, the processor estimates the type of processing, the processing location information, and the unprocessed location information for a person included in the new image to which the identification information is not associated, using a standard estimation model which is an estimation model that has been machine-trained using a plurality of training data including the first image which includes different people, for a person to whom the estimation model is not associated.

[0020] The information processing system according to the 15th embodiment, in the information processing system according to the 3rd embodiment, the processor performs additional processing on each of the new images processed using the estimation model, in accordance with a common processing rule predetermined for each person included in the new image.

[0021] The information processing system according to the 16th embodiment is an information processing system according to any one of the 1st to 4th embodiments, wherein the estimation model is a model in which machine learning has been performed using the training data which further associates the degree of processing performed on the 1st image to obtain the 2nd image with the 1st image, and the processor uses the estimation model to estimate the type of processing, the degree of processing, the processing location information, and the unprocessed location information from the new image, and outputs the type of processing, the degree of processing, the processing location information, and the unprocessed location information for the new image.

[0022] The information processing program according to the 17th embodiment is a program that causes a computer to perform a process of estimating the type of processing, the processing location information, and the unprocessed location information from a new image which is the first image which contains a person and has not been used in the machine learning of the estimation model, and outputting the type of processing, the processing location information, and the unprocessed location information for the new image, using an estimation model which has been machine-learned to output the type of processing, the processing location information, and the unprocessed location information for the first image, using a plurality of training data which associate the type of processing performed on the first image to obtain the second image, processing location information which represents the position of the feature part in the processed first image, and unprocessed location information which represents the position of the feature part in the unprocessed first image, and using an estimation model which has been machine-learned to output the type of processing, the processing location information, and the unprocessed location information for the first image. [Effects of the Invention]

[0023] According to the first aspect and the seventeenth aspect, there is an effect that it is possible to distinguish and estimate a feature part that is better to be processed and a feature part that does not need to be processed for a person included in an image.

[0024] According to the second aspect, there is an effect that it is possible to check the estimated type of processing, processing position information, and unprocessed position information while referring to a new image.

[0025] According to the third aspect, there is an effect that it is possible to check a new image in which image processing represented by the type of processing is executed for the position represented by the processing position information.

[0026] According to the fourth aspect, there is an effect that it is possible to improve the estimation accuracy of the type of processing, processing position information, and unprocessed position information for a new image as compared with the case of using an estimation model generated from learning data including a first image including a plurality of persons.

[0027] According to the fifth aspect, there is an effect that it is possible to grasp from an image the position of a feature part estimated by an information processing system.

[0028] According to the sixth aspect, there is an effect that it is possible to grasp from an image each feature part by distinguishing between a feature part to be processed and a feature part not to be processed.

[0029] According to the seventh aspect, when the estimation result regarding whether or not to process any feature part is different from the user's intention, there is an effect that the user can set a feature part to be processed or a feature part not to be processed according to his / her own intention.

[0030] According to the eighth aspect, there is an effect that it is possible to set a feature part to be processed or a feature part not to be processed in accordance with the user's intention without the user setting each time a feature part to be processed or a feature part not to be processed.

[0031] According to the ninth aspect, this method has the effect of allowing for a more accurate understanding of the location of a feature portion than simply using marks to indicate its position.

[0032] According to the tenth embodiment, it has the effect of being able to understand where and what kind of characteristic attributes are present in a person.

[0033] According to the 11th embodiment, there is an effect that only the main attributes can be output from among the different attributes.

[0034] According to the twelfth embodiment, the characteristic portion can be output for each attribute, which has the effect of being able to output each attribute separately.

[0035] According to the 13th embodiment, the estimation accuracy can be improved compared to the case where the same estimation model is used to estimate the type of processing, processing location information, and unprocessed location information for each individual.

[0036] According to the 14th aspect, even when there is no estimation model associated with a person, the estimation model can estimate the type of processing, processing location information, and unprocessed location information for a person to whom it is not associated.

[0037] According to the 15th embodiment, it has the effect of unifying the atmosphere of each newly processed image that includes the same person.

[0038] According to the 16th embodiment, in addition to the type of processing, processing location information, and unprocessed location information for the new image, the degree of processing can also be estimated. [Brief explanation of the drawing]

[0039] [Figure 1] Figure 1 shows an example of a person's image. [Figure 2] This figure shows an example of the functional configuration of terminals that make up an information processing system. [Figure 3]This figure shows an example of generating training data used in machine learning for estimation models. [Figure 4] This figure shows an example of generating training data using a generative model. [Figure 5] This figure shows an example of the main components of the electrical system of a terminal configured using a computer. [Figure 6] This flowchart shows an example of the flow in the processing of estimated processing information. [Figure 7] This figure shows an example of the editing screen. [Figure 8] This figure shows an example of a new image with a mark superimposed. [Figure 9] This figure shows an example of how a new image is displayed when a mark change operation is accepted. [Figure 10] This is an example of an output image displayed on the editing screen after changing a mark. [Figure 11] This figure shows an example of a new image in which the display format of the mark is changed for each attribute of the feature part. [Figure 12] This figure shows an example of a new image that displays only the marks indicating the location of wrinkles. [Figure 13] This figure shows an example of a new image displaying a string indicating the location of a feature. [Modes for carrying out the invention]

[0040] The embodiments of the disclosure will be described below with reference to the drawings. The same reference numerals are used throughout the drawings for the same components and processes, and redundant explanations are omitted. The dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0041] This document describes the information processing system related to this disclosure. Individuals may have distinctive features that leave a lasting impression on others. For example, moles in visible locations can easily be remembered, along with the person's overall appearance. While these physical features may create a positive impression, some individuals may prefer to conceal them.

[0042] Figure 1 shows an example of a person's image. The person in Image 1 of Figure 1 has two distinctive features, P1 and P2, which are moles. Generally, a mole at the corner of the eye often gives a charming impression. Conversely, a mole at the hairline on the forehead does not usually give a particular impression to others, but the person may be self-conscious about it.

[0043] Therefore, as shown in Image 2 in Figure 1, a situation arises where it is desirable to leave the mole represented by feature area P1 as is, while removing the mole on the forehead represented by feature area P2.

[0044] Based on these requirements, the information processing system estimates the type of processing applied to image 1, the location information of feature parts P that are considered to benefit from image processing, and the location information of feature parts P that are considered to benefit from image processing, all from the unprocessed image 1.

[0045] Hereafter, the location information of feature parts P that are considered to benefit from image processing will be referred to as "processed location information." The location information of feature parts P that are considered to benefit from not being processed will be referred to as "unprocessed location information."

[0046] The positional information of feature segment P is expressed by a specific body part and its relative direction, such as "lower right of the right eye" or "upper left forehead." However, the method of representing the positional information of feature segment P is not limited to this. Any representation that can indicate the position of feature segment P is acceptable.

[0047] Furthermore, when referring to the entire set of feature parts without distinguishing individual feature parts, such as feature parts P1 and P2 in Figure 1, it is referred to as "feature part P." The feature part P relating to this disclosure does not necessarily have to be on a person's face, but is sufficient if it is on the surface of a person's body within a range visible to other people.

[0048] The characteristic feature P is not limited to moles; it can be anything that gives some kind of impression to other people. Therefore, for example, wrinkles, blemishes, pores, beards, acne, fine hairs, hair, teeth, dark circles under the eyes, eyes, nose, mouth, scars, insect bites, and visible blood vessels are all examples of characteristic features P.

[0049] The "images" relating to this disclosure may be still images or videos, and may be in monochrome or color.

[0050] For the sake of explanation, the unprocessed image 1 shown in Figure 1, in which no processing has been applied to the feature portion P, will be referred to as "New Image 1". The image 2 shown in Figure 1, in which processing has been applied to the feature portion P, will be referred to as "Output Image 2". Output Image 2 is an example of the processed New Image 1 related to this disclosure.

[0051] Figure 2 shows an example of the functional configuration of a terminal 10 that constitutes an information processing system. The terminal 10 includes a reception unit 11, an estimation unit 12, an image processing unit 13, and an output unit 14, as well as an estimation model 15.

[0052] The reception unit 11 receives a new image 1 to be processed. The reception unit 11 also receives instructions from the user to the terminal 10.

[0053] The estimation unit 12 estimates processing information from the new image 1 received by the reception unit 11. Processing information includes the type of processing to be performed on the new image 1, processing location information of the feature portion P to be processed, and unprocessed location information of the feature portion P that will not be processed. For the estimation of processing information by the estimation unit 12, for example, an estimation model 15 is used.

[0054] The estimated model 15 is a model generated by machine learning. Machine learning is the process by which a computer learns the causal relationships between data points hidden within sample data through given sample data, and generates a model that represents these causal relationships. Therefore, the larger the number of sample data points provided, the more accurately the causal relationships between data points hidden within the sample data tend to be learned. In this embodiment, the sample data used for machine learning is referred to as "training data 16".

[0055] In order for the estimation unit 12 to estimate processing information using the estimation model 15, it is sufficient to use the estimation model 15 that outputs processing information when a new image 1 is input.

[0056] Figure 3 shows an example of generating training data 16 used for machine learning of the estimation model 15. Assume that a processed image 4 is obtained by a user processing the feature portion P of a person, as shown in Figure 3, using existing image processing software. Specifically, image 4 is obtained by removing the feature portion P3 representing a mole on the cheek and the feature portion P5 representing wrinkles from image 3. Since the user performed the processing themselves, they are aware of the processing information performed on image 3 to obtain image 4.

[0057] Therefore, the user can generate training data 16 by associating image 3 with the processing information performed on image 3 to obtain image 4.

[0058] The user generates training data 16, which associates processing information with image 3, from various images 3. In training data 16, image 3 is the input data, and the processing information is the output data. Note that the people included in image 3 do not necessarily have to be the same person. Also, the age and gender of the people included in image 3 do not necessarily have to be the same.

[0059] The user uses this training data 16 to generate an estimation model 15 through machine learning, such that when an image 3 from the training data 16 is input, the output approaches the content of the processing information contained in the same training data 16. In other words, the estimation model 15 is a model that represents the causal relationship between image 3 and the processing information solely from the relationship between input and output. This method of machine learning an estimation model 15 that represents the relationship between input data and output data using training data 16 in which the desired output data (in this case, processing information) is associated with the input data is called "supervised machine learning."

[0060] Known methods for supervised machine learning include, for example, linear regression, logistic regression, random forest, boosting, support vector machines, deep learning, and autoregressive models. In this disclosure, as an example, machine learning of an estimation model 15 using deep learning is performed. Deep learning is an example of a neural network composed of multiple layers of hidden layers (also called intermediate layers), such as a convolutional neural network (CNN).

[0061] The unprocessed image 3 used to generate the training data 16 is referred to as "unprocessed image 3," and the image 4 obtained by processing unprocessed image 3 is referred to as "processed image 4." Unprocessed image 3 is an example of the first image related to this disclosure. Processed image 4 is an example of the second image related to this disclosure. Both new image 1 and unprocessed image 3 are unprocessed images, but new image 1 is an unprocessed image that was not used in the training data 16.

[0062] The estimated model 15 may be generated on terminal 10, but it is preferable to generate it using a cloud computing service with better processing performance than terminal 10.

[0063] The above describes an example in which a user generates training data 16 by performing data labeling. However, training data 16 may also be generated using a generative model 17 that generates training data 16 from raw image 3 and processed image 4.

[0064] Figure 4 shows an example of generating training data 16 using the generative model 17. As shown in Figure 4, the generative model 17 takes raw image 3 and processed image 4 as input and outputs processing information for raw image 3.

[0065] Such generative models 17 are generated by machine learning using training data that associates processing information with the unprocessed image 3 and the processed image 4. The generative model 17 may also be generated on the terminal 10, similar to the estimation model 15, but it is preferable to generate it using a cloud computing service that has better processing performance than the terminal 10.

[0066] In Figure 2, the image processing unit 13 performs processing on the new image 1 according to the processing information estimated by the estimation unit 12. The image processing unit 13 also performs image processing according to the user instructions received through the reception unit 11.

[0067] The output unit 14 outputs processing information for the new image 1 estimated by the estimation unit 12. In this case, the output unit 14 may output the processing information together with the received new image 1. Alternatively, the output unit 14 may output the new image 1 that has been processed by the image processing unit 13 as output image 2.

[0068] This disclosure describes a terminal 10 that includes an image processing unit 13, but the terminal 10 does not necessarily have to include an image processing unit 13. In this case, the output unit 14 will output processing information for the new image 1.

[0069] In this disclosure, "output" refers to making the information processed by terminal 10 available for user review. Therefore, the forms in which the processed information, new image 1, and output image 2 are transmitted to an external device (not shown) via a communication line, or displayed on a screen, are examples of output for the processed information, new image 1, and output image 2. Furthermore, the forms in which the processed information, new image 1, and output image 2 are printed on paper or the like using a printer, or stored in a storage device for which the user has read access, are also examples of output for the processed information, new image 1, and output image 2. This disclosure will describe the case in which the output unit 14 displays the processed information, new image 1, and output image 2 on a screen.

[0070] Such a terminal 10 is configured, for example, using a computer 20.

[0071] Figure 5 shows an example of the main components of the electrical system of terminal 10 configured using computer 20.

[0072] The computer 20 that constitutes terminal 10 includes a CPU (Central Processing Unit) 21, which is an example of a processor responsible for executing the functions of terminal 10. The computer 20 also includes RAM (Random Access Memory) 22, non-volatile memory 23, and input / output interface (I / O) 24, which are used as temporary workspaces for the CPU 21. The CPU 21, RAM 22, non-volatile memory 23, and I / O 24 are connected to each other via a bus 25.

[0073] Non-volatile memory 23 is an example of a storage device that retains stored information even when the power supplied to it is cut off. For example, semiconductor memory can be used for non-volatile memory 23, but a hard disk may also be used. As in the estimated model 15, information that needs to be retained even when the power to terminal 10 is cut off is stored in non-volatile memory 23.

[0074] For example, a communication unit 26, an input unit 27, and a display unit 28 are connected to I / O24.

[0075] The communication unit 26 is connected to a communication line and is equipped with a communication protocol for sending and receiving data with external devices that are also connected to the communication line.

[0076] The input unit 27 is a device that receives user instructions and notifies the CPU 21. Examples of input units 27 include buttons, touch panels, mice, keyboards, and pointing devices.

[0077] The display unit 28 is an example of a display device that displays information processed by the CPU 21 as an image. The display unit 28 may use, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0078] The units connected to I / O24 are not limited to the communication unit 26, input unit 27, and display unit 28. Depending on the functions of the terminal 10, the necessary units are selected and connected to I / O24.

[0079] Next, we will explain the processing information estimation process performed by terminal 10. Figure 6 is a flowchart showing an example of the flow of the processing information estimation process performed by the CPU 21 of terminal 10 when a new image 1 is received.

[0080] The information processing program that defines the processing of processed information is pre-stored in, for example, the non-volatile memory 23 of terminal 10. The CPU 21 of terminal 10 reads the information processing program stored in the non-volatile memory 23 and executes the processing of processed information. Hereafter, we will describe an example in which terminal 10 receives the new image 1 shown in Figure 1. It is assumed that the estimation model 15 generated by machine learning using training data 16 is pre-stored in the non-volatile memory 23.

[0081] In step S10 of Figure 6, the CPU 21 uses the estimation model 15 stored in the non-volatile memory 23 to estimate the processing information for the received new image 1. Specifically, the CPU 21 inputs the received new image 1 into the estimation model 15. As a result, the estimation model 15 outputs processing information for the new image 1.

[0082] In step S20, the CPU 21 uses a known image recognition method to identify the type of feature portion P located at a position represented by the processed position information and the unprocessed position information included in the processed information. For example, the CPU 21 identifies whether the feature portion P located at a position represented by the processed position information and the unprocessed position information is a mole or a wrinkle. Hereafter, the type of feature portion P will be referred to as the "attribute" of the feature portion P. Moles and wrinkles are examples of attributes of feature portion P.

[0083] In step S30, the CPU 21 performs machining on the feature portion P located at the position represented by the machining position information, according to the machining information output by the estimated model 15, and the machining process indicated by the machining type.

[0084] For example, suppose the position represented by the processing position information is the position of feature portion P2 in the new image 1 in Figure 1, and the position represented by the unprocessed position information is feature portion P1 in the new image 1 in Figure 1. If the processing content indicated by the processing type is removal, the CPU 21 removes only feature portion P2 and not feature portion P1. Since both feature portion P1 and feature portion P2 are moles, as a result, the mole at the corner of the eye is not removed, and output image 2 is generated in which only the mole on the forehead is removed.

[0085] In step S40, the CPU 21 displays an editing screen 30 on the display unit 28, which shows the received new image 1, the processing information estimated in step S10, the attributes of the feature portion P identified in step S20, and the output image 2 generated in step S30.

[0086] Figure 7 shows an example of the editing screen 30. The layout of the editing screen 30 is just one example, and the new image 1, processing information, attributes of feature part P, and output image 2 can be arranged on the editing screen 30 in any way. In the example of the editing screen 30 in Figure 7, the new image 1 is displayed in area 30A, and in area 30B, the attributes and processing information of feature part P are displayed, separated into feature part P to be processed and feature part P not to be processed. Note that in Figure 7, the attributes of feature part P are simply displayed as "Attributes". In addition, the output image 2, which is the new image 1 processed according to the processing information, is displayed in area 30C of the editing screen 30.

[0087] Area 30D of the editing screen 30 displays a processing menu for further processing of the output image 2. The CPU 21 executes the processing corresponding to the processing item selected by the user from the processing menu on the output image 2. For example, if "blur" is selected from the processing menu, the CPU 21 executes a blurring process on the area of ​​the output image 2 specified by the user.

[0088] Based on the above, the CPU 21 completes the processing information estimation process shown in Figure 6. In this way, the terminal 10 distinguishes and estimates feature parts P that should be processed and feature parts P that do not need to be processed for the person included in the new image 1. Therefore, instead of uniformly removing feature parts P extracted from the person, the CPU 21 can retain feature parts P that give a favorable impression or leave a lasting impression on the person.

[0089] In step S10 of Figure 6, the estimation model 15 used to estimate the processed information is a so-called standard estimation model 15A, which is machine-learned using training data 16 generated based on paired raw image 3 and processed image 4 data of various people. Alternatively, an estimation model 15, i.e., an individual estimation model 15B, may be pre-generated for each person, which is machine-learned using training data 16 generated based on paired raw image 3 and processed image 4 data of the same person.

[0090] The standard estimation model 15A represents the greatest common denominator causal relationship between feature parts P of various individuals and the processed information. On the other hand, the individual estimation model 15B represents the causal relationship between feature parts P that exist only in individual individuals included in the unprocessed image 3 and processed image 4 and the processed information.

[0091] In other words, the individual estimation model 15B more accurately represents the causal relationship between feature parts P and processed information than the standard estimation model 15A, with respect to the people included in the raw image 3 and processed image 4 used to generate the training data 16.

[0092] Therefore, if the person in the new image 1 can be identified, the CPU 21 may use an individual estimation model 15B for the person in the new image 1, rather than the standard estimation model 15A, to estimate the processed information. For example, identification information representing the person in the new image 1 is associated with the new image 1. Also, identification information representing the person in the unprocessed image 3 of the training data 16 used for machine learning is associated with the individual estimation model 15B. In this case, the CPU 21 can use the individual estimation model 15B that has the same identification information associated with the new image 1 as the individual estimation model 15B that is pre-prepared for each person, to estimate the processed information.

[0093] If identification information is not associated with the new image 1, the user can input the identification information of the person included in the new image through the input unit 27.

[0094] Furthermore, if the new image 1 contains multiple people, the CPU 21 may estimate the processing information for each person in the new image 1 using an individual estimation model 15B associated with each person.

[0095] However, individual estimation models 15B are not necessarily generated in advance for all individuals included in new image 1. Therefore, if new image 1 includes individuals for whom individual estimation models 15B have not been generated, the CPU 21 should use the standard estimation model 15A to estimate the processing information for those individuals.

[0096] <Examples of display and operation in the editing screen> The user can perform various operations on New Image 1 and Output Image 2 on the editing screen 30. The following describes the editing functions that terminal 10 provides to the user through the editing screen 30, as well as the display format of New Image 1 and Output Image 2 on the editing screen 30.

[0097] In the editing screen 30 shown in Figure 7, area 30A displays the new image 1. The CPU 21 may also superimpose and display marks M at the positions of each characteristic part P represented by the processing position information and the unprocessed position information.

[0098] Figure 8 shows an example of a new image 1 with mark M superimposed. In the example shown in Figure 8, a circle surrounding feature area P is used as mark M, but there are no restrictions on the shape, line type, and color of the mark as long as it can inform the user of the location of feature area P. For example, a triangle or an arrow pointing to feature area P may be used as mark M.

[0099] Furthermore, the CPU 21 may display the mark M representing the position of the feature portion P represented by the unprocessed position information and the mark M representing the position of the feature portion P represented by the processed position information in different display formats.

[0100] For the sake of clarity, to distinguish between the mark M representing the position of feature portion P represented by unprocessed position information and the mark M representing the position of feature portion P represented by processed position information, the former will be referred to as "mark M1" and the latter as "mark M2". Mark M1 is an example of the first mark relating to this disclosure, and mark M2 is an example of the second mark relating to this disclosure. When referring to both mark M1 and mark M2 collectively, they will be referred to as "mark M". Furthermore, the feature portion P represented by unprocessed position information will be described as feature portion P1, and the feature portion P represented by processed position information will be described as feature portion P2.

[0101] In the example shown in Figure 8, a circle with a solid line is used as the mark M1 representing the position of feature part P1. Similarly, a circle with a dotted line is used as the mark M2 representing the position of feature part P2.

[0102] This section describes an example of displaying a mark M representing the position of feature portion P on a new image 1 displayed in area 30A of the editing screen 30. The CPU 21 may also display this mark M representing the position of feature portion P on the output image 2 displayed in area 30C of the editing screen 30. In other words, the CPU 21 may display the mark M on at least one of the new image 1 and the output image 2 displayed on the editing screen 30.

[0103] In output image 2, feature region P2 may be removed. Even in such cases, it is preferable to display mark M2 to indicate to the user the location where the removed feature region P2 was located.

[0104] The CPU 21 displays marks M1 and M2 in the new image 1 shown in Figure 8 according to the processed position information and unprocessed position information included in the processed information estimated in step S10 of Figure 6. However, the discrimination of feature parts P1 and feature parts P2 estimated by the estimation model 15 may differ from human perception. For example, a feature part P that gives a good impression to others may be identified as feature part P2, and conversely, a feature part P that gives a bad impression to others may be identified as feature part P1.

[0105] Therefore, the CPU 21 accepts operations from the user to change marks M1 and M2 displayed on the new image 1. The operation to change marks M1 and M2 involves predetermined operations, such as the user moving the mouse cursor to the mark M they want to change and clicking the mouse.

[0106] Furthermore, user operations on the editing screen 30, including modification operations, are not limited to operations using a mouse. For example, operations using a touch panel or pen tablet may be used instead of a mouse. When using a touch panel, for example, the user can perform a modification by long-pressing the location of the mark M they want to change with their finger. When using a pen tablet, for example, the user can perform a modification by tapping the location of the mark M they want to change twice with the tip of the pen.

[0107] When CPU21 receives a click operation on mark M1, it changes the clicked mark M1 to mark M2. In this case, CPU21 changes the position information of the feature portion P (specifically, feature portion P1) associated with the mark M that was changed from mark M1 to mark M2, so that it is treated as processed position information rather than raw position information.

[0108] Furthermore, if the CPU 21 receives a click operation on mark M2, it changes the clicked mark M2 to mark M1. In this case, the CPU 21 changes the position information of the feature portion P (specifically, feature portion P2) associated with the mark M that has been changed from mark M2 to mark M1, so that it is treated as unprocessed position information rather than processed position information.

[0109] Figure 9 shows an example of how new image 1 is displayed when a change operation on mark M is accepted. The new image 1 shown in Figure 9 is displayed after the user clicks marks M1 and M2 with the mouse on new image 1 shown in Figure 8. Therefore, feature portion P1, which was represented by mark M1 in Figure 8, is displayed as feature portion P2, which is represented by mark M2, in Figure 9. On the other hand, feature portion P2, which was represented by mark M2 in Figure 8, is displayed as feature portion P1, which is represented by mark M1, in Figure 9.

[0110] The CPU 21 also modifies the processing of the output image 2 according to the modification operation of mark M. Figure 10 is an example of the output image 2 displayed in area 30C of the editing screen 30 as a result of the modification operation of mark M shown in Figure 9. When the modification operation of mark M as shown in Figure 9 is performed, the CPU 21 leaves the mole on the forehead, which has become the new feature part P1, as is. On the other hand, the CPU 21 removes the mole at the outer corner of the eye, which has become the new feature part P2. The CPU 21 also modifies the attributes and processing information of the feature parts P displayed in area 30B, which are divided into feature parts P to be processed and feature parts P not to be processed, according to the modification operation of mark M.

[0111] When a user modifies Mark M in this way, there is a high probability that the user will perform the same modification operation on another new image 1 that contains the same person as the new image 1 in which Mark M was modified. However, having the user modify Mark M every time they edit a new image 1 containing the same person degrades usability.

[0112] Therefore, the CPU 21 associates the content of the mark M modification operation performed by the user with the identification information associated with the new image 1 on which the mark M modification operation has been performed, and stores it in the non-volatile memory 23. When the CPU 21 receives a new image 1, it searches the identification information associated with the new image 1 to see if the content of the mark M modification operation is associated with it. If the content of the mark M modification operation is associated with the identification information, the CPU 21 edits the processing information so that the processing information estimated in step S10 of Figure 6 is the same as the editing result of the mark M modification operation associated with the identification information.

[0113] In other words, if the CPU 21 receives another new image 1 that contains the same person as the new image 1 in which the Mark M modification operation was performed in the past, it will transfer the Mark M settings from the new image 1 in which the Mark M modification operation was performed to the other new image 1.

[0114] For example, suppose that the mark M modification operation shown in Figure 9 is performed on new image 1 shown in Figure 8. In this case, for another new image 1 that contains the same person as new image 1 shown in Figure 8, the CPU 21 will process it so that the mole on the forehead remains, but the mole at the corner of the eye is removed, regardless of the estimation result of the estimation model 15.

[0115] On the other hand, as already explained, the editing screen 30 allows the user to perform processing on the output image 2 through the processing menu. The CPU 21 stores the processing performed by the user on a person included in the new image 1 as a common processing rule for that person. In this situation, suppose terminal 10 receives another new image 1 that includes the same person as the new image 1 that the user has processed. In this case, the CPU 21 autonomously performs additional processing on the new image 1 that has been processed based on the processing information estimated using the estimation model 15, in accordance with the common processing rule. Therefore, each output image 2 that includes the same person will be a unified image that follows the common processing rule. The technique of giving an image design and style a sense of unity and consistency is called "tone and manner." In this way, the CPU 21 can also introduce tone and manner to new images 1 that include the same person.

[0116] For example, if a new image 1 containing person A has been converted to sepia in the past, even if the user does not perform any additional processing from the processing menu, CPU 21 will also convert the color tone of other new images 1 containing person A to sepia. Naturally, the user can also instruct CPU 21 not to apply the common processing rules that set the tone and manner.

[0117] Since the processing applied to New Image 1 differs for each person, there are common processing rules used for setting the tone and manner for each person. On the other hand, there are also processing elements that are common to all New Image 1s, regardless of the people included in them.

[0118] For example, if a person's clothing or the background of the person displays registered trademarks such as company names, product names, or logos, or text or images that may cause offense, these texts or images will be obscured by the background color or clothing color. Additionally, if copyrighted material is included in New Image 1, it will be replaced with other material that is permitted for public display, or blurred. Furthermore, if New Image 1 includes landmark buildings or famous places that could lead to the identification of the person's location, these will be obscured, blurred, or replaced.

[0119] A processing rule defining the processing content common to all new images 1 is stored as a standard processing rule in, for example, non-volatile memory 23. The CPU 21 autonomously performs additional processing on the output image 2 according to the standard processing rule. Naturally, the user can also instruct the CPU 21 not to perform additional processing according to the standard processing rule.

[0120] In the above, for example, as shown in Figure 8, the display form of mark M was changed depending on whether mark M represents a feature portion P represented by unprocessed position information or a feature portion P represented by processed position information. In addition to this, the CPU 21 may change the display form of mark M according to the attributes of feature portion P.

[0121] Figure 11 shows an example of a new image 1 in which the display format of the mark M is changed for each attribute of the feature area P. In the example shown in Figure 11, the positions of feature areas P1 and P2, which represent moles, are indicated by marks M1 and M2, which use circles. On the other hand, feature area P6, which represents wrinkles, is indicated by a mark M3, which uses a triangle.

[0122] The CPU 21 may change the display format of the mark M for each attribute of the feature portion P on the output image 2 displayed in area 30C of the editing screen 30. That is, the CPU 21 may change the display format of the mark M superimposed on at least one of the new image 1 and the output image 2 displayed on the editing screen 30 for each attribute of the feature portion P.

[0123] Furthermore, if the new image 1 contains feature parts P with different attributes, the CPU 21 may decide which attribute to display preferentially among the multiple attributes. For example, elderly people often have more feature parts P corresponding to many types of attributes, such as blemishes, wrinkles, and warts, compared to young people. If marks M representing all feature parts P are displayed on the new image 1 at once in this state, the marks M may overlap, making it difficult to distinguish the feature parts P.

[0124] Therefore, CPU21 may display only the mark M corresponding to the attribute with the highest number of occurrences among multiple attributes in the new image 1.

[0125] For example, as shown in Figure 11, suppose that in new image 1, feature areas P1 and P2 are identified as moles, and feature area P6 is identified as a wrinkle by step S20 in Figure 6. Since there are two identified moles and one identified wrinkle, the CPU 21 displays only the marks M1 and M2, which represent the locations of the moles, on new image 1 because there are more moles than wrinkles (see Figure 8).

[0126] In this case, the mark M3 representing the wrinkle location is not displayed in new image 1. Therefore, if the CPU 21 receives a display instruction from the user to display a mark M of another attribute that is not displayed in new image 1, it displays a mark M representing the location of the attribute that has the next highest number after the attribute currently displayed in new image 1.

[0127] The above operation will be explained using a new image 1, as shown in Figure 11, which has characteristic areas P1 and P2 due to moles and a characteristic area P6 due to wrinkles, as an example. As already explained, the CPU 21 displays only the marks M1 and M2 representing the locations of moles, which are more numerous than the wrinkles, on the new image 1 (see Figure 8). Suppose the user then presses, for example, the "Show Other Attributes" button (not shown) displayed on the editing screen 30. Pressing the "Show Other Attributes" button is an example of a display instruction to display marks M of other attributes that are not displayed on the new image 1. When the "Show Other Attributes" button is pressed, the CPU 21 displays only the marks M3 representing the locations of wrinkles, which are the next most numerous after the marks M1 and M2 representing the locations of moles, on the new image 1. Figure 12 is a diagram showing an example of the new image 1 shown in Figure 11, but with only the marks M3 representing the locations of wrinkles displayed.

[0128] Thus, each time the user issues a command to display other attributes, the CPU 21 displays the attributes on the new image 1 in order, starting with the mark M that represents the position of the attribute with the highest number of occurrences among the multiple attributes of the feature portion P identified from the new image 1. Note that the command to display other attributes is an example of an output command related to this disclosure.

[0129] The CPU 21 may also perform the display of the attributes of these feature parts P on the output image 2 displayed in area 30C of the editing screen 30. That is, the CPU 21 controls the order of marks M corresponding to the attributes to be displayed on at least one of the new image 1 and the output image 2 displayed on the editing screen 30, according to the number of each attribute of the feature parts P included in the new image 1.

[0130] Furthermore, along with displaying the mark M, the CPU 21 may superimpose a string representing the position of the human feature portion P contained within the new image 1, which is associated with the mark M, onto the new image 1.

[0131] Figure 13 shows an example of a new image 1 displaying a string representing the location of feature part P. In the example shown in Figure 13, the attributes of feature part P and its location are represented by strings. For example, for feature part P1, it indicates that the attribute of feature part P1 is a mole and that it is located in the lower right corner of the right eye. Similarly, for feature part P2, it indicates that the attribute of feature part P2 is a mole and that it is located in the upper left corner of the forehead. The CPU 21 identifies the location of feature part P by analyzing the positional relationship between the region representing the person included in the new image 1 and the feature part P represented by the processed position information and the unprocessed position information.

[0132] <Variations of the estimated model> The estimation model 15 described above outputs, for a new image 1, the type of processing to be performed, the processing location information of the feature area P to be processed, and the unprocessed location information of the feature area P that will not be processed. Among these, the type of processing to be performed may have varying degrees of processing, such as strong or weak. For example, in blurring processing, the feature area P may be blurred so that it is completely unrecognizable, or it may be blurred to a degree that the feature area P is recognizable.

[0133] Therefore, the CPU 21 may estimate processing information using a modified estimation model 15C that outputs the type of processing to be performed, the extent of processing to be performed, the processing location information of the feature portion P to be processed, and the unprocessed location information of the feature portion P that will not be processed, for the new image 1. In this case, the processing information includes the type of processing to be performed on the new image 1, the extent of processing to be performed on the new image 1, the processing location information of the feature portion P to be processed, and the unprocessed location information of the feature portion P that will not be processed.

[0134] The deformation estimation model 15C is generated by machine learning using training data 16, which associates processing information, including the degree of processing, with the unprocessed image 3.

[0135] Based on the processing information estimated by the deformation estimation model 15C, the CPU 21 performs processing of the type indicated in the processing information on the feature portion P of the new image 1 represented by the processing position information, with the degree of processing indicated in the processing information.

[0136] Although one aspect of the information processing system has been described above using embodiments, the disclosed form of the information processing system is merely an example. The form of the information processing system is not limited to the scope described in the embodiments. Various modifications or improvements can be made to the embodiments without departing from the gist of this disclosure. Such modified or improved forms are also included within the technical scope of the disclosure. For example, the internal processing order in the processing information estimation process shown in Figure 6 may be changed without departing from the gist of this disclosure.

[0137] Furthermore, in the above embodiment, a configuration in which the processing information estimation process is implemented by software was described as an example. However, the same process shown in the flowchart may also be performed by hardware. In this case, the processing speed can be increased compared to the case in which the processing information estimation process is implemented by software.

[0138] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to work in cooperation with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0139] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of hardware such as a CPU, MPU (Micro Processing Unit), FPGA (Field Programmable Gate Array) or other programmable logic devices, ASIC (Application Specific Integrated Circuit) or other dedicated circuits for executing specific processes, GPU (Graphic Processing Unit), or NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a given processor, these multiple hardware components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0140] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage devices). The program may be divided and stored on multiple non-temporary computer-readable media located on physically separate devices. The program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. The program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents. The program of this application may also be provided as a program product.

[0141] Furthermore, terminal 10 may download an information processing program from an external device via the communication unit 26 and store the downloaded information processing program in the non-volatile memory 23.

[0142] The following are additional details regarding this embodiment.

[0143] (Note) (((1))) Equipped with a processor, The aforementioned processor, Data is generated based on each pair of data that associates a first image of a person before processing with a second image of the person after processing the feature portion of the first image, and the type of processing performed on the first image to obtain the second image, processing position information representing the position of the feature portion in the processed first image, and unprocessed position information representing the position of the feature portion in the unprocessed first image are associated with the first image. An estimation model is used in which machine learning is performed to output the type of processing, the processing position information, and the unprocessed position information for the first image, and the type of processing, the processing position information, and the unprocessed position information are estimated from a new image which is the first image that contains a person and has not been used in the machine learning of the estimation model. The type of processing, processing location information, and unprocessed location information for the new image are output. Information processing system.

[0144] (((2))) The processor outputs the type of processing, processing location information, and unprocessed location information for the new image, along with the new image before processing. The information processing system described in (((1))).

[0145] (((3))) The processor refrains from performing image processing on the positions representing the feature portions of a person in the new image, which are represented by the estimated raw position information, while performing image processing on the positions representing the feature portions of a person in the new image, which are represented by the estimated processed position information, which are represented by the estimated processed position information. Further output the processed new image. The information processing system described in (((2))).

[0146] (((4))) The estimation model is a model prepared for each person, in which machine learning has been performed using only the multiple training data sets that include the first image containing the same person, and each estimation model is associated with identification information that identifies the person included in the first image that constitutes the training data used for machine learning. The processor uses the estimation model to which the identification information of the person included in the new image is associated to estimate the type of processing, the processing location information, and the unprocessed location information from the new image. An information processing system described in any one of (((1))) to (((3))).

[0147] (((5))) The processor outputs the position of the person's feature portion in the new image, represented by the estimated processed position information, and a mark representing the position of the person's feature portion in the new image, represented by the estimated unprocessed position information, superimposed on at least one of the new image before processing and the new image after processing. The information processing system described in (((4))).

[0148] (((6))) The processor outputs a first mark, which is a mark representing the location of a person's feature portion in the new image represented by the estimated raw position information, and a second mark, which is a mark representing the location of a person's feature portion in the new image represented by the estimated processed position information, in different display formats. The information processing system described in (((5))).

[0149] (((7))) When the processor receives a predetermined operation on the first mark from the user, it changes the first mark on which the predetermined operation was performed to the second mark, and changes the unprocessed position information associated with the mark that has been changed from the first mark to the second mark to the processed position information, When the user performs the predetermined operation on the second mark, the second mark on which the predetermined operation was performed is changed to the first mark, and the processing position information associated with the mark that has been changed from the second mark to the first mark is changed to the unprocessed position information. The information processing system described in (((6))).

[0150] (((8))) When the processor receives another new image containing the same person as the new image on which the predetermined operation has been performed, it transfers the settings of the first and second marks in the new image on which the predetermined operation has been performed to the other new image. The information processing system described in (((7))).

[0151] (((9))) The processor outputs a new image in which the mark is superimposed with a string representing the position of a person's feature portion in the new image, which is associated with the mark. An information processing system described in any one of (((5))) to (((8))).

[0152] (((10))) The processor identifies the attributes of the feature portions of a person in the new image from the new image, The marks, whose display format is changed for each attribute, are superimposed on at least one of the new image before processing and the new image after processing and output. An information processing system described in any one of (((5))) to (((9))).

[0153] (((11))) The processor outputs the mark corresponding to the attribute that appears most frequently among the attributes identified from the new image, superimposed on at least one of the new image before processing and the new image after processing. The information processing system described in (((10))).

[0154] (((12))) When the processor receives an output instruction from the user to output a mark corresponding to a different attribute than the mark being output, it further superimposes and outputs a mark corresponding to the attribute that is the next most numerous among the attributes identified from the new image, after the attribute represented by the mark being output, onto at least one of the new image before processing and the new image after processing. The information processing system described in (((11))).

[0155] (((13))) If the aforementioned new image includes multiple people, The processor estimates the type of processing, the processing location information, and the unprocessed location information for each person in the new image, using the estimation model to which the identification information of each person is associated. An information processing system described in any one of (((4))) to (((12))).

[0156] (((14))) If there is no estimation model associated with the aforementioned identification information of a person, The processor estimates the type of processing, the processing location information, and the unprocessed location information for a person included in a new image to which the identification information is not associated, using a standard estimation model which is an estimation model that has been machine-trained using a plurality of training data including the first image which includes different people, for a person to which the estimation model is not associated. The information processing system described in (((13))).

[0157] (((15))) The processor performs additional processing on each of the new images processed using the estimation model, according to a common processing rule predetermined for each person included in the new image. An information processing system described in any one of (((1))) to (((14))).

[0158] (((16))) The estimation model is a machine learning model that uses the training data, which further associates the degree of processing performed on the first image to obtain the second image with the first image. The processor uses the estimation model to estimate the type of processing, the degree of processing, the processing location information, and the unprocessed location information from the new image. The following information is output for the new image: the type of processing, the degree of processing, the processing location information, and the unprocessed location information. An information processing system described in any one of (((1))) to (((15))).

[0159] (((17))) Data is generated based on each pair of data that associates a first image of a person before processing with a second image of the person after processing the feature portion of the first image, and the type of processing performed on the first image to obtain the second image, processing position information representing the position of the feature portion in the processed first image, and unprocessed position information representing the position of the feature portion in the unprocessed first image are associated with the first image. An estimation model is used in which machine learning is performed to output the type of processing, the processing position information, and the unprocessed position information for the first image, and the type of processing, the processing position information, and the unprocessed position information are estimated from a new image which is the first image that contains a person and has not been used in the machine learning of the estimation model. An information processing program for causing a computer to perform a process that outputs the type of processing, processing location information, and unprocessed location information for the new image.

[0160] The information processing system related to (((1))) and the information processing program related to (((17))) have the effect of being able to distinguish and estimate which feature parts of a person included in an image should be processed and which feature parts do not need to be processed.

[0161] According to the information processing system described in (((2))), the system has the effect of allowing users to check the estimated type of processing, processing location information, and unprocessed location information while referring to a new image.

[0162] According to the information processing system related to (((3))), it has the effect of being able to confirm a new image in which an image processing represented by the type of processing has been performed on the position represented by the processing position information.

[0163] The information processing system described in (((4))) has the effect of improving the estimation accuracy of the type of processing, processing location information, and unprocessed location information for new images compared to the case where an estimation model generated from training data including a first image containing multiple people is used.

[0164] The information processing system described in (((5))) has the effect of being able to determine the location of the feature portion estimated by the information processing system from the image.

[0165] According to the information processing system described in (((6))), each feature portion can be distinguished from the image into feature portions to be processed and feature portions that are not processed, and this has the effect of being able to grasp them from the image.

[0166] According to the information processing system described in (((7))), if the estimated result regarding which feature parts to process differs from the user's intention, the user can set which feature parts to process or which not to process according to their intention.

[0167] According to the information processing system described in (((8))), the user does not have to set which feature parts to process and which not to process each time, and the system can set which feature parts to process and which not to process according to the user's intentions.

[0168] According to the information processing system related to (((9))), it has the effect of being able to correctly grasp the location of the feature part, rather than indicating the location of the feature part with marks alone.

[0169] According to the information processing system related to (((10))), it has the effect of being able to understand where and what kind of characteristic attributes are present in a person.

[0170] According to the information processing system related to (((11))), it has the effect of being able to output only the main attributes from among different attributes.

[0171] According to the information processing system related to (((12))), the effect is that the feature portion can be output for each attribute.

[0172] The information processing system described in (((13))) has the effect of improving estimation accuracy compared to the case where the same estimation model is used to estimate the type of processing, processing location information, and unprocessed location information for each individual.

[0173] According to the information processing system described in (((14))), even if there is no estimation model associated with a person, the estimation model can estimate the type of processing, processing location information, and unprocessed location information for a person to whom it is not associated.

[0174] According to the information processing system related to (((15))), it has the effect of unifying the appearance of each new image after processing, even if the same person is included.

[0175] According to the information processing system described in (((16))), in addition to the type of processing, processing location information, and unprocessed location information for a new image, the degree of processing can also be estimated. [Explanation of symbols]

[0176] 1 New image 2 Output image 3 Unprocessed image 4. Processed Images 10 devices 11 Reception Department 12 Estimation part 13 Image Processing Section 14 Output section 15 Estimated Models 15A Standard Estimation Model 15B Individual Estimation Models 15C Deformation Estimation Model 16 Training Data 17 Generative Models 20 Computers 21 CPU 22 RAM 23 Non-volatile memory 24 I / O 25 buses 26 Communication Unit 27 Input Units 28 Display Units 30 Editing screen 30A~30D Area on the editing screen M mark P Feature section

Claims

1. Equipped with a processor, The aforementioned processor, Data is generated based on each pair of data that associates a first image of a person before processing with a second image in which the feature parts of the person contained in the first image have been processed, and the type of processing performed on the first image to obtain the second image, processing position information representing the position of the feature parts in the processed first image, and unprocessed position information representing the position of the feature parts in the unprocessed first image are associated with the first image. Using an estimation model that has been machine-learned to output the type of processing, the processing position information, and the unprocessed position information for the first image, the type of processing, the processing position information, and the unprocessed position information are estimated from a new image which is the first image that contains a person and has not been used in the machine learning of the estimation model. The type of processing, processing location information, and unprocessed location information for the new image are output. Information processing system.

2. The processor outputs the type of processing, processing location information, and unprocessed location information for the new image, along with the new image before processing. The information processing system according to claim 1.

3. The processor refrains from performing image processing on the positions representing the feature portions of a person in the new image, which are represented by the estimated raw position information, while performing image processing on the positions representing the feature portions of a person in the new image, which are represented by the estimated processed position information, which are represented by the estimated processed position information. Further output the processed new image. The information processing system according to claim 2.

4. The estimation model is a model prepared for each person, in which machine learning has been performed using only the multiple training data sets that include the first image containing the same person, and each estimation model is associated with identification information that identifies the person included in the first image that constitutes the training data used for machine learning. The processor uses the estimation model to which the identification information of the person included in the new image is associated to estimate the type of processing, the processing location information, and the unprocessed location information from the new image. The information processing system according to claim 1.

5. The processor outputs the position of the person's feature portion in the new image, represented by the estimated processed position information, and a mark representing the position of the person's feature portion in the new image, represented by the estimated unprocessed position information, superimposed on at least one of the new image before processing and the new image after processing. The information processing system according to claim 4.

6. The processor outputs a first mark, which is a mark representing the location of a person's feature portion in the new image represented by the estimated raw position information, and a second mark, which is a mark representing the location of a person's feature portion in the new image represented by the estimated processed position information, in different display formats. The information processing system according to claim 5.

7. When the processor receives a predetermined operation on the first mark from the user, it changes the first mark on which the predetermined operation was performed to the second mark, and changes the unprocessed position information associated with the mark that has been changed from the first mark to the second mark to the processed position information, When the user performs the predetermined operation on the second mark, the second mark on which the predetermined operation was performed is changed to the first mark, and the processing position information associated with the mark that has been changed from the second mark to the first mark is changed to the unprocessed position information. The information processing system according to claim 6.

8. When the processor receives another new image containing the same person as the new image on which the predetermined operation has been performed, it transfers the settings of the first and second marks in the new image on which the predetermined operation has been performed to the other new image. The information processing system according to claim 7.

9. The processor outputs a new image in which the mark is superimposed with a string representing the position of a person's feature portion in the new image, which is associated with the mark. An information processing system according to any one of claims 5 to 7.

10. The processor identifies the attributes of the feature portions of a person in the new image from the new image, The marks, whose display format is changed for each attribute, are superimposed on at least one of the new image before processing and the new image after processing and output. The information processing system according to claim 5.

11. The processor outputs the mark corresponding to the attribute that appears most frequently among the attributes identified from the new image, superimposed on at least one of the new image before processing and the new image after processing. The information processing system according to claim 10.

12. When the processor receives an output instruction from the user to output a mark corresponding to a different attribute than the mark being output, it further superimposes and outputs a mark corresponding to the attribute that is the next most numerous among the attributes identified from the new image, after the attribute represented by the mark being output, onto at least one of the new image before processing and the new image after processing. The information processing system according to claim 11.

13. If the aforementioned new image includes multiple people, The processor estimates the type of processing, the processing location information, and the unprocessed location information for each person in the new image, using the estimation model to which the identification information of each person is associated. The information processing system according to claim 4.

14. If there is no estimation model associated with the aforementioned identification information of a person, The processor, for persons to whom the estimation model is not associated, uses a standard estimation model, which is an estimation model that has been machine-trained using a plurality of training data including the first image containing different persons, to estimate the type of processing, the processing location information, and the unprocessed location information for persons included in the new image to which the identification information is not associated. The information processing system according to claim 13.

15. The processor performs additional processing on each of the new images processed using the estimation model, according to a common processing rule predetermined for each person included in the new image. The information processing system according to claim 3.

16. The estimation model is a machine learning model that uses the training data, which further associates the degree of processing performed on the first image to obtain the second image with the first image. The processor uses the estimation model to estimate the type of processing, the degree of processing, the processing location information, and the unprocessed location information from the new image. The following information is output for the new image: the type of processing, the degree of processing, the processing location information, and the unprocessed location information. An information processing system according to any one of claims 1 to 4.

17. Data is generated based on each pair of data that associates a first image of a person before processing with a second image in which the feature parts of the person contained in the first image have been processed, and the type of processing performed on the first image to obtain the second image, processing position information representing the position of the feature parts in the processed first image, and unprocessed position information representing the position of the feature parts in the unprocessed first image are associated with the first image. Using an estimation model that has been machine-learned to output the type of processing, the processing position information, and the unprocessed position information for the first image, the type of processing, the processing position information, and the unprocessed position information are estimated from a new image which is the first image that contains a person and has not been used in the machine learning of the estimation model. An information processing program for causing a computer to perform a process that outputs the type of processing, processing location information, and unprocessed location information for the new image.

Images