Image processing device, method for operating image processing device, and program for operating image processing device

Abstract

An image processing device comprising a processor, wherein the processor calculates an evaluation value for the quality of an image, and determines, on the basis of the evaluation value, a method for trimming an image to be trimmed which is either the image or a related image of the image.

Description

Image processing device, operation method of image processing device, and operation program of image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, and an operating program for an image processing device.

[0002] Japanese Patent Application Laid-Open No. 2018-014653 describes an image processing device having a detection means for detecting multiple subjects from an image, a setting means for setting multiple coordinates for locating the subjects, a determination means for determining a cropping area so that the subject is positioned on predetermined coordinates of the image if there is one subject, and for calculating evaluation values ​​for the multiple subjects based on the distances from the predetermined coordinates and determining the cropping area based on the evaluation values ​​of the multiple subjects if there are multiple subjects, and a cropping means for cropping the image according to the calculated cropping area.

[0003] One embodiment of the technique of the present disclosure provides an image processing device, an operating method for the image processing device, and an operating program for the image processing device that can more easily determine an appropriate trimming method.

[0004] The image processing device of the present disclosure includes a processor, which calculates an evaluation value of the quality of an image and determines a cropping method for a cropping target image, which is either the image or a related image of the image, based on the evaluation value.

[0005] The processor preferably uses a machine learning model that outputs an evaluation value in response to an image input.

[0006] The machine learning model is preferably trained using a plurality of pieces of training data each consisting of a pair of an image and an evaluation value assigned to the image by a user.

[0007] It is preferable that the processor obtains at least one of attribute information of the user who owns the image to be trimmed, additional information of the image to be trimmed, and specification information of the photo album to be created using the image to be trimmed, and determines a trimming method for the image to be trimmed based on at least one of the attribute information, additional information, and specification information in addition to the evaluation value.

[0008] It is preferable that the processor decides not to crop the image to be cropped if the evaluation value is equal to or greater than a first threshold, decides to perform a first cropping operation in which the proportion of the area occupied by the main subject is less than a third threshold value compared to before cropping if the evaluation value is equal to or greater than a second threshold value and less than the first threshold, and decides to determine the cropping method based on at least one of attribute information, additional information, and specification information if the evaluation value is less than the second threshold.

[0009] If the evaluation value is less than the second threshold, the processor preferably determines to perform either a first trimming or a second trimming in which the ratio is increased by a third threshold or more compared to before the trimming, depending on at least one of the attribute information, the additional information, and the specification information.

[0010] Preferably, if the evaluation value is equal to or greater than a fourth threshold, the processor decides not to crop the image to be cropped; if the evaluation value is equal to or greater than a fifth threshold and less than the fourth threshold, the processor decides to perform a third cropping operation in which the proportion of the area occupied by the main subject is less than a sixth threshold times the size before cropping; and if the evaluation value is less than the fifth threshold, the processor decides to perform a fourth cropping operation in which the proportion is increased to at least the sixth threshold times the size before cropping.

[0011] It is preferable that the images are multiple images belonging to a specified user, the related image to be trimmed is one of the multiple images, and the processor calculates evaluation values ​​from the multiple images and determines a trimming method for the image to be trimmed based on a representative value of the multiple evaluation values ​​calculated from the multiple images.

[0012] The method of operation of the image processing device of the present disclosure includes determining an evaluation value of the quality of an image, and determining a cropping method for a cropping target image, which is either the image or a related image of the image, based on the evaluation value.

[0013] The operating program of the image processing device disclosed herein causes a computer to perform processing including determining an evaluation value of the quality of an image and, based on the evaluation value, determining a trimming method for an image to be trimmed, which is either the image or a related image of the image.

[0014] 1 is a diagram illustrating a user terminal and an image management server. FIG. 1 is a block diagram illustrating computers constituting a user terminal and an image management server. FIG. 2 is a block diagram illustrating a processing unit of a CPU of a user terminal. FIG. 3 is a block diagram illustrating a processing unit of a CPU of an image management server. FIG. 4 is a diagram illustrating data stored in an image DB. FIG. 5 is a diagram illustrating incidental information. FIG. 6 is a diagram illustrating the processing of each processing unit of the image management server when an image storage request is sent from a user terminal. FIG. 7 is a diagram illustrating an image editing screen. FIG. 8 is a diagram illustrating the processing of each processing unit of the image management server when an automatic trimming request is sent from a user terminal. FIG. 9 is a diagram illustrating a detailed configuration of the image editing unit. FIG. 10 is a diagram illustrating how the probability that an image to be trimmed will be adopted for a photo album is calculated using an adoption probability calculation model. FIG. 11 is a diagram illustrating an overview of processing in the learning phase of the adoption probability calculation model. FIG. 12 is a flowchart illustrating the processing procedure of the automatic trimming unit. FIG. 13 is a diagram illustrating decision rules. FIG. 14 is a diagram illustrating how first trimming is performed using a first trimming model. FIG. 15 is a diagram illustrating how second trimming is performed using a second trimming model. FIG. 16 is a diagram illustrating an overview of processing in the learning phase of the first trimming model. FIG. 17 is a diagram illustrating an overview of processing in the learning phase of the second trimming model. FIG. 18 is a diagram illustrating an automatic trimming result display screen. FIG. 19 is a flowchart illustrating the processing procedure of the image management server. 1 is a diagram showing how the probability that an image to be trimmed will be selected in a photo contest is calculated using a selection probability calculation model. FIG. 2 is a diagram showing an overview of processing in the learning phase of the selection probability calculation model. FIG. 3 is a diagram showing how a photo album theme and images to be used in the photo album are selected on an album creation screen. FIG. 4 is a diagram showing how a photo album layout is selected on an album creation screen. FIG. 5 is a diagram showing a photo album creation request. FIG. 6 is a diagram showing decision rules for Modification 2. FIG. 7 is a flowchart showing the processing procedure of the automatic trimming unit of the second embodiment. FIG. 8 is a diagram showing the processing of the automatic trimming unit of the third embodiment. FIG. 9 is a diagram showing the processing of the automatic trimming unit of the fourth embodiment. FIG. 10 is a flowchart showing the processing procedure of the automatic trimming unit of the fifth embodiment. FIG. 11 is a flowchart showing the processing procedure of the automatic trimming unit of the fifth embodiment.

[0015] [First Embodiment] As an example, as shown in FIG. 1 , a user U owns a user terminal 10. The user terminal 10 is a device having a camera function, an image playback / display function, an image editing function, an image transmission / reception function, and the like. The camera function of the user terminal 10 has an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and obtains an image 52 of a subject (see FIG. 5 ) by focusing subject light captured through a lens on the imaging element. Specifically, the user terminal 10 is a smartphone, a tablet terminal, a compact digital camera, a mirrorless single-lens camera, a notebook personal computer, or the like. The user U uses the camera function to capture an image 52 and uses the image editing function to edit the image 52 to his or her liking.

[0016] The user terminal 10 is connected to the image management server 12 via a network 11 so as to be able to communicate with each other. The network 11 is, for example, a wide area network (WAN) such as the Internet or a public communication network. The user terminal 10 transmits (uploads) images 52 to the image management server 12. The user terminal 10 also receives (downloads) images 52 from the image management server 12.

[0017] The image management server 12 is, for example, a server computer, a workstation, or the like, and is an example of an “image processing device” according to the technology of the present disclosure. A plurality of user terminals 10 of a plurality of users U are connected to the image management server 12 via a network 11.

[0018] 2, the computers that make up the user terminal 10 and the image management server 12 basically have the same configuration, and include a storage 20, a memory 21, a CPU (Central Processing Unit) 22, a communication unit 23, a display 24, and an input device 25. These are interconnected via a bus line 26.

[0019] The storage 20 is a hard disk drive built into the computer that constitutes the user terminal 10 and the image management server 12, or connected via a cable or network. Alternatively, the storage 20 is a disk array consisting of multiple hard disk drives. The storage 20 stores control programs such as an operating system, various application programs (hereinafter abbreviated as APs (Application Programs)), and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0020] The memory 21 is a work memory for the CPU 22 to execute processing. The CPU 22 loads a program stored in the storage 20 into the memory 21 and executes processing in accordance with the program. In this way, the CPU 22 comprehensively controls each part of the computer. The CPU 22 is an example of a "processor" according to the technology of the present disclosure. The memory 21 may be built into the CPU 22.

[0021] The communication unit 23 is a network interface that controls the transmission of various information via the network 11, etc. The display 24 displays various screens. The various screens are equipped with operation functions using a GUI (Graphical User Interface). The computers that make up the user terminal 10 and the image management server 12 accept input of operation instructions from an input device 25 via the various screens. The input device 25 is a keyboard, mouse, touch panel, microphone for voice input, etc.

[0022] In the following explanation, the parts of the computer that make up the user terminal 10 (storage 20, CPU 22, display 24, and input device 25) are distinguished by adding the suffix "A" to their symbols, and the parts of the computer that make up the image management server 12 (storage 20 and CPU 22) are distinguished by adding the suffix "B" to their symbols.

[0023] 3, an image AP 30 is stored in the storage 20A of the user terminal 10. The image AP 30 is installed in the user terminal 10 by the user U. The image AP 30 is an AP for playing back, displaying, and editing an image 52 on the user terminal 10. When the image AP 30 is started, the CPU 22A of the user terminal 10 functions as a browser control unit 32 in cooperation with the memory 21 and the like. The browser control unit 32 controls the operation of a web browser dedicated to the image AP 30.

[0024] The browser control unit 32 generates various screens. The browser control unit 32 displays the generated various screens on the display 24A. The browser control unit 32 also accepts various operation instructions input by the user U from the input device 25A via the various screens. The browser control unit 32 transmits various requests according to the operation instructions to the image management server 12.

[0025] 4, an operating program 35 is stored in storage 20B of image management server 12. Operating program 35 is an AP that causes a computer constituting image management server 12 to function as an "image processing device" according to the technology of the present disclosure. In other words, operating program 35 is an example of an "operating program for an image processing device" according to the technology of the present disclosure.

[0026] The storage 20B also stores an image database (hereinafter referred to as DB (Data Base)) 36, an adoption probability calculation model 37, a first trimming model 381, a second trimming model 382, ​​and a decision rule 39. Although not shown in the figure, the storage 20B also stores, as account information of the user U, a user ID (Identification Data) for uniquely identifying the user U, a password set by the user U, and a terminal ID for uniquely identifying the user terminal 10.

[0027] When the operating program 35 is started, the CPU 22B of the image management server 12 cooperates with the memory 21 and the like to function as a request receiving unit 45, an image editing unit 46, a read / write (hereinafter referred to as RW (Read Write)) control unit 47, and a distribution control unit 48.

[0028] The request receiving unit 45 receives various requests from the user terminal 10. The request receiving unit 45 outputs various requests to the image editing unit 46 and / or the RW control unit 47, and the distribution control unit 48.

[0029] The image editing unit 46 performs various image editing operations on the image 52. The image editing unit 46 outputs the image 52 after the image editing to the RW control unit 47.

[0030] The RW control unit 47 controls the storage of various data in storage 20B and the reading of various data from storage 20B. In particular, the RW control unit 47 controls the storage of images 52 in image DB 36 and the reading of images 52 from image DB 36. The RW control unit 47 also reads out the adoption probability calculation model 37, the first trimming model 381, the second trimming model 382, ​​and the decision rule 39 from storage 20B, and outputs the read adoption probability calculation model 37, the first trimming model 381, the second trimming model 382, ​​and the decision rule 39 to the image editing unit 46.

[0031] The distribution control unit 48 controls the distribution of various data to the user terminal 10 .

[0032] As an example, as shown in FIG. 5 , the image DB 36 has a storage area 50 for each user U. A user ID and attribute information 51 are registered in the storage area 50. The attribute information 51 is literally information indicating the attributes of the user U, and includes gender, age, family composition, etc. The attribute information 51 is obtained, for example, by having the user U answer a questionnaire when the user U installs the image AP 30 in the user terminal 10. Alternatively, the attribute information 51 can be obtained by inferring the user U and his / her family from the faces of the user U and his / her family members appearing in the image 52. The attribute information 51 may also include the user U's hometown, current address, hobbies, etc.

[0033] The storage area 50 also stores an image 52 and additional information 53 for the image 52. As shown in FIG. 6 as an example, the image 52 and the additional information 53 are associated with each other by an image ID. The additional information 53 includes multiple items such as the date and time of capture, the location of capture, the photographic equipment, and tags. The date and time of capture is registered as the date and time the image 52 was captured using the camera function of the user terminal 10. The location of capture is registered as the address and / or landmark name determined from the longitude and latitude information of the location where the image 52 was captured, obtained using the GPS (Global Positioning System) function of the user terminal 10. The photographic equipment is registered as the manufacturer, name, and model number of the user terminal 10 that captured the image 52. The tag is a word that succinctly describes the subject appearing in the image 52. Tags include tags manually entered by the user U and tags derived using a machine learning model for subject identification. Although not shown, the additional information 53 also includes items such as exposure value, ISO (International Organization for Standardization) sensitivity, shutter speed, focal length, and whether or not a flash is used.

[0034] 7 , the browser control unit 32 sends an image storage request 60 to the image management server 12 at an appropriate timing, such as when the image AP 30 is launched. The image storage request 60 includes a user ID, an image 52, and additional information 53. The request receiving unit 45 receives the image storage request 60 and outputs the image storage request 60 to the RW control unit 47. The RW control unit 47 stores the image 52 and additional information 53 of the image storage request 60 in a storage area 50 of the image DB 36 corresponding to the user ID.

[0035] As an example, as shown in FIG. 8 , the browser control unit 32 displays an image editing screen 65 on the display 24A in response to an instruction from the user U. The image editing screen 65 displays the image 52 to be edited. A group of image editing instruction buttons 66 is arranged at the bottom of the image editing screen 65. The group of image editing instruction buttons 66 includes various image quality adjustment buttons such as brightness adjustment and saturation adjustment, and various effect buttons such as dynamic, sepia, and monochrome. The group of image editing instruction buttons 66 also includes various display change buttons such as rotation, manual cropping, and automatic cropping. Automatic cropping is a type of cropping in which a cropping frame is automatically specified, unlike manual cropping in which the user U manually specifies a cropping frame.

[0036] When the automatic trimming button 67 is selected on the image editing screen 65, the browser control unit 32 sends an automatic trimming request 70 to the image management server 12, as shown in FIG. 9 as an example. The automatic trimming request 70 includes the user ID and the image ID of the image 52 that was displayed on the image editing screen 65 when the automatic trimming button 67 was selected and that is to be subjected to automatic trimming (hereinafter referred to as the trimming target image 52T). The request receiving unit 45 receives the automatic trimming request 70 and outputs the automatic trimming request 70 to the image editing unit 46 and the RW control unit 47.

[0037] The RW control unit 47 searches for the image 52 corresponding to the image ID of the automatic trimming request 70, i.e., the image to be trimmed 52T, from among the images 52 stored in the storage area 50 corresponding to the user ID of the automatic trimming request 70. The RW control unit 47 outputs the retrieved image to be trimmed 52T, its associated information 53, and further the attribute information 51 to the image editing unit 46.

[0038] 10 , the image editing unit 46 includes various image quality adjustment units such as a brightness adjustment unit 75 that performs processing corresponding to the various image quality adjustment buttons, an effect unit 76 that performs processing corresponding to the various effect buttons, and various display change units such as an automatic trimming unit 77 that performs processing corresponding to the various display change buttons. The image editing unit 46 also includes an album creation unit 78 that creates a photo album. When the automatic trimming button 67 is selected on the image editing screen 65, an automatic trimming request 70 is received by the request receiving unit 45, and the automatic trimming request 70 is input from the request receiving unit 45, the automatic trimming unit 77 performs the processing described below.

[0039] As an example, as shown in FIG. 11 , the automatic trimming unit 77 inputs the trimming target image 52T to the adoption probability calculation model 37. In response to the input of the trimming target image 52T, the adoption probability calculation model 37 outputs a probability 80 that the trimming target image 52T will be adopted in the photo album (hereinafter referred to as the adoption probability). The adoption probability 80 is a numerical value between 0% and 100%. The adoption probability calculation model 37 is configured, for example, by a machine learning model such as a convolutional neural network. The adoption probability is an example of an "evaluation value" according to the technology of the present disclosure. Furthermore, the adoption probability calculation model 37 is an example of a "machine learning model that outputs an evaluation value according to an input of an image" according to the technology of the present disclosure.

[0040] As an example, as shown in FIG. 12 , the adoption probability calculation model 37 is trained using learning data (also referred to as teacher data or training data) 82. The learning data 82 is a pair of a learning trimming target image 52TL and a correct adoption probability 80CA. A plurality of learning data 82 is prepared. The correct adoption probability 80CA is the adoption probability of the learning trimming target image 52TL. The correct adoption probability 80CA is the result of having an unspecified number of users U select whether or not to use the learning trimming target image 52TL in their photo album. For example, if 80 users U out of 100 users U select the learning trimming target image 52TL to use in their photo album, the correct adoption probability 80CA is 80%. The correct adoption probability 80CA is an example of an "evaluation value given to an image by a user" according to the technology disclosed herein.

[0041] In the learning phase, the learning trimming target image 52TL is input to the adoption probability calculation model 37. As a result, the learning adoption probability 80L is output from the adoption probability calculation model 37. The learning adoption probability 80L is then compared with the correct adoption probability 80CA, and a loss calculation is performed for the adoption probability calculation model 37 using a loss function based on the comparison result. Next, the coefficients of the adoption probability calculation model 37 are updated according to the result of the loss calculation, and the adoption probability calculation model 37 is updated according to the updated setting.

[0042] In the learning phase, the above series of processes, including input of the learning trimming target image 52TL to the adoption probability calculation model 37, output of the learning adoption probability 80L from the adoption probability calculation model 37, loss calculation, update setting, and update of the adoption probability calculation model 37, are repeated while exchanging learning data 82. The repetition of the above series of processes is terminated when the calculation accuracy of the learning adoption probability 80L relative to the correct acceptance probability 80CA reaches a preset level, and the adoption probability calculation model 37 at that time is stored in storage 20B and used by the automatic trimming unit 77. Note that learning may be terminated when the above series of processes have been repeated a predetermined number of times, regardless of the calculation accuracy of the learning adoption probability 80L relative to the correct acceptance probability 80CA.

[0043] 13, the automatic trimming unit 77 determines a trimming method for the trimming target image 52T based on the acceptance probability 80. Specifically, if the acceptance probability 80 is 90% or higher (YES in step ST100), the automatic trimming unit 77 determines not to trim the trimming target image 52T (step ST110). In this way, determining the trimming method includes determining not to trim.

[0044] If the adoption probability 80 is equal to or greater than 50% and less than 90% (NO in step ST100, YES in step ST120), the automatic trimming unit 77 determines to perform the first trimming on the trimming target image 52T using the first trimming model 381 (step ST130). If the adoption probability 80 is less than 50% (NO in step ST120), the automatic trimming unit 77 determines the trimming method in accordance with the decision rule 39 (step ST140). 90% is an example of a "first threshold" according to the technology disclosed herein. 50% is an example of a "second threshold" according to the technology disclosed herein.

[0045] 14 , as an example, the decision rules 39 include a decision rule 85 related to attribute information 51 and a decision rule 86 related to incidental information 53. The decision rule 85 related to attribute information 51 and the decision rule 86 related to incidental information 53 are set based on the results of prior marketing. The results of prior marketing may indicate that a user U having specific attribute information 51 tends to prefer the first cropping, or that an image 52 having specific incidental information 53 tends to be subjected to the second cropping.

[0046] The decision rule 85 for the attribute information 51 registers the content of the attribute information 51 and the trimming method corresponding to the content of the attribute information 51. For example, if the content of the attribute information 51 is "female, 20s, no children," "first trimming" is registered as the trimming method. Also, if the content of the attribute information 51 is "female, 20s to 30s, with preschool children," "second trimming" is registered as the trimming method.

[0047] The decision rule 86 for the incidental information 53 registers the content of the incidental information 53 and the trimming method corresponding to the content of the incidental information 53. For example, if the content of the incidental information 53 is "the grade of the photographic equipment is above a threshold level," "first trimming" is registered as the trimming method. Furthermore, if the content of the incidental information 53 is "a human or animal face of a size above a threshold area is included," "second trimming" is registered as the trimming method. Furthermore, if the content of the incidental information 53 is "landscape" or "screenshot," "no trimming" is registered as the trimming method. Whether the grade of the photographic equipment is above a threshold level can be determined from the photographic equipment information in the incidental information 53. Furthermore, whether a human or animal face of a size above a threshold area is included can be determined from the results of a process of recognizing a human or animal face on the image to be trimmed 52T.

[0048] Note that if the trimming method differs between the decision rule 85 for the attribute information 51 and the decision rule 86 for the incidental information 53, the trimming method of the decision rule 85 for the attribute information 51 takes precedence. For example, if the attribute information 51 is "male, 40s, no children" and the incidental information 53 is "landscape," the "first trimming" of the decision rule 85 for the attribute information 51 is adopted rather than the "no trimming" of the decision rule 86 for the incidental information 53. Alternatively, if there are four or more applicable trimming methods, the most common trimming method among them may be adopted.

[0049] As an example, as shown in FIG. 15 , the first trimming model 381 performs a first trimming on the trimming target image 52T. The first trimming is a trimming in which the proportion of the area occupied by the main subject is less than 1.5 times the size before trimming. Therefore, the first trimming can be said to be a trimming that prioritizes composition over the main subject. The first trimming model 381 sets a first trimming frame 901 for the trimming target image 52T, in which the proportion of the area occupied by the main subject is less than 1.5 times the size before trimming. The first trimming model 381 trims the trimming target image 52T using the set first trimming frame 901, and outputs a first trimmed image 911. 1.5 times is an example of the "third threshold multiple" according to the technology of the present disclosure.

[0050] The main subject is a person's face, an animal's face, the front of a vehicle, a building, etc., recognized by image recognition processing. When the trimming target image 52T does not include a person's face, an animal's face, the front of a vehicle, a building, etc., such as a landscape image, the subject appearing in the central region of the trimming target image 52T is determined to be the main subject. In Figure 15, a person's face is included in the trimming target image 52T, so the main subject is the person's face (the same applies to Figure 16).

[0051] As an example, as shown in FIG. 16 , the second trimming model 382 performs a second trimming on the trimming target image 52T. The second trimming is a trimming that increases the proportion of the area occupied by the main subject by at least 1.5 times compared to before trimming. Therefore, compared to the first trimming, the second trimming can be said to be a trimming that prioritizes the main subject over composition. The second trimming model 382 sets a second trimming frame 902 for the trimming target image 52T that increases the proportion of the area occupied by the main subject by at least 1.5 times compared to before trimming. The second trimming model 382 trims the trimming target image 52T using the set second trimming frame 902 and outputs a second trimmed image 912.

[0052] As an example, as shown in FIG. 17 , the first trimming model 381 is trained using first learning data 951. The first learning data 951 is a pair of a training trimming target image 52TL and a correct first trimmed image 911CA. A plurality of first learning data 951 are prepared. The correct first trimmed image 911CA is an image generated by a user U trimming the training trimming target image 52TL, with emphasis on composition rather than the main subject. The correct first trimmed image 911CA is an image in which the proportion of the area occupied by the main subject is less than 1.5 times that of the image before trimming.

[0053] In the learning phase, the training trimming target image 52TL is input to the first trimming model 381. As a result, a training first trimmed image 911L is output from the first trimming model 381. Then, the training first trimmed image 911L is compared with the correct first trimmed image 911CA, and a loss calculation is performed on the first trimming model 381 using a loss function based on the comparison result. Next, the coefficients of the first trimming model 381 are updated according to the result of the loss calculation, and the first trimming model 381 is updated according to the update setting.

[0054] In the learning phase, the above-described series of processes, including input of the learning trimming target image 52TL to the first trimming model 381, output of the learning first trimmed image 911L from the first trimming model 381, loss calculation, update setting, and update of the first trimming model 381, are repeatedly performed while exchanging the first learning data 951. The repetition of the above-described series of processes is terminated when the prediction accuracy of the learning first trimmed image 911L relative to the correct first trimmed image 911CA reaches a preset level, and the first trimming model 381 at that time is stored in the storage 20B and used by the automatic trimming unit 77. Note that learning may be terminated when the above-described series of processes has been repeated a predetermined number of times, regardless of the prediction accuracy of the learning first trimmed image 911L relative to the correct first trimmed image 911CA.

[0055] As an example, as shown in FIG. 18 , the second trimming model 382 is trained using second learning data 952. The second learning data 952 is a set of a learning trimming target image 52TL and a correct second trimmed image 912CA. A plurality of second learning data 952 are prepared. The correct second trimmed image 912CA is an image generated by the user U trimming the learning trimming target image 52TL, with an emphasis on the main subject rather than the composition. The correct second trimmed image 912CA is an image in which the proportion of the area occupied by the main subject is 1.5 times or more compared to before trimming.

[0056] In the learning phase, the training trimming target image 52TL is input to the second trimming model 382. As a result, a training second trimmed image 912L is output from the second trimming model 382. Then, the training second trimmed image 912L is compared with the correct second trimmed image 912CA, and a loss calculation is performed on the second trimming model 382 using a loss function based on the comparison result. Next, the coefficients of the second trimming model 382 are updated according to the result of the loss calculation, and the second trimming model 382 is updated according to the update setting.

[0057] In the learning phase, the above-described series of processes, including input of the learning trimming target image 52TL to the second trimming model 382, ​​output of the learning second trimmed image 912L from the second trimming model 382, ​​loss calculation, update setting, and update of the second trimming model 382, ​​are repeatedly performed while exchanging the second learning data 952. The repetition of the above-described series of processes is terminated when the prediction accuracy of the learning second trimmed image 912L relative to the correct second trimmed image 912CA reaches a preset level, and the second trimming model 382 at that time is stored in the storage 20B and used by the automatic trimming unit 77. Note that learning may be terminated when the above-described series of processes has been repeated a predetermined number of times, regardless of the prediction accuracy of the learning second trimmed image 912L relative to the correct second trimmed image 912CA.

[0058] As an example, as shown in Figure 19, the browser control unit 32 of the user terminal 10 displays an automatic trimming result display screen 100 on the display 24A. On the automatic trimming result display screen 100, if no trimming was performed, the trimming target image 52T is displayed, if a first trimming was performed, the first trimmed image 911 is displayed, and if a second trimming was performed, the second trimmed image 912 is displayed. Figure 19 shows an example in which the second trimming was performed and the second trimmed image 912 is displayed. If the first trimming or the second trimming was performed, the trimming target image 52T is displayed nested in the upper left of the first trimmed image 911 or the second trimmed image 912. Furthermore, when the first cropping or the second cropping is performed, an explanatory sentence indicating the content of the first cropping or the second cropping (in FIG. 19 , the explanatory sentence indicating the content of the second cropping, "Trimming was performed to emphasize the main subject," is shown as an example) is displayed above the first cropped image 911 or the second cropped image 912. When the first cropping is performed, an explanatory sentence such as "Trimming was performed to emphasize the composition" is displayed.

[0059] A cancel button 101 and an OK button 102 are provided at the bottom of the automatic trimming result display screen 100. If the automatic trimming result is not satisfactory, the user U selects the cancel button 101. When the cancel button 101 is selected, the first trimmed image 911 or the second trimmed image 912 is discarded.

[0060] On the other hand, if the result of the automatic trimming is appropriate, the user U selects the OK button 102. When the OK button 102 is selected, an image storage request 60 for the first trimmed image 911 or the second trimmed image 912 is issued from the browser control unit 32. As a result, the first trimmed image 911 or the second trimmed image 912 is associated with the original trimming target image 52T and stored in the image DB 36. Regardless of whether the cancel button 101 or the OK button 102 is selected, the browser control unit 32 transitions the screen from the automatic trimming result display screen 100 to the image editing screen 65. Note that if trimming was not performed, the browser control unit 32 does nothing regardless of whether the cancel button 101 or the OK button 102 is selected.

[0061] Next, the operation of the above configuration will be described with reference to the flowchart shown in Fig. 20 as an example. As shown in Fig. 3, the CPU 22A of the user terminal 10 functions as a browser control unit 32 when the image AP 30 is activated. Also, as shown in Fig. 4, the CPU 22B of the image management server 12 functions as a request receiving unit 45, an image editing unit 46, a RW control unit 47, and a distribution control unit 48 when the operating program 35 is activated.

[0062] User U takes an image 52 using the camera function of user terminal 10. As shown in Figure 7, under the control of browser control unit 32, an image storage request 60 including image 52 and accompanying information 53 is sent to image management server 12.

[0063] In the image management server 12, the image storage request 60 is received by the request receiving unit 45. The image storage request 60 is output from the request receiving unit 45 to the RW control unit 47. Then, under the control of the RW control unit 47, the image 52 and the incidental information 53 are stored in the image DB 36.

[0064] 8, the user U selects the automatic trimming button 67 to perform automatic trimming on the image editing screen 65. This causes the browser control unit 32 to generate an automatic trimming request 70, as shown in FIG. 9, and the automatic trimming request 70 is sent to the image management server 12.

[0065] In the image management server 12, the request receiving unit 45 receives the automatic trimming request 70 (YES in step ST1000). The request receiving unit 45 outputs the automatic trimming request 70 to the image editing unit 46 and the RW control unit 47. Then, under the control of the RW control unit 47, the image 52 corresponding to the automatic trimming request 70, i.e., the image 52T to be trimmed, is read from the image DB 36 (step ST1100). Additionally, the attribute information 51 of the user U who owns the image 52T to be trimmed and the additional information 53 of the image 52T to be trimmed are also read. The image 52, together with the attribute information 51 and the additional information 53, is output from the RW control unit 47 to the image editing unit 46.

[0066] In the automatic trimming section 77 of the image editing unit 46, first, as shown in FIG. 11, the image 52T to be trimmed is input to the adoption probability calculation model 37, and the adoption probability 80 of the image 52T to be trimmed is output from the adoption probability calculation model 37 (step ST1200).

[0067] 13, a trimming method for the trimming target image 52T is determined based on the adoption probability 80 (step ST1300). Specifically, if the adoption probability 80 is 90% or higher, it is determined that the trimming target image 52T will not be trimmed. If the adoption probability 80 is 50% or higher but less than 90%, it is determined that the first trimming is performed on the trimming target image 52T using the first trimming model 381. If the adoption probability 80 is less than 50%, the trimming method is determined in accordance with the decision rule 39.

[0068] If it is decided in step ST1300 that no trimming is to be performed (YES in step ST1400), in the user terminal 10, under the control of the browser control unit 32, an automatic trimming result display screen 100 showing the trimming target image 52T itself is displayed on the display 24A.

[0069] On the other hand, if it is determined in step ST1300 that either the first trimming or the second trimming is to be performed (NO in step ST1400), the trimming target image 52T is trimmed using the determined trimming method (step ST1500). Specifically, if it is determined that the first trimming is to be performed, the trimming target image 52T is input to the first trimming model 381, and a first trimmed image 911 is output from the first trimming model 381, as shown in FIG. 15. If it is determined that the second trimming is to be performed, the trimming target image 52T is input to the second trimming model 382, ​​and a second trimmed image 912 is output from the second trimming model 382, ​​as shown in FIG. 16. The first trimmed image 911 or the second trimmed image 912 is delivered to the user terminal 10 that requested the automatic trimming request 70 under the control of the delivery control unit 48. In the user terminal 10, under the control of the browser control unit 32, the automatic trimming result display screen 100 on which the first trimmed image 911 or the second trimmed image 912 is displayed is displayed on the display 24A.

[0070] As described above, the automatic trimming unit 77 calculates the acceptance probability 80 of the trimming target image 52T and determines the trimming method for the trimming target image 52T based on the acceptance probability 80. Therefore, the trimming target image 52T can be trimmed using a trimming method according to the acceptance probability 80. Using the relatively simple index of the acceptance probability 80, the trimming method for the trimming target image 52T can be easily determined.

[0071] The adoption probability 80 makes it possible to estimate the photography skill of the user U who owns the trimming target image 52T and the degree to which the user U is conscious of the photo album when taking photos. Therefore, by determining the trimming method for the trimming target image 52T based on the adoption probability 80, it is possible to propose a trimming method that suits the user U's taste, such as their photography skill and their consciousness of the photo album.

[0072] 11, the automatic trimming unit 77 uses an adoption probability calculation model 37 that outputs an adoption probability 80 in response to an input of the trimming target image 52T. Therefore, the adoption probability 80 can be easily calculated.

[0073] 12, the acceptance probability calculation model 37 is trained using a plurality of pieces of training data 82 each consisting of a pair of a training trimming target image 52TL and a correct acceptance probability 80CA, which is the acceptance probability 80 assigned to the training trimming target image 52TL by the user U. Therefore, it is possible to calculate a relatively reliable acceptance probability 80.

[0074] 9, the automatic trimming unit 77 acquires attribute information 51 of the user U who owns the image to be trimmed 52T, and the incidental information 53 of the image to be trimmed 52T. As shown in FIGS. 13 and 14, the automatic trimming unit 77 determines the trimming method for the image to be trimmed 52T based on the attribute information 51 and the incidental information 53 in addition to the adoption probability 80. Therefore, it is possible to propose a trimming method that matches the attribute information 51 of the user U and the incidental information 53 of the image to be trimmed 52T.

[0075] 13 , if the acceptance probability 80 is 90% or higher, the automatic trimming unit 77 determines not to trim the trimming target image 52T. If the acceptance probability 80 is 50% or higher but less than 90%, the automatic trimming unit 77 determines to perform a first trimming such that the proportion of the area occupied by the main subject is less than 1.5 times the size before trimming. If the acceptance probability 80 is less than 50%, the automatic trimming unit 77 determines the trimming method based on the attribute information 51 and the incidental information 53.

[0076] If the acceptance probability 80 is 90% or higher, i.e., if the acceptance probability 80 is relatively high, the trimming target image 52T is acceptable as a sufficiently good image without trimming. If the acceptance probability 80 is 50% or higher but less than 90%, i.e., if the acceptance probability 80 is medium, it is better to preserve the original composition of the trimming target image 52T as much as possible while trimming away unnecessary parts that lower the acceptance probability 80. If the acceptance probability 80 is less than 50%, i.e., if the acceptance probability 80 is relatively low, it is considered that the original composition of the trimming target image 52T is not very good, and therefore trimming according to the characteristics of each trimming target image 52T is necessary. Therefore, by determining the trimming method based on the magnitude of the acceptance probability 80 as described above, trimming can be performed using an appropriate trimming method according to the acceptance probability 80, thereby increasing the user U's satisfaction with the automatic trimming process.

[0077] 14, when the adoption probability 80 is less than 50%, the automatic trimming unit 77 determines to perform either the first trimming or the second trimming in accordance with at least one of the attribute information 51 and the additional information 53. Therefore, when the adoption probability 80 is relatively low, trimming can be performed using an appropriate trimming method in accordance with at least one of the attribute information 51 and the additional information 53, thereby increasing the satisfaction of the user U with the automatic trimming process.

[0078] 21 , a winning probability calculation model 105 may be used instead of the winning probability calculation model 37, and a probability 106 that the image to be trimmed 52T will be selected in the photo contest (hereinafter referred to as the winning probability) may be calculated as the evaluation value instead of the winning probability 80.

[0079] The automatic trimming unit 77 inputs the trimming target image 52T to the selection probability calculation model 105. The selection probability calculation model 105 outputs the selection probability 106 in response to the input of the trimming target image 52T. The selection probability 106 is a numerical value between 0% and 100%, similar to the acceptance probability 80. The selection probability calculation model 105, like the acceptance probability calculation model 37, is configured using a machine learning model such as a convolutional neural network. The selection probability is an example of an "evaluation value" according to the technology of the present disclosure. Furthermore, the selection probability calculation model 105 is an example of a "machine learning model that outputs an evaluation value in response to an image input" according to the technology of the present disclosure.

[0080] As an example, as shown in FIG. 22 , the winning probability calculation model 105 is trained using training data 110. The training data 110 is a set of a training trimming target image 52TL and a correct winning probability 106CA. A plurality of training data 110 is prepared. The correct winning probability 106CA is the winning probability of the training trimming target image 52TL. The correct winning probability 106CA is the result of having an unspecified number of users U select whether or not to enter the training trimming target image 52TL in a photo contest. For example, if 20 users U out of 100 users U select that the training trimming target image 52TL should be entered in the photo contest, the correct winning probability 106CA is 20%. The correct winning probability 106CA is an example of an "evaluation value given to an image by a user" according to the technology disclosed herein. The users U involved in generating the probability of winning the correct answer 106CA preferably include users U who are considered to be able to correctly determine whether or not to select the learning trimming target image 52TL for the photo contest. Such users U are, for example, people with relatively high levels of photography skills or people with experience as judges for photo contests.

[0081] In the learning phase, the training trimming target image 52TL is input to the winning probability calculation model 105. As a result, the training winning probability 106L is output from the winning probability calculation model 105. The training winning probability 106L is then compared with the correct winning probability 106CA, and a loss calculation is performed for the winning probability calculation model 105 using a loss function based on the comparison result. Next, the coefficients of the winning probability calculation model 105 are updated according to the result of the loss calculation, and the winning probability calculation model 105 is updated according to the updated setting.

[0082] In the learning phase, the above series of processes, including input of the learning trimming target image 52TL to the winning probability calculation model 105, output of the learning winning probability 106L from the winning probability calculation model 105, loss calculation, update setting, and update of the winning probability calculation model 105, are repeatedly performed while exchanging learning data 110. The repetition of the above series of processes is terminated when the calculation accuracy of the learning winning probability 106L relative to the correct winning probability 106CA reaches a preset level, and the winning probability calculation model 105 at that time is stored in storage 20B and used by the automatic trimming unit 77. Note that learning may be terminated when the above series of processes have been repeated a predetermined number of times, regardless of the calculation accuracy of the learning winning probability 106L relative to the correct winning probability 106CA.

[0083] In this way, the evaluation value is not limited to the acceptance probability 80, but may be the winning probability 106. The winning probability 106 makes it possible to estimate the photography skill of the user U who owns the trimming target image 52T, and the extent to which the user U is conscious of the photo contest when taking photos. Therefore, by determining the trimming method for the trimming target image 52T based on the winning probability 106, it is possible to propose a trimming method that suits the user U's tastes, such as their photography skill and awareness of the photo contest.

[0084] It is also possible to calculate both the probability of being hired 80 and the probability of being selected 106, and use their average, maximum, or minimum representative value as the evaluation value.

[0085] (Modification 2) In the above example, the decision rule 85 regarding the attribute information 51 and the decision rule 86 regarding the supplementary information 53 are shown, but the present invention is not limited to this.

[0086] 23, the browser control unit 32 displays an album creation screen 115 on the display 24A in response to an instruction from the user U. The album creation screen 115 displays a list of images 52 arranged vertically and horizontally. The user U can select images 52 to be used in the photo album from the displayed list of images 52. Check marks 116 are displayed on the images 52 selected by the user U. In this example, all of the images 52 selected by the user U become images 52T to be trimmed.

[0087] The upper part of the album creation screen 115 is provided with a plurality of theme selection buttons 117 for selecting a theme for the photo album. The theme selection buttons 117 include "Travel," "Growth Record," "Weddings, Funerals," and "Sports." The user U selects images 52 to be used in the photo album, selects the desired theme selection button 117, and then selects the OK button 118 located at the bottom of the album creation screen 115. This causes the browser control unit 32 to issue a request 125 (see FIG. 25 ) for creating a photo album using the selected images 52 in accordance with the theme selected with the theme selection button 117. FIG. 23 illustrates an example in which the "Travel" theme selection button 117 is selected.

[0088] As an example, as shown in FIG. 24 , the album creation screen 115 allows the user to select a photo album layout 120. Specifically, the album creation screen 115 displays a list of multiple selectable layouts 120 in response to an instruction from the user U. Radio buttons 121 are provided below each layout 120 for alternatively selecting a layout 120. The layouts 120 range in size from relatively small to relatively large display frames for each image 52. Some layouts 120 have a mixture of relatively large and relatively small display frames. The user U selects the radio button 121 for the desired layout 120 and then selects the OK button 122. This selects the layout 120 desired by the user U.

[0089] 25, the creation request 125 includes a user ID and photo album specification information 126. The specification information 126 registers the image ID of the image 52 selected by the user U on the album creation screen 115 shown in FIG. 23, the theme selected by the user U using the theme selection button 117, and the layout ID of the layout 120 selected by the user U on the album creation screen 115 shown in FIG.

[0090] The creation request 125 is accepted by the request accepting unit 45 of the image management server 12, and is output from the request accepting unit 45 to the album creating unit 78 of the image editing unit 46. The album creating unit 78 then creates a photo album in response to the creation request 125. The created photo album is stored in the storage 20B under the control of the RW control unit 47. The created photo album is then distributed to the user terminal 10 that requested the creation request 125 under the control of the distribution control unit 48.

[0091] 26 , the decision rule 130 of this example includes a decision rule 85 related to the attribute information 51, a decision rule 86 related to the supplementary information 53, and a decision rule 131 related to the specification information 126. Like the other decision rules 85 and 86, the decision rule 131 related to the specification information 126 is set based on the results of prior marketing. The decision rule 131 related to the specification information 126 registers the content of the specification information 126 and a trimming method corresponding to the content of the specification information 126. For example, if the theme of the specification information 126 is "travel," "first trimming" is registered as the trimming method. Furthermore, if the size of the display frame of the layout 120 of the specification information 126 is "small," "second trimming" is registered as the trimming method.

[0092] The photo album themes include a "specific person" for creating a photo album using images 52 of a specific person that has been registered in advance. Although not shown, the decision rule 131 related to the specification information 126 has registered "second trimming" as the trimming method when the theme is "specific person."

[0093] The size of the display frame of the layout 120 is classified as "small" if it is less than a preset threshold, and as "large" if it is equal to or greater than the threshold. For a layout 120 that includes a mixture of relatively large and relatively small display frames, each display frame is classified as "small" or "large," and a trimming method is determined for each image 52 assigned to each display frame. Note that if the trimming method for the theme and the trimming method for the size of the display frame of the layout 120 differ, the trimming method for the theme takes precedence.

[0094] If the acceptance probability 80 is less than 50%, the automatic trimming unit 77 determines the trimming method in accordance with the decision rule 130. That is, if the acceptance probability 80 is less than 50%, the automatic trimming unit 77 determines the trimming method based on at least one of the attribute information 51, the additional information 53, and the specification information 126. Note that the selection probability 106 mentioned in the first modification may be used instead of or in addition to the acceptance probability 80.

[0095] In this way, in the second modification, the trimming method is determined based on the specification information 126 in addition to the attribute information 51 and the incidental information 53. Therefore, it is possible to propose a trimming method that is suited not only to the attribute information 51 of the user U and the incidental information 53 of the image to be trimmed 52T, but also to the specification information 126 of the photo album.

[0096] [Second Embodiment] In the first embodiment, when the adoption probability 80 is less than 50%, the automatic trimming unit 77 determines the trimming method based on at least one of the attribute information 51, the additional information 53, and the specification information 126. However, this is not limiting. As an example, the trimming method may be determined as shown in the flowchart of FIG. 27 .

[0097] 27 , if the acceptance probability 80 is 90% or higher (YES in step ST200), the automatic trimming unit 77 determines not to trim the trimming target image 52T (step ST210), as in the first embodiment. If the acceptance probability 80 is 50% or higher but less than 90% (NO in step ST200, YES in step ST220), the automatic trimming unit 77 also determines to perform a third trimming on the trimming target image 52T, in which the proportion of the area occupied by the main subject is less than 1.5 times the size before trimming (step ST230), as in the first embodiment. On the other hand, if the acceptance probability 80 is less than 50% (NO in step ST220), the automatic trimming unit 77 does not determine the trimming method according to the decision rule 39, but rather determines to perform a fourth trimming on the trimming target image 52T, in which the proportion of the area occupied by the main subject is increased to 1.5 times or more the size before trimming (step ST240). The third trimming is the same as the first trimming in the first embodiment, and the fourth trimming is the same as the second trimming in the first embodiment. 90% is an example of the "fourth threshold" according to the technology of the present disclosure. 50% is an example of the "fifth threshold" according to the technology of the present disclosure. 1.5 times is an example of the "sixth threshold multiplication" according to the technology of the present disclosure. Note that the selection probability of 106 mentioned in Variation 1 may be used instead of or in addition to the acceptance probability of 80.

[0098] Thus, in the second embodiment, if the acceptance probability 80 is 90% or higher, the automatic trimming unit 77 determines not to trim the trimming target image 52T. If the acceptance probability 80 is 50% or higher but less than 90%, the automatic trimming unit 77 determines to perform a third trimming (first trimming) in which the proportion of the area occupied by the main subject is less than 1.5 times the size before trimming. If the acceptance probability 80 is less than 50%, the automatic trimming unit 77 determines to perform a fourth trimming (second trimming) in which the proportion of the area occupied by the main subject is increased to 1.5 times or more the size before trimming. Therefore, as in the first embodiment, trimming can be performed using an appropriate trimming method according to the acceptance probability 80, thereby increasing the user U's satisfaction with the automatic trimming process.

[0099] Third Embodiment In each of the above embodiments, the automatic trimming unit 77 calculates only the adoption probability 80 of the trimming target image 52T, but this is not limited to this. As an example, as shown in FIG. 28 , the automatic trimming unit 77 may use an adoption probability calculation model 37 to calculate the adoption probability 80 from multiple images 52, and determine a trimming method for the trimming target image 52T based on a representative adoption probability 80TV of the multiple adoption probabilities 80 calculated from the multiple images 52. In this case, one of the multiple images 52 becomes the trimming target image 52T. One of the multiple images that becomes the trimming target image 52T is an example of a "related image" according to the technology of the present disclosure.

[0100] The multiple images 52 for which the acceptance probability 80 is calculated are images belonging to the user U whose user ID is included in the automatic trimming request 70. Here, "belonging images" refers to, for example, all of the images 52 stored in the storage area 50 of the user U whose user ID is included in the automatic trimming request 70. The images 52 stored in the storage area 50 include not only images 52 taken by the user U using the camera function of the user terminal 10, but also images 52 given to the user U by family, friends, etc., or images 52 downloaded by the user U via the Internet, etc. All of these images 52 are collectively treated as "belonging images." Of course, it is also possible to exclude images 52 given to the user U by family, friends, etc., and images 52 downloaded by the user U via the Internet, etc., and treat only the images 52 taken by the user U using the camera function of the user terminal 10 as "belonging images." The user U whose user ID is included in the automatic trimming request 70 is an example of a "designated user" according to the technology of the present disclosure.

[0101] The representative hiring probability 80TV is, for example, the average value, mode, or median of multiple hiring probabilities 80. The representative hiring probability 80TV is an example of a "representative value" according to the technology of the present disclosure. For convenience of explanation, FIG. 28 depicts multiple images 52 being input into the hiring probability calculation model 37 at once and multiple hiring probabilities 80 being output from the hiring probability calculation model 37 at once. However, in reality, images 52 are input into the hiring probability calculation model 37 one by one, and the hiring probability 80 is output from the hiring probability calculation model 37 one by one. Note that instead of or in addition to the hiring probability 80, the selection probability 106 described in Variation 1 may be calculated for multiple images 52, and a representative selection probability of the multiple selection probabilities 106 may be calculated.

[0102] As described above, in the third embodiment, the adoption probability 80 is calculated from a plurality of images 52 belonging to a specified user U, and a trimming method for the image to be trimmed 52T is determined based on the representative adoption probability 80TV of the plurality of adoption probabilities 80 calculated from the plurality of images 52. Compared to the adoption probability 80 calculated only from the image to be trimmed 52T, the representative adoption probability 80TV is more reliable as it better reflects the taste of the user U. Therefore, a trimming method that better suits the taste of the user U can be proposed.

[0103] 29 as an example, in the fourth embodiment, the automatic trimming unit 77 inputs the trimming target image 52T to the first trimming model 381 and causes the first trimming model 381 to output a first trimmed image 911. The automatic trimming unit 77 also inputs the trimming target image 52T to the second trimming model 382 and causes the second trimming model 382 to output a second trimmed image 912. The automatic trimming unit 77 inputs the trimming target image 52T, the first trimmed image 911, and the second trimmed image 912 to the adoption probability calculation model 37, respectively, and causes the adoption probability calculation model 37 to output an adoption probability 80 for each of the trimming target image 52T, the first trimmed image 911, and the second trimmed image 912. The automatic trimming unit 77 adopts the image with the highest calculated adoption probability 80. In Figure 29, the adoption probability 80 of the trimming target image 52T itself is 72%, the adoption probability 80 of the first trimmed image 911 is 83%, and the adoption probability 80 of the second trimmed image 912 is 59%, so an example is shown in which the first trimmed image 911 is adopted.

[0104] 29, as in the case of FIG. 28, for convenience of explanation, it is depicted as if the trimming target image 52T, the first trimmed image 911, and the second trimmed image 912 are input to the adoption probability calculation model 37 at one time, and multiple adoption probabilities 80 are output from the adoption probability calculation model 37 at one time, but in reality, the trimming target image 52T, the first trimmed image 911, and the second trimmed image 912 are input one by one to the adoption probability calculation model 37, and the adoption probabilities 80 are output one by one from the adoption probability calculation model 37. Note that instead of or in addition to the adoption probability 80, the selection probability 106 mentioned in Modification 1 may be used.

[0105] In this way, in the fourth embodiment, the image with the highest adoption probability of 80 is adopted from among the image to be trimmed 52T, the first trimmed image 911, and the second trimmed image 912. This makes it possible to propose a trimming method that is more likely to be adopted by photo albums.

[0106] [Fifth Embodiment] In the fifth embodiment, as an example, a trimming method is determined according to the procedure shown in the flowcharts of Figures 30 and 31. First, the RW control unit 47 reads out from the image DB 36 an image 52 belonging to the user U whose user ID is included in the automatic trimming request 70 (hereinafter referred to as the belonging image 52AT) (step ST2000 in Figure 30). The RW control unit 47 outputs the read belonging image 52AT to the image editing unit 46.

[0107] The automatic trimming unit 77 inputs the attributed image 52AT into the first trimming model 381 and the second trimming model 382, ​​and performs first trimming and second trimming on the attributed image 52AT (step ST2100). Next, the automatic trimming unit 77 derives a dissimilarity between the attributed image 52AT and a first trimmed image 911 generated by performing the first trimming on the attributed image 52AT (step ST2200). The dissimilarity is derived using a well-known method such as IoU (Intersection over Union). The automatic trimming unit 77 calculates a representative value (average, mode, median, etc.) of the multiple dissimilarity values ​​derived for the multiple attributed images 52AT.

[0108] If the representative value of the dissimilarity is less than a predetermined threshold value (YES in step ST2300), that is, if the attribute image 52AT and the first cropped image 911 are not significantly different, the automatic cropping unit 77 decides not to crop the image to be cropped 52T (step ST2400).

[0109] On the other hand, if the representative value of the dissimilarity is equal to or greater than the threshold value (NO in step ST2300), that is, if the attributed image 52AT and the first cropped image 911 are relatively different, the automatic cropping unit 77 compares the composition of the attributed image 52AT with the composition of the first cropped image 911 and the composition of the second cropped image 912 (step ST2500 in FIG. 31 ). If this comparison reveals that there are more attributed images 52AT that are similar in composition to the first cropped image 911 (YES in step ST2600), the automatic cropping unit 77 determines to perform the first cropping on the cropping target image 52T (step ST2700). On the other hand, if there are more belonging images 52AT that have a similar composition to the second cropped image 912 (NO in step ST2600), the automatic cropping section 77 decides to perform second cropping on the cropping target image 52T (step ST2800).

[0110] As described above, in the fifth embodiment, the degree of difference between the attribute image 52AT and the first cropped image 911 is derived, and a cropping method is determined based on the degree of difference. Specifically, if the representative value of the degree of difference is less than a threshold, the automatic cropping unit 77 determines not to crop the image to be cropped 52T. If the representative value of the degree of difference is equal to or greater than the threshold and there are more attribute images 52AT similar in composition to the first cropped image 911, the automatic cropping unit 77 determines to perform the first cropping on the image to be cropped 52T. Furthermore, if the representative value of the degree of difference is equal to or greater than the threshold and there are more attribute images 52AT similar in composition to the second cropped image 912, the automatic cropping unit 77 determines to perform the second cropping on the image to be cropped 52T. This makes it possible to suggest a cropping method more preferred by the user U.

[0111] The first to sixth thresholds are not limited to the values ​​exemplified above. For example, the first threshold may be 95%, the second threshold may be 75%, or the third threshold may be 2 instead of the exemplified 1.5.

[0112] The evaluation value of the quality of the image may be determined by taking into account at least one of the following: the image quality of the image to be trimmed 52T (a value derived depending on whether the exposure is correct, whether the image is in focus, etc.), the number of people appearing in the image to be trimmed 52T, the number of faces of the people appearing in the image to be trimmed 52T, the proportion of the area occupied by the faces of the people appearing in the image to be trimmed 52T, and a representative value (sum, average, etc.) of the degree of smiles on the faces of the people appearing in the image to be trimmed 52T.

[0113] The image management server 12 may be responsible for all or part of the functions of the browser control unit 32 of the user terminal 10. Specifically, the image management server 12 generates various screens, such as the image editing screen 65 and the automatic trimming result display screen 100, and delivers and outputs them to the user terminal 10 in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). In this case, the browser control unit 32 of the user terminal 10 reproduces various screens to be displayed on the web browser based on the screen data and displays them on the display 24A. Note that other data description languages, such as JSON (Javascript (registered trademark) Object Notation), may be used instead of XML.

[0114] The hardware configuration of the computer that makes up the image management server 12 can be modified in various ways. For example, the image management server 12 can be configured with multiple computers separated as hardware to improve processing power and reliability. For example, the functions of the request reception unit 45 and image editing unit 46 and the functions of the RW control unit 47 and distribution control unit 48 can be distributed and performed by two computers. In this case, the image management server 12 is configured with two computers. In addition, all or part of the functions of the image management server 12 may be performed by the user terminal 10.

[0115] In this way, the hardware configuration of the computers of the user terminal 10 and the image management server 12 can be changed as appropriate depending on the required performance, such as processing power, safety, reliability, etc. Furthermore, not only the hardware, but also APs such as the image AP 30 and the operating program 35 can be duplicated or stored in multiple storages in order to ensure safety and reliability.

[0116] In each of the above embodiments, the hardware structure of the processing units that execute various processes, such as the browser control unit 32, the request receiving unit 45, the image editing unit 46 (brightness adjustment unit 75, effect unit 76, automatic trimming unit 77, album creation unit 78, etc.), the RW control unit 47, and the distribution control unit 48, can be any of the various processors listed below. The various processors include CPUs 22A and 22B, which are general-purpose processors that execute software (image AP 30 and operating program 35) and function as various processing units, as well as programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacture, such as FPGAs (Field Programmable Gate Arrays), and / or dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed specifically for executing specific processing.

[0117] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0118] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0119] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0120] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0121] [Supplementary Item 1] An image processing device including a processor, wherein the processor calculates an evaluation value for an image quality, and determines a cropping method for an image to be trimmed, which is either the image or a related image of the image, based on the evaluation value. [Supplementary Item 2] The image processing device according to Supplementary Item 1, wherein the processor uses a machine learning model that outputs the evaluation value in response to input of the image. [Supplementary Item 3] The image processing device according to Supplementary Item 2, wherein the machine learning model is trained using a plurality of pieces of training data each consisting of a pair of the image and the evaluation value assigned by a user to the image. [Supplementary Item 4] The image processing device according to any one of Supplementary Items 1 to 3, wherein the processor acquires at least one of attribute information of a user who owns the image to be trimmed, additional information about the image to be trimmed, and specification information for a photo album to be created using the image to be trimmed, and determines a cropping method for the image to be trimmed based on at least one of the attribute information, the additional information, and the specification information in addition to the evaluation value. [Supplementary Item 5] The image processing device according to Supplementary Item 4, wherein the processor, if the evaluation value is equal to or greater than a first threshold, decides not to crop the crop-target image, if the evaluation value is equal to or greater than a second threshold and less than the first threshold, decides to perform a first cropping operation to reduce the area occupied by the main subject to a size less than a third threshold times that before cropping, and if the evaluation value is less than the second threshold, decides a cropping method based on at least one of the attribute information, the additional information, and the specification information. [Supplementary Item 6] The image processing device according to Supplementary Item 5, wherein, if the evaluation value is less than the second threshold, the processor decides to perform either the first cropping operation or a second cropping operation to increase the area occupied by the main subject to at least the third threshold times that before cropping, in accordance with at least one of the attribute information, the additional information, and the specification information.[Supplementary Item 7] The image processing device according to any one of Supplementary Items 1 to 3, wherein the processor: if the evaluation value is equal to or greater than a fourth threshold, decides not to crop the image to be cropped; if the evaluation value is equal to or greater than a fifth threshold and less than the fourth threshold, decides to perform a third cropping operation to reduce the proportion of the area occupied by the main subject to less than a sixth threshold value compared to before cropping; and if the evaluation value is less than the fifth threshold, decides to perform a fourth cropping operation to increase the proportion to at least the sixth threshold value compared to before cropping. [Supplementary Item 8] The image processing device according to any one of Supplementary Items 1 to 7, wherein the images are a plurality of images belonging to a specified user, and the related image to be the image to be cropped is one of the plurality of images, and the processor: calculates the evaluation value from the plurality of images, and determines a cropping method for the image to be cropped based on a representative value of the plurality of evaluation values ​​calculated from the plurality of images.

[0122] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0123] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0124] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0125] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; The processor: Calculate the image quality rating, acquire at least one of attribute information of a user who owns an image to be trimmed, which is either the image or an image related to the image, supplementary information of the image to be trimmed, and specification information of a photo album to be created using the image to be trimmed; If the evaluation value is equal to or greater than a first threshold, it is determined that the image to be trimmed is not to be trimmed; if the evaluation value is equal to or greater than a second threshold and less than the first threshold, it is determined to perform a first cropping operation such that the proportion of the area occupied by the main subject is less than a third threshold value compared to the size before cropping; If the evaluation value is less than the second threshold, a trimming method is determined based on at least one of the attribute information, the additional information, and the specification information. Image processing device.

2. The processor: The image processing device according to claim 1 , wherein a machine learning model is used to output the evaluation value in response to an input of the image.

3. The image processing device according to claim 2 , wherein the machine learning model is trained using a plurality of sets of training data each consisting of the image and the evaluation value assigned by a user to the image.

4. The processor:

2. The image processing device of claim 1, wherein if the evaluation value is less than the second threshold, it is determined to perform either the first trimming or a second trimming that increases the ratio to at least the third threshold value compared to before trimming, depending on at least one of the attribute information, the additional information, and the specification information.

5. the images are a plurality of images belonging to a designated user; the related image to be the trimming target image is one of the plurality of images, The processor: determining the evaluation value from the plurality of images; The image processing apparatus according to claim 1 , wherein a trimming method for the image to be trimmed is determined based on a representative value of the evaluation values ​​obtained from the plurality of images.

6. Obtaining an evaluation value of the quality of the image; Acquiring at least one of attribute information of a user who owns an image to be trimmed, which is either the image or a related image of the image, supplementary information of the image to be trimmed, and specification information of a photo album to be created using the image to be trimmed; and If the evaluation value is equal to or greater than a first threshold, it is determined that the image to be trimmed is not to be trimmed; if the evaluation value is equal to or greater than a second threshold and less than the first threshold, it is determined to perform a first cropping operation such that the proportion of the area occupied by the main subject is less than a third threshold value compared to the size before cropping; If the evaluation value is less than the second threshold, determining a trimming method based on at least one of the attribute information, the additional information, and the specification information; A method of operating an image processing device comprising:

7. Determining an image quality rating; and Acquiring at least one of attribute information of a user who owns an image to be trimmed, which is either the image or a related image of the image, supplementary information of the image to be trimmed, and specification information of a photo album to be created using the image to be trimmed; and If the evaluation value is equal to or greater than a first threshold, it is determined that the image to be trimmed is not to be trimmed; if the evaluation value is equal to or greater than a second threshold and less than the first threshold, it is determined to perform a first cropping operation such that the proportion of the area occupied by the main subject is less than a third threshold value compared to the size before cropping; If the evaluation value is less than the second threshold, determining a trimming method based on at least one of the attribute information, the additional information, and the specification information; An operating program for an image processing device that causes a computer to execute processing including the steps of: