Image evaluation device, image evaluation method, program, and recording medium

Abstract

Provided is an image evaluation device, an image evaluation method, a program, and a recording medium that, when a plurality of processes are performed on an image, make it possible to appropriately evaluate whether the image is an image to be subjected to the processes.　An image evaluation device according to one embodiment of the present invention comprises a processor. The processor is configured to be capable of executing: a first evaluation for evaluating whether to execute a first process on a first target image by means of a first evaluation model constructed by machine learning using information regarding whether or not the first process has been executed on each of a plurality of images; and a second evaluation for evaluating whether to execute a second process on a second target image by means of a second evaluation model constructed by machine learning using information regarding whether or not the second process has been executed on each of two or more images on which the first process has been executed among the plurality of images.

Description

Image evaluation device, image evaluation method, program, and recording medium

[0001] One embodiment of the present invention relates to an image evaluation device, an image evaluation method, a program, and a recording medium for evaluating whether a predetermined process should be performed on an image.

[0002] When a user acquires one or more images by shooting or the like and performs a predetermined process on one or more of the acquired images, it may take time and effort to select the images to be the subject of the predetermined process. Here, examples of the predetermined process on an image include, for example, a process of transferring the acquired image to a predetermined device, and a process for sharing the image through an SNS (Social Networking Service) or the like.

[0003] As an example of a technique for solving the above problem, the invention described in Patent Document 1 can be cited. The invention described in Patent Document 1 is a technique for analyzing an individual's preferences from images posted on an SNS. Specifically described, in the invention described in Patent Document 1, a server receives a plurality of images posted by a user on an SNS, calculates characteristic values of each image, and calculates a tendency value indicating a tendency for the plurality of images from the characteristic values of each image. Then, when the server receives a new image from the user terminal, it selects an image that matches the above tendency value from among them and transmits it to the user terminal.

[0004] Japanese Unexamined Patent Application Publication No. 2019-53536

[0005] On the other hand, among the image group acquired by the user, after performing a first process on some of the images, when a second process is further performed on some of the images on which the first process has been performed. In the case where a plurality of processes are performed step by step like this, even if a tendency value indicating the tendency of the images on which all the plurality of processes are performed is calculated using the invention described in Patent Document 1, the number (absolute number) of images that match the tendency value is expected to be small. Also, intentionally acquiring an image that matches the above tendency value, for example, shooting an image that matches the tendency value itself is generally difficult.

[0006] When performing multiple processing steps on an image, the image to be processed must be selected for each process. Ideally, each selection process should be performed independently. However, the criteria for selecting the image to be processed may change depending on the processing being performed. Therefore, it is necessary to take this into consideration and appropriately select the image to be processed for each process.

[0007] One embodiment of the present invention has been made in view of the above circumstances, and aims to provide an image evaluation device, an image evaluation method, a program, and a recording medium that can appropriately evaluate whether an image is one on which each process should be performed when multiple processes are performed on the image.

[0008] The above objective is achieved by the image evaluation apparatus described in [1] below. [1] An image evaluation apparatus equipped with a processor, wherein the processor is configured to perform a first evaluation, which evaluates whether a first processing should be performed on a first target image using a first evaluation model constructed by machine learning using information on whether a first processing has been performed on each of a plurality of images, and a second evaluation, which evaluates whether a second processing should be performed on a second target image using a second evaluation model constructed by machine learning using information on whether a second processing has been performed on each of two or more images on which a first processing has been performed.

[0009] Furthermore, in one embodiment of the present invention, an image evaluation device described in any of [2] to

[17] below can be provided. [2] The image evaluation device described in [1], wherein the processor is configured to further perform a shooting evaluation, which evaluates whether or not a shooting process should be performed on a display image in a shooting device, using a shooting evaluation model constructed by machine learning using information on whether or not a shooting process has been performed. [3] The image evaluation device described in [2], wherein the shooting process includes a process that is captured in response to a user's shooting operation. [4] The image evaluation device described in any of [1] to [3], wherein the processor performs at least one of the first evaluation and the second evaluation. [5] The image evaluation device described in [2] or [3], wherein the first target image is a captured image, and the second target image is a captured image on which the first processing has been performed. [6] The image evaluation device described in any of [1] to [5], wherein when the processor performs a first evaluation on the first target image, it outputs the result of the first evaluation in association with the first target image. [7] The image evaluation apparatus according to [6], wherein the processor displays the result of the first evaluation superimposed on the first target image when the result of the first evaluation indicates that the first processing should be performed on the first target image. [8] The image evaluation apparatus according to any one of [1] to [7], wherein the processor outputs the result of the second evaluation associated with the second target image when the second evaluation is performed on the second target image. [9] The image evaluation apparatus according to [8], wherein the processor displays the result of the second evaluation superimposed on the second target image when the result of the second evaluation indicates that the second processing should be performed on the second target image.

[10] The image evaluation apparatus according to [2] or [3], wherein the processor performs a shooting evaluation on a displayed image, and the result of the shooting evaluation indicates that a shooting process should be performed on the displayed image when the shooting device displays the result of the shooting evaluation superimposed on the displayed image.

[11] The image evaluation apparatus according to any one of [1] to

[10] , wherein the first evaluation model is reconstructed by relearning using images on which the first processing has been newly performed.

[12] The image evaluation apparatus according to any one of [1] to

[11] , wherein the second evaluation model is reconstructed by relearning using images on which the second processing has been newly performed.

[13] The image evaluation apparatus according to [2] or [3], wherein if new information is obtained regarding whether or not an image capture process has been performed, the image capture evaluation model is reconstructed by relearning using the newly obtained information regarding whether or not an image capture process has been performed.

[14] The image evaluation apparatus according to [2] or [3], wherein at least one of the first evaluation model, the second evaluation model, and the image capture evaluation model is stored in a device connected to the image evaluation apparatus or a device capable of communicating with the image evaluation apparatus.

[15] The image evaluation apparatus according to any one of [1] to

[14] , wherein the processor is configured to further perform a third evaluation, which evaluates whether or not a third processing should be performed on a third target image, or the results if a third processing is performed, using a third evaluation model constructed by machine learning using information regarding the performance of a third processing on each of a plurality of images.

[16] The image evaluation apparatus according to

[15] , wherein if multiple types of third processing are possible, a third evaluation model is constructed for each type of third processing, and the processor is configured to perform a third evaluation on a third target image for each type of third processing.

[17] An image evaluation device according to any one of [1] to

[16] , wherein the first process is the process of transferring an image and storing it in the destination device, and the second process is the process of sharing the image with other users.

[0010] Furthermore, the above-mentioned objectives are achieved by an image evaluation method described in any of the following

[18] to

[20] .

[18] An image evaluation method wherein the processor is capable of performing a first evaluation to evaluate whether a first processing should be performed on a first target image using a first evaluation model constructed by machine learning using information on whether a first processing has been performed on each of a plurality of images, and a second evaluation to evaluate whether a second processing should be performed on a second target image using a second evaluation model constructed by machine learning using information on whether a second processing has been performed on each of two or more images on which a first processing has been performed, and the processor performs at least one of the steps of performing the first evaluation and performing the second evaluation.

[19] The image evaluation method according to

[18] , wherein the processor further performs a step of performing a shooting evaluation to evaluate whether a shooting process should be performed on a display image in a shooting device using a shooting evaluation model constructed by machine learning using information on whether a shooting process has been performed.

[20] The image evaluation method according to

[18] or

[19] , wherein the processor further performs a third evaluation using a third evaluation model constructed by machine learning using information on the performance of the third processing on each of a plurality of images, to determine whether the third processing should be performed on the third target image, or to evaluate the results if the third processing is performed.

[0011] Furthermore, the program according to one embodiment of the present invention is a program that causes a computer to execute each step included in the image evaluation method described in any of

[18] to

[20] . Furthermore, the recording medium according to one embodiment of the present invention is a recording medium that is readable by a processor and on which a program that causes a computer to execute each step included in the image evaluation method described in any of

[18] to

[20] is recorded.

[0012] According to the present invention, when performing multiple processes on an image, it is possible to appropriately evaluate whether an image is one on which each process should be performed.

[0013] This figure shows an image evaluation apparatus and related devices according to one embodiment of the present invention. This figure shows the hardware configuration of the image evaluation apparatus according to one embodiment of the present invention. This is an explanatory diagram of the functions of the image evaluation apparatus according to one embodiment of the present invention. This figure shows an example of the output for the results of the first evaluation. This figure shows an example of the output for the results of the second evaluation. This figure shows an example of the output for the results of the image capture evaluation. This figure shows the machine learning procedure according to one embodiment of the present invention. This figure shows the procedure for the image evaluation method according to one embodiment of the present invention. This figure shows the flow of the image capture evaluation process. This figure shows the flow of the first evaluation process. This figure shows the flow of the second evaluation process. This figure shows the relationship between multiple processes and the images on which each process has been performed.

[0014] The present invention relates to an image evaluation apparatus and an image evaluation method. Furthermore, the present invention can also be applied to computer-executable programs, program products, and recording media on which programs are stored. Specific embodiments of the present invention will be described below with reference to the accompanying drawings. For convenience of explanation, the following description may sometimes be given from the perspective of a GUI (Graphical User Interface).

[0015] Furthermore, the fundamental data processing technologies for realizing the present invention (communication / transmission technologies, data acquisition technologies, data recording technologies, data processing / analysis technologies, machine learning technologies, image processing technologies, image display technologies, and visualization technologies, etc.) are known technologies, and therefore, explanations thereof will be omitted. Regarding the equipment and devices used to apply the above-mentioned known technologies, it is advisable to appropriately select those available at the time of implementing the present invention.

[0016] Furthermore, in this specification, the concept of "device" includes both a single device that performs a specific function and a combination of multiple devices that exist independently and in a distributed manner while cooperating (linking) to perform a specific function. Furthermore, in this specification, the concept of "system" includes both a system composed of multiple devices connected in a manner that allows them to communicate with each other and a system composed of a single device.

[0017] Furthermore, in this specification, "person" means an entity that performs a specific action, and includes individuals, groups, corporations and other legal entities, and organizations, and may also include computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) is a system that realizes intelligent functions such as reasoning, prediction, and judgment using hardware and software resources.

[0018] Furthermore, in this specification, unless otherwise specified, "image" refers to digital image data (hereinafter simply referred to as "image data"). Image data includes, for example, lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format, etc.

[0019] <<About one embodiment of the present invention>> (Outline of this embodiment) An outline of one embodiment of the present invention (hereinafter, this embodiment) will be described below. In this embodiment, a user can evaluate an image using an image evaluation device, and specifically, an evaluation can be performed regarding the capture of an image and the processing of the image.

[0020] Image processing refers to processes performed on images, and specifically includes displaying, transmitting, transferring, copying, converting data formats, deleting, sharing, processing / editing, analyzing, encrypting (encoding) and decrypting images, compressing, correcting, printing, adding to albums (albums as composite images and albums as groups of images), and other image-related processing. Image sharing also includes posting images to social networking services (SNS) (specifically, uploading them to SNS servers) and uploading images to cloud servers to make them available to other users.

[0021] In this embodiment, the image evaluation device (hereinafter referred to as the image evaluation device 10) is mounted on a shooting device having a shooting lens and an image sensor. The shooting device consists of a digital camera or a terminal with a camera, and is a known configuration except that it has the function of the image evaluation device 10. Below, we will describe the case in which the image evaluation device 10 is mounted on a digital camera (hereinafter referred to as the digital camera 12). However, the contents described below can also be applied when the image evaluation device 10 is mounted on a camera-equipped terminal such as a smartphone.

[0022] In this embodiment, the digital camera 12 has a communication function and, as shown in Figure 1, can communicate with other communication devices via a network N such as the Internet and a mobile communication network. Devices that can communicate with the digital camera 12 include a user-accessible terminal (hereinafter referred to as the user terminal 14) and a server.

[0023] The user terminal 14 is a terminal used by the user of the image evaluation device 10, and consists of a smartphone, tablet, PC (Personal Computer), wearable device, or other information and communication terminal. The user terminal 14 is equipped with a display as a display device and can display images and other data transferred from the digital camera 12 on the display. The user terminal 14 is not limited to a device owned by the user, but may also be a device not owned by the user, such as a terminal installed in a store, which can be used by entering account information and logging in.

[0024] The server includes a server for cloud services (hereinafter referred to as the cloud server 16) and a server that stores and distributes information posted to SNS (hereinafter referred to as the SNS server 18). The cloud server 16 receives and stores images transmitted from the digital camera 12 and the user terminal 14. The SNS server 18 receives posting information from the user terminal 14 based on posting operations performed on the user terminal 14 and distributes that posting information. The posting information may include images specified on the user terminal 14 side during the posting operation. Images stored on the cloud server 16 and images included in posting information distributed from the SNS server 18 can be shared by multiple users.

[0025] The digital camera 12, together with the aforementioned communication devices (i.e., the user terminal 14, the cloud server 16, and the SNS server 18), forms an image evaluation system. In other words, the digital camera 12 and the devices connected to the digital camera 12 via the network N cooperate with each other to form an image evaluation system.

[0026] Furthermore, the digital camera 12 displays the live view image as a display image on the display or viewfinder, just like a known digital camera. When the user performs a shooting operation (specifically, pressing the shutter button) while the live view image is being displayed, the digital camera 12 performs a shooting process and captures an image corresponding to the live view image displayed at that time. The captured image (hereinafter also referred to as the captured image) is stored in the memory unit built into the digital camera 12.

[0027] The image may be a still image, a video, or even just a portion of the multiple frames that make up a video. The following explanation will assume the image is a still image. Furthermore, the shooting process may include both a process that takes a picture in response to user input and a process that automatically takes a picture according to pre-set conditions. In the following explanation, the shooting process will be assumed to be a process that takes a picture in response to user input.

[0028] In some cases, multiple processes may be performed in stages on some of the captured images stored in the memory unit of the digital camera 12. Specifically, some of the captured images may be transferred from the digital camera 12 and transmitted via the network N to a user terminal 14 and a cloud server 16, etc., and stored on these devices. Here, the image transfer process corresponds to an example of the "first process" of the present invention, and is a process of transferring images and storing them on the destination device.

[0029] Furthermore, some of the transferred captured images may be shared with other users. Specifically, some of the captured images transferred to the user terminal 14 may be sent from the user terminal 14 to the cloud server 16 and stored on the cloud server 16, or they may be uploaded from the user terminal 14 to the SNS server 18 as information posted to SNS. Here, the image sharing process corresponds to an example of the "second process" of the present invention, and is a process for sharing images with other users.

[0030] Incidentally, the task of selecting images from the captured images to be processed for each type of processing has traditionally been performed by the user of the digital camera 12, but this selection process can be time-consuming and laborious. Furthermore, the criteria for image selection may differ from user to user, and the selection criteria may change depending on the content of the processing.

[0031] In contrast, in this embodiment, the image evaluation device 10 can automatically select an image suitable for each processing step. More specifically, in this embodiment, AI technology can be used to select an appropriate image based on the user's preferences and tastes.

[0032] Furthermore, traditionally, when taking an image, the user, as the photographer, would activate the digital camera, display the live view image, evaluate the quality of the live view image (for example, the subject's condition and composition), and then take the picture at the desired moment. However, if evaluating the quality of the image takes time, there is a risk of missing the appropriate shooting timing.

[0033] In contrast, in this embodiment, the image evaluation device 10 can determine whether or not to take a picture based on the user's preferences and tastes, and inform the user of the appropriate shooting timing based on the determination result. More specifically, in this embodiment, AI technology can be used to determine the appropriate shooting timing based on the user's preferences and tastes.

[0034] Furthermore, the criteria for whether or not to take an image, and the criteria for whether or not to perform a predetermined process on an image, are likely to vary depending on the scene and subject of the image. In addition, the factors that influence these criteria may differ from user to user. In response to this, this embodiment takes the above points into consideration and enables highly accurate and efficient image selection and determination of shooting timing based on the user's unique preferences and tastes.

[0035] (Example of Image Evaluation Device Configuration) As shown in Figure 2, the image evaluation device 10 according to this embodiment comprises a processor 10a, memory 10b, communication interface 10c, and storage 10d. The memory 10b stores an application program for creating image evaluation data (hereinafter referred to as the image evaluation program). The image evaluation program corresponds to the "program" of the present invention and is a program that controls the computer constituting the image evaluation device 10.

[0036] In other words, when the image evaluation program is executed by the processor 10a described above, the computer constituting the image evaluation device 10 performs its function as an image evaluation device 10, specifically performing evaluations related to image capture and the execution of each processing step. The image evaluation program may be obtained by reading it from a program product consisting of a computer-readable recording medium (more specifically, a non-temporary recording medium), or it may be obtained (downloaded) via a network such as the Internet or an intranet.

[0037] The image evaluation device 10 communicates with the user terminal 14, the cloud server 16, and the SNS server 18, etc., through the communication interface 10c, and can send and receive data with these devices. The image evaluation device 10 also further includes an input device 10e and an output device 10f, as shown in Figure 2. The input device 10e includes equipment for receiving user operations, such as a touch panel and cursor buttons. The output device 10f includes a display or a finder.

[0038] Furthermore, the image evaluation device 10 can freely access various types of data stored in the storage 10d. The data stored in the storage 10d includes captured images, temporarily stored live view images (frame images), image capture conditions, information on operations received from the user, and other data necessary for image evaluation. The storage 10d also stores training data used for machine learning and retraining, as well as the learning models constructed by machine learning.

[0039] The storage 10d may be built into or attached to the image evaluation device 10, or it may be configured as a NAS (Network Attached Storage) or the like. Alternatively, the storage 10d may be configured as an external device that is communicatively connected to the image evaluation device 10, such as a database server or online storage.

[0040] In this embodiment, as described above, the image evaluation device 10 is mounted on the digital camera 12, as shown in Figure 2. In other words, the aforementioned hardware components of the image evaluation device 10, namely the processor 10a, memory 10b, and communication interface 10c, are provided in the digital camera 12. The configuration and functions of the digital camera 12 equipped with the image evaluation device 10 are substantially the same as those of a generally known digital camera, except that it has the functions of the image evaluation device 10.

[0041] That is, when a shooting operation is performed, the digital camera 12 shoots a subject within the shooting angle according to preset shooting conditions to obtain (record) an image. The shooting conditions include the exposure time (exposure amount), ISO sensitivity, the focus position at the time of shooting, and the focal length. Further, when acquiring a shot image, the digital camera 12 generates incidental information regarding the shooting date and time, shooting location, shooting conditions, etc., associates the generated incidental information with the shot image, and stores it. For example, the above-described incidental information is written as tag information in a predetermined area of an image data file indicating the shot image.

[0042] The digital camera 12 also includes a display as a display device and a finder (specifically, an electronic viewfinder), and can display the shot image on the display or the finder. Also, the digital camera 12 can display an interface screen (operation screen) for receiving user operations on the display. Further, the digital camera 12 can display the result of evaluation described later on the display.

[0043] The digital camera 12 can also transfer the stored shot images and store them in a device at the transfer destination. Specifically, an image selected from a plurality of shot images can be transmitted through the network N to the user terminal 14 or the cloud server 16.

[0044] The digital camera 12 can also acquire live view images before and after shooting and display the acquired live view images. Specifically, while the user is half-pressing the shooting button of the digital camera 12, the digital camera 12 continuously acquires an image of a subject within the shooting angle, that is, a live view image, at a predetermined frame rate. The live view image is displayed in real time as a through image on the display or the finder of the digital camera 12. Also, in the present embodiment, the digital camera 12 can temporarily store the live view images during a certain period before and after the shooting time, or the live view images every n frames (n is a natural number) in the memory 10b or the storage 10d.

[0045] Furthermore, the digital camera 12 may analyze the captured image to detect the subject area in the image and identify the type of subject present in the detected subject area. If the subject present in the subject area is a person, the digital camera 12 may also identify the facial expression of the person. In addition, the digital camera 12 may identify the resolution (specifically, the degree of blur or blur, etc.) or sharpness of the subject present in the subject area based on the detected subject area. Furthermore, the digital camera 12 may calculate a score for the subject area (i.e., the image of the subject) based on the identified resolution, sharpness, and facial expression, etc. The above functions are realized by applying known image analysis techniques, subject detection techniques, and image processing techniques.

[0046] Furthermore, if a sharing process is performed on the user terminal 14 for images transferred from the digital camera 12, the digital camera 12 can identify the images on which the sharing process was performed through communication with the user terminal 14.

[0047] (Regarding the functions of the image evaluation device) Next, the functions of the image evaluation device 10 according to this embodiment will be described. As shown in Figure 3, the image evaluation device 10 has an image acquisition function, a labeling function, a learning function, a relearning function, an evaluation function, and an output function. These functions are realized through the cooperation of the hardware equipment of the computer constituting the image evaluation device 10, namely the processor 10a, memory 10b, communication interface 10c, and storage 10d mentioned above, and the image evaluation program stored in that computer. Furthermore, some of the above functions (for example, the labeling function, learning function, relearning function, and evaluation function, etc.) may be realized by artificial intelligence (AI). The following describes each of the functions.

[0048] [Image Acquisition Function] The image acquisition function is a function that acquires an image (captured image) captured by a photographing device such as a digital camera 12. The captured images to be acquired are two or more images, which constitute an image group. The captured image may be an image captured by a camera mounted on the user terminal 14. In that case, the image evaluation apparatus 10 acquires (specifically, receives) the captured image from the user terminal 14 via the network N through communication with the user terminal 14. However, it is not limited to this, and among the existing images existing on the network N, an image selected by the user may be acquired (specifically, downloaded) via the network N. Also, in the present embodiment, while the digital camera 12 displays a live view image on a display or the like before and after shooting, the image evaluation apparatus 10 can acquire the live view image during a certain period before and after the shooting time, or the live view image every n frames (n is a natural number).

[0049] [Labeling Function] The labeling function is a function that performs labeling on the captured image or the live view image acquired by the image acquisition function in order to create learning data used in machine learning described later. Specifically, for example, a label of "shooting done" is assigned to the captured image, and a label of "no shooting" is assigned to the live view image (non-captured image) that has not been captured. These labels correspond to information regarding whether or not the shooting process has been performed.

[0050] Also, for a captured image on which a predetermined process has been performed, a label indicating that the predetermined process has been performed is assigned. For example, for a captured image on which a transfer process (first process) has been performed, a label of "transfer done" is assigned, and for a captured image on which the transfer process has not yet been performed, a label of "no transfer" is assigned. These labels correspond to information regarding whether or not the transfer process (first process) has been performed.

[0051] Furthermore, among the captured images that have undergone transfer processing, those that have undergone sharing processing (second processing) are labeled "shared," while those that have not yet undergone sharing processing are labeled "not shared." These labels correspond to information regarding whether or not sharing processing (second processing) has been performed.

[0052] [Learning Function] The learning function is a function that constructs a learning model by performing machine learning using captured images or live view images that have been labeled by the labeling function as learning data. Specifically, the image evaluation device 10 performs machine learning using captured images labeled "taken" and live view images labeled "not taken" as learning data. In other words, the image evaluation device 10 performs machine learning using information on whether or not a shooting process has been performed. Through this machine learning, a learning model (hereinafter referred to as the shooting evaluation model) is constructed to evaluate whether or not a shooting process should be performed on the live view image displayed on the digital camera 12.

[0053] Furthermore, the image evaluation device 10 performs machine learning using captured images labeled "transferred" and captured images labeled "not transferred" as training data. In other words, the image evaluation device 10 performs machine learning using information on whether or not a transfer process (first process) was performed on each of the multiple captured images. Through this machine learning, a learning model (hereinafter referred to as the first evaluation model) is constructed to evaluate whether a transfer process should be performed on the acquired captured images.

[0054] Furthermore, the image evaluation device 10 performs machine learning using captured images labeled "shared" and captured images labeled "not shared" as training data from among the captured images on which the transfer process has been performed. In other words, the image evaluation device 10 performs machine learning using information on whether or not the second process was performed on each of two or more captured images on which the first process was performed from among multiple captured images. Through this machine learning, a learning model (hereinafter referred to as the second evaluation model) is constructed for evaluating whether or not the sharing process should be performed on the captured images.

[0055] Furthermore, there are no particular limitations on the machine learning algorithms or types of machine learning; you should use those that are appropriate for the purpose of machine learning (in other words, the intended use of the learning model constructed by machine learning). The procedures for each of the three machine learning methods mentioned above will be explained in a later section.

[0056] [Retraining Function] The retraining function is a function that performs retraining for the purpose of reconstructing the learning models (i.e., the shooting evaluation model, the first evaluation model, and the second evaluation model) that were built using the machine learning described above. Retraining is performed when new training data is acquired.

[0057] To explain in more detail, for example, when a user takes a picture using the digital camera 12, a new captured image is acquired, and this captured image is labeled "taken". Live view images that were not captured are labeled "not taken". This provides new information on whether or not the shooting process was performed. The image evaluation device 10 can then use this newly acquired information as training data to retrain the shooting evaluation model.

[0058] Furthermore, after the construction of the first evaluation model, if images recorded by the digital camera 12 are transferred to the user terminal 14 or the cloud server 16, those images are labeled "transferred". On the other hand, images that are not transferred are labeled "not transferred". This allows for the acquisition of new information regarding whether or not a transfer process has been performed for each image. The image evaluation device 10 can then use the newly acquired information as training data to retrain the first evaluation model.

[0059] Furthermore, after the construction of the second evaluation model, if an image transferred to the user terminal 14 is shared by being uploaded to the cloud server 16 or sent to the SNS server 18 as posted information, that image will be labeled "shared". On the other hand, images that are not shared after being transferred will be labeled "not shared". This allows the system to acquire new information regarding whether or not a sharing process has been performed on the transferred images. The image evaluation device 10 can then use the newly acquired information as training data to retrain the second evaluation model.

[0060] The procedure for each retraining step is the same as the procedure for performing machine learning (which will be described later). Furthermore, if new images corresponding to training data are acquired, retraining can be performed at any time after acquiring those images.

[0061] [Evaluation Function] The evaluation function is a function that uses a learning model constructed by the learning function or retraining function to perform an evaluation of the image to be evaluated regarding the execution of image capture or processing. Specifically, the image evaluation device 10 (more specifically, the processor 10a) is configured to perform a first evaluation, a second evaluation, and an evaluation for image capture.

[0062] The first evaluation is to evaluate whether a transfer process should be performed on the first target image using the first evaluation model described above. The first target image is the image subject to the first evaluation, and in this embodiment, it is an image (captured image) taken by the digital camera 12. In the first evaluation, the captured image, which is the first target image, is input to the first evaluation model, and an evaluation result is output indicating whether or not a transfer process should be performed on the input captured image.

[0063] The second evaluation involves using the aforementioned second evaluation model to assess whether sharing processing should be performed on the second target image. The second target image is the image subject to the second evaluation, and in this embodiment, it is a captured image on which the transfer processing (first processing) has been performed, i.e., a transferred image. In the second evaluation, the transferred image, which is the second target image, is input to the second evaluation model, and an evaluation result is output indicating whether or not sharing processing should be performed on the input transferred image. Note that the second target image is not limited to a transferred image; for example, it may be a captured image on which the transfer processing has not yet been performed.

[0064] The shooting evaluation involves using the aforementioned shooting evaluation model to evaluate whether or not to perform shooting processing on the live view image displayed on the digital camera 12's screen. The shooting evaluation is performed while the live view image is displayed on the digital camera 12's screen. In the shooting evaluation, each live view image (more specifically, a frame image) is input to the shooting evaluation model, and an evaluation result is output indicating whether or not to perform shooting processing on the input live view image, that is, whether or not to photograph the subject included in the field of view at that time.

[0065] The image evaluation device 10 has the function of performing the three evaluations described above, but the user can specify which evaluation to perform. In other words, when an evaluation specification screen (not shown) is displayed on the digital camera 12's display, the user can specify at least one of the three evaluations through that specification screen. The image evaluation device 10 (more specifically, the processor 10a) receives the user's specification operation and performs the evaluation specified in that operation from among the three evaluations.

[0066] The results of each evaluation can be represented in various forms, such as a flag value consisting of two values ​​(0 and 1), a numerical value such as a score assigned within a range of 0 to 100 points, or a vector value or tensor value containing two or more values.

[0067] [Output Function] The output function is a function that outputs the evaluation results to the user when the evaluation function performs any of the first evaluation, second evaluation, or shooting evaluation.

[0068] Specifically, when the processor 10a of the image evaluation device 10 performs a first evaluation on a first target image, it outputs the result of the first evaluation in association with the first target image. One method of outputting the result of the first evaluation in association with the first target image is, for example, as shown in Figure 4, to display information indicating the result of the first evaluation (hereinafter referred to as first result information Ri1) on the display of the digital camera 12 while the captured image, which is the first target image, is being displayed on the display. First result information Ri1 is displayed when the result of the first evaluation indicates that a transfer process should be performed on the first target image, and specifically, it is text, symbols, marks, icons, or other display objects indicating the result. First result information Ri1 may also be displayed superimposed on the captured image while the captured image is being displayed. As an alternative example of outputting the results of the first evaluation in association with the first target image, the first result information Ri1 may be displayed near the thumbnail of the first target image while the digital camera 12's display is showing a list of thumbnail images of multiple captured images, including the first target image.

[0069] Furthermore, when the processor 10a performs a second evaluation on the second target image, it outputs the result of the second evaluation in association with the second target image. One method for outputting the result of the second evaluation in association with the second target image is, for example, as shown in Figure 5, to display information indicating the result of the second evaluation (hereinafter referred to as second result information Ri2) on the display of the user terminal 14 while the transferred image, which is the second target image, is being displayed on the display. Second result information Ri2 is displayed when the result of the second evaluation is that sharing processing should be performed on the second target image, and specifically consists of text, symbols, marks, icons, or other display objects indicating the result. When displaying second result information Ri2 on the user terminal 14 side, the processor 10a generates display data for second result information Ri2 and transmits the display data to the user terminal 14. Note that second result information Ri2 may also be displayed superimposed on the transferred image while the transferred image is being displayed. As an alternative example of outputting the results of the second evaluation in association with the second target image, the second result information Ri2 may be displayed near the thumbnail of the second target image while the thumbnail of each of the multiple transferred images, including the second target image, is being displayed in a list on the user terminal 14's display. Note that the display device on which the results of the second evaluation are shown is not limited to the user terminal 14's display, but may also be the display of the digital camera 12.

[0070] Furthermore, suppose that a shooting evaluation is performed on the live view image displayed on the digital camera 12's display, and the evaluation result indicates that shooting should be performed on the live view image. In this case, as shown in Figure 6, the processor 10a causes information indicating the result of the shooting evaluation (hereinafter referred to as third result information Ri3) to be displayed on the digital camera 12's display. Third result information Ri3 is text, symbols, marks, icons, or other display objects that indicate that shooting should be performed on the live view image displayed at that time. Note that third result information Ri3 may be displayed superimposed on the live view image as shown in Figure 6.

[0071] The method for outputting evaluation results is not limited to the above; for example, an audio or alarm sound indicating information corresponding to the evaluation results may be played and output from a speaker or similar device.

[0072] (Example of Machine Learning Implementation Procedure) Next, we will explain an example of the implementation procedure for machine learning using the image evaluation device 10 described above. In the following, we will refer to the machine learning for constructing the image capture evaluation model as "image capture learning," the machine learning for constructing the first evaluation model as "first learning," and the machine learning for constructing the second evaluation model as "second learning," and explain the implementation procedure for each of these machine learning processes.

[0073] [Procedure for Performing Training for Photography] When performing training for photography, training data to be used for training for photography is generated. As mentioned above, the training data consists of captured images labeled "Shooting occurred" and live view images (non-shot images) labeled "No shooting occurred". In training for photography, captured images labeled "Shooting occurred" are used as correct images, and live view images labeled "No shooting occurred" are used as incorrect images.

[0074] In the training for image capture, as shown in Figure 7, encoding is performed for both correct and incorrect images. Specifically, the image evaluation device 10 uses the image analysis function of the digital camera 12 to calculate the feature quantities of each image. Image feature quantities include the color, brightness, resolution, data size, position in the field of view, size ratio to the field of view, degree of focus, degree of blur / blur, image composition, and combinations thereof. These feature quantities may be identified, for example, by applying known image analysis techniques to analyze the region in the image where the subject exists.

[0075] However, features may include values ​​other than those listed above. Furthermore, there are no particular limitations on the representation format of the features; for example, features may be one-dimensional or multi-dimensional vector values. Features may also be values ​​output by inputting an image into a feature calculation model pre-built through machine learning. While the features output from a trained model may not be meaningful to humans, they can be used as image features as long as they are uniquely output when a single image is input.

[0076] After calculating feature quantities for both correct and incorrect images, decoding is performed based on these feature quantities, as shown in Figure 7. Specifically, the image evaluation device 10 identifies the correspondence between the feature quantities of each image and the label assigned to that image (in other words, information indicating whether it is a correct or incorrect image), and models this correspondence. Through these steps, the image acquisition training is performed, and as a result, an image acquisition evaluation model is constructed.

[0077] Furthermore, in this embodiment, the training data used for shooting training, namely the captured images and live view images, are acquired based on the same user's operations, specifically operations performed through the same digital camera 12. As a result, the evaluation model (shooting evaluation model) constructed through shooting training reflects the preferences of the user who provides the training data, particularly their preferences regarding shooting (what kind of images they want to capture).

[0078] [Procedure for the First Learning Process] When the first learning process is performed, training data to be used for the first learning process is generated. As mentioned above, the training data consists of captured images labeled "transferred" (transferred images) and captured images labeled "not transferred". In the first learning process, captured images labeled "transferred" are used as correct images, and captured images labeled "not transferred" are used as incorrect images.

[0079] The first learning process is essentially the same as the learning process for capturing images. It involves encoding both correct and incorrect images to calculate the feature vectors for each image, followed by decoding based on the calculated feature vectors to model the correspondence between the feature vectors of each image and the labels assigned to them. This constructs the first evaluation model.

[0080] Furthermore, in this embodiment, the training data used for the first training, i.e., the transferred captured images and the captured images that were not transferred, are acquired based on the same user's operations, more specifically, operations performed through the same digital camera 12. As a result, the evaluation model (first evaluation model) constructed by the first training reflects the preferences of the user who provided the training data. Moreover, the provider of the training data used for the first training and the provider of the training data used for shooting training are both the same user. This allows the user's preferences reflected in the construction of the shooting evaluation model to also be reflected in the construction of the first evaluation model. In particular, the first evaluation model reflects the user's preferences regarding captured images (what kind of images they like).

[0081] [Procedure for the Second Learning Process] When conducting the second learning process, training data to be used for the second learning process is generated. As mentioned above, the training data consists of captured images labeled "shared" and captured images labeled "not shared". In the second learning process, captured images labeled "shared" are used as correct images, and captured images labeled "not shared" are used as incorrect images.

[0082] The second learning process is largely the same as the initial learning process for capturing images. Encoding is performed for both correct and incorrect images to calculate the feature vectors for each image. Then, decoding is performed based on the calculated feature vectors to model the correspondence between the feature vectors of each image and the labels assigned to them. This constructs the second evaluation model.

[0083] Furthermore, in this embodiment, the training data used for the second training, i.e., shared and transferred images, and transferred images that were not shared, are acquired based on the operations of the same user, more specifically, operations performed through the same user terminal 14. As a result, the evaluation model (second evaluation model) constructed by the second training reflects the preferences of the user who provided the training data. Moreover, the source of the training data used for the second training, the training data used for the first training, and the training data used for shooting training is the same user. This allows the user's preferences reflected in the construction of the shooting evaluation model and the first evaluation model to also be reflected in the construction of the second evaluation model. In particular, the second evaluation model reflects the user's preferences regarding image sharing (what kind of images they want to share).

[0084] (Image Evaluation Flow in This Embodiment) Next, the data processing related to image evaluation performed by the image evaluation device 10 according to this embodiment (hereinafter referred to as the image evaluation flow) will be described with reference to Figures 8 to 11.

[0085] The image evaluation flow employs the image evaluation method of the present invention. In other words, each step in the image evaluation flow described below corresponds to a component of the image evaluation method of the present invention. However, the image evaluation flow described below is merely illustrative, and within the scope of the present invention, some steps in the image evaluation flow may be deleted, new steps may be added to the image evaluation flow, or the order in which two steps in the image evaluation flow are performed may be changed.

[0086] As shown in Figure 8, the image evaluation flow includes a learning step (S001), an image capture evaluation step (S002), a first evaluation step (S003), and a second evaluation step (S004). Each of these steps is performed by the processor 10a of the image evaluation device 10. In the image evaluation flow, the processor 10a is capable of performing the image capture evaluation step S002, the first evaluation step S003, and the second evaluation step S004, and performs at least one of the first evaluation step S003 and the second evaluation step S004. At this time, the user can specify which evaluation step to perform. The user can also specify whether or not to perform the image capture evaluation step S002. The following describes each step.

[0087] [Learning Process] The learning process S001 is performed before the subsequent evaluation processes S002 to S004 are carried out. In the learning process, the processor 10a performs the first learning, second learning, and learning for shooting in the manner described above, for the purpose of constructing the first evaluation model, the second evaluation model, and the evaluation model for shooting, respectively. Each evaluation model constructed by each learning process is stored in the storage 10d of the processor 10a.

[0088] [Shooting Evaluation Process] The shooting evaluation process S002 is performed in conjunction with the startup of the digital camera 12. In the shooting evaluation process S002, as shown in Figure 9, the processor 10a performs a shooting evaluation for each live view image displayed on the digital camera 12's display (S011). In the shooting evaluation, the processor 10a evaluates whether or not to perform a shooting process on the live view image by inputting the live view image into the shooting evaluation model.

[0089] If the result of the shooting evaluation indicates that shooting should be performed on the live view image at that time (S012), the processor 10a displays a third result information Ri3 indicating the result, and displays it, for example, superimposed on the live view image (S013). This allows the user of the digital camera 12 to be notified in real time of the optimal timing for shooting subjects or scenes that match the user's preferences. As a result, the user can take pictures without missing a specific moment. In particular, the above notification function is useful when shooting moving subjects or fleeting scenes.

[0090] Then, when the user performs a shooting operation and the digital camera 12 performs the shooting process (S014), a new image is acquired. The processor 10a assigns the label "Shooting" to the new image and stores the image with this label in the storage 10d of the image evaluation device 10 (S015). As a result, new training data is acquired that will be used for retraining to reconstruct the shooting evaluation model.

[0091] When training data for retraining is obtained, the processor 10a performs retraining using that training data (S016). This reconstructs the shooting evaluation model. As a result, the shooting evaluation model better reflects (feeds in) the user's preferences, so in subsequent shooting evaluations, the user can be notified of the timing when it is possible to shoot subjects and scenes that better match the user's preferences.

[0092] The processor 10a may acquire live view images for a certain period before and after the shooting process, i.e., live view images that were not captured, and assign the label "No Capture" to each live view image. In this case, the above retraining may be performed using the live view images labeled "No Capture" as training data. Furthermore, the timing of the retraining is not particularly limited; it may be performed at any time after the training data for retraining has been acquired.

[0093] The above series of steps are repeated each time the user uses the digital camera 12. This updates the shooting evaluation model according to the user's latest preferences. In other words, the more the user uses the digital camera 12 equipped with the image evaluation device 10, the more the retraining is repeated, making it more suitable to the user's preferences. Furthermore, the accuracy of evaluations using the shooting evaluation model improves. As a result, the user's shooting experience can be improved by utilizing the results of evaluations using the shooting evaluation model (shooting evaluation). Consequently, the frequency of use of the digital camera 12 can be increased, or the range of use of the digital camera 12 can be expanded.

[0094] [First Evaluation Step] The first evaluation step S003 is performed, for example, when a captured image is displayed on the display of the digital camera 12. In the first evaluation step S003, as shown in Figure 10, the processor 10a performs a first evaluation on the captured image displayed on the display of the digital camera 12 (S021). In the first evaluation, the processor 10a evaluates whether or not a transfer process should be performed on the captured image by inputting the captured image to be evaluated (first target image) into the first evaluation model.

[0095] If the result of the first evaluation is that a transfer process should be performed on the captured image being evaluated (S022), the processor 10a displays the first result information Ri1 indicating the result on the display of the digital camera 12, for example, by superimposing it on the captured image (S023). This allows the user of the digital camera 12 to be informed of which captured image should be transferred, taking into account the user's preferences.

[0096] Then, when the user performs a transfer operation on the currently displayed captured image, the digital camera 12 performs a transfer process on the captured image (S024). As a result, the captured image selected by the user is transferred to the user terminal 14, or in other words, a newly transferred captured image, i.e., a new transferred image, is acquired (S025).

[0097] The processor 10a assigns the label "transferred" to newly transferred images. As a result, new training data is acquired to be used for retraining to reconstruct the first evaluation model. When training data for retraining is obtained, the processor 10a performs retraining using that training data (S026). This reconstructs the first evaluation model. As a result, the first evaluation model better reflects (feeds in) the user's preferences, so in subsequent first evaluations, it is possible to more appropriately determine whether an image should be transferred or not based on the user's preferences. As a result, images to be transferred can be selected with high accuracy and efficiency.

[0098] The processor 10a may assign the label "Not Transferred" to the captured images that were not transferred, and use these labeled captured images as training data to perform the above retraining. Furthermore, retraining to reconstruct the first evaluation model may be performed at any time after acquiring the training data for retraining.

[0099] By performing the first evaluation step S003 in the manner described above, the system can assist the user by suggesting which images to transfer when the user performs the operation to transfer captured images. Furthermore, through retraining, the first evaluation model is updated according to the user's latest preferences, and repeated retraining makes it more suitable for the user's preferences. In addition, the accuracy of evaluation using the first evaluation model is improved.

[0100] [Second Evaluation Step] The second evaluation step S004 is performed, for example, when the transferred image that has been transferred to the user terminal 14 is displayed on the user terminal 14's display. In the second evaluation step S004, as shown in Figure 11, the processor 10a performs a second evaluation on the transferred image displayed on the user terminal 14's display (S031). In the second evaluation, the processor 10a evaluates whether or not sharing processing should be performed on the transferred image by inputting the transferred image to be evaluated (second target image) into the second evaluation model.

[0101] If the result of the second evaluation indicates that sharing should be performed on the transferred image being evaluated (S032), the processor 10a displays the second result information Ri2, which indicates the result, on the display of the user terminal 14, for example, by superimposing it on the captured image (S033). This allows the user to be informed of images (transferred images) that should be shared with other users, taking into account the user's preferences.

[0102] Then, when the user performs an operation to share the transferred image that is currently being displayed, the user terminal 14 performs the sharing process on the transferred image (S034). As a result, the transferred image selected by the user is uploaded to the cloud server 16 or SNS server 18, or in other words, a newly shared transferred image, i.e., a new shared image, is obtained (S035).

[0103] The processor 10a assigns the label "shared" to the new shared image. As a result, new training data is acquired to be used for retraining to reconstruct the second evaluation model. When training data for retraining is obtained, the processor 10a performs retraining using that training data (S036). This reconstructs the second evaluation model. As a result, the second evaluation model better reflects (feeds in) the user's preferences, so in subsequent second evaluations, it is possible to more appropriately determine whether an image should be shared or not based on the user's preferences. As a result, images to be shared can be selected with high accuracy and efficiency.

[0104] The processor 10a may also assign the label "not shared" to the transferred images that were not shared, and use these transferred images as training data to perform the above retraining. Furthermore, retraining to reconstruct the second evaluation model may be performed at any time after acquiring the training data for retraining.

[0105] By performing the second evaluation step S004 in the manner described above, the system can support the user's operation by suggesting images to share when the user performs an operation to share an image. Furthermore, through retraining, the second evaluation model is updated according to the user's latest preferences, and repeated retraining makes it more suitable for the user's preferences. In addition, the accuracy of evaluation using the second evaluation model is improved.

[0106] (Regarding the effectiveness of this embodiment) According to this embodiment, when multiple processes are performed on an image in stages, it is possible to appropriately evaluate whether the image is one on which each process should be performed.

[0107] For example, when multiple image processing steps are performed sequentially, as shown in Figure 12, an image to be processed is selected at each stage. Also, as can be seen from Figure 12, the later the processing stage, the fewer candidate images there are to process, and the more stringent the selection process becomes.

[0108] To illustrate with a specific example, a user who takes pictures using a digital camera 12 reviews multiple images after taking them, selects the images to be transferred from among the multiple images, and transfers them to a user terminal 14 or the like. The user may also share the transferred images with other users by uploading them to a cloud server 16 or sending them as posts on social media. At each stage of taking, transferring, and sharing, the user evaluates whether to take an image or to perform any processing on the image, but each evaluation can take a considerable amount of time and effort. Furthermore, the time required for evaluation may delay the timing of taking the picture and performing the processing, and as a result, the user may miss the perfect timing for taking a picture.

[0109] Furthermore, while the evaluation criteria for images differ for each process, each process (i.e., capturing, transferring, and sharing) is performed independently. Therefore, it is generally difficult to understand the user's intentions regarding the execution of the process. For example, even if an image previously posted on social media is used as the "correct" image, and the user tries to capture an image with the same or similar characteristics as this correct image, it is difficult for the user to capture an image with the same characteristics as the aforementioned correct image because the number of correct images is small.

[0110] Furthermore, the criteria for evaluating images vary depending on the user's preferences. Conventional systems have been unable to adequately reflect these individual preferences in their evaluations, resulting in a lack of sufficient accuracy in image evaluation.

[0111] In contrast, in this embodiment, the image evaluation device 10 constructs evaluation models for each process (i.e., a first evaluation model and a second evaluation model), and can perform evaluations for each process on whether each process should be performed on the image. Since each evaluation model can be used independently, the image evaluation device 10 can appropriately evaluate whether to transfer and share the image being evaluated. Since the evaluation models are constructed to reflect the user's preferences, the image evaluation device 10 can perform evaluations for each process while taking into account the user's preferences (specifically, the criteria for judging the quality of the image). This reduces the effort required for the user to select the images to be processed for each process.

[0112] Furthermore, in this embodiment, when image processing is performed based on user operations, the evaluation model is retrained and reconstructed with the aim of feeding this information back into the evaluation model. As a result, the more the image evaluation device 10 is used, the better it can understand user preferences and improve the accuracy of image evaluation.

[0113] Furthermore, in this embodiment, the image evaluation device 10 constructs a shooting evaluation model and can perform an evaluation of the live view image using the shooting evaluation model while the live view image is displayed on the digital camera 12's display. This allows the device to evaluate whether or not to perform a shooting process on the displayed live view image and notify the user of the optimal shooting timing based on the evaluation result. As a result, the user can reduce the effort and time required to decide whether or not to take a picture, and can obtain higher quality images, thus improving overall user satisfaction.

[0114] <<Other Embodiments>> Although specific embodiments of the present invention have been described above, the embodiments described above are merely examples given to facilitate understanding of the present invention and do not limit it. That is, the present invention can be modified or improved from the embodiments described below, without departing from its spirit. Furthermore, the present invention includes equivalents thereof. Moreover, embodiments of the present invention may include forms that combine the embodiments described above with one or more of the following modifications.

[0115] (Regarding variations in the use of the evaluation model) In the embodiment described above, three evaluations (specifically, the first evaluation, the second evaluation, and the evaluation for shooting) are possible, and an evaluation model is used in each evaluation. However, it is not necessary to perform all three evaluations; only one evaluation may be performed, or two evaluations may be performed. In other words, one of the first evaluation model, the second evaluation model, and the evaluation model for shooting may be used alone, or two of these may be used in combination. In this case, the evaluation execution speed will be faster. Furthermore, the choice of which evaluation model to use may be determined according to the application of the image evaluation device 10. For example, the device may accept a user's specified operation and use the evaluation model specified by the user. This allows for flexible responses depending on the situation in which an image is selected and processed, i.e., according to the intended use of the image. Examples of combinations of evaluation models and their applications are given in the cases described in (t1) to (t10) below.

[0116] (t1) Perform an evaluation using the shooting evaluation model independently, and by utilizing the evaluation results, predict the scenes and compositions preferred by the user and propose the optimal shooting timing and angle of view to the user in real time. (t2) Perform an evaluation using the shooting evaluation model and an evaluation using the first evaluation model, and by combining the results of these two evaluations, propose images to the user for creating a photobook or album related to the user's experiences or travels. (t3) Perform an evaluation using the first evaluation model and an evaluation using the second evaluation model, and by combining the results of these two evaluations, classify the transferred images (transferred images) based on whether or not they have been shared. (t4) Perform evaluations using the first evaluation model, the second evaluation model, and the shooting evaluation model, and based on the results of these three evaluations, identify the user's shooting skills (shooting methods the user is good at, etc.). (t5) Perform evaluations using the first evaluation model, the second evaluation model, and the shooting evaluation model, and based on the results of these three evaluations, assign scores to the captured images from multiple perspectives.

[0117] (t6) Perform evaluations using the shooting evaluation model and the second evaluation model, and combine the results of these two evaluations to identify excellent images that have not been shared. (t7) Perform evaluations using the shooting evaluation model and the first evaluation model, and combine the results of these two evaluations to automatically generate an image selection filter that reflects the user's preferences. (t8) Perform evaluations using the first evaluation model, the second evaluation model, and the shooting evaluation model, and from the results of these three evaluations, create an album or similar that can identify changes in the user's preferences and interests. (t9) Perform evaluations using the shooting evaluation model and the second evaluation model, and combine the results of these two evaluations to perform AI-based image remastering. (t10) Perform evaluations using the first evaluation model, the second evaluation model, and the shooting evaluation model, and from the results of these three evaluations, identify trends in the user's shooting operations.

[0118] (Regarding the execution of the third evaluation) In the embodiment described above, the image evaluation device 10 (more specifically, the processor 10a) is configured to perform the first evaluation, the second evaluation, and the shooting evaluation. The evaluations that the image evaluation device 10 can perform are not limited to the three evaluations described above, and the image evaluation device 10 may be configured to further perform a third evaluation on a target image (hereinafter referred to as the third target image). In other words, the image evaluation flow described above may further include a step of performing a third evaluation on the third target image.

[0119] The third target image is, for example, an image (captured image) taken by a shooting device such as a digital camera 12, and may be, for example, a captured image before the transfer process is performed, a captured image after the transfer process has been performed but before the sharing process is performed, or a captured image after both the transfer process and the sharing process have been performed.

[0120] The third evaluation is an evaluation of whether the third processing should be performed on the second target image. The third processing is an processing performed on an image, and may be a processing other than the transfer processing (first processing) and the sharing processing (second processing), for example, the processing of deleting an image. The third evaluation is performed using a third evaluation model, which is constructed by machine learning using information on the performance of the third processing on each of multiple images. Specifically, images on which the third processing has been performed are labeled "processed," and images on which the third processing has been performed are labeled "not processed." The third evaluation model is constructed by performing machine learning on images with these labels as training data. Here, the above labels correspond to information on the performance of the third processing.

[0121] Furthermore, the machine learning process for constructing the third evaluation model can be carried out using essentially the same procedure as the machine learning process for constructing the first evaluation model and the second evaluation model. In addition, if new images with the above labels (i.e., training data) are acquired, the third evaluation model may be retrained using these newly acquired images as training data.

[0122] Furthermore, the third process may be a rating process in which a user viewing the captured images through the display of the digital camera 12 assigns a rating score to each captured image. In this case, the third evaluation may be the result of performing the rating process, that is, the value of the rating score. A third evaluation model for performing such a third evaluation is preferably constructed by machine learning using two or more captured images on which the rating process has been performed, and the rating scores assigned to each captured image, as training data.

[0123] Furthermore, if multiple types of third processing (for example, image deletion processing and rating processing) can be performed on the digital camera 12 and the user terminal 14, the third evaluation may be performed for each type of third processing. In other words, a third evaluation model may be constructed for each of the multiple types of third processing, and the image evaluation device 10 (more specifically, the processor 10a) may be configured to perform the third evaluation on the third target image for each type of third processing.

[0124] With the above configuration, when performing a third processing on an image, it is possible to appropriately evaluate whether or not an image should be processed, while maintaining a balance between the accuracy of the evaluation and the speed of the evaluation. Furthermore, with the above configuration, it is possible to respond more flexibly to situations in which images are selected for processing, that is, depending on the intended use of the image.

[0125] (Modifications concerning the construction and storage of evaluation models) In the embodiment described above, the processor 10a of the image evaluation device 10 performs machine learning to construct the first evaluation model, the second evaluation model, and the image capture evaluation model, and also performs retraining to reconstruct each evaluation model. However, it is not limited to this, and the device that performs machine learning to construct the evaluation model and the device that performs retraining to reconstruct the evaluation model may be other devices that can communicate with the image evaluation device 10, such as a user terminal 14 or a server such as a cloud server 16. In this case, the load (computational load) on the image evaluation device 10 due to machine learning can be reduced, thereby improving the flexibility of the entire image evaluation system including the image evaluation device 10.

[0126] Furthermore, in the above-described embodiment, the first evaluation model, the second evaluation model, and the shooting evaluation model are each recorded in the storage 10d of the image evaluation device 10. However, the embodiment is not limited to this, and at least one of the first evaluation model, the second evaluation model, and the shooting evaluation model may be stored in a device connected to the image evaluation device 10 or a device capable of communicating with the image evaluation device 10. Examples of devices connected to the image evaluation device 10 include flash memory and other recording media that can be attached to and detached from the digital camera 12 equipped with the image evaluation device 10. Examples of devices capable of communicating with the image evaluation device 10 include a user terminal 14 or a server such as a cloud server 16.

[0127] Furthermore, if any evaluation model is stored on the server, the image evaluation device 10 can use that evaluation model by communicating with the server. In this case, the evaluation model may also be made available through an API (Application Programming Interface). Also, by storing the evaluation model on the server, each of the multiple image evaluation devices 10 can use the common evaluation model. In addition, devices other than the image evaluation device 10 can also use the evaluation model by communicating with the server.

[0128] Furthermore, in the embodiment described above, processing content was proposed based on the data of a single user, in line with the processing tendencies of that user. However, if data from multiple users (a group of users) is treated as a single unit (for example, if data from all over Japan is treated as a single unit), processing content can be proposed that is in line with the processing tendencies generally performed by that group of users.

[0129] (Modifications of the images to be evaluated) In the embodiments described above, the images to be evaluated were captured images taken by a shooting device such as a digital camera 12, and live view images displayed on the shooting device. However, the images to be evaluated may be images other than captured images and live view images. For example, scanned images obtained using a scanner, captured images such as screenshots, or illustration images created using drawing tools may be set as the images to be evaluated. Alternatively, images generated by image generation AI may be set as the images to be evaluated.

[0130] (Regarding the computer constituting the image evaluation device) Some of the functions of the image evaluation device 10 may be performed by a user terminal 14 or a server (specifically, a cloud server 16 and an SNS server 18, etc.). In that case, the user terminal 14 or the server will cooperate with the digital camera 12 (shooting device) to constitute the image evaluation device 10. Alternatively, the user terminal 14 or the server may constitute the image evaluation device 10 on its own.

[0131] (Regarding the processor of the image evaluation device) In the above-described embodiment, each data processing performed by the image evaluation device 10 is performed on any computer. Furthermore, any computer may perform these data processing operations using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to cooperate with the program to perform the various data processing operations in the above-described embodiment, and can function as a unit or means in the above-described embodiment. Also, the execution order of the data processing by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific application, a workstation, or any other system capable of performing each data processing operation.

[0132] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of programmable logic devices such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), FPGA (Field Programmable Gate Array), dedicated circuits for performing specific data processing such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the hardware components may be a combination of different types of hardware. When multiple hardware components are configured to perform one or more data processing operations of a processor, these components may reside in physically separate devices or in the same device. Furthermore, in any embodiment, the order of the data processing operations performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware components are composed of electrical circuits (circuits) and the like, which are combinations of circuit elements such as semiconductor elements.

[0133] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may also consist of program code and multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located in physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0134] 10 Image evaluation device 10a Processor 10b Memory 10c Communication interface 10d Storage 10e Input device 10f Output device 12 Digital camera 14 User terminal 16 Cloud server 18 SNS server N Network Ri1 First result information Ri2 Second result information Ri3 Third result information

Claims

1. An image evaluation device equipped with a processor, wherein the processor is configured to perform: a first evaluation, which evaluates whether the first processing should be performed on a first target image using a first evaluation model constructed by machine learning using information on whether or not the first processing has been performed on each of a plurality of images; and a second evaluation, which evaluates whether the second processing should be performed on a second target image using a second evaluation model constructed by machine learning using information on whether or not the second processing has been performed on each of two or more of the plurality of images on which the first processing has been performed.

2. The image evaluation apparatus according to claim 1, wherein the processor is further configured to perform an image evaluation, which evaluates whether the image evaluation should be performed on a display image in an image capture device, using an image evaluation model constructed by machine learning using information on whether or not the image capture process has been performed.

3. The image evaluation apparatus according to claim 2, wherein the imaging process includes processing performed in response to the user's imaging operation.

4. The image evaluation apparatus according to claim 1, wherein the processor performs at least one of the first evaluation and the second evaluation.

5. The image evaluation apparatus according to claim 2, wherein the first target image is a captured image, and the second target image is a captured image on which the first processing has been performed.

6. The image evaluation apparatus according to claim 1, wherein the processor outputs the result of the first evaluation in association with the first target image when the first evaluation is performed on the first target image.

7. The image evaluation apparatus according to claim 6, wherein, if the result of the first evaluation is that the first processing should be performed on the first target image, the processor causes the display to superimpose the result of the first evaluation onto the first target image.

8. The image evaluation apparatus according to claim 1, wherein the processor outputs the result of the second evaluation in association with the second target image when the second evaluation is performed on the second target image.

9. The image evaluation apparatus according to claim 8, wherein, if the result of the second evaluation is that the second processing should be performed on the second target image, the processor causes the display to superimpose the result of the second evaluation onto the second target image.

10. The image evaluation apparatus according to claim 2, wherein the processor performs the shooting evaluation on the displayed image, and if the result of the shooting evaluation indicates that the shooting process should be performed on the displayed image, the shooting device displays the result of the shooting evaluation superimposed on the displayed image.

11. The image evaluation apparatus according to claim 1, wherein the first evaluation model is reconstructed by retraining using images on which the first processing has been newly performed.

12. The image evaluation apparatus according to claim 1, wherein the second evaluation model is reconstructed by retraining using images on which the second processing has been newly performed.

13. The image evaluation apparatus according to claim 2, wherein, when new information is obtained regarding whether or not the aforementioned shooting process has been performed, the shooting evaluation model is rebuilt by relearning using the newly obtained information regarding whether or not the aforementioned shooting process has been performed.

14. The image evaluation device according to claim 2, wherein at least one of the first evaluation model, the second evaluation model, and the imaging evaluation model is stored in a device connected to the image evaluation device or a device capable of communicating with the image evaluation device.

15. The image evaluation apparatus according to claim 1, wherein the processor is further configured to perform a third evaluation, which evaluates whether the third processing should be performed on a third target image, or the result if the third processing is performed, using a third evaluation model constructed by machine learning using information on the performance of the third processing on each of the multiple images.

16. The image evaluation apparatus according to claim 15, wherein, if multiple types of the third processing can be performed, the third evaluation model is constructed for each type of the third processing, and the processor is configured to perform the third evaluation on the third target image for each type of the third processing.

17. The image evaluation apparatus according to claim 1, wherein the first process is a process of transferring an image and storing it in a destination device, and the second process is a process for sharing the image with other users.

18. An image evaluation method comprising the steps of: performing a first evaluation on a first target image using a first evaluation model constructed by machine learning using information on whether or not a first processing has been performed on each of a plurality of images; and performing a second evaluation on a second target image using a second evaluation model constructed by machine learning using information on whether or not a second processing has been performed on each of two or more of the plurality of images on which the first processing has been performed, wherein the processor performs at least one of the steps of performing the first evaluation and performing the second evaluation.

19. The image evaluation method according to claim 18, further comprising the step of performing an image evaluation by the processor, which evaluates whether the image evaluation should be performed on a display image in an image capture device using an image evaluation model constructed by machine learning using information on whether or not the image capture process has been performed.

20. The image evaluation method according to claim 18, wherein the processor further performs a third evaluation, which evaluates whether the third processing should be performed on a third target image, or the result if the third processing is performed, using a third evaluation model constructed by machine learning using information on the performance of the third processing on each of the multiple images.

21. A program for causing a computer to perform each step included in the image evaluation method according to any one of claims 18 to 20.

22. A computer-readable recording medium on which a program is recorded causing a computer to perform each step included in the image evaluation method described in any one of claims 18 to 20.

Images