Program, image processing method, image processing device, model generation method, and image processing system
The system addresses the inefficiencies and biases of manual and biased automated image sorting by removing backgrounds and using a learning model to score subject images, ensuring accurate and efficient sorting.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- DAI NIPPON PRINTING CO LTD
- Filing Date
- 2022-07-14
- Publication Date
- 2026-07-07
AI Technical Summary
Manual image sorting is time-consuming and subjective, and existing automated methods using learning models can be biased due to imbalanced training data, leading to inaccurate sorting results.
A system that removes the background region from captured images and uses a learning model to score the suitability of the subject image for sale, eliminating the influence of the background and providing an objective sorting process.
Accurate and efficient image sorting is achieved without manual intervention, reducing workload and ensuring consistent results by focusing on the subject image quality.
Smart Images
Figure 0007885613000001 
Figure 0007885613000002 
Figure 0007885613000003
Abstract
Description
Technical Field
[0001] This application relates to a program, an image processing method, an image processing apparatus, a model generation method, and an image processing system.
Background Art
[0002] Patent Document 1 discloses a sales system that sells images taken by cameramen at sports events, events, etc. as photos and image data. In the sales system as disclosed in Patent Document 1, a large number of images taken by cameramen are sorted into images to be sold and images not to be sold, and such sorting processing is often performed manually.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When sorting images manually, depending on the number of images, it may take a long time to sort, and there is a possibility that differences may occur in the sorting results depending on the sensitivity of the person performing the sorting. Therefore, it is conceivable to sort images using a learning model generated by machine learning. However, if there is a bias in the images used for learning the learning model, it may not be possible to generate a learning model that can accurately sort images. Therefore, there is a problem that it is difficult to accurately sort images without spending time on the sorting process. Patent Document 1 does not mention the process of sorting images.
[0005] An object of the present disclosure is to provide a program or the like that can accurately sort images without taking time for the sorting process.
Means for Solving the Problems
[0006] A program according to one aspect of the present invention acquires a captured image of a subject, removes the background region from the acquired captured image, and inputs the subject image from which the background region has been removed into a learning model that has been trained to output information regarding the suitability of the captured image when the subject image from which the background region has been removed is input. The learning model outputs information regarding the suitability of the captured image, and the program causes the computer to execute a process to sort the captured image into suitability based on the information regarding the suitability of the captured image. [Effects of the Invention]
[0007] In one aspect of the present invention, images can be sorted with high accuracy without requiring time for the sorting process. [Brief explanation of the drawing]
[0008] [Figure 1] This is an explanatory diagram showing an example configuration of an image processing system. [Figure 2] This block diagram shows an example configuration of a server and administrator terminal. [Figure 3] This is an explanatory diagram showing an example of a database record layout stored on the server. [Figure 4] This is an explanatory diagram of the learning model. [Figure 5] This is a flowchart illustrating an example of the process for generating a learning model. [Figure 6] This flowchart shows an example of the image sorting process. [Figure 7] This is an explanatory diagram showing an example of the administrator terminal screen. [Figure 8] This flowchart shows an example of the sorting process procedure in Embodiment 2. [Modes for carrying out the invention]
[0009] The program, image processing method, image processing apparatus, model generation method, and image processing system of this disclosure will be described in detail below with reference to the drawings illustrating their embodiments.
[0010] (Embodiment 1) Figure 1 is an explanatory diagram showing an example configuration of an image processing system. This embodiment describes an image processing system that sorts images taken by a photographer into those to be sold and those to be sold. The image processing system of this embodiment is applicable to a system that sells images taken by photographers on the event organizer's side as photographs and image data to event participants at events such as professional sports matches and concerts. The image processing system of this embodiment includes a server 10, a camera 20, an administrator terminal 30, a photo vending machine 40, and a user terminal 50, and each of these devices is communicated to each other via a network N. The network N may be the Internet or a public telephone network, or it may be a LAN (Local Area Network) built within the facility where the image processing system is installed. In addition, the server 10 and any of the camera 20, administrator terminal 30, or photo vending machine 40 may be configured to directly transmit and receive information via wired communication or wireless communication via a cable.
[0011] Camera 20 is a shooting device that includes an imaging unit having a lens and an image sensor, a communication unit for connecting to the network N, etc. Camera 20 performs the process of taking a picture with the imaging unit in accordance with the operation of the shooting button and acquiring image data (hereinafter referred to as captured image), and the process of transmitting the acquired captured image to the server 10 from the communication unit. Camera 20 is configured to perform the process of acquiring one image (still image) in accordance with one operation of the shooting button, and the process of acquiring, for example, 30 or 15 images (video) per second. Camera 20 may also be configured to transmit the images acquired by shooting to the server 10 sequentially, or it may be configured to store the captured images in a storage unit and transmit them to the server 10 according to the operation of the cameraman. Camera 20 may be a camera that takes pictures while being held by a cameraman, or a camera that takes pictures with the shooting position fixed using a tripod or fixing device, and multiple cameras 20 may be installed in one event venue.
[0012] Server 10 is an image processing device capable of various information processing and information transmission / reception, and can be a server computer, personal computer, etc. Server 10 acquires images captured by camera 20 and sorts the acquired images to determine whether or not they should be sold. In this embodiment, Server 10 uses a learning model 12M (see Figure 2) when sorting whether or not the captured images should be sold. Administrator terminal 30 is a terminal used by an administrator who manages the captured images sold by Server 10, and can be a personal computer, tablet terminal, smartphone, etc. The administrator then sorts the captured images that Server 10 has sorted to be sold using the learning model 12M to determine whether or not they should truly be sold.
[0013] Furthermore, Server 10 processes the sale of captured images sorted for sale by itself or the administrator via Network N. The captured images sold by Server 10 are purchased via Photo Vending Machine 40 or User Terminal 50. Photo Vending Machine 40 is a terminal installed at the venue where the event in which the camera 20 took place was held, and is equipped with a communication unit for communication with Server 10, a touch panel, a payment processing unit, a printing unit, etc. Photo Vending Machine 40 displays the captured images sold by Server 10 on the touch panel and processes the selection of the captured images to be purchased. Photo Vending Machine 40 also processes the payment processing related to the purchase of captured images using the payment processing unit, and prints the processed captured images using the printing unit and provides them to the purchaser. The payment processing may be any payment method such as cash payment, electronic money payment, credit card payment, or app payment. User Terminal 50 is a terminal used by the user who purchases the captured images, and is a smartphone, tablet terminal, etc. User Terminal 50 has a browser installed for browsing websites via Network N, and uses the browser to view and purchase the captured images sold by Server 10 and process the payment related to the purchase. When purchasing images using the user terminal 50, the images are downloaded from the server 10 to the user terminal 50, and the user terminal 50 can print them by sending the images to a printer it can communicate with. In addition to the configuration in which images purchased using the user terminal 50 are downloaded to the user terminal 50, the images may also be downloaded from the server 10 to a printer installed in a convenience store, for example, and printed there, or they may be sent to a designated printing company and the printed photos delivered to the user's home or other location.
[0014] Figure 2 is a block diagram showing an example configuration of a server 10 and an administrator terminal 30. The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, etc., and these units are connected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or an AI chip (AI semiconductor). The control unit 11 executes the information processing and control processing that the server 10 should perform by appropriately executing the program 12P stored in the storage unit 12.
[0015] The storage unit 12 includes RAM (Random Access Memory), flash memory, hard disk, SSD (Solid State Drive), etc. The storage unit 12 stores the program 12P (program product) executed by the control unit 11 and various data. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the program 12P. The program 12P and various data may be written to the storage unit 12 during the manufacturing stage of the server 10, or the control unit 11 may download them from other devices via the communication unit 13 and store them in the storage unit 12. The storage unit 12 also stores, for example, a trained model 12M that has been trained on training data by machine learning. The trained model 12M is a trained model that has been trained to output information indicating whether or not a captured image (more precisely, a subject image from which the background region has been removed) taken by the camera 20 is suitable for sale when it is input. An image suitable for sale is, for example, an image that is in focus on the subject, has a good composition or angle, or is generally appealing to people—in other words, an image that people would want to buy. Below, an image suitable for sale will be referred to as a "good image," and an image unsuitable for sale will be referred to as a "bad image," with the quality of the image being whether or not it is suitable for sale. The learning model 12M is intended to be used as a program module that constitutes artificial intelligence software. The learning model 12M performs a predetermined operation on the input value and outputs the operation result, and the memory unit 12 stores data such as the coefficients and thresholds of the function that defines this operation as the learning model 12M.
[0016] Further, the storage unit 12 stores a photographed image DB 12a, a sorted image DB 12b, a determination result DB 12c, and a sales image DB 12d. The photographed image DB 12a is a database in which photographed images captured by the camera 20 are accumulated. The sorted image DB 12b is a database in which the results of sorting the photographed images into good images or bad images using the learning model 12M are stored. The determination result DB 12c is a database in which the results of the administrator's determination of good or bad for the photographed images sorted into good images using the learning model 12M are stored. The sales image DB 12d is a database in which the photographed images determined to be good images by the administrator and targeted for sale are stored. Part or all of the learning model 12M, the photographed image DB 12a, the sorted image DB 12b, the determination result DB 12c, and the sales image DB 12d may be stored in another storage device connected to the server 10, or may be stored in another storage device to which the server 10 can communicate.
[0017] The communication unit 13 is a communication module for performing processing related to wired communication or wireless communication, and transmits and receives information to and from other devices via the network N. The input unit 14 receives an operation input by the user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various types of information according to an instruction from the control unit 11. Part of the input unit 14 and the display unit 15 may be configured as an integrated touch panel, and the touch panel may also be configured to be externally attached to the server 10.
[0018] In this embodiment, the server 10 may be a multi-computer composed of a plurality of computers, a virtual machine virtually constructed by software, or a cloud server. Further, the input unit 14 and the display unit 15 of the server 10 are not essential, and it may be configured to receive operations through a connected computer, or may be configured to output information to be displayed to an external display device. Further, the server 10 may include a reading unit that reads a non-temporary computer-readable portable storage medium 10a, and the program 12P may be read from the portable storage medium 10a using the reading unit and stored in the storage unit 12. Note that the program 12P may be executed on a single computer, or may be executed on a plurality of computers interconnected via the network N.
[0019] The administrator terminal 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, a display unit 35, etc., and these units are connected via a bus. Each of the control unit 31, the storage unit 32, the communication unit 33, the input unit 34, and the display unit 35 has the same configuration as the control unit 11, the storage unit 12, the communication unit 13, the input unit 14, and the display unit 15 of the server 10, so the description of the configuration is omitted. Since the user terminal 50 has the same configuration as the administrator terminal 30, the illustration and the description of the configuration are omitted. The photo vending machine 40 has a settlement processing unit and a printing unit in addition to the same configuration as the administrator terminal 30, but the detailed description of the configuration is omitted.
[0020] FIG. 3 is an explanatory diagram showing an example of the record layout of the DBs 12a to 12d stored in the server 10. FIG. 3A shows the photographed image DB 12a, FIG. 3B shows the sorted image DB 12b, FIG. 3C shows the determination result DB 12c, and FIG. 3D shows the sales image DB 12d. The photographed image DB 12a, the sorted image DB 12b, the determination result DB 12c, and the sales image DB 12d are each provided for each event to be photographed, and are stored in the storage unit 12 in association with the event ID assigned to each event.
[0021] The captured image DB12a shown in Figure 3A includes columns for image ID, file name, date and time of capture, and location, and stores information about the captured image associated with the image ID. The image ID column stores identification information (image ID) uniquely assigned to the captured image taken by the camera 20. The file name column stores the folder name and file name for reading the captured image stored in the storage unit 12. The captured images acquired from the camera 20 are stored in a predetermined area (image folder) of the storage unit 12. The date and time of capture column stores the date and time the image was taken, and the location column stores information about the location. The location information may include the address of the location, the name of the building at the location, the name of the event being photographed, information indicating the location within the event venue, etc. The contents of the captured image DB12a are not limited to the example shown in Figure 3A; for example, information about the camera 20 and the photographer who took the picture may also be stored.
[0022] The sorting image DB12b shown in Figure 3B includes an image ID column and a sorting result column, and stores the result (good or bad) of the server 10 sorting each captured image into a good or bad image using the learning model 12M, associating it with the image ID of each captured image registered in the captured image DB12a. The judgment result DB12c shown in Figure 3C includes an image ID column and an administrator judgment result column, and stores the result (good or bad) of the administrator sorting each captured image into a good or bad image, associating it with the image ID of each captured image whose sorting result stored in the sorting image DB12b is "good," i.e., each captured image sorted as a good image by the server 10. The sales image DB12d shown in Figure 3D includes an image ID column and a file name column, and stores the folder name and file name for reading the captured images from the storage unit 12, associating it with the image ID of each captured image whose administrator judgment result stored in the judgment result DB12c is "good," i.e., each captured image judged as a good image by the administrator. The contents of the sorting image DB12b, judgment result DB12c, and sales image DB12d are not limited to the examples shown in Figures 3B to 3D.
[0023] Figure 4 is an explanatory diagram of the learning model 12M. Figure 4A shows an example of the configuration of the learning model 12M, and Figure 4B shows an example of a subject image, which is input data for the learning model 12M. The learning model 12M shown in Figure 4A is trained to take a subject image (see the right side of Figure 4B) from which the background region has been removed from a captured image as input, as shown on the left side of Figure 4B, and to perform a calculation to determine whether the captured image is suitable as a target for sale based on the input subject image, and to output the result of the calculation. The subjects extracted as subject images can be, for example, major subjects that are prominently featured in the captured image, subjects that are positioned close to the center of the captured image, etc. The learning model 12M may be constructed using algorithms such as CNN (Convolutional Neural Network), SVM (Support Vector Machine), Transformer, etc., or it may be constructed by combining multiple algorithms.
[0024] The 12M learning model has an input layer into which a subject image is input, a hidden layer that extracts features from the input subject image, and an output layer that outputs information on whether the captured image, including the subject image, is suitable for sale, based on the calculation results of the hidden layer. The input layer has an input node into which the pixel value of each pixel in the subject image is input. The hidden layer calculates an output value based on each pixel value input from the input layer using various functions and thresholds. The output layer (output section) has two output nodes, one corresponding to an image suitable for sale (i.e., a good image) and the other to an image unsuitable for sale (i.e., a bad image). Output node 0 outputs the probability (confidence level) that the captured image should be judged as a good image, and output node 1 outputs the probability (confidence level) that the captured image should be judged as a bad image. The output values from each output node are, for example, between 0 and 1, and the sum of the probabilities output from each output node is 1.0 (100%). In this embodiment, the output value from output node 0 is used as a score indicating the degree to which the captured image is a good image, that is, its suitability as a product for sale.
[0025] With the configuration described above, when a subject image is input, the learning model 12M outputs an output value (confidence score) indicating whether the captured image, before the background region is removed, is a good or bad image for sale. The server 10 obtains the output value from output node 0 of the learning model 12M as a score indicating the degree to which the captured image is a good image for sale. Note that the output layer of the learning model 12M may have only output node 0 instead of having two output nodes.
[0026] The 12M learning model can be generated by machine learning using training data that includes training subject images and information (ground truth labels) indicating whether the captured image before background removal is a good or bad image. The training data is generated, for example, by assigning ground truth labels, which indicate whether an image is suitable for sale (i.e., whether it is a good image or not) to the subject image after the background has been removed from the captured image.
[0027] The learning model 12M learns to receive subject images from the training data as input, such that the output value from the output node corresponding to the correct label (good image or bad image) in the training data approaches 1, and the output value from the other output node approaches 0. During the learning process, the learning model 12M performs calculations in the hidden layer and output layer based on the input subject image and calculates the output value from each output node. The learning model 12M compares the calculated output value of each output node with the value corresponding to the correct label (1 for the output node corresponding to the correct label, and 0 for the other output node) and optimizes the parameters used in the calculations in the hidden layer and output layer so that the two approximate each other. These parameters are the weights (coupling coefficients) between nodes in the hidden layer and output layer. The method of parameter optimization is not particularly limited, but methods such as backpropagation and steepest descent can be used. As a result, a learning model 12M is obtained that, when a subject image is input, predicts whether the captured image before background removal is a good image or a bad image, and outputs the prediction result.
[0028] Server 10 prepares such a learning model 12M in advance and uses it to sort images captured by camera 20 into good images or bad images. The learning model 12M may be trained on another learning device. The trained learning model 12M generated by training on another learning device is downloaded from the learning device to server 10, for example, via network N or portable storage medium 10a, and stored in storage unit 12.
[0029] The following describes the process of generating a learning model 12M by learning from the training data described above. Figure 5 is a flowchart showing an example of the procedure for generating the learning model 12M. The following process is executed by the control unit 11 of the server 10 according to the program 12P stored in the memory unit 12, but it may also be performed by another learning device. In the following process, the control unit 11 first generates training data based on the captured images stored in the memory unit 12, and then trains the learning model 12M using the generated training data. The captured images used as training data are judged as good or bad by an administrator or the like, and the judgment result is attached to the captured images and stored in a predetermined area (predetermined DB) of the memory unit 12.
[0030] The control unit 11 of the server 10 reads the captured image stored in the memory unit 12 and the judgment result (information regarding the suitability of the captured image) in which an administrator or the like has judged the quality of the captured image (S11). The control unit 11 performs background removal processing on the read captured image to remove the background region (S12). The background removal processing can be any processing, for example, processing using the background extraction class (BackgroundSubtrator) of OpenCV, processing using the image background removal tool (remove.bg) provided by Kaleido Inc., processing using a deep learning model such as U2-Net, etc. When using a learning model, a learning model that has been trained to output a subject image when a captured image is input is used, using the captured image and a foreground image (subject image) from which the background has been removed from the captured image. Alternatively, if there is a captured image in which only the background has been captured (background image), the subject image may be generated by removing the background region from the captured image using the background image. Through such background removal processing, the control unit 11 generates a subject image from which the background region has been removed from the captured image.
[0031] The control unit 11 generates training data by assigning correct labels to the generated subject images according to the good / bad judgment result read in step S11, and stores it in the storage unit 12 (S13). Specifically, if the judgment result is good, the control unit 11 assigns a good correct label to the captured image, and if the judgment result is bad, it assigns a bad correct label to the captured image. The control unit 11 stores the generated training data in a training DB (not shown) prepared in the storage unit 12, for example.
[0032] The control unit 11 determines whether there are any unprocessed captured images among the captured images stored in the memory unit 12 that are not used in the training data generation process (S14). If it determines that there are unprocessed captured images (S14: YES), the control unit 11 returns to the process of step S11 and performs the processing of steps S11 to S13 on the unprocessed captured images. The control unit 11 repeats the processing of steps S11 to S14 until it determines that there are no unprocessed captured images. As a result, training data used for training the learning model 12M is generated and stored in the training DB based on the captured images stored in the memory unit 12 and the judgment results for the captured images. In the process described above, the captured images used for generating training data were explained as being stored in the memory unit 12, but the control unit 11 of the server 10 may be configured to acquire each captured image and judgment result from other devices, for example, via the network N.
[0033] If the control unit 11 determines that there are no unprocessed captured images (S14: NO), it trains the learning model 12M using the training data stored in the training DB as described above. The control unit 11 reads one of the training data stored in the training DB through the process described above (S15). Then, the control unit 11 performs the learning process of the learning model 12M based on the read training data (S16). Here, the control unit 11 inputs a captured image included in the training data into the learning model 12M and obtains the output value that is output from the learning model 12M as a result of the input of the captured image. The control unit 11 compares the output value of each output node output from the learning model 12M with a value corresponding to the correct label included in the training data (1 for output nodes corresponding to the correct label, and 0 for other output nodes), and trains the learning model 12M so that the two approximate each other. In the learning process, the learning model 12M optimizes the parameters used for calculations in the hidden layer and output layer. For example, the control unit 11 optimizes parameters such as the weights (coupling coefficients) between nodes in the hidden layer and output layer using a backpropagation method that sequentially updates the learning model 12M from the output layer to the input layer.
[0034] The control unit 11 determines whether there is any unprocessed training data stored in the training DB that has not undergone learning processing (S17). If it determines that there is unprocessed training data (S17: YES), the control unit 11 returns to the process in step S15 and performs the processing in steps S15 to S16 on the unprocessed training data. If it determines that there is no unprocessed training data (S17: NO), the control unit 11 terminates the series of processes.
[0035] The learning process described above generates a learning model 12M that, when a subject image is input, outputs an output value indicating the possibility that the captured image before background removal is a good image, and an output value indicating the possibility that it is a bad image. Therefore, the server 10 can obtain information (information regarding suitability) about whether the captured image is a good or bad image based on the output values from the learning model 12M. Note that in the process described above, the training data generation process in steps S11 to S14 and the learning model 12M generation process in steps S15 to S17 may be performed on separate devices. The learning model 12M can be further optimized by repeatedly performing the learning process using the training data described above. In addition, even an already trained learning model 12M can be retrained using the learning process described above to generate a learning model 12M with further improved discrimination accuracy.
[0036] The following describes the process in this embodiment of the image processing system for sorting images captured using the camera 20 into images to be sold and images not to be sold. Figure 6 is a flowchart showing an example of the image sorting process, and Figure 7 is an explanatory diagram showing an example of the screen of the administrator terminal 30. In Figure 6, the processing performed by the camera 20 is shown on the left, the processing performed by the server 10 is shown in the center, and the processing performed by the administrator terminal 30 is shown on the right.
[0037] In the image processing system of this embodiment, a photographer uses camera 20 to photograph performers and participants of an event and acquire captured images. Camera 20 may be configured to automatically take pictures according to a preset shooting timing. Camera 20 takes pictures using its imaging unit (S21) and transmits the acquired captured images to server 10 via its communication unit (transmission unit) (S22). Camera 20 may be configured to transmit the acquired captured images to server 10 each time it takes a picture, or it may be configured to transmit multiple captured images to server 10 at once. The following description will explain a configuration in which the processing in steps S22 to S35 is performed each time a picture is taken, but the processing in steps S22 to S35 may be performed after multiple pictures have been taken. In this case, in steps S22 to S35, each of the multiple captured images will be processed separately.
[0038] The control unit 11 (acquisition unit) of the server 10 acquires the captured image transmitted from the camera 20 and stores the acquired image in the storage unit 12 (S23). At this time, the control unit 11 stores the captured image in a predetermined area (image folder) of the storage unit 12 and also stores information related to the captured image in the captured image DB 12a. Note that the information of the date and time of shooting and the location of shooting may be acquired from the camera 20 along with the captured image, or may be registered in advance. Next, the control unit 11 (removal unit) performs background removal processing on the acquired image (S24) and generates a subject image from which the background region has been removed from the captured image. The background removal processing can be the same as the process in step S12 in Figure 5. Based on the generated subject image, the control unit 11 calculates a score that indicates the degree to which the captured image before the background region was removed was a good image (S25). Specifically, the control unit 11 inputs the subject image into the learning model 12M and acquires the output value from output node 0 as the score for the captured image. The control unit 11 (sorting unit) sorts the captured images into good or bad images based on the acquired score (S26). For example, if the acquired score is above a predetermined threshold (e.g., 0.7), the control unit 11 sorts the captured image into a good image, and if it is below the predetermined threshold, it sorts the captured image into a bad image. The threshold for sorting captured images into good or bad images is set in advance and stored in the storage unit 12, and can also be changed according to operations via the input unit 14. For example, when the control unit 11 receives an instruction to change the threshold setting via the input unit 14, the threshold stored in the storage unit 12 is updated to the threshold specified in the instruction, thereby changing the threshold setting.
[0039] The control unit 11 stores the sorting result (good or bad) in the sorted image DB 12b, associating it with the image ID of the captured image (S27). Next, the control unit 11 reads the captured images sorted as good from the storage unit 12 based on the contents of the sorted image DB 12b (S28) and sends the read captured images to the administrator terminal 30 (S29). The control unit 31 of the administrator terminal 30 displays the captured images sorted as good, which were sent by the server 10, on the display unit 35 (S30). For example, the control unit 31 displays a screen on the display unit 35 as shown in Figure 7, presenting the captured images acquired from the server 10 to the administrator. The server 10 may also send a subject image obtained by removing the background area from the captured image to the administrator terminal 30 along with the captured image, or in place of the captured image, and the administrator terminal 30 may display the subject image along with the captured image, or in place of the captured image, to present to the administrator. The screen shown in Figure 7 is configured to receive a judgment result on whether the currently displayed captured image is a good or bad image, and is provided with "Good" buttons, "Bad" buttons, and "Hold". The administrator makes a judgment on whether the captured image is a good or bad image by operating the "Good" or "Bad" button on the screen shown in Figure 7 via the input unit 34. If the administrator cannot make a judgment on the captured image, they input that a judgment cannot be made by operating the "Hold" button. The control unit 31 receives the judgment result from the administrator when any of the buttons are operated (S31) and sends the received judgment result from the administrator to the server 10 (S32).
[0040] The control unit 11 of the server 10 receives the judgment result sent by the administrator terminal 30, associates it with the image ID of the captured image, and stores the received judgment result (good or bad) in the judgment result DB 12c (S33). Based on the contents of the judgment result DB 12c, the control unit 11 identifies the captured images that the administrator has classified as good (S34), associates the image ID and file name of the identified captured image, and stores it in the sales image DB 12d (S35). The captured images stored in the sales image DB 12d are sold by the server 10 via the network N through the photo vending machine 40 or user terminal 50. The server 10 also sends a thumbnail list of the captured images to be sold to the photo vending machine 40 or user terminal 50, obtains purchase requests for the captured images received via the thumbnail list from the photo vending machine 40 or user terminal 50, and outputs the captured images that are requested for purchase to the photo vending machine 40 or user terminal 50. The photo vending machine 40 provides the customer with the desired photograph by printing it using the printing unit, which is obtained from the server 10. The sales process for the photographs is a standard procedure and will be omitted here.
[0041] The administrator's judgment results for the captured images stored in the judgment result DB12c through the above-described process can be used when retraining the learning model 12M. Specifically, the control unit 11 of the server 10 can perform the training data generation process and the learning model 12M training process based on the contents of the judgment result DB12c by executing the process shown in Figure 5 on the captured images stored in the judgment result DB12c. This further improves the discrimination accuracy of the learning model 12M. In this embodiment, the learning model 12M is trained using subject images from which the background region has been removed from the captured images and the judgment results of whether the captured images are good or bad as training data, but the system is not limited to this configuration. For example, a learning model trained using training data including captured images and judgment results of whether the captured images are good or bad may be fine-tuned (transfer learning) using training data including subject images and judgment results of whether the captured images are good or bad.
[0042] Through the process described above, the image processing system of this embodiment calculates a score using the learning model 12M that indicates the degree to which the captured image taken by the camera 20 is suitable for sale, and sorts the captured image into good or bad images based on the calculated score. Therefore, manual image sorting is unnecessary, and the workload can be reduced. Furthermore, when images are sorted manually, there is a possibility that the sorting results may differ depending on the subjective judgment of the person doing the sorting, but in this embodiment, sorting is performed based on the score calculated by the learning model 12M, so an objective sorting result can be obtained. In addition, since the learning model 12M calculates a score for the captured image based on the subject image obtained by removing the background region from the captured image, it is not affected by the background region and the discrimination accuracy can be improved. For example, when training with captured images, if there is a bias in the images used as training data, the learning accuracy may decrease. For example, if a learning model is trained using training data in which a predetermined first color is abundant in images judged as good, and a predetermined second color is abundant in images judged as bad, then even an out-of-focus image may be judged as a good image if it contains a lot of the first color. However, in this embodiment, the quality of an image is judged based on the subject image obtained by removing the background region from the captured image, so a decrease in judgment accuracy is suppressed.
[0043] In this embodiment, the threshold used to classify captured images as good or bad images can be changed. Therefore, by changing the threshold according to the number of cameras 20, the number of images captured by the cameras 20, the number of administrators performing the classification, and the time that can be spent on the classification work, it is possible to adjust the number of images that administrators have to classify. In this embodiment, the server 10 uses the learning model 12M to classify the captured images, and the administrators then perform further classification on the captured images classified as good images. As a result, captured images whose scores output by the learning model 12M are below the threshold are excluded from sale without the administrator's review. Therefore, the number of images that administrators have to classify can be reduced, thereby reducing the burden of classification work on administrators. Alternatively, captured images classified as good images using the learning model 12M may be sold as is without further classification by administrators. In this case, the workload on administrators can be further reduced. For example, if a high threshold value (e.g., 0.8 or 0.9) is set, images that have been sorted as good images based on the score output by the learning model 12M are likely to be judged as good images by the administrator as well. In this case, the system may be configured so that the administrator does not perform the sorting.
[0044] In this embodiment, the sorting process for whether or not to sell a captured image is performed based on a score for the subject image obtained by removing the background region from the captured image. Alternatively, the sorting process for whether or not to sell an image may also consider a score for the captured image before the background region is removed, in addition to the score for the subject image. For example, the control unit 11 determines whether the score for the subject image obtained using the learning model 12M is less than a predetermined value (e.g., 0.6), and if it determines that it is less than the predetermined value, it obtains a score for the captured image before the background region of the subject image is removed using the learning model. The control unit 11 then determines whether the score for the acquired captured image is equal to or greater than a predetermined value (e.g., 0.8), and if it determines that it is equal to or greater than the predetermined value, it may be configured to sell the captured image. In this case, the learning model used to calculate the score for the captured image may be the same model as the learning model 12M used to calculate the score for the subject image, or it may be a different model.
[0045] In this embodiment, the administrator terminal 30 can also perform locally any or more of the processes that the server 10 performs among the processes shown in Figure 6: the training data generation process, the learning process of the learning model 12M using the training data, and the processes shown in Figure 6. For example, the administrator terminal 30 may generate training data by executing the process shown in Figure 5, and then generate the learning model 12M using the generated training data and store it in the storage unit 32. This allows the administrator terminal 30 to execute the processes that the server 10 performs in Figure 6. Even with such a configuration, the same processes as in this embodiment are possible and the same effects can be obtained.
[0046] (Embodiment 2) This section describes an image processing system that generates still images from video footage captured by camera 20 and sorts the generated still images to determine whether or not they should be sold. Since the image processing system of this embodiment is implemented using the same equipment as the image processing system of Embodiment 1 shown in Figures 1 and 2, a detailed explanation of the configuration of each device will be omitted.
[0047] Figure 8 is a flowchart showing an example of the sorting process procedure in Embodiment 2. The process shown in Figure 8 is the same as the process shown in Figure 6, but with steps S41 to S42 added before step S23. The steps that are the same as in Figure 6 will not be explained.
[0048] In the image processing system of this embodiment, the camera 20 performs the same processing as in steps S21 to S22 in Figure 6. The images captured by the camera 20 may be still images or moving images. When the control unit 11 of the server 10 acquires an image transmitted from the camera 20, it determines whether the acquired image is a moving image or not (S41). If it is determined that the captured image is not a moving image (S41: NO), i.e., it is a still image, the control unit 11 proceeds to the processing in step S23 and performs the same processing as in Embodiment 1. If it is determined that the captured image is a moving image (S41: YES), the control unit 11 generates still images from the moving image (S42). For example, if the captured image is a moving image containing 30 frames per second, the control unit 11 generates a still image from the captured image every second, generating 30 still images per second. The control unit 11 does not need to generate still images from all frames; for example, it may extract frames at predetermined time intervals (every 0.1 seconds, every 0.5 seconds, etc.) and generate still images from them.
[0049] The control unit 11 stores the still images generated from the video as captured images in the storage unit 12 (S23), and then executes the processing from step S24 onwards. Since multiple captured images (still images) are generated from one video, each of the multiple captured images (still images) is processed in steps S23 to S35. This makes it possible to sort the captured images taken by the camera 20 into whether or not they should be sold, and if the captured images are video, sorting the still images generated from the video into whether or not they should be sold.
[0050] In this embodiment, still images generated from a video are the target of sale, but the video itself may also be the target of sale. For example, the control unit 11 may be configured to generate multiple still images from a video, input each generated still image into a learning model 12M to obtain a score for each still image, and if the average value of the scores for each still image is equal to or greater than a predetermined value (e.g., 0.8), the video may be made available for sale. Alternatively, a learning model may be used to determine whether or not each video is available for sale, by using a learning model that has been trained to output information indicating whether or not the video is suitable for sale (a score indicating the degree to which the video is suitable for sale) when each frame contained in the video is input. In this case, the control unit 11 may be configured to input the video into the learning model, obtain a score for the video from the learning model, and make the video available for sale if the obtained score is equal to or greater than a predetermined value (e.g., 0.8).
[0051] In this embodiment, the same effects as in Embodiment 1 described above can be obtained. Furthermore, in this embodiment, still images generated from video footage captured by the camera 20 can be sold. Therefore, by capturing video footage using the camera 20, still images generated from frames included in the video can be sold, allowing for the collection of a large number of sales targets. In this embodiment as well, the modifications described as appropriate in Embodiment 1 described above can be applied.
[0052] The following additional information is disclosed regarding embodiments including the above embodiments 1 and 2.
[0053] (Note 1) Capture the image of the subject, Remove the background region from the acquired captured image. A trained model, which is designed to output information regarding the suitability of a captured image when a subject image from which the background region has been removed is input, is given the subject image from which the background region has been removed, and the trained model outputs information regarding the suitability of the captured image. Based on the information regarding the suitability of the captured images, the suitability of the captured images is sorted. A program that instructs a computer to perform a process.
[0054] (Note 2) The appropriately sorted captured images are output, The system accepts a judgment on whether the output captured image is suitable or unsuitable. Training data is acquired that includes the aforementioned captured images and the received information regarding suitability. The program described in Appendix 1 that causes the computer to execute the processing.
[0055] (Note 3) The information regarding the suitability of the captured image is a score indicating the degree of appropriateness of the captured image. The threshold for determining whether the captured image is suitable or unsuitable is accepted. Based on the accepted threshold, the acquired images are sorted into whether they are suitable or unsuitable. A program described in Appendix 1 or 2 that causes the computer to perform the processing.
[0056] (Note 4) Obtain a video of the subject, Remove the background area from the frames included in the acquired video. The learning model is input with a subject image obtained by removing the background region from the frame, and the learning model outputs information regarding the suitability of the frame. A program described in any one of the appendices 1 to 3 that causes the computer to perform the processing.
[0057] (Note 5) Capture the image of the subject, Remove the background region from the acquired captured image. A trained model, which is designed to output information regarding the suitability of a captured image when a subject image from which the background region has been removed is input, is given the subject image from which the background region has been removed, and the trained model outputs information regarding the suitability of the captured image. Based on the information regarding the suitability of the captured images, the suitability of the captured images is sorted. An image processing method in which a computer performs the processing.
[0058] (Note 6) An acquisition unit that acquires a captured image of the subject, A removal unit that removes the background region from the acquired captured image, A learning model that has been trained to output information regarding the suitability of a captured image when a subject image from which the background region has been removed is input, and an output unit that takes the acquired subject image from which the background region has been removed as input and outputs information regarding the suitability of the captured image from the learning model, A sorting unit that sorts the captured images based on information regarding the suitability of the captured images. An image processing device equipped with the following features.
[0059] (Note 7) Training data is obtained that includes a subject image from which the background region has been removed from a captured image of the subject, and information regarding the suitability of the captured image. Using the acquired training data, a learning model is generated that outputs information regarding the suitability of the captured image when the subject image is input. A model generation method in which a computer performs the processing.
[0060] (Note 8) The system acquires a photograph of the subject and information regarding the suitability of the photograph. Remove the background region from the acquired captured image. Training data is acquired that includes a subject image obtained by removing the background region from the aforementioned captured image, and information regarding the suitability of the acquired captured image. The model generation method described in Appendix 7, wherein the computer performs the processing.
[0061] (Note 9) An image processing system including a camera and an image processing device, The aforementioned imaging device is It includes a transmission unit that sends captured images of a subject to the image processing device, The aforementioned image processing device is An acquisition unit that acquires the captured image from the aforementioned imaging device, A removal unit that removes the background region from the acquired captured image, A learning model that has been trained to output information regarding the suitability of a captured image when a subject image from which the background region has been removed is input, and an output unit that takes the acquired subject image from which the background region has been removed as input and outputs information regarding the suitability of the captured image from the learning model, The system includes a sorting unit that sorts the captured images based on information regarding the suitability of the captured images. Image processing system.
[0062] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims, not in the sense described above, and all modifications are intended to be in the sense and scope equivalent to the claims. [Explanation of Symbols]
[0063] 10 servers 11 Control Unit 12 Storage section 13 Communications Department 14 Input section 15 Display 20 cameras 30 Administrator terminals 31 Control Unit 32 Storage section 33 Communications Department 40 Photo vending machines 50 User Terminals 12M Learning Model
Claims
1. Capture the image of the subject, Remove the background region from the acquired captured image. A trained model, which is designed to output information regarding the suitability of a captured image for sale when a subject image from which the background region has been removed is input, is then given the acquired subject image from which the background region has been removed, and the trained model outputs information regarding the suitability of the captured image for sale. Based on the information regarding the suitability of the aforementioned captured images for sale, the suitability of the aforementioned captured images for sale is determined. A program that instructs a computer to perform a process.
2. The appropriately sorted captured images are output, The system accepts a judgment on whether the outputted captured image is suitable for sale. Training data is acquired that includes the aforementioned captured images and the received information regarding suitability. The program according to claim 1, which causes the computer to perform the processing.
3. The information regarding the suitability of the aforementioned captured images for sale is a score indicating the appropriateness of the captured images. The threshold for determining whether the aforementioned captured images are suitable for sale is accepted. Based on the accepted threshold, the acquired images are sorted to determine whether they are suitable for sale. The program according to claim 1 or 2, which causes the computer to perform the processing.
4. Obtain a video of the subject, Remove the background area from the frames included in the acquired video. The learning model is input with a subject image obtained by removing the background region from the frame, and the learning model outputs information regarding whether the frame is suitable for sale. The program according to claim 1 or 2, which causes the computer to perform the processing.
5. Capture the image of the subject, Remove the background region from the acquired captured image. A trained model, which is designed to output information regarding the suitability of a captured image for sale when a subject image from which the background region has been removed is input, is then given the acquired subject image from which the background region has been removed, and the trained model outputs information regarding the suitability of the captured image for sale. Based on the information regarding the suitability of the aforementioned captured images for sale, the suitability of the aforementioned captured images for sale is determined. An image processing method in which a computer performs the processing.
6. An acquisition unit that acquires a captured image of the subject, A removal unit that removes the background region from the acquired captured image, A learning model that has been trained to output information regarding the suitability of a photographed image for sale when a subject image from which the background region has been removed is input, and an output unit that takes the acquired subject image from which the background region has been removed as input and outputs information regarding the suitability of the photographed image for sale from the learning model, A sorting unit that sorts the photographed images into those suitable for sale based on information regarding their suitability as saleable items. An image processing device equipped with the following features.
7. Training data is obtained that includes subject images from which the background region has been removed from captured images of the subject, and information regarding the suitability of the captured images for sale. Using the acquired training data, a learning model is generated that, when the subject image is input, outputs information regarding whether the captured image is suitable for sale. A model generation method in which a computer performs the processing.
8. We obtain photographic images of the subject and information regarding whether the photographic images are suitable for sale. Remove the background region from the acquired captured image. Training data is obtained that includes a subject image obtained by removing the background region from the aforementioned captured image, and information regarding the suitability of the acquired captured image as a marketable item. The model generation method according to claim 7, wherein the computer performs the processing.
9. An image processing system including a camera and an image processing device, The aforementioned imaging device is It includes a transmission unit that sends captured images of a subject to the image processing device, The aforementioned image processing device is An acquisition unit that acquires the captured image from the aforementioned imaging device, A removal unit that removes the background region from the acquired captured image, A learning model that has been trained to output information regarding the suitability of a photographed image for sale when a subject image from which the background region has been removed is input, and an output unit that takes the acquired subject image from which the background region has been removed as input and outputs information regarding the suitability of the photographed image for sale from the learning model, The system includes a sorting unit that sorts the captured images based on information regarding their suitability for sale. Image processing system.