Information processing device, information processing method, and program
The information processing device uses machine learning to evaluate and label user-generated images for advertising suitability, addressing the underutilization challenge and enhancing photographer engagement and image collection for advertisements.
Patent Information
- Application Number
- JP2023075648
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-01
- Publication Date
- 2024-11-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The challenge in the UGC market is the difficulty in identifying and utilizing images with high advertising value from user-generated content on social networking services due to the lack of upfront evaluation of their suitability for advertisements, leading to underutilization of images and reduced market expansion.
An information processing device employs machine learning using a learned model to evaluate and label images based on their suitability for advertising, facilitating their use in advertisements and promoting user engagement by providing value recognition to photographers.
The solution enhances the utilization of user-generated images by recognizing their advertising potential, increasing photographer engagement and collaboration, and optimizing image collection for advertisements.
Smart Images

Figure 2024160538000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]
[0002] In recent years, marketing using User Generated Contents (UGC) has become more active. Traditionally, people would select images that were good for advertising from a database of stock photos and create web ads, television ads, poster ads, etc. Here, stock photos are existing photos that have been licensed. There are also cases where people select images posted by users on social networking services (SNS) and use them in advertisements with permission from the posters. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2021-193564 A [Patent Document 2] International Publication No. 2022 / 097709 [Patent Document 3] Patent Publication No. 2021-068238 [Patent Document 4] JP 2019-149171 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, in conventional cases, images suitable for advertising are often extracted manually from images posted on SNS. In addition, users who are photographers rarely recognize that the images they have taken are likely to be used for advertising in the first place. For this reason, some images taken by users who are photographers are not posted, or are not registered in the stock photo database. For this reason, there are potential problems that become a barrier to the expansion of the UGC market. Such problems are thought to be caused by the general difficulty of grasping in advance how much advertising value an image or photo has. For this reason, it is currently difficult to collect information such as so-called SNS-worthy images or photos that are highly rated on SNS, SNS-worthy products and scenery, the locations where such images were obtained, and the locations where photos were taken, and to use them for advertising.
[0005] An object of the embodiment of the present disclosure is to promote utilization of images, to increase the motivation of users who take pictures to provide images, and to cooperate with users who use images. [Means for solving the problem]
[0006] The disclosed embodiment is exemplified by an information processing device in an information system that generates a trained model by machine learning using training data including input data to which a label has been assigned, and executes information processing using the trained model. The information processing device executes the following: acquiring first training data including first input data to which a first label has been assigned, changing the first input data included in the first training data to obtain second input data, and creating second training data by assigning the first label or a second label different from the first label to the second input data depending on the degree of change from the first input data to the second input data. Effect of the Invention
[0007] According to the present information processing device, it is possible to encourage users to utilize images they have taken, to increase the willingness of users who have taken images to provide them, and to promote cooperation with users who use the images. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information system. [Diagram 2] FIG. 2 is a diagram illustrating an example of the flow of data between elements in an information system. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] FIG. 4 is a block diagram illustrating a functional configuration of the information processing device. [Diagram 5] FIG. 5 shows an example of setting parameters set in the teacher data extension unit. [Figure 6] FIG. 6 is a diagram illustrating a table of posted images included in the SNS data. [Figure 7] FIG. 7 is a diagram illustrating an example of an image related information table included in the SNS data. [Figure 8] FIG. 8 is a diagram illustrating a poster attribute table of SNS data. [Figure 9] FIG. 9 is a diagram illustrating an example of an advertisement creator attribute table of the advertisement performance data. [Figure 10] FIG. 10 is a diagram illustrating the procedure by which an AI machine learns the suitability of an advertisement. [Figure 11] FIG. 11 is a diagram illustrating a procedure of machine learning processing by the information processing device. [Figure 12] FIG. 12 is a diagram illustrating the details of the data extension process. [Figure 13] FIG. 13 is a diagram illustrating the procedure of the image determination process E1. [Figure 14] FIG. 14 is a diagram illustrating the procedure of the image determination process E2. [Figure 15] FIG. 15 is a diagram illustrating the details of the teacher data creation process. [Figure 16] FIG. 16 is a diagram illustrating a photographing recommendation process executed by the information processing device. [Figure 17] FIG. 17 is a diagram illustrating an example of an image recommendation process R1 for an advertisement creator, which is executed by the information processing device. [Figure 18] FIG. 18 is a diagram illustrating an example of an image recommendation process R2 for an advertisement creator, which is executed by the information processing device. [Figure 19] FIG. 19 is a diagram illustrating an example of an image recommendation process R3 for acquiring captured images that match a search query input by an advertisement creator. [Figure 20] FIG. 20 is a diagram illustrating an example of a configuration of an information system according to the second embodiment. [Figure 21] FIG. 21 is a diagram illustrating a processing procedure in the information system of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, an information processing apparatus, an information processing method, and a program according to the present embodiment will be described with reference to the drawings.
[0010] First embodiment (system configuration) An information system 10 according to a first embodiment will be described below with reference to Fig. 1 to Fig. 19. Fig. 1 is a diagram illustrating an example of a configuration of the information system 10. Fig. 2 is a diagram illustrating an example of a data flow between elements in the information system 10.
[0011] The information system 10 includes an information processing device 1, an AI machine 2, an SNS server 3, a poster's user device 4-1, an advertisement creator's user device 4-2, and a target location's user device 4-3. The devices and servers included in the information system 10 are connected by a network N1. The poster's user device 4-1, the advertisement creator's user device 4-2, and the target location's user device 4-3 are collectively referred to simply as user devices 4. The network N1 is, for example, a public network such as the Internet.
[0012] The information processing device 1 can also be called a material data server. The information processing device 1 cooperates with an AI machine 2 via a network N1 and transmits material data such as images and photographs to a user device 4. For example, the information processing device 1 uses the AI machine 2 to evaluate images, photos, etc. posted on the SNS server 3, and provides information based on the evaluation results to the user device 4. The images, photos, etc. posted on the SNS server 3 are examples of images that are posted so that they can be viewed on a public network. In this embodiment, "image" includes "photograph" and is a picture or image that can be displayed on a computer display. "Image" also includes image files or image data.
[0013] As illustrated in FIG. 2, the AI machine 2 has a learning system 21, and executes learning using training data including images and labels attached to the images, to generate a trained model 22. Here, the label is, for example, a character string for classifying an image, and is an evaluation value exemplified by, for example, "L1", "L2", . . . , "LN". The label of the training data is, for example, given to each image by a person. For example, the person evaluates the degree to which an image is evaluated on SNS (called "visual appeal") on an N-level scale based on the number of reactions on SNS, and according to the evaluation, the character string is given to each image as a label to create training data. However, N is not limited as long as it is an integer of 2 or more.
[0014] The learning system 21 generates a trained model 22 by learning a large amount of teacher data. Then, the AI machine 2 uses the trained model 22 to evaluate, for example, an image newly posted to the SNS server 3, and provides the evaluation result to the information processing device 1. As described above, the AI machine 2 is, for example, a general-purpose machine learning system that executes deep learning. Therefore, the trained model is not limited to one that simply learns "good looks". For example, in this embodiment, the learning system 21 learns teacher data labeled with an evaluation of suitability for advertisements, and generates a trained model 22 that determines the degree of suitability for advertisements. The degree of suitability for advertisements can be determined by a person based on the subjectivity of a person who assigns the label (for example, an expert in the advertising industry) as well as the degree of similarity to images that have been highly rated in the past or images with high transaction records, and can be assigned to each image.
[0015] The correlation between an image and the degree of suitability for advertising can be understood, for example, as follows. Typically, when the target of advertising is a service or product related to "travel", if the subject of the image includes famous facilities of tourist destinations, famous scenery, famous mountains, islands, rivers, lakes, etc., and animals living there, it can be objectively understood that the image is highly suitable for advertising. Also, when the target of advertising is a vehicle such as a car or ship, if the subject of the image includes desirable scenery, environment, animals and plants living there, and relationships between people riding on the vehicle, it can be assumed that the image is highly suitable for advertising. Naturally, an expert in the advertising industry can be expected to have the ability to attach appropriate labels to appropriate images.
[0016] The SNS server 3 is a system that provides so-called SNS services. The SNS server 3 receives, for example, an image posted by a poster's user device 4-1, and provides the image to other user devices 4 on the SNS. The SNS server 3 also provides an API (Application Programming Interface) to the information processing device 1, and cooperates with the information processing device 1.
[0017] The information processing device 1 can use the API to search for and acquire data posted to the SNS server 3, such as images, acquire information about the user who posted, and acquire the number of reactions to the posted image. Here, the number of reactions refers to, for example, the fact that the posted image has been accessed by other user devices 4, the date and time of access, the number of times it has been accessed during a specific period, the date and time when an evaluation index indicating a high rating was given, the number of times an evaluation index has been given during a specific period, etc. In addition, the fact that it has been accessed is also referred to as a click, and the number of times it has been accessed is also referred to as the number of clicks. In addition, an evaluation index indicating a high rating can also be referred to as an index indicating a "like." Furthermore, the number of times an index indicating a "like" has been given to a post is also referred to as the "number of likes." In addition, in SNS and the like, posted images may be accompanied by text that has been added along with special characters. A character string or a keyword is assigned, and the character string or keyword may be used as a tag called a “hashtag.” Therefore, the information processing device 1 can also search the SNS by “hashtag.”
[0018] The user device 4 is a terminal such as a computer or a smartphone used by a user. Of these, the poster's user device 4-1 is a device used by a user who posts data such as images to the SNS. The poster's user device 4-1 further accesses the information processing device 1 or receives information from the information processing device 1 and executes processing related to the image. For example, the poster's user device 4-1 receives an inquiry from the information processing device 1 about an image posted to the SNS. The inquiry is, for example, about whether or not the image posted to the SNS can be provided as advertising material. The poster's user device 4-1 responds to the inquiry from the information processing device 1 by indicating whether or not the material can be provided in response to a user operation.
[0019] Similarly, the user device 4-2 of the advertisement creator accesses the information processing device 1 or receives information from the information processing device 1. For example, the user device 4-2 of the advertisement creator receives a proposal for an image as advertisement material from the information processing device 1 and acquires the image as the material. The user device 4-2 may also create an advertisement using the acquired image.
[0020] The user device 4-3 at the target location is a user device 4 located near a target location for which the information processing device 1 has determined that it is desirable to obtain an image. The information processing device 1 identifies a target location for which it has been determined that there is a high possibility of obtaining a desirable image, based on images that have been highly evaluated on SNS, past records of images provided to the user device 4-2 of the advertisement creator, or past records of transaction prices with the advertisement creator, etc. Then, the information processing device 1 requests the user device 4 located near the target location, i.e., the user device 4-3 at the target location, to obtain an image or post it on the SNS. The user device 4-3 at the target location provides the image to be used as advertising material to the information processing device 1 via posting on the SNS or directly.
[0021] 3 is a diagram illustrating an example of the hardware configuration of the information processing device 1. The configuration of the information processing device 1 is similar to that of a normal computer. Although the scale of the SNS server 3 and the user device 4 differs, the hardware configuration itself is similar to that of the information processing device 1. The information processing device 1 includes, for example, a CPU 101, a memory 102, an external storage device 103, a display device 104, an input device 105, and a communication device 106.
[0022] The CPU 101 executes a computer program executablely deployed in the memory 102, and provides the functions of the information processing device 1. The CPU 101 is also called a processor. However, the CPU 101 is not limited to a single processor, and may have a multi-processor configuration. Furthermore, the CPU 101 may be a single processor connected via a single socket, and may have a multi-core configuration. Furthermore, at least a part of the processing of the information processing device 1 may be provided by a dedicated processor such as a Digital Signal Processor (DSP), a Graphics Processing Unit (GPU), a numerical calculation processor, a vector processor, or an image processing processor, an Application Specific Integrated Circuit (ASIC), or the like. Furthermore, the information processing At least a part of the information processing device 1 may be a dedicated large scale integration (LSI) such as a Field-Programmable Gate Array (FPGA) or other digital circuit. At least a portion of the circuit may include an analog circuit.
[0023] The memory 102 is also called a main storage device, and stores computer programs executed by the CPU 101, data processed by the CPU 101, etc. The memory 102 is a dynamic random access memory (DRAM), a static random access memory (SRAM), a read only memory (ROM), etc. Furthermore, the external storage device 103 is, for example, a secondary storage device for the memory 102. The CPU 101 executes a computer program, and the CPU The external storage device 103 stores data to be processed by the information processing device 101. The external storage device 103 is a hard disk drive, a solid state drive (SSD), etc. A drive for the medium may be provided. The removable storage medium may be, for example, a Blu-ray disc, a Digital Versatile Disc (DVD), a Compact Disc (CD), a flash memory card, or the like.
[0024] The display device 104 is, for example, a liquid crystal display, an electroluminescence panel, etc. The input device 105 is, for example, a keyboard, a pointing device, etc. In this embodiment, examples of the pointing device include a mouse and a touch panel, etc. The communication device 106 exchanges data with other devices on the network N1.
[0025] Fig. 4 is a block diagram illustrating an example of the functional configuration of the information processing device 1. Fig. 4 also illustrates an AI machine 2, an SNS server 3, a poster's user device 4-1, an advertisement creator's user device 4-2, and a target location's user device 4-3. In the information processing device 1, as described above, the CPU 101 executes a computer program executablely deployed in the memory 102, and provides the processing of each unit in Fig. 4.
[0026] As shown in FIG. 4, the information processing device 1 has an information collecting unit 11, a teacher data setting unit 12, a learning control unit 13, a teacher data extension unit 14, a judgment control unit 15, and an information presenting unit 16. The information processing device 1 also has a database that stores the teacher data group D1, the SNS data D2, the advertising performance data D3, and the like. All of these components do not need to be implemented in the information processing device 1 as a single unit. The information processing device 1 may allocate any of the components in FIG. 4 to another computer that can cooperate with the information processing device 1 through the network N1. For example, the data such as the teacher data group D1, the SNS data D2, and the advertising performance data D3 may be stored in a database managed by the other computer. For example, the AI machine 2 may be a computer integrated with the information processing device 1.
[0027] The information collection unit 11 accesses computers on the network N1, such as the SNS server 3, and monitors posts to the SNS. The information collection unit 11 also searches the SNS server 3 and collects images that meet various conditions, the poster of the image, the posting location, and other information. The information collection unit 11 can access the SNS server 3 using an API provided by the SNS server 3. The information collection unit 11 saves the collected images, data, and other information in the SNS data D2.
[0028] The teacher data setting unit 12 provides a user interface for an administrator user (hereinafter also referred to as the administrator user) to label data such as images. The administrator user sets evaluation values, exemplified by, for example, "L1", "L2", . . ., "LN", etc., for the data such as images displayed by the teacher data setting unit 12. Then, the teacher data setting unit 12 associates the data such as images with the labels received from the administrator user via the user interface and stores them in the teacher data group D1.
[0029] The learning control unit 13 cooperates with the teacher data extension unit 14 to execute machine learning using the AI machine 2, and create a trained model 22. That is, the learning control unit 13 reads out the teacher data group D1, inputs it into the AI machine 2, and executes machine learning. The teacher data extension unit 14 transforms some of the input data (pairs of input data and labels) read out by the learning control unit 13 from the teacher data group D1, and executes data extension. That is, depending on the degree of transformation of the part of the input data, the teacher data extension unit 14 either maintains the label as it is, or changes the label, and attaches the changed label to the transformed input data, thereby creating new teacher data. The learning control unit 13 controls the AI machine using the original teacher data stored in the teacher data group D1 and the teacher data data-extended by the teacher data extension unit 14. The second model performs machine learning.
[0030] The learning control unit 13 creates a trained model 22 in the AI machine 2 by such machine learning. The learning control unit 13 judges newly input data by the created trained model 22. For example, if the created trained model 22 is a model that judges the degree to which an image is suitable for advertising, the learning control unit 13 judges the degree to which a newly input image is suitable for advertising. As described above, the learning control unit 13, the teacher data extension unit 14, and the AI machine 2 form a judgment engine 17 that judges input data. Note that the judgment engine 17 may be a separate computer different from the information processing device 1 and cooperate with the information processing device 1 via the network N1. Also, the judgment engine 17 may be built into the information processing device 1.
[0031] The determination control unit 15 accesses the determination engine 17 from the learning control unit 13 and executes determination of various data, for example, the degree to which an image posted on an SNS is suitable for advertising, or the degree to which an image is highly rated on an SNS, i.e., the so-called "goodness", etc. The information presentation unit 16 presents various information to the user device 4 based on the determination result by the determination control unit 15 and the past performance values stored in the advertising performance data D3, etc.
[0032] For example, the information presenting unit 16 requests permission from the poster's user device 4-1 who posted an image with a degree of suitability for advertising of "L1" to the SNS to use the image with "L1" as an advertisement. Here, L1 is assumed to be the highest evaluation value. Furthermore, when the information presenting unit 16 obtains permission from the poster's user device 4-1, it sends a message to the advertisement creator's user device 4-2 recommending that the image with permission be used as an advertisement. That is, the information presenting unit 16 executes matching between the poster's user device 4-1 and the advertisement creator's user device 4-2, and stores the matching result in the advertisement performance data D3. The matching result includes whether the poster's permission is granted, whether the image has been adopted as an advertisement, and the usage price (also called transaction performance price) of the image if adopted.
[0033] Furthermore, the information presentation unit 16 selects a shooting location where the number of photos having a degree of suitability for advertising of "L1" is greater than a reference value when the image is a photo, based on the advertising performance data D3. Alternatively, the information presentation unit 16 sorts and ranks the shooting locations in order of the number of photos having a degree of suitability for advertising of "L1" taken when the image is a photo, based on the advertising performance data D3. Then, the information presentation unit 16 selects the shooting locations as target locations in order of the highest ranking. Then, the information presentation unit 16 prompts the user devices 4-3 of target locations located near the selected target location to take a photo at the selected target location and post it to an SNS or directly provide the taken photo to the information processing device 1.
[0034] As already mentioned, the teacher data group D1 holds a large number of teacher data including images and labels attached to the images. The teacher data group D1 is divided into subsets of teacher data for learning various judgments. For example, the first subset is a subset of teacher data for learning the "goodness" of an image. The second subset is a subset of teacher data for learning the degree to which an image is suitable for advertising. The SNS data D2 holds information related to images acquired from an SNS, such as the attributes of the poster, the evaluation value of the posted image, and information related to the environment in which the image was taken. The advertising performance data D3 holds information related to images used as advertisements, the attributes of the advertising creator, and the like.
[0035] (Example data) FIG. 5 shows an example of setting parameters set in the teacher data extension unit 14. The teacher data extension unit 14 performs data extension of the teacher data in accordance with the setting parameters. In this example, the number of types of labels in the teacher data is N. In this example, the following is used for data extension: Two types of setting parameters are set. The first example is the setting parameter P1. The setting parameter P1 specifies a change in data that lowers the label by X levels. The setting parameter P1 changes the teacher data at a rate (frequency) of P% of the teacher data input to the AI machine 2, for example. The setting parameter P1 also performs trimming to randomly cut out a range of T1% of the teacher data image, for example. As an example, P=1% and T1=99%. That is, the setting parameter P1 is a case where trimming is performed to remove a relatively large range from the image, and the label is lowered by X levels. For example, when X=1 level, L1 is changed to L2.
[0036] Note that such a change in label is one example. There may be three or more types of labels. The number of stages in which the labels are changed may be two or more. The method of changing the image is not limited to cropping, and may include rotating the image, changing the contrast of the image, changing the brightness of the image, or changing the color of the image.
[0037] In FIG. 5, the second example is the setting parameter P2. The setting parameter P2 does not change the label of the teacher data. For example, the setting parameter P2 changes the teacher data at a rate (frequency) of 100-P% of the teacher data input to the AI machine 2. Also, the setting parameter P2 performs trimming, for example, by randomly cutting out a range of T2% of the image of the teacher data. As an example, T2=99%. That is, the setting parameter P2 is a case where trimming is performed such that a relatively large range of the image remains, and the label of the teacher data is maintained unchanged.
[0038] FIG. 6 is a diagram illustrating a published image table included in the SNS data D2. The published image table records information related to the evaluation of images published on SNS. The published image table has columns for image ID, posting user ID, photogenicity, number of SNS reactions, suitability for advertising, and transaction record price as components. Note that when the information processing device 1 accesses multiple SNSs, the components of the published image table may include information identifying the SNS (such as the URL of the SNS site). In this embodiment, the information identifying the SNS is omitted.
[0039] The image ID is the ID (e.g., file name) of the image posted on the SNS. The image itself corresponding to the image ID may be stored in the SNS data D2. The image itself corresponding to the image ID may be stored in another computer, such as the SNS server 3, instead of the information processing device 1. The posting user ID is the user ID of the poster on the SNS. The posting user ID may be, for example, an email address. The posting user ID is an example of identification information indicating the source of the post. "Good looks" is an evaluation value of the image on the SNS determined by the AI machine 2, i.e., the determination engine 17 having a trained model that has learned the good looks of the image. However, if the evaluation by the determination engine 17 has not been performed, the "good looks" is left blank.
[0040] The number of SNS reactions is the number of clicks on the image of the image ID on the SNS in a predetermined period. However, the number of SNS reactions may be the number of "likes" entered by other users on the SNS in a predetermined period. The predetermined period is, for example, the period from when the image is posted on the SNS to the present. However, the predetermined period may be a fixed period from when the image is posted on the SNS. Also, the predetermined period may be a period starting from when the image is posted on the SNS until the number of SNS reactions per unit time falls below a reference value. The degree of suitability for advertising is an evaluation value determined by the AI machine 2, that is, the determination engine 17 having a trained model that has learned the degree of suitability of the image for advertising. However, if the evaluation by the determination engine 17 has not been performed, the degree of suitability for advertising is left blank. The transaction result price is the actual value of the transaction price when the image is used for advertising. However, if the transaction has not been performed, 6, the image ID, the degree of suitability for advertisement, and the transaction result price may be stored in a separate table in the advertisement result data D3.
[0041] 7 is a diagram illustrating an example of an image-related information table included in the SNS data D2. The image-related information table records information about the shooting conditions or environment of images posted on the SNS. The image-related information table has rows for the following components: image ID, location information, shooting location, season, month, day of the week, public holiday, time, weather, temperature, humidity, and keyword.
[0042] The image ID is the same as that described in FIG. 6. The location information is the location information registered in the SNS that is set in the image posted by the user. The location information is set, for example, as a station, a landmark, a store name, a facility name of a tourist spot, etc., and is shared among users who post to the SNS. The shooting location is the longitude (X coordinate) and latitude (Y coordinate) of the shooting location when the image is a photograph. The season is the season when the photograph was taken. The day of the week is the day of the week (Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday) when the image is taken when the image is a photograph. The holiday is information indicating whether the day when the photograph was taken is a holiday or not. For example, if the day when the photograph was taken is a holiday, "Y" is set in the row of the holiday. The time is the time when the photograph was taken (0 to 24). The weather, temperature, and humidity are the weather, temperature, and humidity when the photograph was taken. Moreover, a keyword is a message presented by a poster or a viewer with respect to an image when the image is posted on the SNS, a keyword summarizing the message, a hashtag, or the like.
[0043] In addition, information other than the weather, temperature, humidity, and keywords in the image-related information table can be collected when the user does not delete data in the Exif (Exchangeable image file format) format included in the data of the photo registered in the SNS. For example, the longitude and latitude of the shooting location can be acquired from the Exif format data. The season, month, day of the week, holiday, and time can be acquired by, for example, comparing the shooting date and time information included in the Exif format data with the calendar information in the information processing device 1. In addition, the weather, temperature, and humidity can be acquired by searching the website of the Japan Meteorological Agency or the website of a weather forecast service on the Internet based on the shooting date and time information included in the Exif format data. In addition, if the information processing device 1 cannot acquire the Exif format data from the photo data of the SNS, it may access the user device 4-1 of the poster and ask for the acquisition of the Exif format data when obtaining permission for advertisement use.
[0044] 8 is a diagram illustrating a poster attribute table of the SNS data D2. The poster table records information illustrating the attributes of posters to SNS. The poster attribute table has, as components, rows of user ID, photogenicity evaluation record, evaluation record value of suitability for advertisement, gender, age, number of followers, SNS reaction number record value, and record value of permission rate for image use.
[0045] The user ID has already been described in the table of posted images in FIG. 6. The photogenicity evaluation result value is the average value of the evaluation value of the image on the SNS for the user of the user ID, which was previously determined by the AI machine 2, i.e., the determination engine 17. For example, when the label is two-level, it is the average value when quantified as L1=1 and L2=-1. The evaluation result value of the suitability for advertisement is calculated in the same manner as the photogenicity evaluation result value. The gender and age are the gender (M: male, W: female) and age of the poster. The number of followers is the number of followers on the poster's SNS. The SNS reaction number result value is the result of the number of SNS reactions, for example, the average number of reactions per image, calculated by dividing the total number of SNS reactions by the number of posted images by each poster. The SNS reaction number has already been described in FIG. 6, and is the number of clicks, likes, etc. for the image of the image ID on the SNS in a specified period. The actual value of the permission rate for image use is the average permission rate (the rate at which use is permitted) when the information processing device 1 asks the user device 4-1 of the contributor about use for advertisements.
[0046] Among these, the past values of the photogenicity evaluation record and the permission rate for image use can be calculated based on past information recorded in the information processing device 1. The gender, age, number of followers, and number of SNS reactions can be acquired from the SNS server 3 via an API. However, when requesting permission from the poster's user device 4-1, the information processing device 1 may also request user attributes such as gender and age and accept the input.
[0047] FIG. 9 is a diagram illustrating an advertisement creator attribute table of the advertisement performance data D3. The advertisement creator attribute table has, as components, rows of advertisement creator ID, gender, age, preference, and summary of past operation log information. The advertisement creator ID is an ID when the advertisement creator uses a service provided by the information processing device 1. The advertisement creator ID may be, for example, the advertisement creator's email address. The gender, age, and preference can be said to be the attribute information of the advertisement creator. The preference is a character string indicating a favorite color, a favorite season, a favorite sport, a favorite dish, a hobby, etc. However, the attribute information of the advertisement creator is not limited to gender, age, and preference. The summary of past operation log information is an excerpt of a character string included in an operation log when the advertisement creator uses a service provided by the information processing device 1. For example, a character string recorded a predetermined number of times or more is extracted as a summary of past operation log information and recorded in the advertisement creator attribute table.
[0048] (Machine Learning) With reference to FIG. 10, the procedure of machine learning in the information processing device 1 is illustrated. FIG. 10 is a diagram illustrating the procedure when the AI machine 2 learns the degree of suitability for advertisement. The learning control unit 13 in FIG. 4 inputs the training data of the training data group D1 to the AI machine 2. The training data group D1 includes, for example, a plurality of subsets of training data in which each of the images IM1, IM2, ..., IMk posted on the SNS is labeled. Each subset includes, for example, training data for learning the "visibility" and training data for learning the degree of suitability for advertisement. The label is a character string indicating the degree to which each of the posted images IM1, IM2, ..., IMk is suitable for advertisement, for example, "L1", "L2", ..., "LN", etc. The "visibility" and the degree of suitability for advertisement are as already described in the learning system 21 in FIG. 2. The administrator user who sets the labels sets labels to the posted images IM1, etc. based on the subjective impression of the posted images IM1, etc., the number of reactions (number of clicks, number of likes) on social media to similar images in the past, the actual trading price, etc.
[0049] The AI machine 2 receives the labeled data as input, propagates the data through each layer of the neural network, and judges the data by a convolution operation between the data (usually vector data) and the weight set for each layer. Then, the AI machine 2 propagates the error between the judgment result and the label in the reverse direction, and adjusts the weight of each layer of the neural network in the direction in which the error becomes smaller. Note that there is no limit to the number of teacher data input when adjusting the weight. For example, the AI machine 2 may adjust the weight by propagating the error in the reverse direction for each input of teacher data. The AI machine 2 may also obtain a subset from the teacher data group D1, and adjust the weight by propagating the error in the reverse direction in units called mini-batches. The AI machine 2 may also obtain, for example, the entire subset of the learning data for learning the degree of suitability for advertising, perform image judgment, and after judging all the learning data of the subset, propagate the error in the reverse direction to adjust the weight.
[0050] Then, when the error for the teacher data input one after another becomes equal to or less than the reference value, the creation of the trained model is completed. As already mentioned, the teacher data input to the AI machine 2 includes the original data read from the teacher data group D1 as well as the data extended by the teacher data extension unit 14.
[0051] As described above, it can be said that the information processing device 1 generates a trained model by machine learning (for example, deep learning) using teacher data including labeled input data.
[0052] (Processing flow) The processing flow of the information processing device 1 will be described with reference to Fig. 11 to Fig. 18. Fig. 11 is a diagram illustrating a processing procedure of machine learning by the information processing device 1 (for example, the learning control unit 13 in Fig. 4). Hereinafter, a first subset of the training data group D1 used for machine learning is training data that learns, for example, whether an image is highly rated on SNS, that is, the "good looks" of an image. Also, a second subset of the training data group D1 used for machine learning is training data that learns, for example, the degree to which an image is suitable for advertising.
[0053] In FIG. 11, a part of the teacher data group D1 is acquired, and machine learning is performed in mini-batches. However, the machine learning of the information processing device 1 is not limited to processing by mini-batches. That is, the information processing device 1 may input a subset of the teacher data group D1 for learning "visibility" at once, and perform batch learning to adjust the weight coefficient in a direction to reduce errors. Also, the information processing device 1 may perform online learning to adjust the weight coefficient in a direction to reduce errors every time each piece of teacher data for learning "visibility" is input from the teacher data group D1.
[0054] In this process, the information processing device 1 sets parameters for the data augmentation process (S1). The parameters for the data augmentation process are illustrated in FIG. 5, for example. Then, the information processing device 1 acquires the next mini-batch of data (S2). Then, the information processing device 1 executes the data augmentation process on the mini-batch of data according to the parameters for the data augmentation process (S3). Then, the information processing device 1 inputs the data augmented mini-batch of data to the AI machine 2, and executes machine learning (S4). The AI machine 2 executes forward processing from the input layer and backward error propagation processing on the neural network. That is, in the forward processing, the AI machine 2 executes input of data, convolution operation by weighting coefficients for the input data, pooling processing, output processing by the fully connected layer, error determination, and the like. In the backward error propagation processing, the AI machine 2 propagates the error determined in the top layer in the backward direction, and executes update processing of the weighting coefficients of each layer of the neural network, and the like.
[0055] Then, the information processing device 1 determines whether all mini-batches have been completed (S5). If unprocessed mini-batches remain, the information processing device 1 returns control to S2 and executes the next mini-batch. Although omitted in FIG. 11, when all mini-batches have been completed, the information processing device 1 determines whether the error has sufficiently converged, and if the error has sufficiently converged, the information processing device 1 completes the formation of the trained model. If the error has not sufficiently converged, the information processing device 1 executes the next machine learning using new training data for continuing the learning.
[0056] FIG. 12 is a diagram illustrating the details of the data extension process (S3 in FIG. 11). In this process, the information processing device 1 applies a deformation exceeding a reference value to a predetermined percentage (P%) of the teacher data included in the mini-batch (S31). The process of S31 can be said to be a process of changing the first input data included in the first teacher data to become the second input data. Here, the predetermined percentage (P%) of the teacher data included in the mini-batch is an example of the first input data. Moreover, the data to which the deformation has been applied is the second input data.
[0057] The deformation exceeding the reference value is, for example, trimming the range of the image to a ratio T1% exceeding the reference value, or rotation of the image exceeding the reference value, or change in contrast or brightness exceeding the reference value.
[0058] Then, the information processing device 1 changes the label in response to the deformation (S32). For example, in the case of a one-step change in teacher data to which four types of labels are assigned, the label is changed from L1 to L2. However, the types of labels are not limited to two, and the step in which the label is changed is not limited to one step. In addition, the label may be not only downgraded but also upgraded depending on the change in contrast, etc. The process of S32 can be said to be assigning a second label different from the first label to the second input data depending on the degree of change from the first input data to the second input data.
[0059] Furthermore, the information processing device 1 applies a deformation less than a reference value to the remaining data (100-P%) of the teacher data included in the mini-batch. At this time, the information processing device 1 maintains the label of the teacher data to which the deformation has been applied as it is (S33). The process of S33 can be said to be maintaining the first label according to the degree of change from the first input data to the second input data. Therefore, the processes of S32 and S33 can be said to be an example of creating second teacher data by assigning the first label or a second label different from the first label to the second input data according to the degree of change from the first input data to the second input data.
[0060] FIG. 13 is a diagram illustrating the procedure of the image determination process E1. The information processing device 1 executes a determination of a new image and a process associated with the determination result, using the trained model created by the machine learning of FIG. 11 and FIG. 12. Note that in FIG. 13, a trained model for determining "good looks" and a trained model for determining the degree of suitability for advertisement are used as examples of trained models. In addition, in the image determination process E1 of FIG. 13, the information processing device 1 judges images posted sequentially on the SNS without setting a priority order for the images posted. The images posted on the SNS are an example of images that have been posted in a viewable manner on a public network in the past.
[0061] In this process, the information processing device 1 (information collecting unit 11 in FIG. 4) monitors the SNS (S11A). The information processing device 1 determines whether or not new images have been posted (S12A). Note that instead of constantly monitoring posts to the SNS, the information processing device 1 may periodically search the SNS for divided periods and collectively process images posted for each divided period.
[0062] If there is no new post (NO in S12A), the information processing device 1 returns control to the process of S11A. Then, if there is a new post (YES in S12A), the information processing device 1 (determination control unit 15 in FIG. 4) judges the photogenicity of the posted image (S13). The photogenicity may be judged, for example, by a trained model that has learned to judge photogenicity. The information processing device 1 may also judge the photogenicity based on the number of reactions to the post on SNS for a predetermined period after posting, for example, the number of clicks or the number of likes. If it is judged not to be photogenic in S13 (NO in S14), the information processing device 1 returns control to the process of S11A. Note that, when evaluating the number of likes, the information processing device 1 may correct the number of likes by the number of followers of the poster. That is, the information processing device 1 may standardize the number of likes by the number of followers by multiplying the number of likes by a correction coefficient that reduces the number of likes for posters with a larger number of followers. It is only natural that posters with a larger number of followers will receive more likes than their "coolness" rating would suggest.
[0063] If it is determined that the posted image looks good (YES in S14), the information processing device 1 then determines the degree to which the posted image is suitable for advertising (S15). The information processing device 1 may determine the degree to which a newly posted image is suitable for advertising, for example, using a trained model that has learned the degree to which past images are suitable for advertising. Note that the information processing device 1 may omit the determination of the degree of beauty in S13 and determine the degree to which the image is suitable for advertising in S15. If it is determined in the process of S15 that the posted image is not suitable for advertising, the information processing device 1 returns control to the process of S11A. Not being suitable for advertising means, for example, using a trained model that has learned the degree to which the image is suitable for advertising. The determination is "L2".
[0064] If the posted image is judged to be suitable for advertisement in the process of S15, the information processing device 1 estimates the expected remuneration or value information when the judged image is used for advertisement (S18). The expected remuneration may be estimated, for example, from the actual usage record of the image judged to be suitable for advertisement in advertisement. For example, if the judgment result in S15 is "L1" in the past, the information processing device 1 may use the actual value and average value of the transaction price in the use in advertisement of the image judged to be suitable for advertisement in the past as "L1" as the expected remuneration. In addition, the value information is, for example, the expected remuneration converted into a numerical value other than the amount. The numerical value other than the amount is, for example, an evaluation value on a 10-point scale. The process of S18 is an example of a process of calculating the expected remuneration or the value information as an advertisement that can be expected as a remuneration for the image based on at least one of the degree to which the judged image is suitable for advertisement, the reaction of viewers on the public network to the image, and the transaction record of the image judged to be suitable for advertisement.
[0065] Next, the information processing device 1 (information presentation unit 16 in FIG. 4) asks the poster of the image if the image can be used for advertising (S19). The poster of the image is identified by, for example, the identification information of the poster's user device 4-1, the poster's user ID on the SNS, etc. The identification information of the user device 4-1 may be, for example, a Medium Access Control (MAC) address, a domain name on the Internet, a mobile phone number, etc. The information processing device 1 may be a mobile phone number of a terminal that functions as a proxy for the image. In this case, the information processing device 1 presents the poster with expected remuneration or value information when the image estimated in S18 is used for an advertisement. Then, the information processing device 1 determines whether or not permission for use in an advertisement has been obtained from the poster of the image (S20). If permission for use in an advertisement has not been obtained, the information processing device 1 returns control to the process of S17.
[0066] On the other hand, if permission for use in an advertisement is obtained, the information processing device 1 transmits a message to the user device 4-2 of the advertisement creator proposing to use the image determined to be suitable for the advertisement in the advertisement (S22). The message may be transmitted by email or may be transmitted as a message on an SNS.
[0067] Then, the information processing device 1 judges whether or not there is a response from the user device 4-2 of the advertisement creator that the proposed image will be adopted in the advertisement (S23). If there is no response within a predetermined period of time, or if there is a response that the image will not be adopted, the information processing device 1 returns control to the process of S17.
[0068] On the other hand, if the proposed image is adopted in the advertisement, the information processing device 1 records the price of the adopted image in the transaction history in the advertisement history data D3 or the published image table in the SNS data D2 (S24). Furthermore, the information processing device 1 creates training data based on the image judgment result in Fig. 13 (S25). As described above, it can be said that the information processing device 1 executes information processing using a trained model.
[0069] Fig. 14 is a diagram illustrating the procedure of the image determination process E2. In Fig. 13, the information processing device 1 constantly monitors posts to the SNS, or accesses the SNS at set periods and determines the posted images. In Fig. 14, instead of the process as in Fig. 13, the information processing device 1 collects images posted to the SNS during a predetermined period, determines images to be prioritized in advance, and processes the images posted in the determined priority order. Hereinafter, such a process is referred to as the image determination process E2.
[0070] In this image evaluation process E2, the information processing device 1 collects information about each poster who has posted to the SNS. Then, the information processing device 1 evaluates the poster based on, for example, the performance score of the appearance of images posted by the poster in the past, the performance score of the suitability for advertisements, the permission record, and the transaction record price when the image was used in an advertisement, and sorts the poster based on the evaluation score (S11B). The performance value evaluated for the image, the performance value of suitability for advertisement, the permission performance, the transaction performance price, etc. are as described in Fig. 8. Furthermore, the evaluation of the poster may be performed, for example, by assigning points for each of the performance value of the appearance of the image posted in the past, the performance value of suitability for advertisement, the permission performance, and the transaction performance price, and sorting the poster by the total points. However, the points may be based on one item such as the performance value of the appearance of the image posted in the past, the performance value of suitability for advertisement, the permission performance, or the transaction performance price.
[0071] Next, the information processing device 1 selects a predetermined number of contributors from the top of the contributors sorted in the process of S11B, and acquires images of the selected contributors from among the posted images (S12B). Then, the process of S13 to S25 is executed for the selected images. The process of S13 to S25 has already been described in FIG. 13. As a result, in the process of S19, the information processing device 1 sorts contributors by the total score, and asks for advertisement use in order from the top contributor. In this case, the information processing device 1 may ask for the contributors by giving priority to the performance value of the photogenicity of the images posted in the past and the performance value of the suitability for advertisement. Therefore, in the process of S19 in FIG. 14, the information processing device 1 executes the image judgment process by the trained model, and asks for the use of the images for advertisement by giving priority to the contributors of the images in order of the evaluation by the judgment process.
[0072] FIG. 15 is a diagram illustrating the details of the teacher data creation process (S25 in FIG. 13 and FIG. 14). Here, an example of the creation of teacher data for learning the degree of suitability for advertisements will be described. In this process, the information processing device 1 (teacher data setting unit 12 in FIG. 4) acquires, for a posted image, the degree of beauty of the image determined by the AI machine 2, the degree of suitability of the image for advertisements determined by the AI machine 2, the expected remuneration estimated in the process of S18 in FIG. 13 and FIG. 14, the presence or absence of permission from the poster to use the image for advertisements, and the transaction record price when used as an advertisement (S251).
[0073] Next, the information processing device 1 presents the information acquired in S251 to the user who is the administrator, and receives the setting of the label (S252). The user who is the administrator may maintain the degree of suitability of the image for advertisement determined by the AI machine 2 as it is. In addition, the user who is the administrator may correct and reset the degree of suitability of the image for advertisement determined by the AI machine 2, by referring to the degree of appearance of the image, the presence or absence of permission for use in advertisement, the transaction record price at the time of use as advertisement, etc., via the user interface of the information processing device 1. Then, the information processing device 1 stores the created teacher data in the teacher data group D1 as teacher data of the degree of suitability for advertisement (S253). In addition, in the creation of teacher data for learning the "degree of appearance", the information processing device 1 (teacher data setting unit 12 in FIG. 4) may present the degree of appearance of the image determined by the AI machine 2, the number of reactions on SNS, etc., for the posted image to the user, and receive the setting of the label for the degree of appearance for each image.
[0074] 16 is a diagram illustrating a photography recommendation process executed by the information processing device 1. In this process, the information processing device 1 selects a target location suitable for photography based on the usage history of posted images in advertisements, and recommends photography to user devices 4-3 at the target location located near the selected target location. In this process, the information processing device 1 acquires information on images that have been used in advertisements from the posted image table (FIG. 6) of the SNS data D2 (S41).
[0075] Then, the information processing device 1 assigns a score to each image based on, for example, the photogenicity determined by the AI machine 2, the suitability for advertising determined by the AI machine 2, or the transaction history price, and sorts the images according to the assigned scores (S42). The score may be based on one item such as photogenicity, suitability for advertising, or transaction history price, or may be a weighted sum of these scores. The information processing device 1 calculates expected remuneration or value information based on the photogenicity, suitability for advertising, or transaction history price, and calculates the expected remuneration or value information. The images may be sorted based on the score information. The process of S42 is an example of a process of setting an evaluation value of a posted image based on expected remuneration or value information, suitability for advertising, or transaction record price.
[0076] Next, the information processing device 1 identifies the shooting locations of the images with the highest scores in S42 (S43). The process of S43 can be said to be an example of a process of identifying the locations where the posted images with evaluation values equal to or higher than a reference value were shot. For example, the information processing device 1 identifies the shooting locations of each of the images with the highest scores in S42. Since the shooting locations are identified by, for example, longitude and latitude, the information processing device 1 collects shooting locations located within a predetermined distance from the shooting locations of the images with the highest scores in S42. When there are multiple collected shooting locations, the center of gravity of the multiple shooting locations may be used as a representative shooting location. In addition, the shooting locations may be tallied based on location information defined in the SNS. As already mentioned, the location information is, for example, a station, a landmark, a store name, a facility name of a tourist spot, etc.
[0077] Furthermore, the information processing device 1 narrows down the locations of the above-mentioned shooting locations or representative locations to locations with a high possibility of shooting based on shooting conditions including the season, date and time when the image was shot, and past weather data for the season and date and time (S44). For example, from among the images corresponding to the shooting locations identified in S43, images are selected that have past weather data for the season, date and time when the image was shot, similar to the current season, date and time, and current weather data. That is, the information processing device 1 narrows down the images from among the images with high scores in S42 to images that can be realized under similar shooting conditions at the present time. Note that, if the images are narrowed down in the process of S44, and the corresponding image and its shooting location cannot be obtained, the information processing device 1 may omit the process of S44.
[0078] Then, the information processing device 1 identifies a user device 4-3 at a target location that is located near the photography location aggregated in the process of S43 or the photography location set as a representative (hereinafter referred to as a target location), or the target location narrowed down in the process of S44, and transmits a message encouraging photography to the user device 4-3 at the target location (S45). The message may be transmitted by email or a message on an SNS. The process of S45 can be considered as an example of a process of transmitting information encouraging a device located near the identified location to post an image viewable on a public network.
[0079] 17 is a diagram illustrating an example of an image recommendation process R1 for an advertisement creator executed by the information processing device 1. In this process, when an advertisement theme or keyword is given, the information processing device 1 presents the corresponding image to the advertisement creator. In this process, the information processing device 1 (information presentation unit 16 in FIG. 4) acquires a theme or keyword related to the advertisement from the advertisement creator's user device 4-2 via, for example, a user interface (S51). The process of S51 is an example of acquiring a condition including a theme or keyword related to the advertisement.
[0080] Next, the information processing device 1 searches for the keyword row in the image-related information table of the SNS data D2, and collects the image IDs of images corresponding to the theme or keyword (S52). The process of S52 is an example of collecting images that satisfy a condition from among images viewable on a public network. Then, the information processing device 1 estimates the poster's expected remuneration or value information for each collected image ID from its photogenicity, its suitability for advertising, the number of reactions on the SNS, the transaction record price, etc. (S53). The relationship between the poster's expected remuneration and the photogenicity, its suitability for advertising, the number of reactions on the SNS, the transaction record price, etc. can be determined by, for example, regression analysis.
[0081] Next, the information processing device 1 calculates the cost of advertisement production based on the expected remuneration and value information (S54). The cost of advertisement production is calculated, for example, by multiplying the poster's expected remuneration or value information by a predetermined coefficient, or by processing the poster's expected remuneration or value information with a predetermined function. Then, the information processing device 1 presents the image corresponding to the collected image ID and the cost to the advertisement creator, and recommends the image (S55).
[0082] 18 is a diagram illustrating an example of an image recommendation process R2 for advertisement creators executed by the information processing device 1. In this process, the information processing device 1 (information presentation unit 16 in FIG. 4) acquires attribute information of the advertisement creator to be processed next from the advertisement creator attribute table of the advertising performance data D3, or log information or a summary of the log information when the advertisement creator searched for an image (S61). The advertisement creator's attribute information includes gender, age, preferences, etc.
[0083] Next, the information processing device 1 acquires information on the published image and the photographer that matches the attribute information or log information, etc. (S62). The information processing device 1 may search the image-related information table of the SNS data D2, for example, with keywords related to the attribute information or log information of the advertisement creator. Furthermore, since the information processing device 1 searches with the attribute information or log information and keywords related thereto, the information processing device 1 may prepare a thesaurus that defines words and words related thereto. The information processing device 1 may then acquire one or more keywords related to the attribute information or log information of the advertisement creator from the thesaurus, and search the image-related information table of the SNS data D2 with the acquired keywords.
[0084] Then, when the information processing device 1 finds the acquired keyword in the image-related information table of the SNS data D2, it determines that the image specified by the image ID of that row is suitable for the advertisement creator. The information processing device 1 collects such suitable images from the image-related information table of the SNS data D2. Then, the information processing device 1 determines the advertisement creator currently being processed as the advertisement creator suitable for the collected published images (S63). Then, the information processing device 1 presents the image suitable for the advertisement creator and information on the photographer of the suitable image, and presents information on the advertisement creator suitable for the image to the photographer of the image (S64). In other words, the published images and the photographers of the published images are matched with the advertisement creators.
[0085] Next, the information processing device 1 determines whether or not there are any advertisement creators remaining to be processed next (S65). If there are any advertisement creators remaining to be processed next, the information processing device 1 returns control to the process of S61. If there are no advertisement creators remaining to be processed next, the information processing device 1 ends the process.
[0086] (Effects of the embodiment) In this embodiment, the information processing device 1 maintains the label of the teacher data or changes the label according to the transformation of the teacher data. That is, the information processing device 1 creates second teacher data by assigning a first label or a second label different from the first label to the second input data according to the degree of change from the first input data to the second input data. Therefore, even when the information processing device 1 causes the AI machine 2 to perform machine learning using the teacher data group D1, the information processing device 1 can transform the data, i.e., expand the data, and assign appropriate labels to appropriately generate new teacher data. As a result, the information processing device 1 can input a large amount of appropriate teacher data to the AI machine 2 and cause it to perform machine learning.
[0087] In addition, since the training data of the training data group D1 includes an image and the degree of suitability for an advertisement attached to the image, a trained model that learns the degree of suitability for an advertisement can be generated.
[0088] Furthermore, the information processing device 1 calculates the expected remuneration for the image posted on the SNS or the advertising value information based on the degree to which the image is suitable for advertising, and presents the calculated expected remuneration for the image to related parties such as the poster. As a result, the information processing device 1 allows the poster to recognize the advertising value of the image.
[0089] Furthermore, in the process of S18, the information processing device 1 determines the expected remuneration or value information based on at least one of the images previously posted as viewable on the SNS, which is a public network, the reactions of viewers on the public network to the images, and the transaction results of advertising the images. Therefore, the information processing device 1 can determine appropriate expected remuneration or value information.
[0090] 13 and 14, the information processing device 1 presents the poster with expected remuneration or value information when the image is used for advertising in the process of S18, so that the poster can obtain specific information regarding the remuneration when deciding whether or not to permit the use for advertising. Therefore, the information processing device 1 allows the poster to appropriately decide whether or not to permit the use of the image for advertising.
[0091] Furthermore, the information processing device 1 executes image determination processing using a trained model, as in the image determination processing E2 in Fig. 14, and asks the posters of the images to use the images for advertisements, giving priority to those who have a higher evaluation in the determination processing. Therefore, the information processing device 1 can ask the posters who are more likely to post images suitable for advertisements to use the images, giving priority to those who have a higher possibility of posting images suitable for advertisements.
[0092] 16, the information processing device 1 sets a score as an evaluation value of the posted image based on the expected remuneration or value information, the suitability for advertisement, or the transaction record price. The information processing device 1 aggregates the shooting locations of the images with the highest scores in S42, identifies the target location, and transmits a message recommending shooting to the user device 4-3 at the identified target location. Therefore, the information processing device 1 allows the user to effectively post images with high expected remuneration or value information, the suitability for advertisement, or the transaction record price.
[0093] Furthermore, as in S44, the information processing device 1 selects images corresponding to the shooting locations identified in S43, which are similar in season, date and time, and past weather data for the season and date and time when the images were shot to the current season, date and time, and past weather data for the season and date and time. Therefore, the information processing device 1 can cause the poster to post images with a high degree of photogenicity, suitability for advertising determined by the AI machine 2, transaction record price, expected remuneration, or value information with a higher probability.
[0094] Furthermore, the information processing device 1 acquires conditions including a theme or keyword related to the advertisement and collects images corresponding to the theme or keyword, so that it is possible to obtain an image suitable for the given theme or keyword. Furthermore, the information processing device 1 calculates the cost of advertisement production based on the expected remuneration and value information from the collected images and presents it to the advertisement creator, so that it is possible to support the advertisement creator by presenting the image desired by the advertisement creator together with the cost.
[0095] Furthermore, the information processing device 1 acquires the attribute information of the advertisement creator or the log information or a summary of the log information when the advertisement creator searches for an image from the advertisement creator attribute table of the advertisement performance data D3. Then, the information processing device 1 acquires the image posted on the SNS and the information of the photographer that matches the attribute information or the log information, and can match the advertisement creator with the photographer.
[0096] (Modification) In the above Fig. 18, taken images and photographers that match the advertisement creator are collected based on the attribute information of the advertisement creator or log information when the advertisement creator searches for images. Instead of such a process, taken images that match the advertisement creator may be searched for when the advertisement creator searches for images. Fig. 19 is a diagram illustrating an image recommendation process R3 that acquires taken images that match a query when the advertisement creator inputs a search query.
[0097] In this process, the information processing device 1 (information presentation unit 16 in FIG. 4) performs image search by the advertisement creator. The advertisement creator acquires a query for searching from the user device 4-2 of the advertisement creator (S71). The query is in natural language, for example, "beautiful beach," "wild animals," "beautiful flowers," and the like.
[0098] Next, the information processing device 1 acquires information on posted images and photographers that match the query (S72). For example, the information processing device 1 may search the image-related information table of the SNS data D2 using keywords (nouns, adjectives, etc.) included in the query. Furthermore, since the information processing device 1 searches using attribute information or log information and keywords related thereto, the information processing device 1 may prepare a thesaurus that defines words and words related thereto. The information processing device 1 may then acquire one or more keywords related to the query entered by the advertisement creator from the thesaurus, and search the image-related information table of the SNS data D2 using the acquired keywords.
[0099] Then, when the information processing device 1 finds the acquired keyword in the image-related information table of the SNS data D2, it determines that the image identified by the image ID of that row matches the advertisement creator's query. The information processing device 1 acquires such matching images from the image-related information table of the SNS data D2. Then, the information processing device 1 presents the matching images and information on the photographers of the matching images to the advertisement creator (S73). Then, the information processing device 1 waits for a response from the advertisement creator. If the advertisement creator is not satisfied with the presented images (NO in S74), the information processing device 1 returns the process to S72, acquires the next published image and information on the photographer that matches the query, and continues the process.
[0100] On the other hand, when the information processing device 1 receives a response from the user device 4-2 of the advertisement creator that the presented image is acceptable (YES in S74), the information processing device 1 presents the advertisement creator to the photographer of the image, i.e., the user device 4-1 of the poster, and waits for an agreement between the two (S75). Then, the information processing device 1 waits for a response from the photographer of the image. When the information processing device 1 does not receive a response from the photographer of the image, i.e., the user device 4-1 of the poster, that the advertisement creator and the photographer have been agreed upon (NO in S76), the information processing device 1 returns the process to S72, and next acquires information on the published image and the photographer that matches the query, and continues the process. When the information processing device 1 receives a response from the photographer of the image, i.e., the user device 4-1 of the poster, that the advertisement creator and the photographer have been agreed upon (YES in S76), the information processing device 1 ends the process. That is, the information processing device 1 can match the published image and the photographer of the published image with the query of the advertisement creator.
[0101] <Second embodiment> In the first embodiment, the information processing device 1, for example, data-extends the mini-batch teacher data, inputs the data-extended teacher data to the AI machine 2, and executes learning. In this embodiment, the information processing device 1 evaluates images using a trained model before the update, and selects labeling image candidates that are suitable for advertisements. Here, the information processing device 1 re-evaluates the labeling image candidates selected in the trained model before the update based on the evaluation of the administrator user, creates updated teacher data, and inputs it again to the AI machine 2. Then, the AI machine 2 re-learns the updated teacher data again, and creates the updated trained model. The information processing device 1 of this embodiment sequentially updates the trained model through a loop of creating a trained model, evaluating the judgment result by the trained model by the administrator user, updating the teacher data based on the evaluation, and re-learning, thereby improving the accuracy of judgment.
[0102] FIG. 20 is a diagram illustrating a configuration of an information system 10A according to the second embodiment. In this embodiment, the information system 10A also includes an information processing device 1 and an AI machine 2. In this embodiment, the AI machine 2 also includes a learning system 21, and generates a trained model. However, in this embodiment, the learning system 21 generates a trained model after the update based on a pre-update trained model 22-1. Generate trained model 22-2.
[0103] Therefore, the AI machine 2 first evaluates the image candidates D10 using the pre-update trained model 22-1, selects a highly rated image, and sets it as the labeling image candidate D11. Here, the image candidate D10 is an image that will be a candidate for the training data after the update. The image candidate D10 is, for example, an image acquired from the SNS server 3. The image candidate D10 may be, for example, an image included in the training data used to train the pre-update trained model. In addition, a highly rated image is, for example, an image that looks good or is highly suitable for advertising.
[0104] The information processing device 1 provides an administrator user with a user interface as a labeling server. That is, the information processing device 1 presents a labeling image candidate D11 that is a highly rated image, and receives the suitability from the administrator user. Then, the information processing device 1 sets a label to the labeling image candidate according to the suitability received from the administrator user, and creates updated teacher data D12. Then, the information processing device 1 inputs the updated teacher data D12 to the AI machine 2, executes learning, and creates an updated trained model 22-2.
[0105] 21 is a diagram illustrating a processing procedure in the information system 10A of the second embodiment. In this processing, first, the AI machine 2 scores image candidates D10 using a pre-update trained model (S101). Next, the AI machine 2 extracts high-scoring images from the scored images as labeling image candidates D11 (S102).
[0106] Next, the information processing device 1, as a labeling server, accepts human evaluations of the extracted labeling image candidates D11 (S103). Images with high photogenicity or high advertising evaluations are relatively few, and in order to efficiently train the AI machine 2, the processes from S103 onward (for example, processes including S105, S106, and S107) are effective. Furthermore, in addition to the processes from S101 to S102, the information processing device 1 accepts natural language for transforming the image candidate D10, and transforms the image candidate D10 according to the natural language using the AI machine 2 (S104).
[0107] Here, the natural language is, for example, a description indicating the specification for transforming the image candidate D10. For example, when "change tulip to carnation" is specified for an image depicting a tulip flower, the AI machine 2 creates an image candidate D10 depicting a carnation. This is realized, for example, by flowing data from the top fully connected layer to the input layer side in the reverse direction in the hierarchy of the neural network. For example, it is assumed that the AI machine 2 has a trained model that recognizes carnations. When an image is input from the input layer to this trained model, the AI machine 2 determines whether the image is a "carnation" or not, and if the image includes a carnation, outputs the natural language "carnation" from the fully connected layer. On the other hand, when the natural language "carnation" is input to the fully connected layer of this trained model, the data is sequentially converted in the input layer direction, making it possible to generate an image including "carnation". Note that the natural language is not limited to the above, and for example, when "change the time to evening" is specified, the landscape photo is changed to one including a sunset. Also, for example, when the natural language is specified as "a painting in the style of Picasso," the image is changed to resemble a painting by Picasso.
[0108] The AI machine 2 may execute the process of S104 without human intervention. For example, the AI machine 2 recognizes the image candidate D10, and generates the natural language "beautiful tulips and coffee cups on a table" from the recognized image. Here, it is assumed that the AI machine 2 has already learned images of objects such as "tulips" and "coffee cups." It is also assumed that the AI machine 2 has learned to classify "tulips" into "beautiful tulips," "withered tulips," "withered tulips," and so on.
[0109] The AI machine 2 has a "word list related to similar objects that look good". For example, for the word "tulip", "carnation" is registered in the "word list related to similar objects that look good". However, the "word list related to similar objects that look good" may also register "words" together with conditions. For example, for the word "tulip", "carnation" is registered in the "word list related to similar objects that look good" with the time conditions of "May" and "early summer". Also, for example, for the word "tulip", "cherry blossom" is registered in the "word list related to similar objects that look good" with the time conditions of "March to April" and "spring". When the condition that the current time is "May" or "early summer" is satisfied, the AI machine 2 converts the natural language "beautiful tulips and coffee cups on the table" to "beautiful carnations and coffee cups on the table". Furthermore, when the condition that the current season is "March to April" or "Spring" is satisfied, AI machine 2 converts the natural language "beautiful tulips and a coffee cup on the table" to "beautiful cherry blossoms and a coffee cup on the table." Then, AI machine 2 may change image candidate D10 according to the converted natural language.
[0110] Then, the information processing device 1 presents the candidate images transformed by the above process and receives the suitability from the administrator user (S105).Then, the information processing device 1 sets a label to the labeling image candidate D11 extracted in the process of S102 based on the evaluation in S103, and creates updated teacher data D12.The information processing device 1 also sets a label to the image candidate transformed in the process of S104 according to the suitability received from the administrator user in the process of S105, and creates updated teacher data D12 (S106).
[0111] Then, the information processing device 1 inputs the updated teacher data, which is an image candidate reflecting human evaluation, as a learning image, and causes the AI machine 2 to execute re-learning (S107). Then, at the time of the next performance improvement, the AI machine 2 deploys the created trained model as an updated trained model and uses it for image judgment (S108). Note that in FIG. 20, the information processing device 1 may provide the processes of S101, S102, S103, and S106 to S108, and the processes of S104 to S108, independently as separate functions.
[0112] As described above, in this embodiment, the information processing device 1 creates a trained model trained using the first teacher data before the update, and judges image candidates that are candidates for updated teacher data using the trained model before the update. Then, the information processing device 1 accepts the suitability of the result judged using the trained model before the update. Furthermore, based on the accepted suitability, the information processing device 1 resets the updated label to the image candidate to create updated teacher data. Through such processing, the information processing device 1 can improve the accuracy of judgment of the trained model.
[0113] Furthermore, in this embodiment, the information processing device 1 receives a natural language from the AI machine 2 indicating a direction or a specification of the change to change the image candidate, and creates a changed image candidate by changing, for example, the first input data included in the first teacher data according to the natural language. Then, the information processing device 1 receives a label setting changed according to the change in the updated image candidate. Therefore, the information processing device 1 can flexibly and freely perform data extension by freely transforming the image candidate using natural language and attaching an appropriate label.
[0114] <Computer-readable recording medium> A program that causes a computer or other machine or device (hereinafter referred to as a computer, etc.) to realize any of the above functions can be recorded on a recording medium that can be read by the computer, etc. The functions can be provided by having a computer or the like read and execute the program from this recording medium.
[0115] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Examples of such recording media that can be removed from a computer include flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, Blu-ray disks, DATs, 8mm tapes, and memory cards such as flash memory. Examples of recording media that are fixed to a computer include hard disks and ROMs (read-only memories). Furthermore, SSDs (Solid State Drives) are also recording media that can be removed from a computer, and can be read by a computer. The present invention can also be used as a recording medium fixed to a computer. [Explanation of symbols]
[0116] 1. Information processing device 2. AI Machines 3. SNS Server 4 User Device 11 Information Gathering Department 12 Teacher data setting section 13 Learning control section 14 Training Data Expansion Section 15 Judgment control section 16 Information presentation section
Claims
1. An information processing device that generates a trained model by machine learning using teacher data including labeled input data, and executes information processing using the trained model, Obtaining first teacher data including first input data to which a first label is assigned; creating second teacher data by changing the first input data included in the first teacher data to obtain second input data, and assigning the first label or a second label different from the first label to the second input data according to a degree of change from the first input data to the second input data; An information processing device comprising a control unit that executes the above.
2. the information processing is a determination process for determining a degree to which an image posted so as to be viewable on a public network is suitable for advertising; the first input data is an image to be determined by the determination process, The information processing device according to claim 1 , wherein the first label and the second label are information indicating a degree to which the image is suitable for the advertisement.
3. The information processing device of claim 2, wherein the control unit calculates an expected remuneration for the image or value information as an advertisement based on at least one of the degree to which the image is suitable for the advertisement as determined by the determination process, the response of viewers to the image on the public network, and the trading history of images whose suitability for the advertisement has already been determined.
4. The information processing device according to claim 3 , wherein the control unit presents the expected reward or value information to a poster identified by a device that posted the image or identification information indicating the posting source when the image is posted on the public network.
5. The information processing device according to claim 2 , wherein the control unit prioritizes posters of images highly evaluated in the determination process and transmits to the posters of the images an inquiry for obtaining permission to use the images as advertisements.
6. The control unit sets an evaluation value of the posted image based on the expected remuneration or value information, the suitability for the advertisement, or a transaction record price, and identifies a location where the image having the evaluation value equal to or above a standard was taken; The information processing device according to claim 3 , further comprising: transmitting information to a device located near the specified location, the information encouraging the posting of an image viewable on the public network.
7. The information processing device according to claim 6, wherein the control unit selects one of the locations where the image was taken based on shooting conditions including the season, date and time when the image was taken, and past weather data for the season and date and time, and transmits information to a device located near the selected location encouraging the posting of an image that can be viewed on the public network.
8. The control unit is Obtaining terms including a theme or keyword related to an advertisement; collecting images that satisfy the condition from among the images; The information processing device according to claim 3 , further comprising: a calculation of a cost for providing an image that satisfies the condition based on the expected remuneration or value information; and recommending the image together with the cost to an advertisement creator.
9. Ad creator attribute information or log information from when the ad creator searched for images in the past To obtain determining an advertisement creator that matches the posted image by acquiring an image that matches the attribute information or the log information of the advertisement creator from among images that are posted and viewable on the public network; Obtaining the photographer of the posted image; 3. The information processing device according to claim 2, further comprising: presenting the published image and the photographer of the published image to the advertisement creator; and presenting a suitable advertisement creator to the photographer of the published image.
10. Creating a trained model trained using at least the first teacher data; Accepting an evaluation of a result of determining the first input data by the trained model; The information processing device according to claim 1 , further comprising: creating updated first teacher data by resetting the updated first label to the first input data based on the received evaluation.
11. Receiving natural language included in the first teacher data, the natural language indicating a direction for changing the first input data; changing the first input data included in the first teacher data according to the natural language to create the changed first input data; receiving an evaluation of the changed first input data; Further execute The information processing apparatus according to claim 10 , wherein creating the updated first teacher data includes setting the first label based on an evaluation of the changed first input data.
12. In an information system that generates a trained model by machine learning using teacher data including labeled input data, and executes information processing using the trained model, the computer comprising: Obtaining first teacher data including first input data to which a first label is assigned; creating second teacher data by changing the first input data included in the first teacher data to obtain second input data, and assigning the first label or a second label different from the first label to the second input data according to a degree of change from the first input data to the second input data; An information processing method for performing the above.
13. In an information system for generating a trained model by machine learning using teacher data including labeled input data, and for executing information processing using the trained model, the information system includes: Obtaining first teacher data including first input data to which a first label is assigned; creating second teacher data by changing the first input data included in the first teacher data to obtain second input data, and assigning the first label or a second label different from the first label to the second input data according to a degree of change from the first input data to the second input data; A program for executing.
Citation Information
Patent Citations
Calculation device, calculation method, and calculation program
JP2019149171A
Program, device and method for selecting items based on compensated effects, and item effect estimation program
JP2021068238A
Machine learning method, machine learning system, and non-transitory computer-readable storage medium
JP2021193564A
Data augmentation method, learning device, and program
WO2022097709A1
