Information processing device, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program that can improve a recognition accuracy.SOLUTION: An information processing apparatus according to the present application includes a collection unit and a granting unit. The collection unit collects a pair of a posted image posted by a user and an image explanation related to the posted image. The granting unit grants a label related to a landmark to a posted image when a result of comparing local feature amounts of a reference image in which the landmark related to the image explanation is captured and of the posted image satisfies a predetermined condition.SELECTED DRAWING: Figure 3

Description

【Technical Field】 【0001】 The present invention relates to an information processing apparatus, an information processing method, and an information processing program. 【Background Art】 【0002】 Conventionally, a technique for recognizing a subject shown in an image input from a user has been proposed (see, for example, Patent Document 1). 【Prior Art Document】 【Patent Document】 【0003】 【Patent Document 1】 Japanese Unexamined Patent Application Publication No. 2021-149953 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0004】 However, in the prior art, there is room for improvement, for example, in accurately recognizing a relatively large subject such as a landmark. 【0005】 The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of improving recognition accuracy. 【Means for Solving the Problems】 【0006】 The information processing apparatus according to the present application includes a collection unit and an assignment unit. The collection unit collects a set of a posted image posted by a user and an image description related to the posted image. The assignment unit assigns a label related to the landmark to the posted image for which a comparison result between local feature amounts of a reference image in which a landmark related to the image description is photographed and the posted image satisfies a predetermined condition. 【Effects of the Invention】 【0007】 According to one embodiment, the recognition accuracy can be improved. [Brief explanation of the drawing] 【0008】 [Figure 1A] Figure 1A shows the first process performed by the information processing device according to this embodiment. [Figure 1B] Figure 1B shows the second process performed by the information processing device according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 shows an example of an image database. [Figure 5] Figure 5 is a flowchart showing the processing procedure of process 1 executed by the information processing device according to the embodiment. [Figure 6] Figure 6 is a flowchart showing the processing procedure for processing step 2 executed by the information processing device according to the embodiment. [Figure 7] Figure 7 shows an example of a hardware configuration. [Modes for carrying out the invention] 【0009】 The following describes in detail, with reference to the drawings, embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted. 【0010】 (Embodiment) First, the processes performed by the information processing device according to the embodiment will be described using Figures 1A and 1B. Figures 1A and 1B are diagrams showing processes 1 and 2 performed by the information processing device according to the embodiment. Figures 1A and 1B show an example of the operation of the information processing system S, which includes the information processing device 1 according to the embodiment. 【0011】 As shown in Figures 1A and 1B, the information processing system S according to this embodiment includes an information processing device 1 and a user terminal 100. 【0012】 First, let's explain the first process shown in Figure 1A. In the first process shown in Figure 1A, the information processing system S according to the embodiment collects a set of a posted image and an image description submitted by a user, and assigns a label related to the landmark to posted images for which the comparison result of the local features of the reference image in which the landmark related to the image description was captured and the posted image satisfies predetermined conditions. 【0013】 Specifically, the information processing device 1 first collects a pair of a posted image and an image description from the user via the user terminal 100 (step S1). Figure 1A shows an example of collecting a pair of a posted image of Lake XX with a relatively well-known mountain XX in the distance, and an image description set by the user, "Lake XX". 【0014】 In this case, if image recognition is performed on the entire image of such a posted image, there is a risk of misidentification as "Mount XX". As a result, if such posted images are labeled as "Mount XX" and used as training data for machine learning, the accuracy of the learning model may decrease. 【0015】 Therefore, this disclosure improves recognition accuracy by using local features of the submitted image. Specifically, the information processing device 1 compares the local features of the submitted image with a reference image of the landmark associated with the image description (step S2). The reference image is an image to which labels related to the landmark have been assigned, and is registered in the image database 41 (see Figure 3), which will be described later. 【0016】 Specifically, for each of the posted image and the reference image, the information processing apparatus 1 detects feature points with large density changes in the image by a known feature point detection method, and calculates local feature amounts obtained by vectorizing the region around the feature points using pixel values or differential values. That is, the information processing apparatus 1 compares partial regions (i.e., landmark regions) having feature amounts in the posted image and the reference image with each other. 【0017】 Subsequently, when the comparison result satisfies a predetermined condition, the information processing apparatus 1 assigns a label related to the landmark to the posted image (step S3). Specifically, when the degree of coincidence of the local feature amounts is equal to or greater than a predetermined threshold value, the information processing apparatus 1 determines that the predetermined condition is satisfied and assigns a label related to the landmark. Further, when a plurality of local feature amounts are obtained from each of the posted image and the reference image, the information processing apparatus 1 may determine whether the degree of coincidence of the positional relationship of each local feature amount is equal to or greater than a predetermined threshold value. 【0018】 Then, the information processing apparatus 1 stores the posted image with the assigned label in the image DB 41 as a reference image (step S4). 【0019】 As described above, according to the information processing apparatus 1 according to the embodiment, the recognition accuracy can be improved by performing image recognition by comparing the local feature amounts of the posted image and the reference image. 【0020】 Next, processing part 2 will be described with reference to FIG. 1B. In processing part 2 shown in FIG. 1B, for example, when the user wants to know the identity of a subject captured, by inputting a query image of such a subject, information about the subject is provided. 【0021】 Specifically, the information processing apparatus 1 first receives a query image from the user via the user terminal 100 (step S11). The query image may be an image captured by the user using the camera function of the user terminal 100, or may be an image downloaded from the Internet or the like. 【0022】 Next, the information processing device 1 generates a query vector based on the features of the query image (step S12). The query vector is generated, for example, by the output of a model generated by machine learning. Such a model is trained, for example, to output similar vectors from images of the same subject. The information processing device 1 generates the query vector using the model and also inputs each image included in the image DB 41 into the model to generate vectors for it as well. 【0023】 Then, the information processing device 1 extracts images with vectors similar to the query vector from the image DB 41 as similar images (step S13). For example, the information processing device 1 extracts images with vectors whose degree of agreement with the query vector is above a predetermined threshold as similar images. 【0024】 Next, the information processing device 1 determines whether the query image and the similar image contain the same subject based on the comparison of local features between the query image and the similar image (step S14). Specifically, the information processing device 1 determines that the query image and the similar image contain the same subject if the degree of agreement of the local features is above a predetermined threshold. In other words, it determines that the landmark of the label assigned to the similar image is included in the query image. 【0025】 Then, if the information processing device 1 determines that the query image and similar images contain the same subject, it provides the user with information about the similar images (step S15). For example, the information processing device 1 provides the user with similar images. The information processing device 1 may also provide information about the landmarks of the labels assigned to the similar images. 【0026】 Thus, according to the information processing device 1 of this embodiment, the recognition accuracy can be improved by performing image recognition by comparing local features between similar images and query images. 【0027】 Next, an example of the configuration of the information processing system S according to the embodiment will be described using Figure 2. Figure 2 is a block diagram showing an example of the configuration of the information processing system S according to the embodiment. As shown in Figure 2, in the information processing system S according to the embodiment, the information processing device 1 and a plurality of user terminals 100 are connected to a network N by wire or wireless. The network N is, for example, a network such as the Internet, a WAN (Wide Area Network), or a LAN (Local Area Network). 【0028】 The information processing device 1 is a server device that executes the information processing method according to the embodiment. The information processing device 1 collects pairs of posted images and image descriptions posted by users, and assigns labels related to landmarks to posted images that satisfy predetermined conditions when the comparison result of the local features of the reference image in which the landmark related to the image description was captured and the posted image is found. The information processing device 1 also determines whether the same subject is included in a query image received from a user based on the comparison result of the local features of the query image and similar images, and provides information about similar images based on the determination result. 【0029】 Furthermore, the information processing device 1 is an information processing device that interacts with multiple user terminals 100 and provides various API (Application Programming Interface) services for various applications (hereinafter referred to as "apps") and various data to the multiple user terminals 100, and is realized by a server device or a cloud system. 【0030】 Furthermore, the information processing device 1 may be an information processing device that provides some kind of web service online to multiple user terminals 100. For example, the information processing device 1 may provide services such as internet connection, search services, SNS (Social Networking Service), e-commerce (EC), electronic payment, online games, online banking, online trading, accommodation / ticket reservations, video / music distribution, news, maps, route search, route guidance, route information, service information, and weather forecasts as web services. In practice, the information processing device 1 may cooperate with various servers that provide the above-mentioned web services and act as an intermediary for web services, or it may be responsible for processing web services. 【0031】 The user terminal 100 is a terminal device owned by the user. The user terminal 100 can be any type of terminal device, such as a smartphone, desktop PC, notebook PC, or tablet PC. The user terminal 100 transmits various types of information to the information processing device 1, etc., and receives information provided by the information processing device 1, etc. 【0032】 Next, with reference to Figure 3, an example configuration of the information processing device 1 will be described. 【0033】 Figure 3 is a diagram showing an example configuration of an information processing device 1 according to an embodiment. As shown in Figure 3, the information processing device 1 has a communication unit 2, a control unit 3, and a storage unit 4. The control unit 3 includes a collection unit 31, an assignment unit 32, a reception unit 33, a generation unit 34, an extraction unit 35, a determination unit 36, and a provision unit 37. The storage unit 4 stores an image DB 41. 【0034】 The communication unit 2 is implemented, for example, by a NIC (Network Interface Card). The communication unit 2 is connected to the network by wire or wireless connection. 【0035】 The control unit 3 is a controller and is implemented by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 1, using RAM or the like as a working area. Alternatively, the control unit 3 is a controller and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or GPGPU (General Purpose Graphic Processing Unit). 【0036】 The memory unit 4 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. 【0037】 Image DB41 is a database containing images of predetermined subjects, such as landmarks. Image DB41 is updated by the control unit 3. 【0038】 Figure 4 shows an example of the image database 41. As shown in Figure 4, the image database 41 includes items such as "image ID," "landmark," "vector," and "image." 【0039】 "Image ID" is identification information that identifies the image. "Landmark" is information about landmarks that appear in the image. "Vector" is vector information generated by the generation unit 34, which will be described later. "Image" is information about the image. 【0040】 Next, we will describe the functions of the control unit 3 of the information processing device 1 (collection unit 31, assignment unit 32, reception unit 33, generation unit 34, extraction unit 35, determination unit 36, and provision unit 37). 【0041】 The collection unit 31 collects pairs of posted images and image descriptions. Specifically, the collection unit 31 collects pairs of posted images and image descriptions entered by the user. The collection unit 31 may also collect images and content posted by users to other services such as SNS (Social Networking Service) as pairs of posted images and image descriptions. 【0042】 The labeling unit 32 assigns a landmark label to submitted images whose local feature comparison result with a reference image in which the image description and associated landmarks are captured satisfies predetermined conditions. Specifically, the labeling unit 32 selects images as reference images in which the degree of agreement between the image description text and the landmark item in the image DB 41 is above a threshold. The labeling unit 32 may also select images of landmarks similar to the selected reference image as reference images. Similar reference images include, for example, landmarks that have a similar shape to the landmark selected as the reference image, or landmarks located within a predetermined range from the location of the landmark selected as the reference image. 【0043】 The labeling unit 32 determines that a predetermined condition is met when the degree of agreement of the local features is equal to or greater than a predetermined threshold, and assigns a label to the landmark. Furthermore, if multiple local features are obtained from both the submitted image and the reference image, the labeling unit 32 may determine whether the degree of agreement of the positional relationship of each local feature is equal to or greater than a predetermined threshold, and if it determines that it is equal to or greater than the threshold, it may assign a label to the landmark. 【0044】 The labeling unit 32 stores the submitted image to which a label has been assigned as the reference image in the image database 41. 【0045】 The reception unit 33 receives a query image from the user, which serves as the search query. The query image may be an image taken by the user using the camera function of the user terminal 100, or an image downloaded from the internet or elsewhere. 【0046】 The generation unit 34 generates a query vector based on the features of the received query image. The query vector is generated, for example, by the output of a model generated by machine learning. Such a model is trained, for example, to output similar vectors from images of the same subject. The training data for the model uses, for example, the pixel value (luminance and color information) of each pixel, the difference in pixel information of surrounding pixels, etc. The generation unit 34 generates the query vector using the model and also inputs each image included in the image DB 41 into the model to generate vectors. 【0047】 The extraction unit 35 extracts images with vectors similar to the query vector from the image DB 41 as similar images. Specifically, the extraction unit 35 extracts images with vectors whose degree of agreement with the query vector is above a predetermined threshold as similar images. 【0048】 The determination unit 36 ​​determines whether the query image and the similar image contain the same subject (label landmark) based on the comparison results of local features extracted from each of the query image and the similar image (posted image to which a label has been assigned). Specifically, the determination unit 36 ​​determines that the query image and the similar image contain the same subject if the degree of agreement of the local features is above a predetermined threshold. In other words, it determines that the landmark of the label assigned to the similar image is included in the query image. Furthermore, if multiple local features are obtained from each of the query image and the similar image, the determination unit 36 ​​determines whether the degree of agreement of the positional relationship of each local feature is above a predetermined threshold, and determines that the same subject is included if it is above the threshold. 【0049】 The providing unit 37 provides information about similar images that the determination unit 36 ​​has determined to contain the same subject. For example, the providing unit 37 provides similar images to the user. The providing unit 37 may also provide information about the landmarks of the labels assigned to the similar images. 【0050】 Next, the processing steps of the processes executed by the information processing device 1 according to the embodiment will be described using Figures 5 and 6. Figures 5 and 6 are flowcharts showing the processing steps of processes 1 and 2 executed by the information processing device 1 according to the embodiment. 【0051】 First, the processing procedure for process 1 will be explained using Figure 5. As shown in Figure 5, the control unit 3 first collects a pair of posted images and image descriptions from the user (step S101). 【0052】 Next, the control unit 3 compares the local features of the reference image of the landmarks related to the image description with those of the submitted image (step S102). 【0053】 Next, the control unit 3 assigns a landmark label to the posted image if the comparison result satisfies predetermined conditions (step S103). 【0054】 Next, the control unit 3 registers the submitted image with the label assigned as the reference image in the image DB 41 (step S104), and then terminates the process. 【0055】 Next, the processing procedure for process 2 will be explained using Figure 6. As shown in Figure 6, the control unit 3 receives a query image from the user (step S201). 【0056】 Next, the control unit 3 generates a query vector based on the query image (step S202). 【0057】 Next, the control unit 3 extracts similar images from the image DB 41 based on the query vector (step S203). 【0058】 Next, the control unit 3 determines whether the similar image and the query image contain the same subject (step S204). 【0059】 Next, the control unit 3 provides information about similar images that have been determined to contain the same subject (step S205), and then terminates the process. 【0060】 〔others〕 Furthermore, some of the processes described as being performed automatically in the above embodiments can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above documents and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown. 【0061】 Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. 【0062】 For example, some or all of the storage unit 4 shown in Figure 3 may be stored in a storage server or the like, rather than being held by each device. In this case, each device obtains various information by accessing the storage server. 【0063】 [Hardware configuration] Furthermore, the information processing device 1 according to the above embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 7. Figure 7 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090. 【0064】 The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and is implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, etc. 【0065】 Output IF1060 is an interface for transmitting information to be output to output devices 1010, which output various types of information such as monitors and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). Input IF1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, and scanners. It is implemented using, for example, USB. 【0066】 The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductor memory. Furthermore, the input device 1020 may also be an external storage medium such as a USB memory stick. 【0067】 Network IF1080 receives data from other devices via network N and sends it to the arithmetic unit 1030, and also transmits data generated by the arithmetic unit 1030 to other devices via network N. 【0068】 The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program. 【0069】 For example, when computer 1000 functions as information processing device 1, the arithmetic unit 1030 of computer 1000 realizes the functions of control unit 3 by executing a program loaded on primary storage device 1040. 【0070】 〔effect〕 As described above, the information processing device 1 according to the embodiment comprises a generation unit 34, an extraction unit 35, a determination unit 36, and a provision unit 37. The generation unit 34 generates a query vector based on the feature quantities of the query image which will be the search query. The extraction unit 35 extracts images with vectors similar to the query vector from a pre-created image database 41 as similar images. The determination unit 36 ​​determines whether the query image and the similar images contain the same subject based on the comparison results of the local feature quantities extracted from the query image and the similar images, respectively. The provision unit 37 provides information about the similar images which have been determined to contain the same subject. 【0071】 According to the information processing device 1 of each embodiment described above, recognition accuracy can be improved. 【0072】 Furthermore, the information processing device 1 according to this embodiment includes a collection unit 31 and an assignment unit 32. The collection unit 31 collects pairs of posted images submitted by users and image descriptions related to those posted images. The assignment unit 32 assigns labels related to landmarks to posted images for which the comparison result of the local features of a reference image in which the landmark associated with the image description is captured and the posted image satisfies predetermined conditions. 【0073】 According to the information processing device 1 of each embodiment described above, recognition accuracy can be improved. 【0074】 Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention. 【0075】 〔others〕 Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown. 【0076】 Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. 【0077】 Furthermore, each of the processes described in the embodiments above can be combined as appropriate, provided that the processing content does not contradict each other. 【0078】 Furthermore, the terms "section, module, unit" used above can be replaced with "means" or "circuit," etc. For example, control unit 3 can be replaced with control means or control circuit. [Explanation of symbols] 【0079】 1. Information Processing Device 2 Communications Department 3. Control Unit 4 Storage section 31 Collection Department 32 Granting section 33 Reception Department 34 Generation part 35 Extraction part 36 Judgment section 37 Providing Department 41 Image Database 100 user terminals S Information Processing System

Claims

[Claim 1] A collection unit that collects pairs of user-submitted images and image descriptions related to those images, A labeling unit assigns a label to a submitted image whose local features satisfy predetermined conditions when a comparison result between a reference image in which a landmark related to the image description is captured and the submitted image satisfies the submitted image. Equipped with, The aforementioned attachment unit, An image in which the degree of agreement between the text of the image description and the information of the landmark registered in the image database is equal to or greater than a threshold is selected as the reference image. Information processing device. [Claim 2] The aforementioned attachment unit, An image of the landmark that is similar to the selected reference image is selected as the reference image. The information processing apparatus according to claim 1. [Claim 3] The aforementioned attachment unit, If the degree of agreement of the local features is equal to or greater than a predetermined threshold, it is determined that the predetermined conditions are met, and a label is assigned to the landmark. The information processing apparatus according to claim 1. [Claim 4] The aforementioned attachment unit, If multiple local features are obtained from the submitted image and the reference image, it is determined whether the degree of agreement in the positional relationship of each local feature is equal to or greater than a predetermined threshold, and if it is determined to be equal to or greater than the threshold, a label relating to the landmark is assigned. The information processing apparatus according to claim 1. [Claim 5] A reception unit that accepts query images that will serve as search queries, A determination unit determines whether the same landmark is included in the query image and the posted image based on the comparison result of the local features of the query image and the posted image to which the label has been assigned. The information processing apparatus according to claim 1, further comprising: [Claim 6] A method of information processing performed by a computer, A collection process that collects pairs of user-submitted images and image descriptions related to those images, A labeling step of assigning a label to a posted image in which a reference image in which a landmark related to the image description is captured meets predetermined conditions, by comparing the local features of the reference image with the posted image, Includes, The aforementioned application step is, An image in which the degree of agreement between the text of the image description and the information of the landmark registered in the image database is equal to or greater than a threshold is selected as the reference image. Information processing methods. [Claim 7] A collection procedure for collecting pairs of user-submitted images and their corresponding image descriptions, A labeling procedure for a submitted image in which a reference image in which a landmark related to the image description is captured meets predetermined conditions for the result of comparing the local features of the submitted image with the reference image, and for the submitted image in which the comparison result of the reference image meets predetermined conditions, Have the computer run it, The aforementioned granting procedure is, An image in which the degree of agreement between the text of the image description and the information of the landmark registered in the image database is equal to or greater than a threshold is selected as the reference image. Information processing program.

Images