Search system, search method, and computer program

Abstract

A search system (10) comprises: a wording generation unit (110) that generates wording corresponding to an object included in an image by using a trained model; an information assignment unit (120) that assigns the image with the wording corresponding to the object as adjective information for the object; a query acquisition unit (130) that acquires a search query; and a search unit (140) that searches for an image corresponding to the search query from among a plurality of images on the basis of the search query and the adjective information. According to the search system, it is possible to implement a search that utilizes various characteristics relating to an object in an image.

Description

[Technical field]

[0001] The present invention relates to the technical fields of a search system, a search method, and a computer program for searching, for example, images. [Background technology]

[0002] As this type of system, a system that searches for a desired image from among multiple images is known. For example, Patent Document 1 discloses a technique for extracting matching images after comparing the score of an evaluation expression of an image with a predetermined threshold value. Patent Document 2 discloses a technique for extracting feature words and searching for description information of an image. Patent Document 3 discloses a technique for searching for an image using the feature amount of the image and the adjective pair evaluation value.

[0003] As other related techniques, Patent Document 4 discloses a technique for extracting features for each word string by performing sequence processing on acquired text. Patent Document 5 discloses a technique for classifying sets of image features and text features into multiple classes. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-151588 A [Patent Document 2] Special Publication No. 2019-536122 [Patent Document 3] JP 2016-218708 A [Patent Document 4] JP 2020-157168 A [Patent Document 5] JP 2015-041225 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to search for images, information indicating the state or appearance of objects contained in the images is sometimes added. However, it may not be easy to add appropriate information by analyzing the images, for example.

[0006] The present invention has been made in consideration of the above problems, and has an objective of providing a search system, a search method, and a computer program that are capable of performing searches using various properties of objects in an image. [Means for solving the problem]

[0007] One aspect of the search system of the present invention includes a sentence generation unit that generates sentences corresponding to objects included in an image using a trained model, an information assignment unit that assigns the sentences corresponding to the objects to the image as adjective information of the objects, a query acquisition unit that acquires a search query, and a search unit that searches for an image corresponding to the search query from among a plurality of images based on the search query and the adjective information.

[0008] One aspect of the search method of the present invention generates sentences corresponding to objects contained in an image using a trained model, assigns the sentences corresponding to the objects to the image as adjective information for the objects, obtains a search query, and searches for images corresponding to the search query from among a plurality of images based on the search query and the adjective information.

[0009] One aspect of the computer program of the present invention operates a computer to generate sentences corresponding to objects contained in an image using a trained model, assign the sentences corresponding to the objects to the image as adjective information for the objects, obtain a search query, and search for an image corresponding to the search query from among a plurality of images based on the search query and the adjective information. Effect of the Invention

[0010] According to one aspect of each of the above-described search system, search method, and computer program, it is possible to realize a search that utilizes various properties related to objects in an image. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a hardware configuration of a search system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a functional configuration of a search system according to a first embodiment. [Diagram 3] 5 is a flowchart showing a flow of an information adding operation of the search system according to the first embodiment. [Figure 4] 3 is a diagram showing an example of a set of images and texts used for training of a sentence generation unit according to the first embodiment. FIG. [Diagram 5] 4 is a flowchart showing the flow of a search operation of the search system according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing the functional configuration of a search system according to a second embodiment. [Figure 7] 13 is a flowchart showing the flow of an information adding operation of the search system according to the second embodiment. [Figure 8] FIG. 11 is a conceptual diagram showing a specific operation of the sentence generation unit according to the second embodiment. [Figure 9] FIG. 13 is a block diagram showing the functional configuration of a search system according to a third embodiment. [Figure 10] 13 is a flowchart showing the flow of a search operation of the search system according to the third embodiment. [Figure 11] FIG. 13 is a block diagram showing the functional configuration of a search system according to a fourth embodiment. [Figure 12] 13 is a flowchart showing a flow of an information adding operation of the search system according to the fourth embodiment. [Figure 13] FIG. 13 is a conceptual diagram showing a specific operation of an object detection unit according to the fourth embodiment. [Figure 14]FIG. 13 is a block diagram showing a functional configuration of an information imparting system according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of a search system, a search method, and a computer program will be described with reference to the drawings.

[0013] First Embodiment The search system according to the first embodiment will be described with reference to FIGS.

[0014] (Hardware configuration) First, the hardware configuration of the search system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the search system according to the first embodiment.

[0015] 1, a search system 10 according to the first embodiment includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The search system 10 may further include an input device 15 and an output device 16. The processor 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16 are connected via a data bus 17.

[0016] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may obtain (i.e., read) a computer program from a device (not shown) disposed outside the search system 10 via a network interface. The processor 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the computer program that the processor 11 reads. In particular, in this embodiment, when the processor 11 executes the computer program that the processor 11 reads, functional blocks are realized in the processor 11 for executing a process of generating a sentence from an image and adding adjective information, and a process of searching for an image using the adjective information. Examples of the processor 11 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit). The processor 11 may be one of the above examples, or a plurality of processors may be used in parallel.

[0017] The RAM 12 temporarily stores a computer program executed by the processor 11. The RAM 12 temporarily stores data that is temporarily used by the processor 11 when the processor 11 is executing the computer program. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0018] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a programmable ROM (P-ROM).

[0019] The storage device 14 stores data that the search system 10 stores long-term. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0020] The input device 15 is a device that receives input instructions from a user of the search system 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel. The input device 15 may be a dedicated controller (operation terminal). The input device 15 may also include a terminal owned by a user (for example, a smartphone, a tablet terminal, etc.). The input device 15 may be, for example, a device that includes a microphone and is capable of voice input.

[0021] The output device 16 is a device that outputs information related to the search system 10 to the outside. For example, the output device 16 may be a display device (e.g., a display) that can display information related to the search system 10. The display device here may be a television monitor, a personal computer monitor, a smartphone monitor, a tablet terminal monitor, or a monitor of another mobile terminal. The display device may also be a large monitor or digital signage installed in various facilities such as a store. The output device 16 may also be a device that outputs information in a format other than an image. For example, the output device 16 may be a speaker that outputs information related to the search system 10 by voice.

[0022] (Functional configuration) Next, the functional configuration of the search system 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the search system according to the first embodiment.

[0023] As shown in FIG. 2, the search system 10 according to the first embodiment includes a sentence generation unit 110, an information assignment unit 120, a query acquisition unit 130, and a search unit 140 as processing blocks for realizing the functions. Each of the sentence generation unit 110, the information assignment unit 120, the query acquisition unit 130, and the search unit 140 may be realized, for example, by the above-mentioned processor 11 (see FIG. 1). The search system 10 is configured to be able to read and rewrite a plurality of images stored in the image storage unit 50 as appropriate. Note that, although the image storage unit 50 is an external device of the search system 10 here, the image storage unit 50 may be provided within the search system 10. In this case, the image storage unit 50 may be realized, for example, by the above-mentioned storage device 14 (see FIG. 1).

[0024] The sentence generating unit 110 is configured to be able to generate sentences corresponding to objects included in an image using a trained model. The "sentences corresponding to objects" here are sentences that indicate what kind of object the object included in the image is, and include adjectival information (for example, general adjectives as well as words that describe objects). The sentences generated by the sentence generating unit 110 may be multiple. The amount of sentences generated by the sentence generating unit 110 may be set in advance by a system administrator or a user, or may be appropriately determined based on the analysis results of the image. The trained model that generates sentences will be described in detail in another embodiment described later. In the following example, the sentences corresponding to objects generated by the sentence generating unit 110 will be described using Japanese sentences as an example. The sentences corresponding to objects generated by the sentence generating unit 110 are configured to be output to the information adding unit 120.

[0025] The information adding unit 120 is configured to add a sentence corresponding to an object generated by the sentence generating unit 110 to an image as adjective information. More specifically, the information adding unit 120 links an object included in an image with a sentence corresponding to the object and stores the object in the image storage unit 50. The "adjective information" here is information that represents the state or appearance of an object. For example, if the object included in the image is a "food", the adjective information may include information that represents the taste (sweetness, spiciness, saltiness, etc.), smell, temperature (hotness, coldness), etc. of the food. Alternatively, if the object included in the image is an "item (e.g., a product sold on a shopping site or in a store)", the adjective information may include information that represents the texture, touch, etc. of the item. The adjective information may also include information that represents the degree of the above information (i.e., information that represents the state or appearance of an object). For example, the adjective information that represents the spiciness of a dish may be not only "spicy", but also "very spicy", "slightly spicy", "mild spicy", etc. The adjective information may also be information including multiple adjectives, such as "slightly spicy but rich in flavor." The adjective information may furthermore be information including subtle nuances based on personal sensibility, rather than being a uniform expression. The adjective information may also be subjective information rather than objective information (for example, information including personal impressions of the person who captured the image or the person who viewed the image). Note that the adjective information described above is merely an example, and expressions other than these may also be included in the adjective information.

[0026] The query acquisition unit 130 is configured to be able to acquire a search query input by a user who is searching for an image. The query acquisition unit 130 acquires a search query input using, for example, the input device 15 (see FIG. 1 ). The search query here may be in a natural language. For example, the search query may include multiple words, such as "the rich ramen I ate in Tokyo two years ago" or "the super spicy curry I ate in Sapporo in October." The search query acquired by the query acquisition unit 130 is configured to be output to the search unit 140.

[0027] The search unit 140 is configured to be able to search for an image corresponding to the search query from among the multiple images stored in the image storage unit 50 based on the search query acquired by the query acquisition unit 130 and the adjective information assigned to the image by the information assignment unit 120 (for example, by comparing the search query with the adjective information). The search unit 140 may have a function of outputting an image corresponding to the search query as a search result. In this case, the search unit 140 may output the search result using the above-mentioned output device 16. In addition, the search unit 140 may output one image that best matches the search query, or may output multiple images that match the search query. A specific search method by the search unit 140 will be described in detail in another embodiment described later.

[0028] (Information addition operation) Next, an operation of adding adjective information by the search system 10 according to the first embodiment (hereinafter, appropriately referred to as an "information adding operation") will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of the information adding operation of the search system according to the first embodiment.

[0029] As shown in FIG. 3, when the information adding operation by the search system 10 according to the first embodiment is started, the search system 10 first acquires an image from the image storage unit 50 (step S101). The image acquired here is an image, among the multiple images stored in the image storage unit 50, to which no adjective information has yet been added (for example, to which the information adding operation has not yet been executed). The image may be acquired from a source other than the image storage unit 50. For example, the image may be automatically acquired from the Internet (for example, a shopping site, a review site, etc.). Alternatively, the image may be directly input to the search system 10 by a system administrator, a user, etc.

[0030] Next, the sentence generation unit 110 uses the acquired image to generate a sentence corresponding to an object included in the image (step S102). Then, the information attachment unit 120 attaches the sentence generated by the sentence generation unit 110 to the image as adjective information (step S103).

[0031] The above-mentioned series of processes may be executed consecutively for each of the multiple images. That is, after executing a process of generating a sentence for a first image and assigning the sentence as adjective information, a process of generating a sentence for a second image and assigning the sentence as adjective information may be executed. The information assignment operation may be executed for all images stored in the image storage unit 50 by repeatedly executing the information assignment operation in this manner.

[0032] (Learning data) Next, learning data (i.e., training data) used for learning of the sentence generation unit 110 will be specifically described with reference to Fig. 4. Fig. 4 is a diagram showing an example of a set of images and texts used for learning of the sentence generation unit according to the first embodiment.

[0033] In order to execute the above-mentioned information adding operation (see FIG. 3), the sentence generating unit 110 has a trained model for generating sentences from images. This trained model is configured by, for example, a neural network, and is machine-learned using training data before the information adding operation is started.

[0034] As shown in Fig. 4, the trained model may use a set of images and sentences (i.e., text data) corresponding to objects contained in the images as training data. In the example shown in the figure, images of ramen and curry are set with text data including impressions of eating the ramen and curry. By using such training data, it is possible to generate a model that generates sentences including adjectival information about a dish when an image including the dish is input.

[0035] The above training data is an example, and images including objects other than food may be used as the training data. Furthermore, text data including sentences explaining the state of an object may be used as the training data, instead of text data including impressions about the object. In other words, the type of training data is not particularly limited as long as it is a set of an image including some object and text data including sentences corresponding to the object.

[0036] (Search behavior) Next, an operation of searching for an image by the search system 10 according to the first embodiment (hereinafter, appropriately referred to as a "search operation") will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of the search operation of the search system according to the first embodiment.

[0037] 5, when a search operation by the search system 10 according to the first embodiment is started, the query acquisition unit 130 first acquires a search query (step S201). The acquired search query is output to the search unit 140.

[0038] Next, the search unit 140 compares the search query acquired by the query acquisition unit 130 with the adjective information assigned to the image (step S202). Then, the search unit 140 outputs the image corresponding to the search query as a search result (step S203). Note that the search unit 140 is not limited to comparing the search query with the adjective information, and may output the search result based on the search query and the adjective information.

[0039] In addition to the adjective information, the search unit 140 may perform a search using other information related to an image or an object. Specifically, the search unit 140 may perform a search using at least one of time information indicating the time when the image was captured, location information indicating the location where the image was captured, and name information indicating the name of the object. In this case, the time information may be obtained from a time stamp of the image. The location information may be obtained from a Global Positioning System (GPS). The name information may be obtained from object detection information from the image (which will be described in detail in other embodiments later).

[0040] Furthermore, the search target of the search unit 140 may be a plurality of images included in the video data (i.e., images of each frame of the video data). In this case, an image corresponding to the search query may be output as a search result, or video data including an image corresponding to the search query may be output as a search result.

[0041] (Technical effect) Next, technical effects obtained by the search system 10 according to the first embodiment will be described.

[0042] As described in Fig. 1 to Fig. 5, in the search system 10 according to the first embodiment, sentences corresponding to objects included in an image are automatically generated and assigned as adjective information. Then, the adjective information is used to search for images. In this way, it is possible to appropriately search for an image desired by a user using the adjective information assigned as a sentence.

[0043] If the adjective information is registered in the dictionary in advance, a search can be performed using the adjective information without generating a sentence as in this embodiment, but it is difficult to register adjective information that cannot be expressed in a single expression (for example, "It is spicy, but has the sweetness of vegetables") in the dictionary one by one. However, according to the search system 10 of this embodiment, an automatically generated sentence is added as the adjective information, so that an image search can be performed using the adjective information that cannot be expressed in a single expression.

[0044] Furthermore, according to the search system 10 of this embodiment, it is possible to use, as adjective information, information containing subtle nuances based on personal sensibilities, or unique information experienced by an individual on the spot, rather than uniform adjective information. Although it is possible to have the user record such information, recording such information each time is a very time-consuming task for the user. However, according to the search system 10 of this embodiment, sentences are automatically generated by a trained model, so there is no increase in the user's time and effort.

[0045] <Second embodiment> A search system 10 according to the second embodiment will be described with reference to Fig. 6 to Fig. 8. The second embodiment differs from the first embodiment in only some of its configurations and operations, and the other parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit appropriate descriptions of other overlapping parts.

[0046] (Functional configuration) First, the functional configuration of the search system 10 according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the functional configuration of the search system according to the second embodiment. In Fig. 6, the same elements as those shown in Fig. 2 are denoted by the same reference numerals.

[0047] 6, the search system 10 according to the second embodiment includes, as processing blocks for realizing its functions, a sentence generation unit 110, an information assignment unit 120, a query acquisition unit 130, and a search unit 140. In particular, the sentence generation unit 110 according to the second embodiment is configured to include two models, an extraction model 111 and a generation model 112, as trained models.

[0048] The extraction model 111 is configured to be able to extract features of an object included in an input image from the image. The features here indicate the features of the object and can be used when generating a sentence corresponding to the object. The extraction model 111 may be configured as a CNN (Convolutional Neural Network) such as ResNet (Residual Network) or EfficientNet. Alternatively, the extraction model 111 may be configured as an image feature extractor such as a color histogram or an edge. Note that a method for extracting features from an image using such a model can be appropriately adopted from existing technologies, and therefore a detailed description thereof will be omitted here.

[0049] The generation model 112 is configured to be able to generate a sentence corresponding to an object from the feature amount extracted by the extraction model 111. The generation model 112 may be configured as, for example, a Long Short Term Memory (LSTM) decoder. The generation model 112 may also be configured as a Transformer. Note that a method for generating a sentence from a feature amount using such a model can be appropriately implemented using existing technology, and therefore a detailed description thereof will be omitted here.

[0050] (Information addition operation) Next, an information adding operation by the search system 10 according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of the information adding operation of the search system according to the second embodiment. In Fig. 7, the same processes as those shown in Fig. 3 are denoted by the same reference numerals.

[0051] As shown in FIG. 7, when the information adding operation by the search system 10 according to the second embodiment is started, the search system 10 first acquires an image from the image storage unit 50 (step S101).

[0052] Next, the sentence generation unit 110 extracts features of the object from the image using the extraction model 111 (step S121). Then, the sentence generation unit 110 generates sentences corresponding to the object from the features using the generation model 112 (step S122).

[0053] Thereafter, the information adding unit 120 adds the sentence generated by the sentence generating unit 110 to the image as adjective information (step S103).

[0054] (Specific operation example) Next, a specific operation example of the search system 10 according to the second embodiment (particularly, the operation of the sentence generation unit 110) will be described with reference to Fig. 8. Fig. 8 is a conceptual diagram showing a specific operation of the sentence generation unit according to the second embodiment. Note that the following description will be given using an example in which the extraction model 111 is configured as a CNN and the generation model 112 is configured as an LSTM decoder.

[0055] As shown in FIG. 8, it is assumed that an object image (here, an image of ramen) is input to the sentence generation unit 110 according to the second embodiment. In this case, first, the extraction model 111 extracts the feature amount of the object from the image. Note that, as shown in the figure, if an object label (for example, information indicating the name of the object) is input together with the object image, the information on the object label may be integrated with the feature amount extracted by the extraction model 111. The feature amount extracted by the extraction model 111 is output to the generation model 112.

[0056] Next, the generation model 112 generates sentences from the features extracted by the extraction model 111. In the example shown in Fig. 8, the words "Korezo" are output from h1 of the generation model 112 (i.e., the LSTM decoder), "The family" from h2, and "Toiu" from h3. The generation model 112 combines the words output in this way to generate sentences corresponding to the objects.

[0057] (Technical effect) Next, technical effects obtained by the search system 10 according to the second embodiment will be described.

[0058] 6 to 8, in the search system 10 according to the second embodiment, the sentence generation unit 110 includes the extraction model 111 and the generation model 112, so that it is possible to generate sentences that appropriately correspond to objects from images. Note that the extraction model 111 and the generation model 112 may be trained separately, or the two may be trained together.

[0059] <Third embodiment> A search system 10 according to the third embodiment will be described with reference to Figures 9 and 10. The third embodiment differs from the first and second embodiments in some configurations and operations, and other parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first and second embodiments, and will omit descriptions of other overlapping parts as appropriate.

[0060] (Functional configuration) First, the functional configuration of the search system 10 according to the third embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the functional configuration of the search system according to the third embodiment. In Fig. 9, the same elements as those shown in Fig. 2 are denoted by the same reference numerals.

[0061] 9, the search system 10 according to the third embodiment includes, as processing blocks for realizing its functions, a sentence generation unit 110, an information assignment unit 120, a query acquisition unit 130, and a search unit 140. In particular, the search unit 140 according to the third embodiment includes a word extraction unit 141, a feature vector generation unit 142, and a similarity calculation unit 143.

[0062] The word extraction unit 141 extracts words that can be used for search from the search query acquired by the query acquisition unit 130 and the adjective information assigned to the image. The word extraction unit 141 may extract a plurality of words from each of the search query and the adjective information. The words extracted by the word extraction unit 141 may be adjectives included in the search query and the adjective information, or may be words other than adjectives. Note that, for the adjective information assigned to the image, words may be extracted in advance (for example, before starting the search operation). In this case, the extracted words may be stored in addition to or in place of the sentences that have been stored as the adjective information up until that point. Information on the words extracted by the word extraction unit 141 is configured to be output to the feature vector generation unit 142.

[0063] The feature vector generating unit 142 is configured to generate a feature vector from the words extracted by the word extracting unit 141. Specifically, the feature vector generating unit 142 generates a feature vector of the search query (hereinafter, appropriately referred to as a "query vector") from the words extracted from the search query, and generates a feature vector of the adjective information (hereinafter, appropriately referred to as a "target vector") from the words extracted from the adjective information. Note that a specific method for generating a feature vector from a word can be appropriately adopted from existing technologies, and therefore a detailed description thereof will be omitted here. The feature vector generating unit 142 may generate one feature vector from one word, or may generate one feature vector from multiple words (i.e., a feature vector corresponding to multiple words). In addition, when the word extracting unit 141 does not extract words, the feature vector generating unit 142 may generate a feature vector from the search query or the adjective information itself (i.e., a sentence not divided into words). The feature vectors (that is, the query vector and the target vector) generated by the feature vector generation unit 142 are output to the similarity calculation unit 143.

[0064] The similarity calculation unit 143 is configured to be able to calculate the similarity between the query vector and the target vector generated by the feature vector generation unit 142. As a specific method for calculating the similarity, an existing technology can be appropriately adopted, and one example is a method for calculating cosine similarity. The similarity calculation unit 143 calculates the similarity between the query vector and the target vector corresponding to each of the multiple images, and searches for an image corresponding to the search query based on the similarity. For example, the similarity calculation unit 143 outputs an image with the highest similarity as a search result. Alternatively, the similarity calculation unit 143 may be configured to output a predetermined number of images as search results in descending order of similarity.

[0065] (Search operation) Next, a search operation by the search system 10 according to the third embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the search operation of the search system according to the third embodiment. In Fig. 10, the same processes as those shown in Fig. 5 are denoted by the same reference numerals.

[0066] 10, when a search operation by the search system 10 according to the third embodiment is started, the query acquisition unit 130 first acquires a search query (step S201). The acquired search query is output to the search unit 140.

[0067] Next, the word extraction unit 141 in the search unit 140 extracts words that can be used for searching from the acquired search query and the adjective information assigned to the image (step S231). Then, the feature vector generation unit 142 generates feature vectors (i.e., a query vector and a target vector) from the words extracted by the word extraction unit 141 (step S232). Then, the similarity calculation unit 143 calculates the similarity between the query vector and the target vector, and searches for an image corresponding to the search query (step S233).

[0068] After that, the search unit 140 outputs an image corresponding to the search query as a search result (step S203).

[0069] (Technical effect) Next, technical effects obtained by the search system 10 according to the third embodiment will be described.

[0070] As described in Fig. 9 and Fig. 10, in the search system 10 according to the third embodiment, a search is performed using the similarity between feature vectors generated from the search query and the adjective information. In this way, the input search query can be appropriately compared with the adjective information added to the image. As a result, it becomes possible to appropriately search for an image desired by the user.

[0071] <Fourth embodiment> A search system 10 according to the fourth embodiment will be described with reference to Fig. 11 to Fig. 13. The fourth embodiment differs from the first to third embodiments in part of its configuration and operation, and the remaining parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first to third embodiments, and will omit a description of the other overlapping parts as appropriate.

[0072] (Functional configuration) First, the functional configuration of the search system 10 according to the fourth embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of the search system according to the fourth embodiment. In Fig. 11, the same elements as those shown in Fig. 2 are denoted by the same reference numerals.

[0073] As shown in Fig. 11, the search system 10 according to the fourth embodiment includes, as processing blocks for realizing its functions, an object detection unit 150, a sentence generation unit 110, an information assignment unit 120, a query acquisition unit 130, and a search unit 140. That is, the search system 10 according to the fourth embodiment is configured to further include an object detection unit 150 in addition to the configuration of the first embodiment (see Fig. 2). The object detection unit 150 may be realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0074] The object detection unit 150 is configured to be able to detect an object from an image. Specifically, the object detection unit 150 is configured to be able to detect an area in an image where an object exists, and detect the name and type of the object. Note that a specific method for detecting an object from an image can be appropriately adopted from existing technologies, and therefore a detailed description thereof will be omitted here. The object detection unit 150 may be configured as, for example, Faster R-CNN.

[0075] (Information addition operation) Next, an information adding operation by the search system 10 according to the fourth embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of the information adding operation of the search system according to the fourth embodiment. In Fig. 12, the same processes as those shown in Fig. 3 are denoted by the same reference numerals.

[0076] As shown in FIG. 12, when the information adding operation by the search system 10 according to the fourth embodiment is started, the search system 10 first acquires an image from the image storage unit 50 (step S101).

[0077] Next, the object detection unit 150 detects an object from the image (step S141). Then, the sentence generation unit 110 generates a sentence corresponding to the object detected by the object detection unit 150 (step S102).

[0078] Thereafter, the information adding unit 120 adds the sentence generated by the sentence generating unit 110 to the image as adjective information (step S103).

[0079] (Specific operation example) Next, a specific operation example of the search system 10 according to the fourth embodiment (particularly, the operation of the object detection unit 150) will be described with reference to Fig. 13. Fig. 13 is a conceptual diagram showing a specific operation of the object detection unit according to the fourth embodiment. Note that, in the following, the description will be given using an example in which the object detection unit 150 is configured as Faster R-CNN.

[0080] As shown in FIG. 13, assume that an image (here, an image including curry in the right-hand region) is input to object detection unit 150 according to the fourth embodiment. In this case, object detection unit 150 first extracts an area including an object from the image (for example, a rectangular region as shown in the figure). Then, object detection unit 150 detects that the extracted object is curry. That is, object detection unit 150 detects the name of the extracted object.

[0081] When an input image contains multiple objects, object detection unit 150 may detect each of the multiple objects. In other words, object detection unit 150 may detect multiple objects from one image.

[0082] (Technical effect) Next, technical effects obtained by the search system 10 according to the fourth embodiment will be described.

[0083] 11 to 13, in the search system 10 according to the 43rd embodiment, an object included in an image is detected by the object detection unit 150. In this way, it is possible to accurately recognize an object included in an image. As a result, it is possible to appropriately generate a sentence corresponding to an object included in an image.

[0084] <Fifth embodiment> The information imparting system according to the fifth embodiment will be described with reference to Fig. 14. The information imparting system according to the fifth embodiment differs from the search systems according to the first to fourth embodiments in only a part of its configuration and operation, and other parts may be generally similar. Therefore, the following will describe in detail the parts that differ from the first to fourth embodiments, and will omit appropriate descriptions of other overlapping parts.

[0085] (Functional configuration) First, the functional configuration of the information imparting system according to the fifth embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the functional configuration of the information imparting system according to the fifth embodiment. In Fig. 14, the same elements as those shown in Fig. 2 are denoted by the same reference numerals.

[0086] As shown in Fig. 14, the information imparting system 20 according to the fifth embodiment is configured to include a sentence generating unit 110 and an information imparting unit 120 as processing blocks for realizing the functions. That is, the information imparting system 20 according to the fifth embodiment is configured to include only components related to the information imparting operation among the configuration of the search system according to the first embodiment (see Fig. 2). Note that the operation of the information imparting system 20 according to the fifth embodiment may be the same as the information imparting operation (see Fig. 3) executed by the search system 10 according to the first embodiment.

[0087] (Technical effect) Next, technical effects obtained by the information imparting system 20 according to the fifth embodiment will be described.

[0088] As described in Fig. 14, in the information assignment system 20 according to the fifth embodiment, a sentence corresponding to an object included in an image is automatically generated and assigned as adjective information. In this way, it is possible to execute various processes using the adjective information assigned as a sentence.

[0089] The scope of each embodiment also includes a processing method in which a program that operates the configuration of each embodiment to realize the functions of the above-mentioned embodiments is recorded on a recording medium, the program recorded on the recording medium is read as code, and the program is executed on a computer. That is, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0090] Examples of recording media that can be used include floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs. In addition, the scope of each embodiment does not include those that execute processing by themselves recorded on the recording media, but also those that execute processing by operating on an OS in cooperation with other software and functions of an extension board.

[0091] This disclosure may be modified as appropriate without departing from the spirit or concept of the invention as can be read from the claims and the entire specification, and search systems, search methods, and computer programs incorporating such modifications are also included in the technical concept of this disclosure.

[0092] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0093] (Appendix 1) The search system described in Appendix 1 is a search system characterized by comprising a sentence generation unit that generates sentences corresponding to objects included in an image using a trained model, an information addition unit that adds the sentences corresponding to the objects to the image as adjective information of the objects, a query acquisition unit that acquires a search query, and a search unit that searches for an image corresponding to the search query from among a plurality of images based on the search query and the adjective information.

[0094] (Appendix 2) The search system according to Supplementary Note 2 is the search system according to Supplementary Note 1, characterized in that the adjective information is information expressing a state or appearance of the object.

[0095] (Appendix 3) The search system described in Appendix 3 is the search system described in Appendix 2, characterized in that the object is a dish and the adjective information is information including at least one of the taste, smell, and temperature of the dish.

[0096] (Appendix 4) The search system described in Appendix 4 is the search system described in Appendix 2, characterized in that the object is an article, and the adjective information is information including at least one of the texture and tactile sensation of the article.

[0097] (Appendix 5) The search system described in Supplementary Note 5 is the search system described in any one of Supplementary Notes 1 to 4, characterized in that the search query is in a natural language.

[0098] (Appendix 6) The search system described in Appendix 6 is the search system described in any one of Appendixes 1 to 5, characterized in that the trained model includes an extraction model that extracts features of the object from the image, and a generation model that generates sentences corresponding to the object from the features of the object.

[0099] (Appendix 7) The search system described in Appendix 7 is the search system described in any one of Appendixes 1 to 6, characterized in that the search unit searches for images that correspond to the search query based on the similarity between a feature vector generated from the search query and a feature vector generated from the adjective information.

[0100] (Appendix 8) The search system described in Appendix 8 is the search system described in Appendix 7, characterized in that the search unit extracts words that can be used for search from the search query and the adjective information, and generates the feature vector based on the extracted words.

[0101] (Appendix 9) The search system described in Appendix 9 is the search system described in any one of Appendixes 1 to 8, further comprising an object detection unit that detects the object from the image, and the sentence generation unit generates a sentence corresponding to the object detected by the object detection unit.

[0102] (Appendix 10) The search system described in Appendix 10 is the search system described in any one of Appendixes 1 to 9, characterized in that the search unit searches for images that correspond to the search query using, in addition to the adjective information, at least one of time information indicating the time when the image was captured, location information indicating the location where the image was captured, and name information indicating the name of the object.

[0103] (Appendix 11) The search system described in Supplementary Note 11 is the search system described in any one of Supplements 1 to 10, characterized in that the search unit searches for an image that corresponds to the search query from among a plurality of images that constitute the video data.

[0104] (Appendix 12) The search method described in Supplementary Note 12 is a search method characterized by generating sentences corresponding to objects included in an image using a trained model, assigning the sentences corresponding to the objects to the image as adjective information of the objects, obtaining a search query, and searching for an image corresponding to the search query from among a plurality of images based on the search query and the adjective information.

[0105] (Appendix 13) The computer program described in Appendix 13 is a computer program characterized by operating a computer to generate sentences corresponding to objects included in an image using a trained model, assign the sentences corresponding to the objects to the image as adjective information of the objects, obtain a search query, and search for an image corresponding to the search query from among a plurality of images based on the search query and the adjective information.

[0106] (Appendix 14) The recording medium described in Appendix 14 is a recording medium having the computer program described in Appendix 13 recorded thereon. [Explanation of symbols]

[0107] 10 Search System 11 CPU 50 Image storage unit 110 Sentence generation section 111 Extraction Model 112 Generative Model 120 Information Provision Department 130 Query Acquisition Unit 140 Search Department 141 Word Extraction Unit 142 Feature Vector Generation Unit 143 Similarity calculation unit 150 Object detection unit

Claims

1. a sentence generation unit that generates a sentence including a plurality of words corresponding to an object included in an image by using a trained model; an information adding unit that adds a sentence including the plurality of words to the image as adjective information of the object; a query acquisition unit that acquires a search query; a search unit that searches the images for an image corresponding to the search query based on the search query and the adjective information; A search system comprising:

2. 2. The search system according to claim 1, wherein the adjective information is information expressing a state or a condition of the object.

3. The object is a dish, The adjective information includes at least one of the taste, the smell, and the temperature of the food.

3. The search system according to claim 2.

4. the object is an article, The adjective information is information including at least one of the texture and the tactile sensation of the article.

3. The search system according to claim 2.

5. The search system according to claim 1 , wherein the search query is in natural language.

6. The search system according to any one of claims 1 to 5, characterized in that the trained model includes an extraction model that extracts features of the object from the image, and a generation model that generates a sentence including the multiple words from the features of the object.

7. The search system according to any one of claims 1 to 6, characterized in that the search unit searches for an image corresponding to the search query based on a similarity between a feature vector generated from the search query and a feature vector generated from the adjective information.

8. The search system according to claim 7 , wherein the search unit extracts words that can be used for a search from the search query and the adjective information, and generates the feature vector based on the extracted words.

9. A sentence containing multiple words corresponding to objects contained in an image is generated using a trained model. adding a sentence including the plurality of words to the image as adjective information of the object; Get the search query, Based on the search query and the adjective information, an image corresponding to the search query is searched for from among the plurality of images. A search method comprising:

10. A sentence containing multiple words corresponding to objects contained in an image is generated using a trained model. adding a sentence including the plurality of words to the image as adjective information of the object; Get the search query, Based on the search query and the adjective information, an image corresponding to the search query is searched for from among the plurality of images. A computer program for causing a computer to operate in such a manner.

Images