Query modification system, search system, storage medium, and computer program product

Abstract

The present application provides a query modification system, a search system, and a storage medium. The query modification system has a processor that, in a case where a first query image input by a user contains a portion that contradicts a first condition, generates a second query image by modifying the portion in accordance with the first condition, wherein the first condition is a condition associated with a search target.

Description

TECHNICAL FIELD

[0001] The present application relates to a query modification system, a search system, and a storage medium. BACKGROUND

[0002] There is a technique of searching for an image associated with a query image in a case where the query image is input as a search condition. A representative example of the associated image is an image similar to the query image.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT DOCUMENTS

[0005] Patent Document 1: Japanese Patent Application Publication No. 2016-218578 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, the query image is input in order to find an image closer to the user's image, and the query image does not necessarily satisfy the condition associated with the search target. Therefore, if the search is performed using only the features extracted from the query image that does not satisfy the condition, there is a possibility that an image corresponding to a feature that contradicts the condition associated with the search target is searched for.

[0008] One way of the non-limiting embodiments of the present disclosure relates to a technique of easily searching for an image satisfying a condition associated with a search target when the condition exists.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] [1] According to one way of the present disclosure, there is provided a query modification system having a processor that, in a case where a portion that contradicts a first condition is included in a first query image input by a user, modifies the portion according to the first condition to generate a second query image, wherein the first condition is a condition associated with a search target.

[0011] [2] In the query modification system according to [1], the first condition can also be used as a search query.

[0012] [3] In the query modification system according to [2], the first condition can also be associated with features extracted from the first query image.

[0013] [4] In the query modification system according to [1], the processor can also maintain features other than the portion in the first query image in a case where the portion is modified.

[0014] [5] In the query modification system according to [4], the structural feature can also be determined in accordance with a category of content included in the first query image.

[0015] [6] In the query modification system according to [4], the structural feature can also be set by each user.

[0016] [7] In the query modification system according to [1] or [4], the processor can also not modify the portion that contradicts the first condition, if the portion satisfies a second condition that is predetermined.

[0017] [8] In the query modification system according to [7], the second condition can also be a condition in which the portion that contradicts the first condition corresponds to a structural feature that appears in the first query image.

[0018] [9] In the query modification system according to [1], the processor can also notify the user of the portion that contradicts the first condition before modifying the first query image.

[0019]

[10] In the query modification system according to [1] or [9], the processor can also ask the user whether to generate the second query image before modifying the first query image.

[0020]

[11] In the query modification system according to [1] or [9], if the portion that contradicts the first condition includes a portion that is excluded from the modification target, the processor can also ask the user for a change in the first condition associated with the portion that is excluded from the modification target.

[0021]

[12] In the query modification system according to [1], the processor can also ask the user whether to change the first condition that corresponds to the contradiction, if the first condition can be changed.

[0022]

[13] In the query modification system according to [1], the processor presents the generated second query image to the user.

[0023]

[14] In the query modification system according to

[13] , the processor can also arrange the second query image and the first query image on the same screen.

[0024]

[15] In the query modification system according to

[14] , the processor can also add a mark that indicates a portion that is modified to either or both of the first query image and the second query image.

[0025]

[16] In the query modification system according to

[13] , the processor can also ask the user about use of the generated second query image in the search before performing the search.

[0026]

[17] In the query modification system according to [1], the first condition can also include a law.

[0027]

[18] In the query modification system according to

[17] , the processor can also give priority to the law in a case where the input of the user as the first condition contradicts the law.

[0028]

[19] According to another aspect of the present disclosure, there is provided a search system having a processor that, in a case where a first query image input by a user includes a portion that contradicts a first condition associated with a search target, modifies the portion according to the first condition, generates a second query image, and searches a database using the generated second query image.

[0029]

[20] According to another aspect of the present disclosure, there is provided a storage medium storing a program that causes a computer to function to accept a first query image input by a user, and in a case where the first query image includes a portion that contradicts a first condition associated with a search target, modify the portion according to the first condition and generate a second query image.

[0030] Effects of Invention

[0031] According to [1], when there is a condition associated with a search target, it is possible to easily search for an image that satisfies the condition.

[0032] According to [2], by reducing contradiction between a condition associated with a search target and a query image, it is possible to easily search for an image that satisfies the condition.

[0033] According to [3], it is possible to easily determine a portion that becomes an object of modification.

[0034] According to [4], it is possible to generate a second query image that maintains the user's intention included in the first query image as much as possible.

[0035] According to [5], it is possible to generate a second query image that maintains a structural feature of the first query image.

[0036] According to [6], it is possible to generate a second query image that maintains the user's intention included in the first query image as much as possible.

[0037] According to [7], even a feature that contradicts a condition, it is possible to prepare an exception that is excluded from an object of modification.

[0038] According to [8], in a case where an impression on a structure of the first search image is influenced by a feature that contradicts the condition, it is possible to exclude from the modified object.

[0039] According to [9], it is possible to notify the user in advance that there is a portion that contradicts other conditions in the first search image.

[0040] According to

[10] , it is possible to perform modification of the first search image on the premise of an instruction of the user.

[0041] According to

[11] , it is possible to change the condition according to a selection of the user.

[0042] According to

[12] , it is possible to change the first condition according to a selection of the user.

[0043] According to

[13] , it is possible to easily confirm the modified portion.

[0044] According to

[14] , it is possible to easily confirm the modification content.

[0045] According to

[15] , it is possible to make the user easily confirm the modification content.

[0046] According to

[16] , it is possible to make the user confirm before performing the search.

[0047] According to

[17] , even in a case where the user's restriction is satisfied, it is possible to modify the search image when a law is not satisfied.

[0048] According to

[18] , it is possible to reduce in advance a case where a search is performed again due to a contradiction with a law.

[0049] According to

[19] , when there is a condition associated with a search object, it is possible to easily search for an image that satisfies the condition.

[0050] According to

[20] , when there is a condition associated with a search object, it is possible to easily search for an image that satisfies the condition. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a diagram showing a structure example of an image search system used in the embodiment.

[0052] Figure 2 is a diagram explaining a structure on a function of a computer that generates a data set recorded in a database.

[0053] Figure 3 is a diagram explaining a structure on a function of a search server that searches for an image similar or associated with an image input as a search from a database.

[0054] Figure 4 is a diagram that describes information input as a query from a terminal operated by a user.

[0055] Figure 5 is a diagram that describes an example of extraction of features by the feature extraction section.

[0056] Figure 6 is a diagram that describes an example of the functional structure of the pre-processing section.

[0057] Figure 7 is a flowchart that describes an example of the processing performed by the pre-processing section.

[0058] Figure 8 is a diagram that shows an example of a screen for notifying of a determined contradiction.

[0059] Figure 9 is a diagram that shows an example of a table for determining whether or not it is possible to change premise information.

[0060] Figure 10 is a diagram that shows an example of a screen that represents a modified query image.

[0061] Figure 11 is a diagram that shows an example of another screen that represents a modified query image.

[0062] Figure 12 is a diagram that shows an example of a screen that represents a modified query image.

[0063] Figure 13 is a diagram that describes an example of modification of a query image in a case where a building in which a three-story building is included in the query image is specified as a location in the premise information that allows only a two-story building. Figure 13 (A) of FIG. 10 shows a query image before modification, Figure 13 (B) of FIG. 10 shows a query image after modification. DETAILED DESCRIPTION

[0064] Hereinafter, an embodiment of the present application will be described with reference to the accompanying drawings.

[0065] <EMBODIMENT>

[0066] Hereinafter, an image retrieval system that is assumed to be used in a planning studio or a design studio will be described.

[0067] In a planning studio or the like, records of cases that have been handled in the past are stored. In the information here, in addition to images such as design plans, design drawings, and the like, documents such as records of complaints from customers, cases of accidents, and internal comments of the company are included. The image retrieval system described in the present embodiment assists in the efficiency of design business by utilizing these pieces of information.

[0068] <system configuration>

[0069] Figure 1 is a diagram showing a configuration example of an image retrieval system 1 used in the embodiment.

[0070] Figure 1 The image retrieval system 1 shown is composed of a retrieval server 10 that retrieves an image similar or related to an input query, a database (= Data Base) 20 that stores data of an image (hereinafter, referred to as "image data") as a retrieval target, a terminal 30 operated by a user who inputs a query, and a network 40 that connects them so as to be communicable. Note that the network can be a local area network or the Internet. The image retrieval system 1 is an example of a retrieval system.

[0071] Figure 1 The retrieval server 10 shown has a processor 11 that performs a retrieval and other processes by executing a program, a storage device 12 that stores the program and various data, a network interface (= Interface) 13 that realizes communication with the outside, and other signal lines 14 such as a bus that connect them.

[0072] The processor 11 is composed of, for example, a CPU. The storage device 12 is composed of a ROM (= Read Only Memory) that stores, for example, a BIOS (= Basic Input Output System) and the like, a RAM (= Random Access Memory) that serves as a work area, and a hard disk device that stores a basic program or an application program and the like. Note that it is not excluded that the ROM or the RAM is included in a part of the processor 11. The processor 11 and the storage device 12 constitute a computer.

[0073] In the database 20 shown, Figure 1 In the database 20 shown, in addition to images such as design plans and design drawings, documents such as records of complaints from customers, cases of accidents, and internal comments of the company are recorded. These information are collectively referred to as "past cases".

[0074] A tag for retrieval is attached to each of the information constituting the past cases. The tag is obtained by a set of features (hereinafter, referred to as "features") included in each of the information. In the embodiment, the set of features is also referred to as a data set.

[0075] Figure 1 The terminal 30 shown is a so-called computer. The terminal 30 can be a desktop computer, a notebook computer, or a portable computer such as a smartphone or a wearable terminal. In Figure 1 In the embodiment, only one terminal 30 is shown, but the number is arbitrary.

[0076] Furthermore, the retrieval server 10 does not need to be a single unit; it can be multiple computers working together. In this embodiment, the retrieval server 10 will also be referred to as an example of a retrieval system. Additionally, the retrieval server 10 is also an example of a query / modification system.

[0077] <Functional Structure>

[0078] Figure 2 This is a diagram illustrating the functional structure of the computer 50 that generates the dataset of records in database 20.

[0079] The hardware structure of the computer 50 and Figure 1 The retrieval server 10 shown is the same. That is, the computer 50 is equipped with an interface for communication with the processor, storage device, and database 20.

[0080] If computer 50 reads past events from database 20, it performs preprocessing in preprocessing unit 51, which is prepared to extract features classified as structural expressions (hereinafter referred to as "structural information"), and provides the result of the preprocessing to structural information extraction unit 52. Figure 2 In the case of a structural information extraction unit 52, one or more features contained in past instances are deduced by using the derivation of the derivation model learned from each feature, and the set of deduced features is output to the database 20 as a dataset attached to the past instances.

[0081] In this implementation, a derivation model is prepared for each feature. The derivation model is generated beforehand using machine learning or similar methods. Figure 2 In this context, derivation using a derivation model is referred to as AI (Artificial Intelligence) derivation.

[0082] If the computer 50 reads past cases from the database 20, it performs preprocessing in the preprocessing unit 53, which is prepared to extract features classified as perceptual expressions (hereinafter referred to as "perceptual information"), and provides the result of the preprocessing to the perceptual information extraction unit 54. In this embodiment, perceptual information refers to features that do not contain structural or quantitative expressions. In other words, perceptual information refers to features that contain qualitative or subjective expressions.

[0083] exist Figure 2In this case, the sensibility information extraction section 54 derives one or more sensibility information included in the past case by using derivation of a derivation model learned with the sensibility information, and outputs the derived set of sensibility information as a data set attached to the past case to the database 20. In the case of the present embodiment, a derivation model is prepared for each sensibility information. The derivation model is generated in advance by machine learning or the like.

[0084] Thus, in the past case stored in the database 20, one or more features belonging to both or one of the structure information and the sensibility information are attached.

[0085] Figure 3 is a diagram illustrating a functional configuration of the retrieval server 10 that retrieves an image similar or related to an image input as a query (hereinafter, referred to as "query image") from the database 20. The query image input by the user is an example of the first query image.

[0086] The retrieval server 10 functions as the following components: a classification section 101 that classifies the query image by object; a pre-processing section 102 that performs a predetermined process on the query image; a feature extraction section 103 that extracts a feature amount (hereinafter, referred to as "feature") included in the query image; a normalization section 104 that modifies a difference in expression of a text including a structure expression (hereinafter, referred to as "structure information text"); a normalization section 105 that modifies a difference in expression of a text including a sensibility expression (hereinafter, referred to as "sensibility information text"); a corresponding feature classification section 106 that classifies a feature corresponding to a string constituting the structure information text or the sensibility information text; a feature modification section 107 that modifies a feature provided to a retrieval engine 108; and the retrieval engine 108 that retrieves a case having high relevance to the modified feature from the database 20.

[0087] These functions are realized by execution of a program by the processor 11 (refer to Figure 1 ). In addition, the structure of the retrieval server 10 other than the retrieval engine 108 is an example of a retrieval condition determination system.

[0088] The retrieval server 10 in the present embodiment inputs premise information, image information, structure information text, and sensibility information text as a query (hereinafter, also referred to as "retrieval query") from the terminal 30 (refer to Figure 1 ).

[0089] However, these four kinds of information need not all be input as a query. Also, in the case of the present embodiment, the structural information text and the emotional information text need not be explicitly distinguished. In fact, there is no constraint on the kind of expression used to input the text. Thus, the user can input a request for a past case desired to be searched for by a character string without distinguishing between the two.

[0090] Figure 4 is a diagram explaining the information input as a query from the terminal 30 operated by the user.

[0091] The premise information is structural information or quantitative information that is more preferential than other queries in the query input by the user. Also, among the premise information, there are laws and the like. The premise information is an example of a condition associated with the search target. However, the laws and the like need not be input by the user.

[0092] In the case of the present embodiment, an image related to a building is set as the search target.

[0093] Thus, the premise information is obtained, for example, by the address, the land area, the land use condition, the surrounding environment, the category of the object, the budget, the presence or absence of a yard, the presence or absence of a vehicle, the presence or absence of a garage, the composition of family members, and the number of families. The category of the object is, for example, a high-rise building, an apartment, or a detached house.

[0094] The image information is so-called query images. The image information is obtained, for example, by a hand-drawn picture, a photograph, a flyer, or computer graphics. In the case of the present embodiment, the image information is less preferential than other kinds of queries.

[0095] The structural information text is text containing structural expressions. The structural information text is, for example, two-family house, 10-minute walk, 3LDK, or wooden house.

[0096] The emotional information text is text containing emotional expressions. The emotional information text is, for example, open feeling, gathering of family members, overall space, or comfort of wood.

[0097] Also, there are cases where the structural information text and the emotional information text are input without being explicitly distinguished. Text in which structural expressions and emotional expressions are mixed is, for example, "bright living room with an open feeling". "Living room" is a structural expression because it is a noun that can be explicitly determined, and "open feeling" or "bright" is an emotional expression because it is an adjective that indicates a state of the senses.

[0098] Returning to the explanation of Figure 3 .

[0099] The classification unit 101 classifies the query images input by the user according to the object. In this embodiment, the query images are classified as images of a living room, images of a kitchen, and images of an exterior. Of course, the candidates for classification are not limited to three types. Other candidates include children's room, bedroom, bathroom, toilet, entrance hall, and courtyard. The classification unit 101 attaches the classification results as attributes of each query image.

[0100] The preprocessing unit 102 performs pre-defined processing on the input query image. Pre-defined processing includes, for example, size adjustment, contrast adjustment, edge enhancement, and noise removal.

[0101] In addition, Figure 3 The preprocessing unit 102 shown includes a function to remove portions of the query image that contradict other conditions, as part of the preprocessing. For example, if a garage is specified as unnecessary in the prerequisite information but is included in the query image, the preprocessing unit 102 modifies the query image to remove the garage portion. Here, "contradiction" is used to mean that it does not occur simultaneously or coexist. In this embodiment, the contradictory portion is also referred to as the mismatched portion.

[0102] However, even parts that contradict other conditions are sometimes excluded from the modification. In this embodiment, modifications to parts of the query image that contradict other determined conditions are performed according to the user's instructions, and the data is then output to the next segment.

[0103] The feature extraction unit 103 extracts one or more features contained in the query image by comparing them with a derivation model prepared according to each feature using machine learning or the like.

[0104] Figure 5 This diagram illustrates an example of feature extraction using the feature extraction unit 103. Figure 5 In this case, features such as "high ceiling", "high windows", "ladder", "floor", "comfort of wood", "open", "ceiling fan", and "spotlight" are extracted from the living room photo that is used as the input image for the query.

[0105] Return to Figure 3 Explanation.

[0106] The normalization section 104 modifies the expression of structural information text input by users as queries. For example, it incorporates the uniformity of character types and fluctuations in spelling or marking.

[0107] Normalization unit 105 modifies the differences in the expression of the user's emotional information text as query input. Furthermore, normalization unit 104 also modifies the deviations in each person's expression.

[0108] The corresponding feature classification unit 106 classifies the following cases: whether the string constituting the structural information text or the perceptual information text corresponds to the structural feature, or corresponds to the perceptual feature, or corresponds to both the structural feature and the perceptual feature.

[0109] The feature modification unit 107 performs a process to modify the features provided to the search engine 108 in order to easily obtain search results that match the user's intent. In this embodiment, the feature modification unit 107 removes contradictions between extracted features. For example, the feature modification unit 107 performs modifications to remove features extracted from the query image that contradict the premise information. Furthermore, the feature modification unit 107 performs modifications to remove contradictory features between multiple query images, for example.

[0110] <Functional Structure of Pre-processing Unit 102>

[0111] Here, the details of the processing functions performed by the preprocessing unit 102 in the aforementioned functional structure of the retrieval server 10 will be explained.

[0112] Figure 6 This diagram illustrates an example of the functional structure constituting the preprocessing unit 102.

[0113] exist Figure 6 The preprocessing unit 102 shown includes the following components: an image analysis unit 121 that analyzes the query image input by the user; a prerequisite information analysis unit 122 that analyzes prerequisite information that is a condition associated with the search object; a comparison unit 123 that compares the analysis results of the image analysis unit 121 with the analysis results of the prerequisite information analysis unit 122; and a contradiction correction unit 124 that corrects contradictions determined by the comparison results.

[0114] The preprocessing unit 102 here is another example of a query and modification system.

[0115] In this embodiment, the image analysis unit 121 extracts information related to the structure of buildings contained in the query image and outputs the extracted information to the comparison unit 123. A building is an example of a retrieval object. The information related to the structure of the building is an example of features extracted from the first query image.

[0116] In this embodiment, information related to the structure is extracted, such as information related to the building's shape, land use information, surrounding environment, presence or absence of a garage, and presence or absence of a courtyard. Information related to the building's shape includes, for example, the number of floors, the shape of the roof, the proportion of windows, and the ratio of ceiling height.

[0117] In the case of the present embodiment, the number of stories refers to information such as a single-story building, a two-story building, a three-story building, and the like. However, since the number of stories is extracted in an appearance manner from the query image, it does not necessarily coincide with the number of stories of the actual building.

[0118] In the case of the present embodiment, the shape of the roof refers to information such as a gable roof, a hipped roof, a one-side slope roof, and the like. Of course, unless the query image is a six-faced image, the accurate shape of the roof cannot be known.

[0119] In the case of the present embodiment, the proportion of the window refers to the proportion of the area of the window to the area of the wall surface of the building. Even if the scale and the like are not clear, the proportion of the area of the window to the area of the wall surface can be calculated. Further, the number of windows can also be extracted from information related to the window.

[0120] In the case of the present embodiment, the height of the ceiling refers to a height assumed from the ratio between the height of the other object reflected or depicted in the query image. Further, a ceiling having a ratio exceeding a predetermined threshold is referred to as a relatively high ceiling, and a ceiling having a ratio less than the threshold is referred to as a relatively low ceiling. However, instead of the height of the ceiling, the height of the story can also be extracted.

[0121] Further, the land-use information has, for example, the area of the land, the difference in height between the road and the land.

[0122] Further, the surrounding environment has, for example, a residential area, a commercial area, and the outskirts.

[0123] Further, the presence or absence of a garage indicates the presence or absence of a structure or space assumed to be used as a garage in the query image. Further, in the case where there is a garage, the number of vehicles that can be parked in the garage is also extracted. Of course, the number is an estimated value.

[0124] Further, the presence or absence of a yard refers to the presence or absence of an empty space considered to be a yard within the land regardless of whether or not it is actually used as a garage.

[0125] Further, depending on the content of the query image, the layout of the rooms such as 1DK, 2LDK, 3LDK, and the like can also be extracted.

[0126] The premise information analysis section 122 in the present embodiment extracts information included in the premise information, and outputs the extracted information to the comparison section 123. The premise information is an example of the first condition.

[0127] In the case of the present embodiment, since the target is a building, for example, the budget, the land area, the composition of family members, the land use conditions, the surrounding environment, the presence or absence of a vehicle, the presence or absence of a yard, the address, and the like are extracted. Since this information is input by the user in a text manner, or with the user's selection attached to the text, it is extracted by item. In addition, in the case where the user has a vehicle, the number of vehicles is also extracted.

[0128] The comparison section 123 in the present embodiment compares mutually corresponding or associated information between the information extracted from the query image by the image analysis section 121 and the information extracted from the premise information by the premise information analysis section 122.

[0129] However, mutually corresponding or associated information does not necessarily exist. Thus, the comparison in the comparison section 123 is a case where mutually corresponding or associated information exists in both the image analysis section 121 and the premise information analysis section 122.

[0130] In addition, mutually corresponding information refers to a case where the content represented by the extracted information is the same. For example, the relationship between the presence or absence of a garage extracted from the query image and the presence or absence of a garage extracted from the premise information is an example of mutually corresponding information. Further, the relationship between the land area extracted from the premise information and the assumed value of the land area extracted from the query image is also an example of mutually corresponding information.

[0131] On the other hand, mutually associated information refers to a case where, even if the items or content of expression are different, they are in a relationship that includes information that can be compared. For example, the relationship between the number of vehicles owned by the premise information and the number of parking spaces allowed extracted from the query image is an example of mutually associated information. Also, the relationship between the presence or absence of land and the information of the location extracted from the premise information and the information of the number of stories and the information of the assumed height extracted from the query image is an example of mutually associated information. This is because, if the presence or absence of land and the information of the location extracted from the premise information are known, the upper limit of the number of stories allowed by law in the corresponding area is known. Further, the budget extracted from the premise information and the room layout information extracted from the query image are also examples of mutually associated information. This is because, as long as the room layout information is known, the cost required for the construction of the building can be estimated.

[0132] The contradiction modification section 124 in the present embodiment performs modification for removing a contradiction from the query image in the case where mutually corresponding or associated information is contradictory.

[0133] In the case where the information corresponding to each other or associated with each other is contradictory, for example, there are cases where a flat is expected in the premise information, but the query image is a two-story building; the number of vehicles owned in the premise information is three, but the query image can park one vehicle; and the difference between the land area specified in the premise information and the estimated value of the land area estimated from the query image is more than a predetermined threshold. The threshold value can be obtained as an initial value or can be set by the user. In the case of the present embodiment, the threshold value is 20% of the land area specified in the premise information.

[0134] The query image from which the contradiction is removed, i.e., the modified query image is an example of the second query image.

[0135] In the case of the present embodiment, the structural features of the query image input by the user are maintained as much as possible in the modified query image.

[0136] Specifically, even if there are other parts associated with the contradictory part, the contradiction modification section 124 excludes the other parts associated with the contradictory part from the modification target. For example, in the case where the query image is a three-story building, although a three-story building is prohibited in the area specified in the premise information, the modification content is determined in such a manner that the structural features of the query image are left. For example, in the case where the number and area of windows are extracted as the structural features of the query image, modification is performed on the query image in such a manner that the number and area of windows are maintained.

[0137] The contradiction modification section 124 in the present embodiment basically performs modification to remove the contradiction from the query image, but in the case where the contradictory part corresponds to the structural features extracted from the query image, the part is sometimes excluded from the modification target.

[0138] For example, in the case where the query image is a two-story building, even if a flat building is specified in the premise information, the query image is not modified to a flat. Also, for example, in the case where the shape of the roof of the building of the query image is a triangular shape, even if a flat roof is specified in the premise information, the query image is not modified to a flat roof.

[0139] This is because any feature is a structural feature of the query image, and if the feature is modified, the impression of the query image is greatly changed. In other words, this is because the possibility that the feature intended by the user is lost by modification is high.

[0140] However, the method of eliminating the contradiction is not limited to modification of the query image. For example, the contradiction can also be eliminated by modifying the premise information.

[0141] Therefore, the contradiction modification unit 124 in this embodiment also includes the following function: when the premise information associated with the contradiction can be changed, it queries the user to change the premise information. Of course, there is also premise information that cannot be changed. For example, if the user does not own land, the location designated as premise information can be changed. Similarly, if the user does not own land, the land area designated as premise information can be changed. Furthermore, the number of vehicles parked in the parking lot and the presence or absence of a yard can sometimes be changed.

[0142] <Specific examples of processing performed in the pre-processing unit>

[0143] The following uses Figure 7 For the pre-processing unit 102 (see reference) Figure 3 The processor 11 (see reference) performs the function Figure 1 Here is an example of the processing action performed.

[0144] Figure 7 This is a flowchart illustrating an example of the processing performed by the preprocessing unit 102. Additionally, in Figure 7 The notation S used in this context indicates a step.

[0145] First, the preprocessing unit 102 receives prerequisite information and a query image (step 1). In this embodiment, the prerequisite information includes not only the information entered by the user, but also laws and regulations. Furthermore, the query image processed in this embodiment includes not only the query image entered by the user, but also an image selected by the user from the previously performed search results.

[0146] In this embodiment, only one image is queried, but multiple images can also be entered.

[0147] Next, the preprocessing unit 102 extracts information such as budget, land area, family composition, land use conditions, surrounding environment, presence or absence of vehicles, presence or absence of a yard, and address from the accepted prerequisite information (step 2). In this embodiment, the information to be extracted from the prerequisite information is predetermined. However, the information to be extracted can be individually set by the user or initially set. Furthermore, the information to be extracted can also be determined based on the objects contained in the query image.

[0148] Next, the preprocessing unit 102 extracts information such as the building shape, land use information, surrounding environment, presence or absence of garage, and presence or absence of courtyard from the received query image (step 3). The building shape here includes, for example, the number of floors, the shape of the roof, the proportion of windows, and the ratio of ceiling height. These are, of course, just examples.

[0149] Alternatively, the order of steps 2 and 3 can be changed, or they can be executed in parallel.

[0150] If the extraction of information ends, the pre-processing section 102 compares the information extracted from the premise information with the information extracted from the query image (step 4).

[0151] Next, the pre-processing section 102 determines whether there is a contradiction between the compared information (step 5). As described above, the determination of the contradiction is premised on the case where the information corresponding to or associated with each other is extracted from the premise information and the query image, respectively.

[0152] In the case where none of the contradictory information exists, the pre-processing section 102 obtains a negative result in step 5. In the case where a negative result is obtained in step 5, the pre-processing section 102 ends the pre-processing. Specifically, the query image input by the user is output to the feature extraction section 103 (refer to Figure 3 ).

[0153] On the other hand, in the case where the contradictory information is found, the pre-processing section 102 obtains a positive result in step 5. In the case where a positive result is obtained in step 5, the pre-processing section 102 determines the contradictory information (step 6). If the determination of all the contradictory information ends, the pre-processing section 102 notifies the user of the determined information (step 7).

[0154] Figure 8 is a drawing showing an example of a screen 200 for notifying the determined contradictory. The screen 200 is displayed on the terminal 30 operated by the user (refer to Figure 1 ).

[0155] Figure 8 The screen 200 shown in the drawing includes a title bar 201, an explanation text 202, a display bar 203 of the query image, a button 204 operated in the case where the search is expected to be continued, and a button 205 operated in the case where the search is terminated. In the case where the search is expected to be continued, a "Yes" label is attached to the button 204. Also, a "No" label is attached to the button 205. Figure 8 In the case where the search is expected to be continued, the title bar 201 shows the summary of the content required of the user through the screen 200. Here, the user is required to "confirm the contradiction".

[0156] Figure 8 In the case where the search is terminated, the title bar 201 shows the summary of the content required of the user through the screen 200. Here, the user is required to "confirm the contradiction".

[0157] The specific content of the contradiction and the operation required of the user are described in the explanation text 202. In the example of Figure 8 , it is described as "The portion (the portion encircled with a frame) contradictory to the premise information (no garage) is included in the input query image. In the case where the search is continued, click "Yes". The modification plan of the query image in which the contradiction is eliminated is suggested."

[0158] ​The explanatory text 202 corresponds to the display field 203 of the search image. Therefore, a portion recognized as a garage in the search image is shown by a frame line 203A. A manner capable of distinguishing from other portions in the search image is used in the display of the frame line 203A. For example, the hue, thickness, brightness, and the like of the frame line 203A are defined so as to be largely different from the hue, edge component, brightness, and the like of the search image side. For example, a red thick frame line 203A is used on a white wall building. However, the display of the frame line 203A is arbitrary. The display of the frame line 203A can also be limited to a case where the user desires the display.

[0159] In the explanatory text 202, it is also required to confirm whether or not a modification to eliminate the contradictory portion from the search image in the display can be suggested.

[0160] Returning to Figure 7 the explanation of the explanatory text 202.

[0161] After notifying the user in step 7, the pre-processing section 102 determines whether or not to continue the processing (step 8).

[0162] The determination here is a determination of whether to operate the "Yes" button 204 (refer to Figure 8 ) or the "No" button 205 (refer to Figure 8 ). In a case where the operation of the button 204 is detected, the pre-processing section 102 obtains a positive result in step 8. On the other hand, in a case where the operation of the button 205 is detected, the pre-processing section 102 obtains a negative result in step 8.

[0163] In a case where a negative result is obtained in step 8, the pre-processing section 102 does not output the current search image to the feature extraction section 103 (refer to Figure 3 ) and ends the pre-processing. In this case, the state where the user re-enters the search image or the premise information is restored.

[0164] On the other hand, in a case where a positive result is obtained in step 8, the pre-processing section 102 modifies the search image according to the premise information (step 9).

[0165] The modification here is for the purpose of eliminating the determined contradiction. In the case of the present embodiment, the object of the modification is the search image entered or selected by the user. Thus, other images satisfying the premise information are not output as a result of the modification instead of the current search image. By setting the search image entered or selected by the user as the object of the modification, the original intention of the user is also easily reflected in the modified search image.

[0166] Examples of the modification content include deletion, addition of other images, synthesis of other images, enlargement, reduction, adjustment or change of hue, adjustment or change of brightness, and editing.

[0167] The editing includes, for example, reconstruction of the query image, change of the aspect ratio of the building, and the like. For example, in a case where the number of windows is more than a predetermined threshold, the modification that maintains the state where the number of windows after the modification is also more than the threshold is included in the editing. Also, for example, in a case where the ceiling is higher than a predetermined threshold, the modification that maintains the state where the ceiling after the modification is also higher than the threshold is included in the editing.

[0168] However, the modification for eliminating the contradiction performed by the pre-processing section 102 in the present embodiment has several constraints. The constraint here is an example of the second condition.

[0169] One of the constraints is that the portion where the contradiction is found is associated with a structural feature of the query image. In this case, the modification for eliminating the contradiction is not performed. This is because the structural feature included in the query image has a high likelihood of being a feature intended by the user. In other words, this is because a large change in the structural feature is accompanied by a large change in the impression of the query image.

[0170] For example, since the modification of the query image of a two-story building to a bungalow is accompanied by a large change in the structural feature, even if the portion is contradictory, the modification is prohibited.

[0171] On the other hand, in a case where a query image of a three-story building is input to an area where a three-story building has not been built, modification to a two-story building is allowed. This is because the change from a three-story building to a two-story building is different from the change to a bungalow, and is common in that it is a multi-story building. Whether or not the criterion corresponds to the constraint is specified in advance.

[0172] Another constraint is to maintain the structural feature other than the portion where the contradiction is found as much as possible. When the modification is performed on the contradictory portion, the influence of the modification sometimes also reaches other portions.

[0173] For example, in a case where the plot area specified in the premise information contradicts the sense of size of the building determined from the query image, the sense of size of the building in the query image needs to be modified according to the plot area. In this case, the modification is limited only to the sense of size of the building, and the structural features of the building such as the number of windows and the shape of the windows, the shape of the roof, and the like are maintained as much as possible as objects of the modification.

[0174] When the modification of the query image is completed, the pre-processing section 102 displays the query image after the modification (step 10).

[0175] Also, the pre-processing section 102 determines whether or not the premise information for eliminating the contradiction can be changed (step 11). In a case where the premise information can be changed, the pre-processing section 102 obtains a positive result in step 11. On the other hand, in a case where the premise information cannot be changed, the pre-processing section 102 obtains a negative result in step 11.

[0176] In the case where the affirmative result is obtained in step 11, the pre-processing section 102 further determines whether to change the premise information (step 12).

[0177] In the case of changing the premise information, the pre-processing section 102 obtains the affirmative result in step 12, and returns to step 1. That is, it returns to the screen for accepting the input of the premise information.

[0178] On the other hand, in the case of not changing the premise information, the pre-processing section 102 obtains the negative result in step 12. In the case of obtaining the negative result in step 12, the pre-processing section 102 further determines whether to use the modified query image for the search (step 13).

[0179] In addition, in the case of obtaining the negative result in step 11, the pre-processing section 102 also performs the determination of step 13.

[0180] In the case of using the modified query image for the search, the pre-processing section 102 obtains the affirmative result in step 13. In the case of obtaining the affirmative result in step 13, the pre-processing section 102 ends the pre-processing. Specifically, it outputs the modified query image to the feature extraction section 103 (refer to Figure 3 ).

[0181] On the other hand, in the case of not using the modified query image for the search, the pre-processing section 102 obtains the negative result in step 13. In the case of obtaining the negative result in step 13, the pre-processing section 102 returns to step 3. In this case, the user inputs or selects a new query image.

[0182] In addition, in the example of Figure 7 , either of steps 10 and 11 can be performed first.

[0183] Figure 9 is a diagram showing an example of a table 250 for determining whether to change the premise information.

[0184] Figure 9 The table 250 shown in FIG. 10 is composed of a column 251 of the premise information, a column 252 of the feature extracted from the query image, a column 253 of the contradictory content, and a column 254 of the possibility of changing the premise information.

[0185] Five examples of the contradiction are shown in Figure 9 .

[0186] The first contradictory case is the case of the first row from the top of Table 250. This case is a case where the number of parking spaces extracted from the query image is lower than the number of vehicles owned. For example, it is a case where the number of vehicles owned is two, and the number of parking spaces extracted from the query image is one. In this case, the change of the premise information is set to "OK". However, in a case where the concept of not easily reducing the number of vehicles is adopted, the change of the premise information can also be set to "NG".

[0187] The second contradictory case is the case of the second row from the top of Table 250. This case is a case where a yard is expected in the premise information, but a yard is not extracted from the query image, or a case where a yard is not expected in the premise information, but a yard is extracted from the query image. This case can also be either case. Therefore, in a case where the yard is not expected in the premise information, the change of the premise information is set to "OK". Figure 9

[0188] The third contradictory case is the case of the third row from the top of Table 250. This case is a case where the number of floors extracted from the query image exceeds the upper limit of the number of floors set for the region where the site is located. For example, it is a case where the number of floors extracted from the query image is a three-story building, but the two-story building specified in the premise information is the upper limit by law and regulation.

[0189] In this case, information on the presence or absence of land or the site is input as the premise information. Also, information on the number of floors of the building or information on the assumed height is obtained from the query image.

[0190] Even in a case where the number of floors of the building within the query image is not explicitly known, the height can be estimated when information on a standard that can be used for the height, such as a vehicle or a person, is included. Also, if the height of the building is known, the approximate number of floors can also be assumed.

[0191] In this case, the change of the premise information is set to "NG" in a case where the land is already owned. On the other hand, the change of the premise information is set to "OK" in a case where the land is not yet owned.

[0192] The fourth contradictory case is the case of the fourth row from the top of Table 250. This case is a case where it is possible to exceed the budget in a case where a house is built based on the room layout information extracted from the query image. For example, it is a case where the room layout extracted from the query image is 5LDK, and the budget amount specified in the premise information corresponds to 2LDK. In this case, the change of the premise information is set to "OK".

[0193] ​The 5th contradictory case is the case of the 5th row from the top of the table 250. This case is a case where the assumed value of the land area calculated from the query image exceeds the land area. For example, it is a case where the assumed value of the land area extracted from the query image is 300 square meters, but the land area specified in the premise information is 200 square meters.

[0194] In this case, the presence or absence of the land and the land area are input as the premise information. Also, the assumed land of the land area is extracted from the query image.

[0195] In this case, in the case of already owning the land, the change of the premise information is set to "not possible". On the other hand, in the case of not owning the land, the change of the premise information is set to "possible".

[0196] In addition, Figure 9 The case shown is an example, and the conclusion regarding the change possibility is also an example.

[0197] <Display Associated with Preprocessing>

[0198] Hereinafter, an example of the screen displayed in association with steps 9 to 13 (refer to Figure 7 ) will be described.

[0199] <Example 1>

[0200] Figure 10 is a drawing showing an example of a screen 300 that displays the modified query image. The screen 300 is used in the case of arranging and displaying the query image before and after modification. The screen 300 is displayed on the terminal 30 operated by the user (refer to Figure 1 ).

[0201] Figure 10 The screen 300 shown includes a title bar 301, an explanatory text 302, a display column 303 of the query image before modification, a display column 304 of the query image after modification, an inquiry sentence 305, a button 306 operated in the case of using the query image after modification for retrieval, and a button 307 operated in the case of not using the query image after modification for retrieval. In the case of Figure 10 , the label "OK" is attached to the button 306. Also, the label "NG" is attached to the button 307.

[0202] In the case of Figure 10 , the title bar 301 shows a summary of the content required of the user through the screen 300. Here, the user is required to "confirm the retrieval condition". The retrieval condition here includes both the premise information and the query image.

[0203] In the explanation 302, there is described the search condition in which a contradiction is found and the modification of the query image generated by the system side in order to eliminate the contradiction. In Figure 10 , it is described that "in accordance with the premise information that there is no garage, the input query image was modified as follows."

[0204] In the case of Figure 10 , the display field 303 of the query image before modification and the display field 304 of the query image after modification are arranged and displayed. By arranging and displaying the query image before and after modification, the modified part is easily confirmed. However, it is also possible to display only the query image after modification. The query image after modification is an example of the second query image.

[0205] In the case of Figure 10 , the building in which the garage is removed is displayed in the display field 304 of the query image after modification. However, the appearance of the building other than the garage maintains the structural feature.

[0206] However, the building displayed in the display field 304 is not only the image in which the garage circled by the frame line 303A is removed from the building before modification, but also the image in which the white outer wall that should exist below the roof is added. This is because, without adding the outer wall, the building becomes an unnatural appearance.

[0207] This modification is realized by the execution of the inference model that learns the database of the building or the processing according to the preprogrammed condition.

[0208] In the case of Figure 10 , below the display field 304 of the query image after modification, the inquiry sentence 305 is displayed to confirm to the user whether to use the query image after modification for the search.

[0209] In addition, Figure 10 the screen 300 illustrated is an example in the case where the premise information cannot be changed. For example, in the case where the user owns the land. Therefore, the buttons 306 and 307 are two alternatives of whether to use the query image after modification for the search.

[0210] Figure 11 is a drawing illustrating an example of another screen 300A that displays the query image after modification. In Figure 11 , the symbols corresponding to the corresponding parts of Figure 10 are illustrated.

[0211] The screen 300A is different from the screen 300 (refer to Figure 9 ) in the following points: the buttons for inputting the response to the inquiry sentence 305 are the button 306 labeled "OK" and the button 308 labeled "change premise information."

[0212] In Figure 11 the case of the screen 300A, the user is notified by the display of the button 308 that the premise information can be changed. The detection of the operation of the button 308 indicates that the affirmative result is obtained in step 12 (refer to Figure 7 ).

[0213] In addition, even in Figure 11 the case of the screen 300A, the button 307 (refer to Figure 10 ) labeled "NG" can be configured for the user who neither desires to change the premise information nor desires to modify the query image.

[0214] Example 2

[0215] Figure 12 is a drawing illustrating an example of the screen 300B that displays the modified query image. In Figure 12 , the symbols corresponding to the corresponding portions of Figure 10 are annotated to be illustrated.

[0216] Figure 12 The content of the query image of the screen 300B illustrated in Figure 10 is different from that of the screen 300 (refer to ).

[0217] In Figure 12 the query image illustrated, a building in a suburban area with a surplus of land is shown. Further, a large garage capable of parking two or more vehicles is provided next to the building. Also, in Figure 12 the query image illustrated, the land on the front side of the building is vacant. From the above, it can be estimated that the building included in Figure 12 the query image is built on a fairly wide land.

[0218] On the other hand, in Figure 12 , the land area specified in the premise information is exceptionally smaller than the estimated value of the land area extracted from the query image. Therefore, "land area" is shown in the description 302B as an item in which a contradiction is found.

[0219] In Figure 12 , since the contradiction between the premise information and the query image is the land area, the display field 303 of the query image input by the user is entirely circled with the frame line 303B.

[0220] Further, in Figure 12The display column 304 of the modified query image in the modification example 2 shows a building surrounded by a boundary wall. In addition, the size of the modified building is modified in accordance with the land area and the shape of the land shown in the premise information. Specifically, the lateral width of the building is narrowed. However, the design as a structural feature of the building is maintained.

[0221] Since Figure 13 The example shown in FIG. 6 allows the premise information to be changed, and thus the button 306 labeled "OK" and the button 307 labeled "NG" are displayed.

[0222] Example 3

[0223] Figure 13 is a diagram for explaining a modification example of a query image in a case where a building including a three-story building in the query image is allowed to have only a two-story building in the location specified in the premise information. Figure 13 (A) of FIG. 7 shows the query image before the modification, Figure 13 (B) of FIG. 7 shows the query image after the modification.

[0224] In the case of the modification example 2, Figure 13 In the case of the modification example 2,

[0225] In addition, in the modification of the query image, it is also possible to delete the three-story portion and install a roof on the two-story portion. However, one of the structural features of the query image before the modification is the number of windows. Therefore, in the case of the modification example 2, Figure 11 the number of windows of the three-story portion is maintained as a structural feature.

[0226] Other Embodiments

[0227] The embodiments of the present application have been described above, but the technical scope of the present application is not limited to the above-described embodiments. Various changes that can be made to the above-described embodiments based on the concept of the present application, which is defined in the claims, will be inferred as being included in the technical scope of the present application.

[0228] In the above-described embodiments, the image retrieval system that is assumed to be used in a planning studio or a design studio is exemplified, but the field of use is not limited to the construction industry as long as it is an image retrieval system that inputs a query image and text information. For example, it is also possible to be applied to web retrieval or document retrieval.

[0229] In the case of the image analysis section 121 in the foregoing embodiment, if the contents of the input query image are the same, the extracted structural features are the same regardless of the user's differences, but the extracted features can also differ for each user who is the performer of the search. For example, in the case where there is a tendency for each user to focus on a structural feature in the query image, the features extracted by the image analysis section 121 can also be changed according to the tendency. This is because a structural feature that the user focuses on with a high tendency has a large influence on the search result.

[0230] In the foregoing embodiment, structural features were extracted from a query image that represents the appearance of a building, but in the case of a query image that represents the interior of the same building, the setting can also be made to extract structural features that are different from those of a query image that represents the appearance of a building. In other words, the extracted structural features can also be set according to the kind of content included in the query image. According to this setting, unique features are extracted for each kind of content included in the query image, and it is easy to find contradictions.

[0231] In the foregoing embodiment, in the case where a contradiction is found between the structural features extracted from the query image and the premise information, the user is notified of the existence of the contradiction, but the query image can also be modified without notifying the user of the contradiction.

[0232] In the foregoing embodiment, the query image that is modified according to the premise information to resolve the determined contradiction is presented to the user before the search is performed, but the search can also be started without presenting the query image to the user.

[0233] In the foregoing embodiment, in the case where the premise information associated with the determined contradiction can be changed, the user is provided with an opportunity to change the premise information, but such an opportunity can also not be provided.

[0234] In the case of the foregoing embodiment, the pre-processing section 102 does not perform modification to remove contradictions between both or one of the structural information text and the emotional information text, but both or one of the structural information text and the emotional information text can also be included in the contradiction removal target in the pre-processing section 102.

[0235] In the foregoing embodiment, the button 308 that accepts a change in the premise information in the case where the premise information can be changed is displayed (refer to ​ ), but a function that prompts the input of another query image according to the contents of the contradiction can also be provided. Also, in the case where the condition input by the user contradicts a law or regulation, the law or regulation can also be given priority.

[0236] In addition, the processor in each of the aforementioned embodiments refers to a broad processor, and includes not only a general-purpose processor (for example, a CPU (= Central Processing Unit) or the like) but also a dedicated processor (for example, a GPU (= Graphical Processing Unit), an ASIC (= Application Specific Integrated Circuit), an FPGA (= Field Programmable Gate Array), a program logic device, or the like).

[0237] Also, the operation of the processor in each of the aforementioned embodiments can be performed by one processor alone, but can also be performed in cooperation with a plurality of processors existing at physically separated locations. Also, the order of execution of each operation in the processor is not limited only to the order described in each of the aforementioned embodiments, and can be changed individually.

Claims

1. A query modification system having a processor, In a case where a portion that contradicts the first condition is included in the first query image input by the user and the portion does not satisfy a second condition that is specified in advance, the processor generates a second query image by modifying the portion in accordance with the first condition, the first condition is a condition associated with a search target, the processor does not modify a portion in the first query image that contradicts the first condition and that satisfies the second condition.

2. The query modification system according to claim 1, wherein the first condition is used as a search query.

3. The query modification system according to claim 2, wherein the first condition is associated with a feature extracted from the first query image.

4. The query modification system according to claim 1, wherein the processor maintains a structural feature other than the portion in the first query image in a case where the portion is modified.

5. The query modification system according to claim 4, wherein the structural feature is determined in accordance with a category of content included in the first query image.

6. The query modification system according to claim 4, wherein the structural feature is set by each user.

7. The query modification system according to claim 1, wherein the second condition is a condition in which the portion that contradicts the first condition corresponds to a structural feature in the first query image.

8. The query modification system according to claim 1, wherein the processor notifies a user of information of the portion that contradicts the first condition before modifying the first query image in a case where the first query image includes the portion that contradicts the first condition and that does not satisfy the second condition.

9. The query modification system according to claim 1 or 8, wherein the processor asks the user whether to generate the second query image before modifying the first query image.

10. The query modification system according to claim 1 or 8, wherein the processor asks the user of a change of the first condition associated with the portion excluded from the modified object in a case where the portion that contradicts the first condition includes the portion excluded from the modified object.

11. The query modification system according to claim 1, wherein the processor asks the user whether to change the first condition corresponding to the contradiction in a case where the first condition can be changed.

12. The query modification system according to claim 1, wherein the processor prompts the user of the generated second query image.

13. The query modification system according to claim 12, wherein the processor arranges the second query image and the first query image on the same screen.

14. The query modification system according to claim 13, wherein the processor adds a mark indicating a modified portion to one or both of the first query image and the second query image.

15. The query modification system according to claim 12, wherein the processor asks the user of use of the generated second query image in a search before performing the search.

16. The query modification system according to claim 1, wherein the first condition includes a law or regulation.

17. The query modification system according to claim 16, wherein the processor prioritizes the law or regulation in the case where the user's input as the first condition contradicts the law or regulation.

18. A search system having a processor, In a case where a portion that contradicts the first condition is included in the first query image input by the user and the portion does not satisfy a second condition that is specified in advance, the processor generates a second query image by modifying the portion in accordance with the first condition, and searches the database using the generated second query image, wherein the first condition is a condition associated with a search target, in the case where a portion of the first query image that includes a contradiction with the first condition and that satisfies the second condition is not modified, the first query image is used to search a database.

19. A storage medium storing a program that causes a computer to function as: accepting a first query image input by a user; and In a case where the portion which is included in the first search image and which is contradictory to the first condition does not satisfy a second condition which is predetermined, a second search image is generated by modifying the portion according to the first condition, the first condition is a condition associated with a search target, and in the case where a portion of the first query image that includes a contradiction with the first condition and that satisfies the second condition is not modified.

20. A computer program product including a program that causes a computer to function as: accepting a first query image input by a user; and In a case where the portion which is included in the first search image and which is contradictory to the first condition does not satisfy a second condition which is predetermined, a second search image is generated by modifying the portion according to the first condition, the first condition is a condition associated with a search target, and in the case where a portion of the first query image that includes a contradiction with the first condition and that satisfies the second condition is not modified.

Images