Document searching system

Abstract

To retrieve a character string or an image in a PDF file at higher speed.SOLUTION: An in-document retrieval system 1 includes a database construction unit 11 that extracts character strings included in a document realized as a PDF file prior to retrieval and constructs a database 40 in which each of the extracted character strings and position information of the character string in the document are registered in association with each other outside the document, a retrieval query input reception unit 12 that receives an input of a retrieval character string, and a retrieval unit 13 that retrieves a character string matching or related to the retrieval character string from the database 40, extracts position information in the document associated with the retrieved character string, and presents content of the retrieved character string and the position information of the retrieved character string in the document in association with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a document search system. [Background technology]

[0002] Portable Document Format (PDF) files have traditionally been widely used across a wide range of business fields. However, PDF files are a file format established for typesetting, and are not suitable for searching the contents of files, making them less searchable. While viewers for PDF files are commercially available, searching for a string of characters within a PDF file using such a viewer can take a significant amount of time if the PDF file is large. In this regard, Patent Document 1 discloses an information processing device that includes a setting file storage means that stores document type information in association with a setting file that describes item names and location information of the item values ​​that indicate input areas for the item values ​​of each of the item names; a form generation means that acquires the setting file associated with the document type information from the setting file storage means based on a PDF file that includes the document type information and generates a form using the acquired setting file; a form assignment means that assigns the form generated by the form generation means to the PDF file; and an item value extraction means that uses the form to extract item values ​​for each item name from the PDF file and stores the extracted item values ​​in the form.

[0003] However, in Patent Document 1, the form generated based on the PDF file is stored in the PDF file itself, so when searching for a character string in the PDF file, it is ultimately necessary to access the PDF file each time. Therefore, even if Patent Document 1 reduces the time required for searching when the PDF file size is large, for example, the amount of reduction is considered to be limited. Furthermore, commercially available viewers for viewing PDF files, such as those mentioned above, basically only support searching for character strings, and are unable to search for images within PDF files. It is desirable to search for text or images in PDF files more quickly. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7346760 Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a document search system that can quickly search for character strings or images in a PDF file. [Means for solving the problem]

[0006] The present invention provides an in-document search system that, when a search string is input as a search query, searches a document realized as a PDF file for a string that matches or is related to the search string, the in-document search system comprising: a database construction unit that extracts the strings included in the document prior to the search and constructs a database outside the document in which each of the extracted strings is associated with location information of the string in the document, an input acceptance unit that accepts input of the search string, and a search unit that searches the database for strings that match or are related to the search string, extracts the location information in the document that is associated with the searched string, and presents the content of the searched string in association with the location information of the searched string in the document. According to the above configuration, the database construction unit extracts character strings contained in a document realized as a PDF file prior to a search, and then constructs a database outside the document in which each extracted character string is registered in association with its location information within the document. When actually searching for a character string in a document, the input accepting unit accepts input of the search character string as a search query. The searching unit searches a database constructed outside the document for a character string that matches or is related to the search character string. As a result, the searching unit extracts location information in the document that is associated with the character string found in the database as matching or related to the search character string. The extracted location information indicates the location of the searched character string that is associated with the searched character string. Therefore, the searching unit can present the content of the searched character string and the location information in the document in association with each other, for example, by highlighting the portion of the document that is indicated as the location information. In the above series of processes, information about character strings in a document is stored in a database constructed outside the document, and searches for character strings in the document are performed in the database constructed outside the document, rather than targeting the document. In this way, because each process is not performed on the document, character string searches can be performed more quickly even in cases where a search would take a long time if the document were the target, such as when the document file size is large. In this way, it becomes possible to search for character strings in PDF files more quickly.

[0007] In one aspect of the present invention, the database construction unit extracts character objects contained in the document, and when a second character object is located within a search range set for one of the character objects, extracts the character string by repeatedly concatenating the second character object to the first character object, and the search range is set so that it is centered on the first character object, extends in the direction in which the character object following the first character object is expected to appear, and gradually becomes wider as it moves away from the first character object. When a document contains text that is scattered throughout the document as individual character objects for each character that makes up the text, it may not be possible to search it using commercially available browsing software, for example. Even if a search were possible, the search speed may be slow because the connections between the character objects must be evaluated each time a search is performed to determine whether multiple consecutive character objects match or are related to the search string. In contrast, with the above configuration, a search range is set for character objects that are individually scattered throughout a document. The search range is set so that it is centered on one character object, extends in the direction in which another character object following the one character object is expected to appear, and gradually widens as it moves away from the one character object. If a second character object is located within this search range, the second character object is concatenated to the first character object. By repeating this process, a concatenated character string can be extracted. Therefore, even if character objects are individually scattered throughout a document, it is possible to extract connected character strings from the character objects. Furthermore, the concatenation process described above only needs to be performed once when constructing the database. In other words, when searching for a character string, it is sufficient to refer to the database in which the concatenation process has been completed and the results have been registered, and since there is no need to perform the concatenation process on each document, character string searches can be performed quickly.

[0008] In another aspect of the present invention, the database construction unit performs morphological analysis on each of the extracted character strings to generate substrings, and associates each of the generated substrings with the location information of the substring in the document and registers it in the database. The character string extracted from a document by the database construction unit may be long. In this case, when a search is performed, the long character string may be searched as a related character string, for example, because it partially matches the search character string. In this case, for example, when the search unit associates the content of the searched character string with location information in the document and presents the associated content, the long character string extracted from the document is presented in association with the search character string itself, rather than the search character string itself. Therefore, for example, when this associated presentation highlights a portion indicated as location information, the long character string is highlighted. Therefore, in order to identify which part of the highlighted long character string the search character string is in, the user must visually search for the search character string. In contrast, with the above configuration, the database construction unit generates meaningful substrings, such as parts of speech, for character strings extracted from a document by morphological analysis, and associates the generated substrings with location information of the substrings and registers them in the database. Thus, the character strings extracted from the document are broken down into smaller units, such as parts of speech, and the character strings and their location information are associated and registered in the database. Therefore, when a long character string is extracted from a document and registered in the database by the database construction unit, if a search is performed using a substring generated by morphological analysis and contained in the long character string as a search string, the substring is extracted by the search, and location information in the document corresponding to the substring is presented in association with the substring. In this way, the location information presented as search results is more precisely limited to the shorter substrings after morphological analysis has been applied, rather than the string before morphological analysis has been applied, allowing for a more precise understanding of the location of the search string.

[0009] In another aspect of the present invention, the search unit determines that the character string registered in the database is related to the search character string if the search character string partially matches the character string registered in the database, or if either the Levenshtein distance or the Jaro-Winkler distance between the character string registered in the database and the search character string is equal to or less than a similar character string determination threshold. According to the above configuration, it is possible to appropriately search for character strings related to a search character string.

[0010] In another aspect of the present invention, the database is provided individually for each of the plurality of documents, and the search unit searches the database for each of the plurality of documents for a character string that matches or is related to the search character string, extracts the location information within the document that is associated with the searched character string, and displays the content of the searched character string, the identification information of the document that contains the character string, and the location information within the document in association with each other. According to the above configuration, a search is performed on a database constructed outside the document, so that even when searching multiple documents, a search for a character string can be performed more quickly.

[0011] The present invention also provides an in-document search system that, when a search image is input as a search query, searches for images in a document realized as a PDF file that match or are related to the search image, the in-document search system comprising: a database construction unit that, prior to the search, extracts the images included in the document and constructs a database outside the document in which, for each of the extracted images, image features, which are features generated based on the image, are associated with location information of the image in the document; an input acceptance unit that accepts input of the search image; and a search unit that searches the database for image features that match or are related to search image features, which are features generated based on the search image, extracts the location information in the document that is associated with the searched image features, and presents the content of the image corresponding to the searched image features in association with the location information in the document. According to the above configuration, the database construction unit extracts images contained in the document realized as a PDF file prior to searching, and then constructs a database outside the document in which, for each extracted image, image features generated based on the image are associated with the image's location information within the document. When actually searching for an image in a document, the input accepting unit accepts input of a search image as a search query. The searching unit searches a database constructed outside the document for image features that match or are related to search image features, which are features generated based on the search image. As a result, the searching unit extracts location information in the document associated with the image features found in the database as matching or related to the search image features. The extracted location information indicates the location of the image that is the basis of the searched image features and is associated with the searched image features. Therefore, the searching unit can, for example, highlight the portion of the document indicated as the location information, thereby displaying the content of the image corresponding to the searched image features in association with the location information in the document. In the above series of processes, information about images in a document is stored in a database constructed outside the document, and a search for images in the document is performed on the database constructed outside the document, rather than on the document itself. In this way, because each process is not performed on the document itself, image searches can be performed more quickly even in cases where a search on the document itself would take a long time, for example, because the document file size is large. In this way, images within PDF files can be searched for more quickly by entering the image as a search query.

[0012] In one aspect of the present invention, the search unit searches for image features that match or are related to the search image features by extracting, from the image features registered in the database, image features whose distance as an index of similarity between them and the search image features is less than or equal to a similar image determination threshold. According to the above configuration, it is possible to appropriately search for images.

[0013] The present invention also provides an in-document search system that, when a search string is input as a search query, searches for images related to the search string within a document realized as a PDF file, the system comprising: a database construction unit that, prior to the search, extracts the images included in the document, generates captions corresponding to each of the extracted images, extracts features from the captions to create caption features, and constructs a database outside the document in which the captions, the caption features, and location information of the images within the document are associated and registered; an input acceptance unit that accepts input of the search string; and a search unit that searches the database for captions that match or are related to the search string, or caption features that match or are related to search string features that are features generated based on the search string, extracts the location information within the document that is associated with the found caption or caption feature, and displays the content of the image corresponding to the found caption or caption feature in association with the location information within the document. According to the above configuration, prior to a search, the database construction unit extracts images contained in a document realized as a PDF file, generates a caption for each extracted image, extracts features from the caption to create caption features, and constructs a database outside the document in which the captions, caption features, and location information of the images in the document are registered in association with each other. When actually searching for images in a document, the input accepting unit accepts input of a search string as a search query. The searching unit searches a database constructed external to the document for captions that match or are related to the search string, or for caption features that match or are related to search string features, which are features generated based on the search string. As a result, the searching unit extracts location information in the document associated with captions found in the database as matching or related to the search string, or caption features found in the database as matching or related to the search string features. The extracted location information indicates the location of the image associated with the found caption or caption feature. Therefore, the searching unit can, for example, highlight the portion of the document indicated as the location information, thereby displaying the content of the image corresponding to the found caption or caption feature in association with the location information in the document. In the above series of processes, information about images in a document is stored in a database constructed outside the document, and a search for images in the document is performed on the database constructed outside the document, rather than on the document itself. In this way, because each process is not performed on the document itself, image searches can be performed more quickly even in cases where a search on the document itself would take a long time, for example, because the document file size is large. In this way, images in PDF files can be searched for more quickly by entering a string (text, natural language) as a search query.

[0014] In one aspect of the present invention, the document is an architectural document or a drawing drawn for construction. Architectural documents or drawings drawn for architecture contain various images, diagonally arranged text, circularly arranged text, etc., and can have a huge number of pages. For this reason, directly searching through drawings converted into PDF files takes time, and text with special layouts may not be searchable. According to the above configuration, it is possible to quickly search for special layouts and images contained in architectural documents or drawings drawn for construction. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a document search system that can search for character strings and images in PDF files at higher speed. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a schematic configuration of a document search system according to an embodiment of the present invention; [Figure 2] 10 is a flowchart of a database construction method using the document search system of the embodiment. [Figure 3] 10 is a flowchart of a database construction method using the document search system of the embodiment. [Figure 4] FIG. 10 is a conceptual diagram for explaining a search range when searching for a concatenated character string. [Figure 5] FIG. 10 is a conceptual diagram for explaining a method for searching for a concatenated character string. [Figure 6] FIG. 10 is a conceptual diagram for explaining a method for searching for a connected character string having an inclination. [Figure 7] FIG. 10 is a conceptual diagram for explaining a method for searching for a connected character string having an inclination. [Figure 8] 10 is a flowchart of a character string search method performed by the document search system of the embodiment. [Figure 9] 10 is a flowchart of an image search method (when the search query is an image) performed by the document search system of the embodiment. [Figure 10] 10 is a flowchart of an image search method (when the search query is a character string) performed by the document search system of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. A block diagram showing a schematic configuration of a document search system according to an embodiment of the present invention is shown in Fig. 1. The document search system 1 functionally comprises a document input accepting unit 10, a database constructing unit 11, a search query input accepting unit (input accepting unit) 12, and a search unit 13. The document input receiving unit 10 receives document input from a user. The input document is a PDF file.

[0018] The database construction unit 11 constructs a database based on the input documents. The database construction unit 11 includes a character string DB creation unit 20, an image DB creation unit 30, and a database (DB) 40. In this embodiment, the database 40 is constructed as a relational database. The database 40 is not constructed to be included in the document, but is constructed outside the document, for example, as a file separate from the document.

[0019] Prior to the search, the character string DB creation unit 20 extracts character strings contained in the document input from the document input acceptance unit 10, and registers each of the extracted character strings in the database 40 in association with the position information of the character string in the document (page number and coordinates on the page). The character string DB creation unit 20 includes a character object extraction unit 21, a concatenated character string extraction unit 22, and a word / phrase extraction unit 23. The character object extraction unit 21 extracts a character object as a single character included in a document. The concatenated character string extraction unit 22 analyzes whether the extracted character objects can be concatenated with each other based on the coordinates of the extracted character objects on the page, determines that the character objects are concatenable if predetermined concatenation conditions are satisfied, and concatenates the character objects to generate a concatenated character string. The concatenated character string extraction unit 22 associates the generated concatenated character string with the character string's position information in the document (page number and coordinates on the page) as a character string contained in the document and registers the character string in the database 40. Note that concatenable character objects are two character objects that exist within a predetermined distance from each other (a search range S described below) and that can be included together in a single sentence. The word / phrase extraction unit 23 performs morphological analysis on the concatenated character string generated, divides it into substrings such as words and phrases, and registers the substrings divided as words and phrases in the database 40 as character strings contained in the document, in association with the position information of the character string (substring) in the document (page number and coordinates on the page).

[0020] Prior to searching, the image DB creation unit 30 extracts images contained in the document input from the document input acceptance unit 10, and registers in the database 40 image features (feature vectors) that are features generated based on the extracted images, captions generated from the contents of the images, caption features (embedded vectors) generated by extracting features from the captions, and location information of the images in the document (page number and coordinates within the page) in association with each other. The image DB creation unit 30 includes an image extraction unit 31, an image feature amount generation unit 32, and a caption generation unit 33. The image extraction unit 31 extracts an image contained in a document, and obtains the content of the image and the position information of the image within the document.

[0021] The image feature generation unit 32 calculates image features that represent the characteristics of the extracted image from the content of the image, and associates the image features with the position information of the image within the document and registers them in the database 40. In this embodiment, the image features are image feature vectors obtained through a trained model for extracting image features, such as deep convolutional neural networks (CNNs), that converts an image into a vector of fixed dimension. The image feature generation unit 32 generates an image feature vector using the image feature extraction trained model described above, and registers this as an image feature in the database 40 in association with the position information of the image within the document.

[0022] The caption generation unit 33 uses a caption generation (Image Captioning) technique to generate natural language sentences describing the characteristics of each image for the content of the extracted images, i.e., captions. For this caption generation, a deep learning-based large language model (LLM) such as Bootstrap Language Image Pre-training (BLIP) that receives multimodal input and generates natural language is preferably used. However, any other method may be used, such as a deep learning model using CNN+Long Short-Term Memory (LSTM) or the results of manual captioning. The caption generation unit 33 inputs each of the extracted images into the above-described caption generation trained model to generate captions. Furthermore, the caption generation unit 33 extracts features from the generated captions to generate caption features. In this embodiment, the caption generation unit 33 applies the captions generated by the caption generation trained model to a caption feature extraction trained model constructed by LLM or the like to convert the captions into feature vectors (embedding vectors) reflecting the features of the captions, and uses these as caption features. The caption generation unit 33 associates, for each image, the image features associated by the image feature generation unit 32 with the position information of the image in the document, as well as the caption of the image with the caption features extracted and generated from the caption, and registers them in the database 40.

[0023] The search query input receiving unit 12 receives input of a search character string and a search image as a search query from a user.

[0024] The search unit 13 includes a search execution unit 50 and a search result presentation unit 51 . When searching for a character string, when a search character string is input as a search query from search query input receiving unit 12, search execution unit 50 searches database 40 for character strings that match or are related to the search character string, extracts location information within a document that is associated with the character string found in database 40 as matching or related to the search character string, and outputs the location information within the document to search result presentation unit 51. At the same time, search execution unit 50 outputs the content of the character string found as matching or related to the search character string to search result presentation unit 51. The search result presentation unit 51 receives the contents of the searched character strings and the position information of the character strings in the document from the search execution unit 50. The search result presentation unit 51 presents the correspondence to the user as a list 60 so that the contents of the searched character strings and the position information of the character strings in the document are displayed in association with each other. In this case, when the user selects any correspondence relationship in the list 60, the search result presentation unit 51 opens the document 61 and, for example, highlights and displays the portion of the document 61 that corresponds to the location information of the selected correspondence relationship, thereby displaying the content of the searched character string and the location information of the character string in the document 61 in association with each other as the search result. In this way, the search result presentation unit 51 presents the search result to the user.

[0025] Furthermore, when searching for an image, the search execution unit 50 inputs a search image as a search query from the search query input receiving unit 12, and generates search image features by inputting the search image to the same image feature extraction trained model used when extracting image features in the image feature generation unit 32. The search execution unit 50 uses a technique such as a neighbor vector search to search the database 40 for image features whose distance as an index of similarity to the search image features is equal to or less than a predetermined similar image determination threshold, i.e., image features that match or are related to the search image features, extracts location information within a document associated with the image features found in the database 40 as matching or related to the search image features, and outputs the location information within the document to the search result presentation unit 51. At the same time, the search execution unit 50 acquires an image corresponding to the image features found as matching or related to the search image features based on the location information within the document of the image, and outputs the content of the image to the search result presentation unit 51 as the content of the searched image. The search result presentation unit 51 receives the contents of the searched images and the location information of the images in the document from the search execution unit 50. The search result presentation unit 51 presents the correspondence to the user as a list 60 so that the contents of the searched images and the location information of the images in the document are displayed in association with each other. In this case, when the user selects any correspondence relationship in the list 60, the search result presentation unit 51 opens the document 61 and, for example, highlights and displays the portion of the document 61 that corresponds to the location information corresponding to the selected correspondence relationship, thereby displaying, as a search result, the content of the image corresponding to the searched image feature (the content of the searched image) in association with the location information of the image in the document 61. In this way, the search result presentation unit 51 presents the search results to the user.

[0026] Furthermore, when searching for an image, when a search string is input as a search query from search query input receiving unit 12, search execution unit 50 searches database 40 for captions that match or are related to the search string, extracts location information within the document associated with the caption found in database 40 as matching or related to the search string, and outputs the location information within the document to search result presentation unit 51. At the same time, search execution unit 50 obtains an image corresponding to the caption found as matching or related to the search string based on the location information within the document of the image, and outputs the content of the image to search result presentation unit 51 as the content of the searched image. Furthermore, the search execution unit 50 generates search string features by inputting the search string into the same caption feature extraction trained model used when the caption generation unit 33 extracted the caption features. The search execution unit 50 searches the database 40 for caption features whose distance as an index of similarity to the search string features is equal to or less than a predetermined threshold, i.e., for caption features that match or are related to the search string features, extracts location information within the document associated with the caption features found in the database 40 as matching or related to the search string features, and outputs the location information within the document to the search result presentation unit 51. At the same time, the search execution unit 50 obtains images corresponding to the caption features found as matching or related to the search string features based on the location information within the document of the images, and outputs the content of the images to the search result presentation unit 51 as the content of the searched image. The search result presentation unit 51 receives the contents of the searched images and the location information of the images in the document from the search execution unit 50. The search result presentation unit 51 presents the correspondence to the user as a list 60 so that the contents of the searched images and the location information of the images in the document are displayed in association with each other. In this case, when the user selects any correspondence relationship in the list 60, the search result presentation unit 51 opens the document 61 and, for example, highlights and displays the portion of the document 61 that corresponds to the location information corresponding to the selected correspondence relationship, thereby displaying, as a search result, the content of the image corresponding to the retrieved caption or caption feature (the content of the retrieved image) in association with the location information of the image in the document 61. In this way, the search result presentation unit 51 presents the search results to the user.

[0027] The above-described document search system 1 is composed of a computer device, and hardware including a CPU, memory, hard disk drive, and other storage devices constituting the computer device cooperates with a computer program pre-installed in the computer device to execute predetermined processing. The document search service provided by the document search system 1 according to this embodiment may also be implemented as a web application provided via a network (such as the Internet) to run on a web browser launched on a user's terminal. In this case, for example, a user accesses a web page provided by the document search system 1 via a network (such as the Internet) using a user terminal (not shown), and drags and drops a document to be searched into a predetermined window on the web page (corresponding to the document input accepting unit 10). When the document is dragged and dropped into the predetermined window, the document search system 1 extracts character strings and images contained in the document and builds a database 40, as described above. Once the database 40 is constructed, the user inputs a search string or a search image as a search query in a predetermined window of a predetermined web page (corresponding to the search query input accepting unit 12). When the search string or search image is input in the predetermined window, the document search system 1, as described above, searches the database 40 for strings and images that match or are related to the search string or search image, and outputs the strings and images that match or are related to the search string or search image in association with location information within the document. The user terminal displays a list 60 of the searched strings and images and their location information within the document as search results. When the user selects (specifies) a desired search result from the list of search results, the search result (string or image) is highlighted (for example, highlighted) on the document 61 on the user terminal (corresponding to the search result presentation unit 51).

[0028] Hereinafter, the flow of processing in the document search system 1 will be explained, and each component included in the document search system 1 will be explained in detail. 2 and 3 are flowcharts for explaining the operation of the database construction process by the document search system 1. FIG. 3 is a diagram following FIG. The document input receiving unit 10 receives document input from a user (step S10). This document may include a PDF file that has only been scanned from a paper medium and has no text objects placed on it, a PDF file that has been scanned in advance and then processed with OCR (optical character recognition) to place text objects on it, or a PDF file that has been output from a predetermined application with text objects placed on it. Therefore, the database construction unit 11 determines whether all pages of the input document (PDF) have been OCR processed (whether text objects are placed on them) (step S12). If the document has not been OCR processed (NO in step S12), the database construction unit 11 performs preprocessing by performing OCR processing on pages of the document that have not been OCR processed, and placing a transparent string of characters at the position of the characters on the image so that the parts of the document that correspond to the characters can be accessed as character objects (step S14).

[0029] Next, the database creator 11 determines whether or not the OCR process has been completed for all pages of the document (step S16). When the OCR process is completed for all pages of the document (YES in step S16), the database constructor 11 performs a character object connection analysis process to convert the character objects that are individually and scattered throughout the document into coherent character strings such as words or sentences. The connection analysis process analyzes whether the character objects are considered to be part of a continuous string of characters based on the positional relationship (the distance between characters) of the character objects that are individually and scattered, that is, arranged separately, and if they are considered to be part of a continuous string of characters, connects the character objects to extract a single continuous string of characters. The connection analysis process is implemented as follows.

[0030] The character object extraction unit 21 first determines whether or not a character object is included in the page of the document (step S18). If a character object is not included in the page (NO in step S18), the page is skipped and the process moves to the next page (NO in step S30, return to step S18). On the other hand, if a character object is included in the page (YES in step S18), the character object extraction unit 21 extracts the character object included in the page of the document (step S20). Next, the concatenated character string extraction unit 22 analyzes whether the character objects can be connected to each other based on the coordinates of the extracted character objects on the page (concatenation analysis process), and if they can be connected, extracts those character objects as a single continuous string of characters, i.e., a concatenated character string (step S22). The concatenated character string extraction unit 22 registers the concatenated character string as a character string extracted from the document in the database 40 in association with the position information of the concatenated character string in the document (step S24).

[0031] Here, a process of analyzing whether character objects can be connected to each other and extracting connected character strings will be described. FIG. 4 is a conceptual diagram for explaining the search range when searching for a concatenated character string. In the concatenation analysis process, most technical documents are written in languages ​​that are written horizontally from left to right (LTR), and most Japanese text is written in either LTR or vertical format. Therefore, as shown in Figure 4, a fan-shaped search range S is set from the center of the character object on the page being analyzed (hereinafter referred to as the target page) to the right of the page if the text is written horizontally, or to the bottom of the page if the text is written vertically. Among the character objects within this fan-shaped search range S, the character object closest to and located to the character object for which search range S is set (i.e., the leftmost character object for horizontal writing, or the topmost character object for vertical writing) is selected as the character to be concatenated with the character object for which search range S is set. Then, two characters are concatenated in order: the character object for which search range S is set and the character object closest to and located to the character object for which search range S is set. Repeating this process generates a concatenated string.

[0032] FIG. 5 is a conceptual diagram for explaining a method for searching for a concatenated character string. For example, as shown in FIG. 5( a), assume that character objects "A," "Ru," "Mi," "Ni," "U," and "Mu" are each arranged as individual character objects on a document. The concatenated character string extraction unit 22 first selects the character object "A" located at the top left of the target page and sets a fan-shaped search range S with a length r that extends from the center of the character object "A" to the right. The search range S is set so that it is centered on the character object and extends in the direction where the next character object following the character object is expected to appear, and gradually widens as it moves away from the character object. Of the character objects present inside this fan-shaped search range S, the leftmost character object "Ru" is concatenated as a character concatenated with the character object "A" to generate a concatenated character string. In the illustrated example, the concatenated character string at this point is "Aru." This process is repeated until no characters are found in the concatenation direction, and the individually arranged character objects are converted into a concatenated character string. In the example shown in Figure 5(a), a single connected string of characters, "aluminum," is generated from the individually arranged character objects "a," "ru," "mi," "ni," "u," and "mu."

[0033] In the example shown in FIG. 5(b), the character objects "A," "Ru," "Mi," "Ni," "U," and "Mu" are arranged diagonally. Similarly, in this situation, a search range S is first set to the right starting from "A," and it is determined whether the adjacent character object "Ru" is connected to "A." Since the adjacent character object "Ru" is included in the search range S, "Ru" is connected to "A." Similarly, by continuing "Ru," "Mi," "Ni," "U," and "Mu," "Aluminum" is formed into a single connected character string. In this way, by setting the search range S so that its width gradually increases with increasing distance from the character objects, even character strings arranged diagonally can be extracted as a single connected character string. Typically, in technical documents, strings of characters, including Japanese characters, are often written horizontally in LTR format. Therefore, it is preferable to first perform a horizontal concatenation analysis, and then perform a vertical concatenation analysis on character objects for which concatenation as a string is not confirmed as a result, but this is not limited to this.

[0034] 6 and 7 are conceptual diagrams for explaining a method for searching for a connected character string having an inclination. The direction in which the fan-shaped search range expands takes into account the angle of rotation when the character object is rotated at an angle, as shown in Fig. 6. To take into account the angle θ of rotation of the character object, the search range S(θ;φ) is defined as follows:

number

[0035] Of the character objects present within the search range S as described above, the one with the smallest Euclidean distance to (x, y) is concatenated as the next character in the character string, and that character object is used as the next starting point to repeatedly extract character strings until there are no more concatenated character objects. By repeating this process until there are no more unconcatenated character objects within the search range S of all character objects and it becomes impossible to continue the concatenation process, character strings present in documents such as drawings with complex layouts can be extracted.

[0036] In the example shown in FIG. 7, the character objects "A," "Ru," "Mi," "Ni," "U," and "Mu" are arranged diagonally, each at an angle θ. Even in this state, as in the above, a search range S is set starting from "A" in a direction based on its inclination, and it is determined whether the adjacent character object "Ru" is connected to "A." Since the adjacent character object "Ru" is included in the search range S, "Ru" is connected to "A." Similarly, by continuing "Ru," "Mi," "Ni," "U," and "Mu," "Aluminum" is formed into a single connected character string. In this way, even when the character objects are arranged diagonally and at an angle, they can be extracted as a single connected character string.

[0037] The concatenated character string extracted by the above process is extracted as the character string with the longest concatenated length among the character strings present in the document. Therefore, for example, when searching for a certain search string, this long character string may be found as a character string containing the search string, for example, by searching for a partial match. In particular, in this case, even if the search result is displayed in an emphasized manner, it may not be immediately clear at a glance where in the long character string the search string actually entered as the search target is located. In particular, in documents written in Japanese, in order to precisely and accurately identify search results in units of small character strings such as words, in this embodiment, the extracted connected character string is divided into smaller sub-character strings, and the divided sub-character strings are registered as character strings in the database 40. In particular, in this embodiment, word segmentation based on morphological analysis is performed on the extracted connected character string, and the connected character string is divided into sub-character strings such as words and phrases.

[0038] The word / phrase extraction unit 23 performs morphological analysis on the concatenated character string, divides it into partial character strings such as words and phrases (step S26), and registers the contents of the partial character strings divided into words and phrases in association with the position information of the partial character strings of the words, phrases, etc. in the document in the database 40 (step S28). If the division into substrings of words and phrases results in a sequence of multiple nouns, the resulting sequence is generally registered in the database 40 as a single noun. For example, if a page of a document contains the string "PDF files are extremely widely used regardless of the business field," the concatenated string extraction unit 22 extracts "PDF files are extremely widely used regardless of the business field" as a single concatenated string. Next, the word and phrase extraction unit 23 performs morphological analysis on the concatenated string to divide it into strings such as "PDF," "file," "business," "field," "regardless," "extremely," "widely," and "used," and extracts them as substrings of words and phrases. However, since "PDF" and "file" are words formed by combining two or more nouns, they are treated as "PDF file," and similarly, "business" and "field" are treated as "business field." Therefore, for the above example sentence, the nouns "PDF file," "business field," and "document file format" are ultimately extracted as substrings. However, the present invention is not limited to this, and as described above, the results of the morphological analysis divided into individual noun phrases (in the above example, "PDF," "file," "business," and "area") may be registered in database 40. Alternatively, both the concatenation of consecutive nouns and the results of the morphological analysis may be registered in database 40.

[0039] In the document search system 1 according to this embodiment, the concatenated character string itself extracted by the concatenated character string extraction unit 22 before being divided into partial character strings such as words and phrases is also stored in the database 40. In this way, even if the word / phrase extraction unit 23 fails to extract partial character strings such as words and phrases, and a character string that is linguistically correct is not registered in the database 40, and therefore the small unit character string that should be generated by morphological analysis is not found in the database 40 when searching for a character string, the concatenated character string before being divided can at least be found. This makes it possible to prevent a situation in which a character string matching the search character string is not found due to a failure of morphological analysis, even though it is present in the document.

[0040] Next, the character string DB creation unit 20 determines whether processing has been completed for all pages (step S30), and if processing has not been completed for all pages (NO in step S30), returns to step S18 and repeats the above-mentioned processing for the next page. As a result, by performing a concatenation analysis process and a morphological analysis on the character strings placed on all pages of the document (PDF), the concatenated character strings and each of the substrings generated from the concatenated character strings are stored as character strings in the database 40.

[0041] When database 40 is realized as a relational database, the attributes included in the string table (table name: texttable), which is a table related to strings and created by string DB creation unit 20, can be set, for example, as follows: id: String identification number page : The page number that contains the string text : the contents of the string top-left: The coordinates of the top left corner of the smallest rectangle that circumscribes the text (hereafter referred to as the circumscribed rectangle) on the page bot-left : The coordinates of the bottom left corner of the bounding rectangle on the page bot-right: The coordinates of the bottom right corner of the bounding rectangle on the page top-right: The coordinates of the top right corner of the bounding rectangle within the page. The character string DB creation unit 20 associates each extracted character string with position information of the character string in the document and registers them in the database 40. In this embodiment, the position information of the character string in the document is the page number in the document and coordinates within the page. In the above example, "text" corresponds to the extracted character string, "page" corresponds to the page number in the document, and "top-left," "bottom-left," "bottom-right," and "top-right" correspond to coordinates within the page. In this way, the character string DB creation unit 20 registers the coordinates of the four corners of the circumscribing rectangle that circumscribes the character string in the database 40. In this way, if the character string is provided at an angle within the document, and as a result the circumscribing rectangle is also at an angle, it becomes possible to represent this angle. This circumscribing rectangle is used later when a character string is searched for, by highlighting the area enclosed by the four corners and displaying it in a differentiated manner from the rest.

[0042] When processing has been completed for all pages (YES in step S30), the image extraction unit 31 first determines whether or not the page of the document contains an image (step S32). If the page does not contain an image (NO in step S32), the page is skipped and the process moves to the next page (NO in step S46, return to step S32).

[0043] On the other hand, if an image is included in the page (YES in step S32), the document search system 1 constructs the database 40 for image search according to the following procedure. The image extraction unit 31 extracts an image included in the document (step S34). When the image extraction unit 31 extracts a set of lines or shapes, such as those expressed as vector data, it generates an image by cutting out a portion corresponding to this set, and executes subsequent processes using the generated image as an image extracted from the document corresponding to the set. The image extraction unit 31 acquires the content of the image and the position information of the image (step S36). Next, the image feature generation unit 32 calculates image features representing the features of the image from the content of the image (step S38). Specifically, the image feature generation unit 32 inputs the image to the image feature extraction trained model as already described, and generates an image feature vector as the image feature.

[0044] The image feature generating unit 32 associates the image feature with the position information of the image in the document for each image, and registers them in the database 40 (step S40). When the database 40 is realized as a relational database, the attributes included in the image table (table name: imagetable), which is a table related to images and created by the image DB creation unit 30, can be set, for example, as follows: id: Image identification number page : The page number that contains the image imagefeature: image feature top-left: The coordinates of the top left corner of the smallest rectangle that circumscribes the image (hereafter referred to as the circumscribed rectangle) on the page bot-left : The coordinates of the bottom left corner of the bounding rectangle on the page bot-right: The coordinates of the bottom right corner of the bounding rectangle on the page top-right: The coordinates of the top right corner of the bounding rectangle within the page. In this embodiment, the position information of an image in a document is the page number in the document and the coordinates within the page. In the above example, "imagefeature" corresponds to the image feature of the extracted image, "page" corresponds to the page number in the document, and "top-left", "bottom-left", "bottom-right", and "top-right" correspond to the coordinates within the page. In this way, the image DB creation unit 30 registers the coordinates of the four corners of the circumscribing rectangle that circumscribes the image in the database 40. In this way, if the image is tilted within the document and as a result the circumscribing rectangle is also tilted, it becomes possible to represent this tilt. As with text strings, the bounding rectangle is used later when the image is searched for, to highlight the area enclosed by the four corners and distinguish it from the rest.

[0045] Next, the document search system 1 constructs a database 40 for searching for images using natural language in the following procedure. The caption generation unit 33 generates a natural language sentence (caption) describing each image using a caption generation technique for the content of the extracted image (step S42). Specifically, the caption generation unit 33 inputs the image into the caption generation trained model as already described to generate a caption. Next, the caption generation unit 33 extracts features from the generated caption to generate caption features (step S44). Specifically, the caption generation unit 33 inputs the caption to the caption feature extraction trained model as already described, to generate caption features.

[0046] Then, for each image, the caption generation unit 33 associates the image features associated by the image feature generation unit 32 with the position information of the image in the document, as well as the caption of the image with the caption features generated from the caption, and registers them in the database 40. Specifically, the caption generation unit 33 updates the database 40 by adding and registering, for example, the following two attributes to the image table imagetable described in the image feature generation unit 32 (step S46). caption : caption captionfeature:Caption feature

[0047] Thereafter, the image DB creation unit 30 determines whether or not the processing has been completed for all pages (step S48), and if the processing has not been completed for all pages (NO in step S48), the process returns to step S32 and repeats the above-described processing for the next page. On the other hand, if the processing has been completed for all pages (YES in step S48), the process ends.

[0048] FIG. 8 is a flowchart of a character string search method performed by the document search system of the embodiment. When a user inputs a search string as a search query from the search query input receiving unit 12 (step S50), the search execution unit 50 acquires the search criteria (exact match / partial match, etc.) (step S52) and searches for character strings in the database 40 according to the search criteria (step S54). If the search criteria indicate an exact match, the search execution unit 50 searches the database 40 for character strings that exactly match the search string. If the search criteria indicate a partial match, the search execution unit 50 searches the database 40 for character strings that partially contain the search string as character strings related to the search string. In addition, the search execution unit 50 may perform a search that takes ambiguity into consideration, such as extracting character strings whose distance between ambiguous character strings, such as the Levenshtein distance or the Jaro-Winkler distance, is equal to or less than a certain threshold for determining similar character strings, and then searches the database 40 for the extracted character strings that are similar to the search string as character strings related to the search string. For example, if "construction book" is entered as a search query and both the strings "construction book" and "This is a construction book" are stored in database 40, the search can be configured so that an exact match will search for only "construction book", and a partial match will search for both "construction book" and "This is a construction book".

[0049] Next, the search execution unit 50 outputs location information in the documents stored in the database 40 corresponding to the character strings found to match or be related to the search character string to the search result presentation unit 51 (step S56). At the same time, the search execution unit 50 outputs the contents of the character strings found to match or be related to the search character string to the search result presentation unit 51. The search result presentation unit 51 receives the contents of the searched character strings and the position information of the character strings in the document from the search execution unit 50. The search result presentation unit 51 presents the correspondence to the user as a list 60, for example, so that the contents of the searched character strings and the position information of the character strings in the document are displayed in association with each other (step S58). For example, when the user selects any correspondence relationship in the list 60, the search result presentation unit 51 opens the document 61 and displays, for example, in an highlighted manner, a portion of the document 61 that corresponds to the location information of the selected correspondence relationship. In this way, the search result presentation unit 51 presents the search result to the user by displaying the content of the searched character string and the location information of the character string in the document 61 in association with each other as the search result. In this embodiment, the searched character string is highlighted in the document by opening the document to the page number (page) registered in database 40 corresponding to the searched character string, and then coloring the part surrounded by the four corners (top-left, bottom-left, bottom-right, top-right) registered in database 40 corresponding to the searched character string in a different color from the other parts.

[0050] Here, as described above, if "construction books" is entered as a search query and both the strings "construction books" and "This is a construction book" are stored in database 40, both "construction books" and "This is a construction book" may be searched for. In such cases, it is possible that the bounding rectangle of one character string completely encompasses the bounding rectangle of the other character string. For example, in the above example, the bounding rectangle of "This is a construction document" completely encompasses the bounding rectangle of "Construction Document." In this embodiment, as described above, the searched character string is highlighted in a different color from the other parts based on the coordinates of the corners of the circumscribing rectangle. Therefore, in the above case, even though "construction documents" is searched for, "This is a construction document" which includes "construction documents" is also searched for, so not only "construction documents" but also "This is a construction document" is highlighted, and the highlighting for "construction documents" is hidden, resulting in the user being unable to identify the exact location of the searched character string. To prevent such a situation from occurring, in the above case, it is preferable to present to the user only search results (character strings) at smaller included positions among the search results.

[0051] Therefore, in the search process of step S54, if a search result indicates that a certain string has a circumscribing rectangle that encompasses the circumscribing rectangle of the searched string on each page, post-processing is performed to remove the larger, i.e., the encompassing, circumscribing rectangle from the search results. In the above example, "This is a construction document," whose circumscribing rectangle includes "construction document," is removed from the search results, leaving only "Business area" as the search result. This prevents "This is a construction document," whose circumscribing rectangle encompasses "construction document," from being highlighted when a search is performed using "construction document." This allows the user to more accurately identify the location of the search string.

[0052] Next, the search unit 13 determines whether or not the user has selected (specified) another search result from the list of search results (step S62), and if the user has selected (specified) another search result (YES in step S62), the search unit 13 returns to step S60 and repeats the above-mentioned process. On the other hand, if the user does not select (specify) another search result (NO in step S62), it is determined whether or not a new search has been instructed by the user (step S64), and if a new search has been instructed (YES in step S64), the process returns to step S50 and repeats the above-described process with the newly input search query (character string).On the other hand, if the user has not instructed a new search (NO in step S64), the process ends.

[0053] 9 is a flowchart of an image search method (when the search query is an image) by the document search system of the above embodiment. When an image is input as the search query, the image search is realized by the following procedure. When the user inputs a search image as a search query from the search query input receiving unit 12 (step S70), the search execution unit 50 inputs the search image into the same image feature extraction trained model that was used when extracting image features in the image feature generation unit 32, and calculates search image features, which are image features of the input search query (image) (step S72). Next, the search execution unit 50 calculates the distance as an index of similarity between the search image features and the image features stored in the database 40, and searches for image features that are equal to or less than a predetermined similar image determination threshold as image features that match or are related to the search image features (step S74). This similarity calculation may use the Euclidean inner product of vectors, cosine similarity, or other similarity indices, or an approximate calculation method such as vector neighborhood search using vector quantization. The similarity threshold used in this similarity comparison is a parameter that controls how similar images are presented as search results, and may be user-specifiable.

[0054] Next, the search execution unit 50 outputs location information in the document stored in the database 40 corresponding to the image feature found to match or be related to the search image feature to the search result presentation unit 51 (step S75). At the same time, the search execution unit 50 acquires an image corresponding to the image feature found to match or be related to the search image feature based on the location information in the document of the image, and outputs the content of the image as the content of the searched image to the search result presentation unit 51. Alternatively, when extracting an image from a document, the content of the image itself may be registered in the database 40, and the content of the image may then be acquired from the database 40. The search result presentation unit 51 receives the content of the image and the position information of the image in the document from the search execution unit 50. The search result presentation unit 51 presents the correspondence to the user as a list 60 so that the content of the image and the position information of the image in the document are displayed in association with each other (step S76). For example, when the user selects any correspondence relationship in the list 60, the search result presentation unit 51 opens the document 61 and displays, for example, in an emphasized manner, a portion of the document 61 that corresponds to the location information of the selected correspondence relationship. In this way, the search result presentation unit 51 presents the search result to the user by displaying, as the search result, the content of the image corresponding to the searched image feature (the content of the searched image) and the location information of the image in the document 61 in association with each other. In this embodiment, the highlighting of the image in the document is performed by opening the document to the page with the page number (page) registered in database 40 corresponding to the searched image feature, and then coloring the area surrounded by the four corners (top-left, bottom-left, bottom-right, top-right) registered in database 40 corresponding to the searched image feature in a different color from the other areas.

[0055] Next, the search unit 13 determines whether the user has selected (specified) another search result from the list of search results (step S80), and if the user has selected (specified) another search result (YES in step S80), it returns to step S78 and repeats the above-mentioned processing. On the other hand, if the user does not select (specify) another search result (NO in step S80), it is determined whether or not a new search has been instructed by the user (step S82), and if a new search has been instructed (YES in step S82), the process returns to step S70 and repeats the above-described process with the newly input search query (image). If the user does not then instruct a new search (NO in step S82), the process ends.

[0056] 10 is a flowchart of an image search method (when the search query is a character string) by the document search system of the embodiment. When a character string is input as the search query, an image search is realized by the following procedure. When the user inputs a search string as a search query from the search query input receiving unit 12 (step S110), the search execution unit 50 searches the database 40 for captions that match or are related to the input search string (step S112). The search execution unit 50 can treat captions that partially match the search string as captions related to the search string, for example. Regarding the input of a search string in step S50 of the string search method using Figure 8 and the input of a search string in step S110 of this image search method (when the search query is a string), both involve inputting a search string (text, natural language) as a search query, but because the objects being searched are different, it is desirable to provide different interfaces for inputting the search query. The search execution unit 50 outputs, as a first search result, location information in the document stored in the database 40 corresponding to the caption found to match or be related to the search string, to the search result presentation unit 51 (step S113). At the same time, the search execution unit 50 acquires an image corresponding to the caption found to match or be related to the search string based on the location information in the document of the image, and outputs the content of the image as the content of the searched image to the search result presentation unit 51, including it in the first search result. Alternatively, when extracting an image from a document, the content of the image itself may be registered in the database 40, and the content of the image may then be acquired from the database 40.

[0057] Next, the search execution unit 50 inputs the search string into the same caption feature extraction trained model that was used when the caption generation unit 33 extracted the caption features, and calculates search string features, which are features of the input search query (string) (step S114). Next, the search execution unit 50 calculates the distance as an index of similarity between the search string features and the caption features stored in the database 40, and searches for caption features that are equal to or less than a predetermined threshold as caption features that match or are related to the search string features (step S116). The search execution unit 50 outputs, as a second search result, location information in the document stored in the database 40 corresponding to the caption feature found to match or be related to the search string feature, to the search result presentation unit 51 (step S117). At the same time, the search execution unit 50 acquires an image corresponding to the caption feature found to match or be related to the search string feature based on the location information in the document of the image, and outputs the content of the image as the content of the searched image to the search result presentation unit 51, including it in the second search result. Alternatively, when extracting an image from a document, the content of the image itself may be registered in the database 40, and the content of the image may then be acquired from the database 40.

[0058] The search result presentation unit 51 receives the content of the images and the position information of the images in the document as the first search result and the second search result from the search execution unit 50. The search result presentation unit 51 presents the correspondence to the user in a list 60 that includes the first search result and the second search result, for example, so that the content of the images and the position information of the images in the document are displayed in association with each other (step S118). For example, when the user selects any correspondence relationship in the list 60, the search result presentation unit 51 opens the document 61 and displays, for example, in an emphasized manner, a portion of the document 61 that corresponds to the location information of the selected correspondence relationship (step S120). In this way, the search result presentation unit 51 presents the search result to the user by displaying, as the search result, the content of the image corresponding to the retrieved caption or caption feature (the content of the retrieved image) and the location information of the image in the document 61 in association with each other. In this embodiment, the highlighting of the image in the document is performed by opening the document to the page with the page number (page) registered in database 40 corresponding to the searched caption or caption feature, and then coloring the part surrounded by the four corners (top-left, bot-left, bot-right, top-right) registered in database 40 corresponding to the searched caption or caption feature in a different color from the other parts.

[0059] Next, the search unit 13 determines whether or not another search result has been selected (specified) by the user from the list of search results (step S122), and if another search result has been selected (specified) by the user (YES in step S122), returns to step S120 and repeats the above-mentioned processing. On the other hand, if the user does not select (specify) another search result (NO in step S122), it is determined whether or not a new search has been instructed by the user (step S124). If a new search has been instructed (YES in step S124), the process returns to step S110 and the above-described process is repeated with the newly input search query (character string). If the user does not then instruct a new search (NO in step S124), the process ends.

[0060] According to the above-described configuration, the document search system 1 is a system that searches for strings that match or are related to the search string in a document realized as a PDF file when a search string is input as a search query, and is equipped with: a database construction unit 11 that extracts strings contained in the document prior to the search and constructs a database 40 outside the document in which each of the extracted strings is associated with location information of the string in the document and registered; a search query input acceptance unit (input acceptance unit) 12 that accepts input of the search string; and a search unit 13 that searches the database 40 for strings that match or are related to the search string, extracts location information in the document that is associated with the searched string, and presents the content of the searched string in association with the location information of the searched string in the document. According to the above configuration, prior to a search, database construction unit 11 extracts character strings contained in a document realized as a PDF file, and then constructs database 40 outside the document in which each extracted character string is associated with location information of the character string within the document and registered. When actually searching for a character string in a document, a search query input receiving unit (input receiving unit) 12 receives input of a search character string as a search query. A search unit 13 searches a database 40 constructed outside the document for a character string that matches or is related to the search character string. As a result, the search unit 13 extracts location information in the document that is associated with the character string that is searched for in the database 40 as matching or related to the search character string. The extracted location information indicates the location of the searched character string that is associated with the searched character string. Therefore, the search unit 13 can present the content of the searched character string and the location information in the document in association with each other, for example, by highlighting a portion in the document that is indicated as location information. In the series of processes described above, information about character strings in a document is stored in a database 40 constructed outside the document, and searches for character strings in a document are performed not on the document itself but on the database 40 constructed outside the document. In this way, because each process is not performed on the document itself, character string searches can be performed more quickly even in cases where a search would take a long time if the document were the target, such as when the document file size is large. In this way, it becomes possible to search for character strings in PDF files more quickly.

[0061] In addition, the database construction unit 11 extracts character objects contained in a document, and when a second character object is located within a search range S set for one character object, it repeatedly connects the second character object to the first character object to extract a character string, and the search range S is set so that it is centered on the first character object, extends in the direction in which a character object following the first character object is expected to appear, and gradually becomes wider as it moves away from the first character object. When a document contains text that is scattered throughout the document as individual character objects for each character that makes up the text, it may not be possible to search it using commercially available browsing software, for example. Even if a search were possible, the search speed may be slow because the connections between the character objects must be evaluated each time a search is performed to determine whether multiple consecutive character objects match or are related to the search string. In contrast, with the above configuration, a search range S is set for character objects that are individually scattered throughout a document. The search range S is set so that it is centered on one character object, extends in the direction in which another character object following that one character object is expected to appear, and gradually widens as it moves away from the one character object. Then, if a second character object is located within this search range S, the second character object is concatenated to the first character object. By repeating this process, a concatenated character string can be extracted. Therefore, even if character objects are individually scattered throughout a document, it is possible to extract connected character strings from the character objects. Furthermore, the concatenation process described above only needs to be performed once when constructing the database 40. In other words, when searching for a character string, it is sufficient to refer to the database 40 in which the concatenation process has been completed and the results have been registered, and since there is no need to perform the concatenation process on a document each time, character string searches can be performed quickly.

[0062] In addition, the database construction unit 11 performs morphological analysis on each of the extracted character strings to generate substrings, and registers each of the generated substrings in the database 40 in association with the location information of the substring in the document. The character string extracted from a document by the database construction unit 11 may be long. In this case, when a search is performed, the long character string may be searched as a related character string, for example, because it partially matches the search character string. In this case, for example, when the search unit associates the content of the searched character string with location information in the document and presents the associated content, the long character string extracted from the document is presented in association with the search character string itself, rather than the search character string itself. Therefore, for example, when this associated presentation highlights a portion indicated as location information, the long character string is highlighted. Therefore, in order to identify which part of the highlighted long character string the search character string is in, the user must visually search for the search character string. In contrast, with the above configuration, the database construction unit 11 generates meaningful substrings such as parts of speech from character strings extracted from a document by morphological analysis, and associates the generated substrings with location information of the substrings, and registers them in the database 40. As a result, the character strings extracted from the document are broken down into smaller units such as parts of speech, and the character strings are associated with their location information and registered in the database 40. Therefore, in a situation where the database construction unit 11 extracts a long character string from a document and registers it in the database 40, if a search is performed using a substring generated by morphological analysis that is included in the long character string as a search string, the substring is extracted by the search, and location information in the document that corresponds to the substring is presented in association with the substring. In this way, the location information presented as search results is more precisely limited to the shorter substrings after morphological analysis has been applied, rather than the string before morphological analysis has been applied, allowing for a more precise understanding of the location of the search string.

[0063] Furthermore, the search unit 13 determines that a character string registered in the database 40 is related to the search character string if the search character string partially matches a character string registered in the database 40, or if either the Levenshtein distance or the Jaro-Winkler distance between the character string registered in the database 40 and the search character string is equal to or less than a similar character string determination threshold. According to the above configuration, it is possible to appropriately search for character strings related to a search character string.

[0064] Furthermore, the above-described intra-document search system 1 is an intra-document search system 1 that, when a search image is input as a search query, searches for images in a document realized as a PDF file that match or are related to the search image, and is equipped with a database construction unit 11 that extracts images contained in the document prior to the search and constructs a database 40 outside the document in which, for each extracted image, image features, which are features generated based on the image, are associated with location information of the image in the document, and registers them; a search query input acceptance unit (input acceptance unit) 12 that accepts input of the search image; and a search unit 13 that searches the database 40 for image features that match or are related to the search image features, which are features generated based on the search image, extracts location information in the document that is associated with the searched image features, and presents the content of the image corresponding to the searched image features in association with the location information in the document. According to the above configuration, prior to a search, database construction unit 11 extracts images contained in a document realized as a PDF file. Then, for each extracted image, database 40 is constructed outside the document, in which image feature amounts, which are feature amounts generated based on the image, and position information of the image within the document are registered in association with each other. When actually searching for an image in a document, a search query input receiving unit (input receiving unit) 12 receives input of a search image as a search query. A search unit 13 searches a database 40 constructed outside the document for image features that match or are related to search image features, which are features generated based on the search image. As a result, the search unit 13 extracts location information in the document associated with the image features searched from the database 40 as matching or related to the search image features. The extracted location information indicates the location of the image that is associated with the searched image features and is the basis of the searched image features. Therefore, the search unit 13 can display the content of the image that corresponds to the searched image features in association with the location information in the document, for example, by highlighting the part of the document indicated as the location information. In the series of processes described above, information about images in a document is stored in a database 40 constructed outside the document, and a search for images in the document is performed not on the document itself but on the database 40 constructed outside the document. In this way, because each process is not performed on the document itself, even in cases where a search would take a long time if the document were the target, such as when the document file size is large, the image search can be performed more quickly. In this way, images within PDF files can be searched for more quickly by entering the image as a search query.

[0065] In addition, the search unit 13 searches for image features that match or are related to the search image features by extracting, from the image features registered in the database 40, image features whose distance as an index of similarity between them and the search image features is less than or equal to a similar image determination threshold. According to the above configuration, it is possible to appropriately search for images.

[0066] Furthermore, the above-described intra-document search system 1 is an intra-document search system 1 that, when a search string is input as a search query, searches for images related to the search string in a document realized as a PDF file, and is equipped with: a database construction unit 11 that, prior to the search, extracts images included in the document, generates captions corresponding to each of the extracted images, extracts features from the captions to create caption features, and constructs a database 40 outside the document in which the captions, the caption features, and location information of the images in the document are associated and registered; a search query input acceptance unit (input acceptance unit) 12 that accepts input of the search string; and a search unit 13 that searches the database 40 for captions that match or are related to the search string, or caption features that match or are related to search string features, which are features generated based on the search string, extracts location information in the document that is associated with the searched caption or caption feature, and displays the content of the image corresponding to the searched caption or caption feature in association with the location information in the document. According to the above configuration, prior to a search, database construction unit 11 extracts images contained in a document realized as a PDF file. Then, for each extracted image, a caption corresponding to the image is generated, feature amounts are extracted from the caption to form caption feature amounts, and a database 40 is constructed outside the document in which the captions, caption feature amounts, and position information of the images in the document are registered in association with each other. When actually searching for an image in a document, a search query input receiving unit (input receiving unit) 12 receives input of a search string as a search query. A search unit 13 searches a database 40 constructed outside the document for captions that match or are related to the search string, or for caption features that match or are related to search string features, which are features generated based on the search string. As a result, the search unit 13 extracts location information in the document associated with the captions searched from the database 40 as matching or related to the search string, or the caption features searched from the database 40 as matching or related to the search string features. The extracted location information indicates the location of the image associated with the searched caption or caption feature. Therefore, the search unit 13 can display the content of the image corresponding to the searched caption or caption feature in association with the location information in the document, for example, by highlighting the portion of the document indicated as the location information. In the series of processes described above, information about images in a document is stored in a database 40 constructed outside the document, and a search for images in the document is performed not on the document itself but on the database 40 constructed outside the document. In this way, because each process is not performed on the document itself, even in cases where a search would take a long time if the document were the target, such as when the document file size is large, the image search can be performed more quickly. In this way, images in PDF files can be searched for more quickly by entering a string (text, natural language) as a search query.

[0067] The document search system of the present invention is not limited to the above-described embodiment explained with reference to the drawings, and various other modifications are conceivable within the technical scope thereof. For example, the search range S is not limited to a fan shape, but may also be a trapezoid. By setting the search range S to a trapezoid shape, similar to the fan shape, it is possible to perform connection analysis on character strings with special layouts specific to architectural documents, such as characters that are diagonally oriented or arc-shaped, and to ensure their searchability. It is preferable that the length by which the trapezoidal search range S is expanded, the ratio of the lengths of the short side and the long side, etc. be adjusted according to the size of the character objects, and the magnification may be a parameter adjusted by the user depending on how far apart character objects are to be connected.

[0068] Furthermore, in the above embodiment, database 40 is constructed based on character strings and images contained in a single document, and the constructed database 40 is used to search for a single document. However, instead of this, database 40 may be constructed for multiple documents based on character strings and images for each document, and the constructed database 40 may be used to search for multiple documents. In this case, by constructing each database 40 for multiple documents using a relational database as in the above embodiment, searches based on the input search query can be easily performed across multiple databases 40. In this way, when performing a search across multiple documents, when displaying the search results, it is desirable to display document identification information, such as the document name or file name, in association with the search results so that it is clear which document contained the searched string.

[0069] That is, in this case, database 40 is provided individually for each of the multiple documents, and search unit 13 searches database 40 for each of the multiple documents for a character string that matches or is related to the search character string, extracts location information within the document that is associated with the searched character string, and displays the content of the searched character string, the identification information of the document that contains the character string, and the location information within the document in association with each other. According to the above configuration, a search is performed on a database 40 constructed outside the document, so that even when searching multiple documents, a search for a character string can be performed more quickly.

[0070] Furthermore, in the above embodiment, the search result presentation unit 51 presents the search results to the user in a list 60, and when the user selects a specific search result from the list 60, the portion of the document corresponding to the location information is highlighted and displayed, but this is not limited to this. The search result presentation unit 51 may not create the list 60, but may highlight and display the portion of the document corresponding to the location information of the first search result, and then sequentially display the next search result in response to a user operation such as pressing a button on the input device.

[0071] Furthermore, in the above embodiment, when the search result presentation unit 51 displays the contents of the searched character string or image in association with the location information of the character string or image within the document, it has been explained that the part indicated by the location information is highlighted by, for example, coloring the part surrounded by the four corners (top-left, bottom-left, bottom-right, top-right) in a different color from the other parts, but this may take any form. For example, the search result presentation unit 51 may highlight the searched character string or image by drawing a rectangular frame that surrounds the searched character string or image by connecting adjacent corners of the four corners with lines. Alternatively, the search result presentation unit 51 may highlight the searched character string or image by connecting the two lower corners (bottom-left and bottom-right) of the four corners with lines and underlining the searched character string or image.

[0072] In addition to this, the present invention is not limited to the above-described embodiments and modified examples described with reference to the drawings, and it is possible to select and discard the configurations listed in the above-described embodiments and modified examples, or to change them to other configurations as appropriate, as long as this does not deviate from the gist of the present invention. [Explanation of symbols]

[0073] 1 Document Search System 20 Character String DB Creation Department 11 Database Construction Department 30 Image DB Creation Department 12 Search query input reception unit (input reception unit) 40 Database 13 Search section S Search range

Claims

1. A document search system that, when a search string is input as a search query, searches for a string that matches or is related to the search string in a document realized as a PDF file, comprising: a database construction unit that extracts the character strings contained in the document prior to the search, and constructs a database outside the document in which each of the extracted character strings is associated with location information of the character string in the document and registered; an input receiving unit that receives an input of the search string; a search unit that searches the database for character strings that match or are related to the search character string, extracts the location information in the document that is associated with the searched character string, and presents the content of the searched character string and the location information in the document of the searched character string in association with each other; A document search system comprising:

2. the database construction unit extracts character objects included in the document, and when a second character object is located within a search range set for one of the character objects, repeatedly connects the second character object to the first character object, thereby extracting the character string; The search range is set so as to have the one character object as its center, extend in a direction in which the character object following the one character object is expected to appear, and gradually widen as it moves away from the one character object.

2. The document search system according to claim 1.

3. The database construction unit performs morphological analysis on each of the extracted character strings to generate subcharacter strings, and associates each of the generated subcharacter strings with the position information of the subcharacter string in the document, and registers the subcharacter strings in the database.

2. The document search system according to claim 1.

4. The search unit determines that the character string registered in the database is related to the search character string if the search character string partially matches the character string registered in the database, or if either the Levenshtein distance or the Jaro-Winkler distance between the character string registered in the database and the search character string is equal to or less than a similar character string determination threshold.

2. The document search system according to claim 1.

5. the database is provided individually corresponding to each of the plurality of documents, The search unit searches the database for each of the plurality of documents for the character string that matches or is related to the search character string, extracts the location information in the document that is associated with the searched character string, and displays the content of the searched character string, identification information of the document that includes the character string, and the location information in the document in association with each other.

2. The document search system according to claim 1.

6. A document search system that, when a search image is input as a search query, searches for images that match or are related to the search image in a document realized as a PDF file, comprising: a database construction unit that extracts the images included in the document prior to the search, and constructs a database outside the document in which, for each of the extracted images, image features that are features generated based on the image are associated with location information of the image within the document and registered; an input receiving unit that receives an input of the search image; a search unit that searches the database for image features that match or relate to search image features, which are features generated based on the search image, extracts the location information in the document that is associated with the searched image features, and presents the content of the image that corresponds to the searched image features and the location information in the document in association with each other; A document search system comprising:

7. The search unit searches for image features that match or are related to the search image features by extracting, from the image features registered in the database, image features whose distance as an index of similarity between the image features and the search image features is equal to or less than a similar image determination threshold.

7. The document search system according to claim 6.

8. A document search system that, when a search string is input as a search query, searches for images related to the search string in a document realized as a PDF file, comprising: a database construction unit that, prior to the search, extracts the images included in the document, generates a caption corresponding to each of the extracted images, extracts features from the captions to set them as caption features, and constructs a database outside the document in which the captions, the caption features, and position information of the images in the document are associated and registered; an input receiving unit that receives an input of the search string; a search unit that searches the database for the caption that matches or is related to the search string, or the caption feature that matches or is related to a search string feature that is a feature generated based on the search string, extracts the position information within the document that is associated with the retrieved caption or caption feature, and displays the content of the image that corresponds to the retrieved caption or caption feature and the position information within the document in association with each other; A document search system comprising:

9. The document is a construction document or a drawing drawn in construction.

9. The document search system according to claim 1, wherein the document search system is a search system for searching a document.

Images