Search device, search method, and control program

Abstract

To provide a retrieval device, a retrieval method, and a control program for excellently retrieving an available recognition module from among existing recognition modules.SOLUTION: In a retrieval system in which a retrieval device, an arithmetic device, and a database device are connected to a network, a first processing circuit of the retrieval device includes: a storage unit configured to store a table in which target information related to a character string, an image feature quantity, or an object of a recognition target is registered for each of multiple recognition modules that output information related to the recognition target when an image is input; an acquisition unit 121 configured to acquire the input image, a recognition unit 122 configured to recognize a character string, an image feature quantity, or an object included in the input image; an extraction unit 123 configured to extract, from among the multiple recognition modules, a recognition module in which the target information registered in the table corresponds to the character string, the image feature quantity, or the object recognized by the recognition unit, as a recognition module candidate corresponding to the input image; and an output control unit 125 configured to output information about the recognition module candidate.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a search device, a search method, and a control program. [Background technology]

[0002] In recent years, work style reform has been promoted, and there is a demand for the efficiency of various operations. On the other hand, image recognition technology has improved in various fields such as autonomous driving, road inspection, character recognition, and cancer detection, and automatic recognition technology by computers is used in various operations. Therefore, various types of recognition modules have been developed in various industries. However, there was no mechanism for using a recognition module developed for a specific operation in other operations, and each company had to develop a new recognition module every time it tried to improve the efficiency of a new operation.

[0003] An information processing system has been disclosed that prepares a model version that pairs learning results and an inference method for each task to be managed, and executes the selected model version in a selected execution environment (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 163823 Summary of the Invention [Problem to be solved by the invention]

[0005] When using automatic computer recognition technology, it is desirable to be able to efficiently search for available recognition modules from among existing recognition modules.

[0006] An object of the present invention is to provide a search device, a search method, and a control program that are capable of effectively searching for a usable recognition module from among existing recognition modules. [Means for solving the problem]

[0007] A search device according to one aspect of the present invention comprises: A character string recognition module that outputs information about a character string to be recognized when an image is input, an image feature recognition module that outputs information about an image feature to be recognized when an image is input, and an object recognition module that outputs information about an object to be recognized when an image is input. a storage unit that stores a table in which target information related to each of the character strings, image features, or objects to be recognized is registered; an acquisition unit that acquires an input image; and a recognition unit that recognizes the character strings, image features, or objects included in the input image. Character string recognition module, image feature recognition module, and object recognition module The apparatus has an extraction unit that extracts, from the target information registered in the table, a recognition module corresponding to a character string, image feature, or object recognized by the recognition unit as a recognition module candidate corresponding to the input image, and an output unit that outputs information regarding the recognition module candidate.

[0008] In addition, in a search method according to one aspect of the present invention, a computer A character string recognition module that outputs information about a character string to be recognized when an image is input, an image feature recognition module that outputs information about an image feature to be recognized when an image is input, and an object recognition module that outputs information about an object to be recognized when an image is input. a table in which target information relating to each of the character strings, image features, or objects to be recognized is registered; an input image is acquired; and the character strings, image features, or objects included in the input image are recognized; Character string recognition module, image feature recognition module, and object recognition module From the target information registered in the table, a recognition module corresponding to the recognized character string, image feature, or object is extracted as a candidate recognition module corresponding to the input image, and information regarding the candidate recognition module is output.

[0009] A control program according to one aspect of the present invention is a control program for a computer having a storage unit, A character string recognition module that outputs information about a character string to be recognized when an image is input, an image feature recognition module that outputs information about an image feature to be recognized when an image is input, and an object recognition module that outputs information about an object to be recognized when an image is input. a table in which target information relating to each of the character strings, image features, or objects to be recognized is registered in a storage unit; an input image is acquired; and the character strings, image features, or objects included in the input image are recognized; Character string recognition module, image feature recognition module, and object recognition module The computer is caused to extract, from the target information registered in the table, a recognition module corresponding to the recognized character string, image feature, or object as a recognition module candidate corresponding to the input image, and output information regarding the recognition module candidate. Effect of the Invention

[0010] According to the present invention, the search device, search method, and control program are capable of effectively searching for a usable recognition module from among existing recognition modules. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of a schematic configuration of a search system 1 according to an embodiment. [Diagram 2] FIG. 13 illustrates an example of a data structure of a candidate table. [Diagram 3] 2 is a diagram showing a schematic configuration of a first storage device 110 and a first processing circuit 120. FIG. [Figure 4] 13 is a flowchart showing an example of a search process operation. [Diagram 5] FIG. 5 is a schematic diagram showing an example of a search screen 500. [Figure 6] FIG. 5 is a schematic diagram showing an example of a search screen 500. [Figure 7] 13 is a flowchart illustrating a verification process. [Figure 8] FIG. 8 is a schematic diagram showing an example of a verification screen 800. [Figure 9] FIG. 8 is a schematic diagram showing an example of a verification screen 800. [Figure 10] 13 is a flowchart illustrating a registration process. [Figure 11] FIG. 11 is a schematic diagram showing an example of a registration screen 1100. [Figure 12] FIG. 11 is a schematic diagram showing an example of a registration screen 1100. [Figure 13] 13 is a diagram showing a schematic configuration of another first processing circuit 420. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, a search system according to one aspect of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present disclosure is not limited to the embodiments, but extends to the inventions described in the claims and their equivalents.

[0013] FIG. 1 is a diagram showing an example of a schematic configuration of a search system 1 according to an embodiment.

[0014] The search system 1 includes a search device 100, a calculation device 200, and one or more database devices 300.

[0015] The search device 100, the calculation device 200, and the database device 300 each have a wired or wireless communication function, are connected to a network N, and communicate with each other via the network N.

[0016] The search device 100 is an information processing device such as a server, a PC (Personal Computer), a notebook PC, a tablet PC, or a multi-function mobile phone (a so-called smartphone). In response to a request from a user of the arithmetic device 200, the search device 100 searches for a recognition module that matches the purpose of the user of the arithmetic device 200 from among a plurality of recognition modules stored in the database device 300. Each recognition module is a module that outputs information about a recognition target when an image is input. The recognition target is a predetermined character string, an image feature amount (feature vector), an object, or the like.

[0017] When the recognition target is a predetermined character string, the recognition module is a character recognition engine that recognizes a predetermined character string from an image using OCR (Optical Character Recognition) technology. When the recognition target is a predetermined image feature, the recognition module is a feature calculation engine that calculates a predetermined image feature in an image. The image feature is SIFT (Scale-Invariant Feature Transform), AKAZE (Accelerated-KAZE), ORB (Oriented FAST and Rotated BRIEF), etc. When the recognition target is a predetermined object, the recognition module is a learning model that is pre-trained to output information about the recognition target when an image is input. The information about the recognition target is the probability that the image contains a predetermined object, information indicating whether the image contains a predetermined object or not, or information indicating the object contained in the image, and information indicating the area in which each object is contained in the image. The learning model is trained by a predetermined method such as YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), M2Det, etc.

[0018] The search device 100 includes a first communication device 101, a first input device 102, a first display device 103, a first storage device 110, and a first processing circuit 120. Each unit of the search device 100 will be described in detail below.

[0019] The first communication device 101 is an example of an output unit, and has a wired communication interface circuit such as TCP / IP (Transmission Control Protocol / Internet Protocol). The first communication device 101 is communicatively connected to a network N according to a communication method such as Ethernet (registered trademark). The first communication device 101 supplies data received from the arithmetic device 200 or the database device 300 via the network N to the first processing circuit 120. The first communication device 101 also transmits data supplied from the first processing circuit 120 to the arithmetic device 200 or the database device 300 via the network N. The first communication device 101 may be any device that can communicate with an external device. For example, the first communication device 101 may communicate with the arithmetic device 200 or the database device 300 via an access point (not shown) according to a wireless LAN (Local Area Network) communication method.

[0020] The first input device 102 includes an input device such as a touch panel input device, a keyboard, a mouse, etc., and an interface circuit for acquiring signals from the input devices. The first input device 102 receives input data input by a user, and outputs a signal corresponding to the received input data to the first processing circuit 120.

[0021] The first display device 103 has a display configured of liquid crystal, organic EL (Electro-Luminescence), or the like, and an interface circuit that outputs image data or various information to the display. The first display device 103 is connected to the first processing circuit 120, and displays image data output from the first processing circuit 120 on the display. The first input device 102 and the first display device 103 may be integrated together using a touch panel display.

[0022] The first storage device 110 is an example of a storage unit. The first storage device 110 has a memory device such as a random access memory (RAM) or a read only memory (ROM), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or an optical disk. The first storage device 110 also stores computer programs, databases, tables, and the like used for various processes of the search device 100. The computer programs may be installed in the first storage device 110 from a computer-readable portable recording medium using a known setup program or the like. The portable recording medium is, for example, a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or the like.

[0023] The first storage device 110 stores a candidate table and the like as data. In the candidate table, target information related to the recognition targets of each of the multiple recognition modules is registered. The candidate table will be described in detail later. In addition, the first storage device 110 stores, as data, multiple learning images including each object for each of multiple types of objects.

[0024] The first processing circuit 120 operates based on a program stored in advance in the first storage device 110. The first processing circuit 120 is, for example, a CPU (Central Processing Unit). Note that the first processing circuit 120 may be a DSP (digital signal processor), an LSI (large scale integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0025] The first processing circuit 120 is connected to the first communication device 101, the first input device 102, the first display device 103, and the first storage device 110, and controls each of these units. The first processing circuit 120 performs data transmission / reception control via the first communication device 101, input control of the first input device 102, output control of the first display device 103, and control of the first storage device 110. The first processing circuit 120 also acquires an input image from the first communication device 101, and recognizes a character string, image feature, or object contained in the input image. The first processing circuit 120 extracts a recognition module corresponding to the recognized character string, image feature, or object from target information registered in a candidate table stored in the first storage device 110, and transmits the extraction result to the arithmetic device 200 via the first communication device 101.

[0026] The arithmetic device 200 is an information processing device such as a PC, a notebook PC, a tablet PC, or a multi-function mobile phone. The arithmetic device 200 is used by a user and recognizes a character string, an image feature, or an object contained in an image using a recognition module searched for by the search device 100. The arithmetic device 200 includes a second communication device 201, a second input device 202, a second display device 203, a second storage device 210, and a second processing circuit 220. Each unit of the arithmetic device 200 will be described in detail below.

[0027] The second communication device 201 has a communication interface circuit similar to that of the first communication device 101, and is communicatively connected to the network N according to the same communication method as that of the first communication device 101. The second communication device 201 supplies data received from the retrieval device 100 or the database device 300 via the network N to the second processing circuit 220. In addition, the second communication device 201 transmits the data supplied from the second processing circuit 220 to the retrieval device 100 or the database device 300 via the network N.

[0028] The second input device 202 has an input device and an interface circuit similar to those of the first input device 102, receives input data input by a user, and outputs to the second processing circuit 220 a signal according to the received input data.

[0029] The second display device 203 has a display and an interface circuit similar to those of the first display device 103, and displays the image data output from the second processing circuit 220 on the display.

[0030] The second storage device 210 has a memory device, a fixed disk device, a storage device, or the like similar to the first storage device. The second storage device 210 also stores computer programs, databases, tables, and the like used for various processes of the arithmetic device 200. The computer programs may be installed in the second storage device 210 from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM using a known setup program or the like.

[0031] The second processing circuit 220 operates based on a program previously stored in the second storage device 210. The second processing circuit 220 is, for example, a CPU. Note that the second processing circuit 220 may be, for example, a DSP, an LSI, an ASIC, an FPGA, or the like.

[0032] The second processing circuit 220 is connected to the second communication device 201, the second input device 202, the second display device 203, and the second storage device 210, and controls each of these units. The second processing circuit 220 controls data transmission and reception via the second communication device 201, input control of the second input device 202, output control of the second display device 203, and control of the second storage device 210. The second processing circuit 220 also receives a recognition module from the search device 100 via the second communication device 201, recognizes a character string, image feature amount, or object included in an image using the received recognition module, and displays the recognition result on the second display device 203.

[0033] The database device 300 is an information processing device such as a server or a PC. The database device 300 is provided in the form of, for example, cloud computing. The database device 300 stores a plurality of recognition modules registered from the search device 100 or another computer. The database device 300 includes a third communication device 301, a third storage device 310, and a third processing circuit 320. Each unit of the database device 300 will be described in detail below.

[0034] The third communication device 301 has a communication interface circuit similar to that of the first communication device 101, and is communicatively connected to the network N according to the same communication method as that of the first communication device 101. The third communication device 301 supplies data received from the retrieval device 100 or the arithmetic device 200 via the network N to the third processing circuit 320. In addition, the third communication device 301 transmits data supplied from the third processing circuit 320 to the retrieval device 100 or the arithmetic device 200 via the network N.

[0035] The third storage device 310 has a memory device, a fixed disk device, a storage device, or the like similar to the first storage device. The third storage device 310 also stores computer programs, databases, tables, and the like used for various processes of the database device 300. The computer programs may be installed in the third storage device 310 from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM using a known setup program or the like.

[0036] The third processing circuit 320 operates based on a program previously stored in the third storage device 310. The third processing circuit 320 is, for example, a CPU. Note that, as the third processing circuit 320, a DSP, an LSI, an ASIC, an FPGA, or the like may be used.

[0037] The third processing circuit 320 is connected to the third communication device 301 and the third storage device 310 and controls each of these units. The third processing circuit 320 controls data transmission and reception via the third communication device 301 and the third storage device 310. The third processing circuit 320 also receives a recognition module from the search device 100 or another computer via the third communication device 301 and stores it in the third storage device 310. The third processing circuit 320 transmits the recognition module stored in the third storage device 310 to the search device 100 or the calculation device 200 via the third communication device 301 in accordance with a signal received from the search device 100 or the calculation device 200 via the third communication device 301.

[0038] FIG. 2 is a diagram showing an example of the data structure of a candidate table stored in the first storage device 110 of the search device 100. As shown in FIG.

[0039] As shown in FIG. 2, the candidate table stores, for each of a plurality of recognition targets, identification information (target ID), target type, image type, target information, and identification information (candidate ID) of a recognition module candidate.

[0040] The object type is the type of each recognition target. If the recognition target is a predetermined character string, the object type is a character string, and if the recognition target is a predetermined image feature or a predetermined object, the object type is an image.

[0041] The image type is the type of image (objects included in the entire image) in which each recognition target is included, and is the object, space, etc. to which each recognition target belongs. When the recognition target is a predetermined character string, the image type is the type of document (document) in which the character string is written (invoice, contract, etc.). When the recognition target is a predetermined image feature, the image type is the name of an object having the image feature, etc. For example, when the image feature is the feature of a mark printed on a component (LSI, IC) mounted on a printed circuit board, the image type is the printed circuit board or the component (LSI, IC), etc. When the recognition target is a predetermined object, the image type is the name of a space in which the object is included, etc. For example, when the object is a car on a road, the image type is the road, etc.

[0042] The target information is information related to the recognition target. When the recognition target is a specified character string, the target information indicates the character string itself. The target information may be a parameter name such as billing amount or address. For example, when the target information is the parameter name "billing amount", the numerical value indicating "billing amount" is the recognition target. The number of character strings indicated by the target information is not limited to one, and may be multiple.

[0043] When the recognition target is a predetermined image feature, the target information indicates the image feature itself. The number of image features indicated by the target information is not limited to one, and may be multiple. When the target information is an image feature, the type of the image feature (SIFT, AKAZE, ORB, etc.) may be further set as the target information.

[0044] When the recognition target is a predetermined object, the target information is a general learning model for detecting the object. Each learning model is pre-trained by deep learning or the like to output information about each object when an image is input. The information about each object is the probability that each object is included in the image, information indicating whether or not each object is included in the image or the object included in the image, and information indicating the area in which each object is included in the image. When the target information is a learning model, information indicating the object to be recognized and the format of data output by each learning model (probability that each object is included in the image, information indicating whether or not each object is included in the image or the object included in the image, etc.) are further set as the target information.

[0045] In addition, as the target information, an area in the image that includes each character string, image feature, or object may be set. In this case, the entire area in the image may be set as the area that includes the character string, image feature, or object.

[0046] The recognition module candidates are recognition modules corresponding to each recognition target, and are one or more recognition modules registered as recognition modules capable of recognizing each recognition target.

[0047] FIG. 3 is a diagram showing a schematic configuration of the first storage device 110 and the first processing circuit 120 of the search device 100. As shown in FIG.

[0048] 3, the first storage device 110 stores various programs such as an acquisition program 111, a recognition program 112, an extraction program 113, a verification program 114, an output control program 115, and a registration program 116. Each of these programs is a functional module implemented by software that runs on a processor. The first processing circuit 120 reads each program stored in the first storage device 110 and operates according to the read program. As a result, the first processing circuit 120 functions as an acquisition unit 121, a recognition unit 122, an extraction unit 123, a verification unit 124, an output control unit 125, and a registration unit 126.

[0049] FIG. 4 is a flowchart showing an example of the operation of the search process by the search device 100.

[0050] An example of the operation of the search process will be described below with reference to the flowchart shown in Fig. 4. The flow of the operation described below is executed mainly by the first processing circuit 120 in cooperation with each element of the search device 100 based on a program stored in advance in the first storage device 110.

[0051] First, the acquisition unit 121 generates search screen display data for displaying a search screen used to perform a search, and transmits the search screen display data to the arithmetic device 200 via the first communication device 101 (step S101). The second processing circuit 220 of the arithmetic device 200 receives the search screen display data from the search device 100 via the second communication device 201, and displays the search screen on the second display device 203 based on the received search screen display data.

[0052] 5 and 6 are schematic diagrams showing an example of a search screen 500 displayed on the second display device 203. FIG.

[0053] As shown in FIGS. 5 and 6, a search screen 500 includes an image display area 501, an area designation button 502, an object type designation box 503, a search button 504, and the like.

[0054] The image display area 501 is an area for displaying an input image designated by a user. Before an input image is designated by a user, a blank (white space) is displayed in the image display area 501. When an input image is designated by a user by an operation such as drag-and-drop on the image display area 501, the designated input image is displayed in the image display area 501. In the search screen 500 of FIG. 5, an image 511 of a printed board is designated as an input image, and in the search screen 500 of FIG. 6, an image 611 of an invoice is designated as an input image. The arithmetic device 200 may have an imaging device such as a camera, as in a mobile phone, a notebook computer, etc., and the search screen 500 may include a capture button. In this case, when the user presses the capture button, the arithmetic device 200 executes imaging by the imaging device and displays the captured image in the image display area 501 as an input image.

[0055] The area designation button 502 is a button for designating a target area including a recognition target in the input image displayed in the image display area 501. The user selects an area including a recognition target in the input image displayed in the image display area 501 and presses the area designation button 502 to designate the target area. The number of target areas designated is not limited to one, and may be two or more. In the search screen 500 of FIG. 5, a local area 512 including a predetermined LSI in an image 511 is designated as the target area. In the search screen 500 of FIG. 6, no target area is designated. In this case, the entire area in the image 611 becomes the target area.

[0056] The object type designation box 503 is a box for designating the object type of the recognition target contained in the input image displayed in the image display area 501. In the search screen 500 of Fig. 5, an image is designated as the object type. In the search screen 500 of Fig. 6, the object type is not designated. In this case, a search is performed on the recognition modules corresponding to all object types (images and character strings).

[0057] The search button 504 is a button for instructing (starting) a search for a recognition module for recognizing a recognition target from an input image. When the user presses the search button 504 using the second input device 202, the second processing circuit 220 generates a search request signal requesting a search for a recognition module, and transmits the search request signal to the search device 100 via the second communication device 201. The search request signal includes the input image, target area, and / or target type specified by the user.

[0058] Next, when the search button 504 is pressed on the search screen 500, the acquisition unit 121 receives a search request signal from the arithmetic device 200 via the first communication device 101, and acquires the input image, the target area, and / or the target type included in the received search request signal (step S102). As a result, the acquisition unit 121 accepts the user's designation of the target area in the input image that includes the recognition target and / or the user's designation of the target type of the recognition target included in the input image.

[0059] Next, the recognition unit 122 determines whether or not to recognize a character string (step S103). If a character string is specified as the object type, or if the object type is not specified, the recognition unit 122 determines to recognize a character string, and if an image is specified as the object type, the recognition unit 122 determines not to recognize a character string. If it is determined not to recognize a character string, the recognition unit 122 proceeds to step S107.

[0060] On the other hand, if it is decided to recognize the character string, the recognition unit 122 recognizes the character string included in the input image (step S104). The recognition unit 122 recognizes the character string included in the input image using general OCR software. If a target area is specified, the recognition unit 122 recognizes the character string in the specified target area, and if a target area is not specified, the recognition unit 122 recognizes the character string in the entire area of ​​the input image.

[0061] Next, the extraction unit 123 refers to the candidate table and determines whether target information corresponding to the character string recognized by the recognition unit 122 is registered (step S105). When all character strings registered as target information are included in the character string recognized by the recognition unit 122, the extraction unit 123 determines that the target information corresponds to the recognized character string. When an area including each character string in an image is set as target information, the extraction unit 123 determines that the target information corresponds to the recognized character string when each character string registered as target information is recognized within the set area of ​​the input image.

[0062] In addition, the extraction unit 123 may determine that the target information corresponds to the recognized character string when at least a part of the character string registered as the target information is included in the character string recognized by the recognition unit 122. In addition, the extraction unit 123 may determine that the target information corresponds to the recognized character string when the character string recognized by the recognition unit 122 includes a character string similar to the character string registered as the target information. The similar character string is a character string that may have the same meaning, such as "billing amount" and "total amount", and is registered in advance. In addition, when a parameter name is set as the target information, the extraction unit 123 determines that the target information corresponds to the recognized character string when the recognized character string satisfies the rule assigned to the parameter name. For example, when the parameter name "billing amount" is set as the target information, the target information is determined to correspond to the recognized character string when the recognized character string includes numbers, "yen", "￥", etc. In the case where the target information corresponding to the character string recognized by the recognition unit 122 is not registered, the extraction unit 123 does not extract a recognition module candidate and moves the process to step S107.

[0063] On the other hand, if target information corresponding to the character string recognized by the recognition unit 122 is registered, the extraction unit 123 extracts all recognition module candidates registered in association with the target information (step S106).

[0064] Next, the recognition unit 122 determines whether or not to recognize the image (step S107). If an image is specified as the object type, or if the object type is not specified, the recognition unit 122 determines to recognize the image, and if a character string is specified as the object type, the recognition unit 122 determines not to recognize the image. If it is determined not to recognize the image, the recognition unit 122 proceeds to step S114.

[0065] On the other hand, when it is decided to recognize the image, the recognition unit 122 recognizes the image feature amount contained in the input image (step S108). The recognition unit 122 calculates each image feature amount contained in the input image using general feature amount calculation software for calculating each image feature amount calculated by each feature amount calculation engine registered in the candidate table. When a target area is specified, the recognition unit 122 calculates the image feature amount for the specified target area, and when a target area is not specified, the recognition unit 122 calculates the image feature amount for the entire area in the input image. Note that the recognition unit 122 may divide the input image into a plurality of areas having a predetermined size and calculate the image feature amount for each divided area.

[0066] Next, the extraction unit 123 refers to the candidate table and determines whether target information corresponding to the image feature amount recognized by the recognition unit 122 is registered (step S109). The extraction unit 123 calculates the similarity between each image feature amount registered as target information and each image feature amount calculated by the recognition unit 122, and determines that each image feature amount is similar when the calculated similarity is equal to or greater than a predetermined threshold. The similarity is, for example, an inner product of each image feature amount (feature vector). The extraction unit 123 determines that the target information corresponds to the calculated image feature amount when all image feature amounts registered as target information are similar to any image feature amount calculated by the recognition unit 122. When an area including each image feature amount is set as target information in the image, the extraction unit 123 determines that the target information corresponds to the recognized image feature amount when each image feature amount registered as target information is similar to an image feature amount in the set area of ​​the input image.

[0067] In addition, when at least a part of image features registered as target information is similar to the image features calculated by the recognition unit 122, the extraction unit 123 may determine that the target information corresponds to the calculated image features. When target information corresponding to the image features recognized by the recognition unit 122 is not registered, the extraction unit 123 proceeds to step S111 without extracting a recognition module candidate.

[0068] On the other hand, if target information corresponding to the image feature recognized by the recognition unit 122 is registered, the extraction unit 123 extracts all recognition module candidates registered in association with the target information (step S110).

[0069] Next, the recognition unit 122 recognizes objects included in the input image (step S111). The recognition unit 122 recognizes objects to be recognized that are registered as target information from the input image, using each learning model registered as target information in the candidate table. When a target region is specified, the recognition unit 122 cuts out the specified target region from the input image and inputs it to each learning model, and obtains output information output from each learning model. On the other hand, when a target region is not specified, the recognition unit 122 inputs the entire input image to each learning model, and obtains output information output from each learning model.

[0070] When the learning model is trained to output the probability that the image contains the object to be recognized, the recognition unit 122 determines that the input image contains the object to be recognized when the acquired output information is equal to or greater than a predetermined threshold, and detects the object. When the learning model is trained to output whether the image contains the object to be recognized or not, the recognition unit 122 determines that the input image contains the object to be recognized when the acquired output information indicates that the object to be recognized is included, and detects the object. When the learning model is trained to output information indicating an object included in an image, the recognition unit 122 determines that the input image contains the object to be recognized when the acquired output information includes information indicating the object to be recognized, and detects the object.

[0071] Next, the extraction unit 123 refers to the candidate table and determines whether target information corresponding to the object recognized by the recognition unit 122 is registered (step S112). When an object to be recognized is detected from the input image by the learning model registered as target information, the extraction unit 123 determines that the target information corresponds to the object recognized by the recognition unit 122. When an area including each object in the image is set as target information, and each object registered as target information is detected within the set area of ​​the input image, the extraction unit 123 determines that the target information corresponds to the object recognized by the recognition unit 122. When target information corresponding to the object recognized by the recognition unit 122 is not registered, the extraction unit 123 does not extract a recognition module candidate and proceeds to step S114.

[0072] On the other hand, if target information corresponding to the object recognized by the recognition unit 122 is registered, the extraction unit 123 extracts all recognition module candidates registered in association with the target information (step S113).

[0073] In this way, the recognition unit 122 recognizes a character string, an image feature, or an object included in the input image. In particular, when the acquisition unit 121 receives a user's designation of a target area in the input image that includes a recognition target, the recognition unit 122 recognizes a character string, an image feature, or an object within the designated target area of ​​the input image. This allows the recognition unit 122 to recognize a necessary character string, image feature, or object from the input image with higher accuracy. Also, when the acquisition unit 121 receives a user's designation of a type of a recognition target included in the input image, the recognition unit 122 recognizes a character string, image feature, or object corresponding to the designated type. This allows the recognition unit 122 to recognize a necessary character string, image feature, or object from the input image with higher accuracy.

[0074] Furthermore, the extraction unit 123 extracts, from among the multiple recognition modules, a recognition module whose target information registered in the candidate table corresponds to a character string, an image feature, or an object recognized by the recognition unit 122, as a recognition module candidate corresponding to the input image. This allows the extraction unit 123 to extract a recognition module that matches the user's purpose from among various recognition modules.

[0075] Next, the extraction unit 123 determines whether or not a recognition module candidate has been extracted (step S114).

[0076] When at least one recognition module candidate is extracted by the extraction unit 123, the verification unit 124 executes a verification process (step S115) and ends the series of steps. The verification process will be described later in detail.

[0077] On the other hand, if no recognition module candidates are extracted by the extraction unit 123, the registration unit 126 executes a registration process (step S116) and ends the series of steps. The registration process will be described later in detail.

[0078] The acquisition unit 121 may not accept the designation of the target area and / or the target type. In that case, the process of step S103 and / or step S107 is omitted, and the process of steps S104 to S106 and / or steps S108 to S113 is always executed. Also, any one or two of steps S104 to S106, S108 to S110, and S111 to S113 may be omitted.

[0079] 7 is a flowchart showing an example of the operation of the verification process. The flow of the operation shown in FIG. 7 is executed in step S115 of the flowchart shown in FIG.

[0080] First, for each of the recognition module candidates extracted by the extraction unit 123, the verification unit 124 inputs an input image to each of the recognition module candidates and obtains output information output from each of the recognition module candidates (step S201).

[0081] The verification unit 124 transmits an acquisition instruction signal, which instructs acquisition of each recognition module candidate extracted by the extraction unit 123, to the database device 300 via the first communication device 101. The acquisition instruction signal includes identification information of each recognition module candidate (recognition module). The third processing circuit 320 of the database device 300 receives the acquisition instruction signal from the search device 100 via the third communication device 301. The third processing circuit 320 reads out the recognition module specified in the received acquisition instruction signal from the third storage device 310 and transmits it to the search device 100 via the third communication device 301. The verification unit 124 receives each recognition module candidate from the database device 300 via the first communication device 101, and inputs an input image to each received recognition module candidate to acquire output information.

[0082] The verification unit 124 may transmit an acquisition instruction signal, which instructs acquisition of output information of each recognition module candidate, to the database device 300 via the first communication device 101. In this case, the acquisition instruction signal includes identification information of each recognition module candidate (recognition module) and an input image. The third processing circuit 320 of the database device 300 receives the acquisition instruction signal from the search device 100 via the third communication device 301. The third processing circuit 320 inputs the input image to the recognition module specified in the received acquisition instruction signal to acquire output information, and transmits it to the search device 100 via the third communication device 301. The verification unit 124 receives each output information from the database device 300 via the first communication device 101.

[0083] Next, the output control unit 125 generates verification screen display data for displaying the acquired output information, and outputs it by transmitting it to the arithmetic device 200 via the first communication device 101 (step S202). The output control unit 125 refers to the candidate table, specifies the identification information of each recognition module candidate and the image type of the recognition target of each recognition module candidate, and generates the verification screen display data to display it together with the output information. The second processing circuit 220 of the arithmetic device 200 receives the verification screen display data from the search device 100 via the second communication device 201, and displays the verification screen on the second display device 203 based on the received verification screen display data. The identification information of the recognition module candidate, the image type of the recognition target of the recognition module candidate, and / or the output information of the recognition module candidate, which are included in the verification screen display data, are examples of information about the recognition module candidate.

[0084] 8 and 9 are schematic diagrams showing an example of a verification screen 800 displayed on the second display device 203. FIG.

[0085] A verification screen 800 shown in Fig. 8 is an example of a verification screen when a search is performed on the search screen 500 shown in Fig. 5, and a verification screen 800 shown in Fig. 9 is an example of a verification screen when a search is performed on the search screen 500 shown in Fig. 6. As shown in Fig. 8 and Fig. 9, the verification screen 800 includes an image display area 501, an area designation button 502, an object type designation box 503, a search button 504, etc., similar to the search screen 500. The verification screen 800 further includes a selection button 801, a result display area 802, an acquisition button 803, a registration button 804, etc.

[0086] The selection button 801 is a button that allows the user to select a recognition module candidate when multiple recognition module candidates are extracted.

[0087] The result display area 802 is an area for displaying the identification information of each recognition module candidate, the image type of the recognition target of each recognition module candidate, and the output information (tag) output from each recognition module candidate. Information about the recognition module candidate selected using the selection button 801 is displayed in the result display area 802. Note that if the recognition module candidate fails in recognition and the verification unit 124 is unable to obtain output information from the recognition module candidate, a blank (white space) is displayed as the output information output from that recognition module candidate.

[0088] The acquisition button 803 is a button for acquiring a recognition module. When the user presses the acquisition button 803, the second processing circuit 220 identifies the recognition module candidate selected using the selection button 801, and transmits an acquisition request signal requesting acquisition of the identified recognition module candidate to the search device 100 via the second communication device 201. The acquisition request signal includes identification information of the identified recognition module candidate.

[0089] The registration button 804 is a button for registering a new recognition module using the input image displayed in the image display area 501. When the user presses the registration button 804, the second processing circuit 220 transmits a registration process request signal, which requests execution of registration process of the new recognition module, to the search device 100 via the second communication device 201.

[0090] Next, the verification unit 124 determines whether or not acquisition of the recognition module has been requested (step S203). When the verification unit 124 receives an acquisition request signal from the arithmetic device 200 via the first communication device 101, it determines that acquisition of the recognition module has been requested.

[0091] When acquisition of a recognition module is requested, the output control unit 125 outputs the recognition module corresponding to the identification information included in the received acquisition request signal by transmitting it to the arithmetic device 200 via the first communication device 101 (step S204), and ends the series of steps. When the recognition module is not stored in the first storage device 110, the output control unit 125 transmits an acquisition instruction signal requesting acquisition of the recognition module to the database device 300 via the first communication device 101. The output control unit 125 receives the recognition module from the database device 300 via the first communication device 101, and transmits the received recognition module to the arithmetic device 200 via the first communication device 101. The recognition module transmitted by the output control unit 125 is an example of information regarding candidate recognition modules.

[0092] The second processing circuit 220 of the arithmetic device 200 receives the recognition module from the search device 100 via the second communication device 201, and stores the received recognition module in the second storage device 210. After that, the second processing circuit 220 acquires an input image by receiving it from another information processing device via the second communication device 201, or by accepting it from a user using an interface device (not shown). The arithmetic device 200 may have an imaging device such as a camera, and the second processing circuit 220 may acquire an image captured by the imaging device as an input image. The second processing circuit 220 inputs the acquired input image to the recognition module and acquires information output from the recognition module, thereby recognizing a character string, image feature amount, or object in the input image. The second processing circuit 220 displays the recognition result on the second display device 203, or transmits it to another information processing device via the second communication device 201.

[0093] On the other hand, if acquisition of a recognition module is not requested, the verification unit 124 determines whether or not registration of a recognition module is requested (step S205). If the verification unit 124 receives a registration process request signal from the arithmetic device 200 via the first communication device 101, it determines that registration of a recognition module is requested. If registration of a recognition module is not requested, the verification unit 124 returns the process to step S203.

[0094] On the other hand, if registration of the recognition module is requested, the registration unit 126 executes the registration process (step S206), and ends the series of steps.

[0095] That is, the registration unit 126 executes the registration process when the user cannot find a recognition module that outputs the recognition result expected by referring to the verification screen. Note that the registration unit 126 may automatically execute the registration process when the verification unit 124 cannot obtain output information from all the recognition module candidates extracted by the extraction unit 123.

[0096] Fig. 10 is a flowchart showing an example of the operation of the registration process. The flow of the operation shown in Fig. 10 is executed in step S116 of the flowchart shown in Fig. 4 and in step S206 of the flowchart shown in Fig. 7.

[0097] First, the registration unit 126 generates registration screen display data for displaying a registration screen used to register a new recognition module, and transmits the same to the arithmetic device 200 via the first communication device 101 (step S301). The second processing circuit 220 of the arithmetic device 200 receives the registration screen display data from the retrieval device 100 via the second communication device 201, and displays the registration screen on the second display device 203 based on the received registration screen display data.

[0098] 11 and 12 are schematic diagrams showing an example of a registration screen 1100 displayed on the second display device 203. FIG.

[0099] A registration screen 1100 shown in Fig. 11 is an example of a registration screen when a search is executed on the search screen 500 shown in Fig. 5, and a registration screen 1100 shown in Fig. 12 is an example of a registration screen when a search is executed on the search screen 500 shown in Fig. 6. As shown in Fig. 11 and Fig. 12, the registration screen 1100 includes an image display area 1101, a first area designation button 1102, a second area designation button 1103, an object type designation box 1104, an image type designation box 1105, a generate button 1106, a registration button 1107, a result display area 1108, and the like.

[0100] The image display area 1101 is an area for displaying an input image specified by a user on the search screen 500. The registration screen 1100 in Fig. 11 displays an image 511 of a printed board, and the registration screen 1100 in Fig. 12 displays an image 611 of an invoice.

[0101] The first area designation button 1102 is a button for designating a first area in an input image displayed in the image display area 1101. When the recognition target is an object, the first area is an area in which a learning model, which is a recognition module, performs learning. The user selects an area including the recognition target in the input image displayed in the image display area 1101 and presses the first area designation button 1102 to designate the first area. The number of first areas designated is not limited to one, and may be two or more. In the registration screen 1100 of FIG. 11, a local area 1112 including a predetermined LSI in the image 511 is designated as the first area. In the registration screen 1100 of FIG. 12, the first area is not designated. In this case, the entire area in the image 611 becomes the first area. When the user newly designates an area including the recognition target, the registration unit 126 can generate a recognition module that matches the user's purpose.

[0102] The second area designation button 1103 is a button for designating a second area including a recognition target in the input image displayed in the image display area 1101. The second area is an area in which the recognition module recognizes the recognition target when the recognition target is a character string or an image feature. The user selects an area including the recognition target in the input image displayed in the image display area 1101 and presses the second area designation button 1103 to designate the second area. The number of second areas to be designated is not limited to one, and may be two or more. In the registration screen 1100 of FIG. 11, a local area 1113 including a predetermined mark in the image 511 is designated as the second area. In the registration screen 1100 of FIG. 12, a local area 1213 including a predetermined character string in the image 611 and a local area 1214 including a predetermined amount are designated as the second area. When the user newly designates an area including the recognition target, the registration unit 126 can generate a recognition module that matches the user's purpose.

[0103] In addition, the user specifies the attributes of the recognition target included in each of the specified first and second regions. If the first and second regions are not specified, the attributes of the recognition target included in all regions are specified. If the recognition target is a character string, the character string itself or the parameter name of the character string (billing amount, address, name, etc.) is specified as the attribute. If the recognition target is an image feature, the type of image feature (SIFT, AKAZE, ORB) is specified as the attribute. If the recognition target is an object, the object name is specified as the attribute. In the registration screen 1100 of FIG. 11, "LSI" is specified as the attribute of the first region 1112, and "SIFT" is specified as the attribute of the second region 1113. In addition, in the registration screen 1100 of FIG. 12, "bill" is specified as the recognition target of the first region 1112, and the parameter name "billing amount" is specified as the recognition target of the second region 1113.

[0104] The object type designation box 1104 is a button for designating the object type of the recognition target to be recognized from the input image displayed in the image display area 1101. In the registration screen 1100 of Fig. 11, an image is designated as the object type. In the registration screen 1100 of Fig. 12, a character string is designated as the object type.

[0105] The image type designation box 1105 is a button for designating the type (items contained in the entire image) of the input image displayed in the image display area 1101. In the registration screen 1100 of Fig. 11, "printed board" is designated as the image type. In the registration screen 1100 of Fig. 12, "invoice" is designated as the image type.

[0106] The generate button 1106 is a button for generating a new recognition module using the input image displayed in the image display area 1101. When the user presses the generate button 1106 using the second input device 202, the second processing circuit 220 generates a generation request signal requesting the generation of a new recognition module, and transmits the signal to the search device 100 via the second communication device 201. The generation request signal includes the input image and learning definition data such as the first area, second area, attributes of each area, target type, and / or image type specified by the user.

[0107] The registration button 1107 is a button for registering a newly generated recognition module. When the user presses the registration button 1107 using the second input device 202, the second processing circuit 220 generates a registration request signal requesting registration of the newly generated recognition module, and transmits the signal to the search device 100 via the second communication device 201.

[0108] The result display area 1108 is an area for displaying output information (tags) etc. output from a newly generated recognition module. Before a new recognition module is generated, the result display area 1108 displays a blank space.

[0109] Next, when the generate button 1106 is pressed on the registration screen 1100, the acquisition unit 121 receives a generation request signal from the arithmetic device 200 via the first communication device 101. The acquisition unit 121 acquires the input image, the first region, the second region, the attributes of each region, the object type, and / or the image type included in the generation request signal. As a result, the acquisition unit 121 accepts designation by the user of the region in the input image including the recognition target, the attributes of the recognition target included in each region, the object type of the recognition target included in the input image, and / or the image type of the input image (step S302).

[0110] Next, the registration unit 126 determines whether or not a first area has been designated by the user (step S303).

[0111] If the first region is not specified, the registration unit 126 determines whether the target type specified by the user is a character string (step S304).

[0112] If the specified object type is a character string, the registration unit 126 generates a new recognition module that outputs a character string included in the specified second region when an image is input (step S305). The character string output by the recognition module is an example of information related to the recognition object.

[0113] The registration unit 126 inputs an input image to the generated recognition module and acquires a character string output by the recognition module. The registration unit 126 updates the registration screen display data so as to display the acquired character string, and transmits the data to the arithmetic device 200 via the first communication device 101. The second processing circuit 220 of the arithmetic device 200 receives the updated registration screen display data from the search device 100 via the second communication device 201, and displays the updated registration screen on the second display device 203 based on the received registration screen display data. In the example shown in FIG. 12, the character strings included in the specified second area 1213 and second area 1214 are displayed in the result display area 1108 as output information (tags) output from the newly generated recognition module.

[0114] Next, when the registration button 1107 is pressed on the registration screen 1100, the registration unit 126 receives a registration request signal from the arithmetic device 200 via the first communication device 101, and registers the newly generated recognition module in the database device 300 (step S306). The registration unit 126 transmits a registration instruction signal to the database device 300 via the first communication device 101, which instructs the database device 300 to register the newly generated recognition module. The registration instruction signal includes the newly generated recognition module. The database device 300 receives the registration instruction signal from the search device 100 via the third communication device 301, and stores and registers the recognition module included in the received registration instruction signal in the third storage device 310.

[0115] Next, the registration unit 126 registers information about the newly generated recognition module in the candidate table (step S307). The registration unit 126 adds a new item to the candidate table, assigns a new target ID, registers identification information of the newly generated recognition module as identification information of the recognition module candidate, and registers the target type and image type included in the generation request signal. The registration unit 126 also registers information about the character string to be recognized in the candidate table as target information of the recognition module. The registration unit 126 uses each second area included in the generation request signal and the attribute of the recognition target included in each second area (character string or parameter name) as information about the character string to be recognized.

[0116] Next, the registration unit 126 transmits the newly generated recognition module to the arithmetic device 200 via the first communication device 101 (step S308), and the series of steps ends. Similar to the process of step S204, the second processing circuit 220 of the arithmetic device 200 receives the newly generated recognition module from the search device 100 via the second communication device 201.

[0117] On the other hand, if the specified object type is not a character string but an image in step S304, the registration unit 126 generates a new recognition module that outputs image features included in the specified second region when an image is input (step S309). The registration unit 126 generates a recognition module that outputs image features of an attribute (type) specified by the user. The registration unit 126 may generate recognition modules that each output image features of one or more predetermined types. The image features output by the recognition module are an example of information related to the recognition target.

[0118] The registration unit 126 inputs an input image to the generated recognition module and acquires image features output by the recognition module. The registration unit 126 updates the registration screen display data so as to display the acquired image features, and transmits the same to the arithmetic device 200 via the first communication device 101. The second processing circuit 220 of the arithmetic device 200 receives the updated registration screen display data from the search device 100 via the second communication device 201, and displays the updated registration screen on the second display device 203 based on the received registration screen display data. In the example shown in FIG. 11, the image features included in the specified second area 1113 are displayed in the result display area 1108 as output information (tags) output from the newly generated recognition module.

[0119] Next, when the registration button 1107 is pressed on the registration screen 1100, the registration unit 126 registers the recognition module in the database device 300 in the same manner as in the process of step S306 (step S310).

[0120] Next, the registration unit 126 registers information about the newly generated recognition module in the candidate table in the same manner as in the process of step S307 (step S311). The registration unit 126 registers information about the image feature of the recognition target in the candidate table as target information of the recognition module. The registration unit 126 uses each second region included in the generation request signal, the image feature calculated in each second region, and the attribute (type of image feature) of the recognition target included in each second region as information about the image feature of the recognition target.

[0121] Next, the registration unit 126 transmits the newly generated recognition module to the arithmetic device 200 via the first communication device 101 (step S308), and the series of steps ends.

[0122] On the other hand, if a first region is specified by the user in step S303, the registration unit 126 determines whether or not a predetermined number or more of learning images including an object included in the specified first region are stored in the first storage device 110 (step S312). The predetermined number is set to a number sufficient for generating a learning model with a certain level of accuracy. The registration unit 126 determines whether or not a predetermined number or more of learning images including an object included in the specified first region are stored depending on whether or not a predetermined number or more of learning images including an object indicated in an attribute (object name) included in the generation request signal are stored.

[0123] If the predetermined number or more of learning images including the object included in the first region are not stored, the registration unit 126 executes the processes of steps S309 to S311 and S308, and ends the series of steps. That is, in this case, when an image is input, the registration unit 126 newly generates a recognition module that outputs an image feature included in the specified second region, registers it in the database device 300, and registers information about the recognition module in the candidate table.

[0124] On the other hand, if a predetermined number or more of learning images including an object included in the first region are stored, the registration unit 126 newly generates a recognition module that outputs information about the object included in the specified first region when an image is input (step S313). The registration unit 126 generates a learning model as a recognition module that has been trained to output information about the object when an image is input using learning images including an object included in the specified first region by a predetermined method such as YOLO, SSD, M2Det, etc. The information about the object is an example of information about the recognition target, and is the probability that the image includes a predetermined object, information indicating whether the image includes a predetermined object or not, or information indicating the object included in the image, and information indicating the area in which each object is included in the image.

[0125] The registration unit 126 inputs an input image to the generated recognition module and acquires output information output by the recognition module. The registration unit 126 updates the registration screen display data to display the acquired output information, and transmits the updated registration screen display data to the arithmetic device 200 via the first communication device 101. The second processing circuit 220 of the arithmetic device 200 receives the updated registration screen display data from the retrieval device 100 via the second communication device 201, and displays the updated registration screen on the second display device 203 based on the received registration screen display data.

[0126] In this way, the registration unit 126 generates a learning model when a predetermined number or more of learning images including an object included in the first region are stored, and generates a recognition module that calculates image features when the predetermined number or more of such learning images are not stored. That is, the registration unit 126 generates a learning model when an accurate learning model can be generated, and generates a recognition module that calculates image features when an accurate learning model cannot be generated. This allows the registration unit 126 to generate a recognition module with the highest possible performance within the scope of the resources (learning images) it possesses.

[0127] Next, when the registration button 1107 is pressed on the registration screen 1100, the registration unit 126 registers the recognition module in the database device 300 in the same manner as in the process of step S306 (step S314).

[0128] Next, the registration unit 126 registers information about the newly generated recognition module in the candidate table in the same manner as the process of step S307 (step S315). The registration unit 126 registers information about the object to be recognized in the candidate table as target information of the recognition module. The registration unit 126 uses each of the first regions included in the generation request signal and the newly generated recognition module itself corresponding to each of the first regions as information about the object to be recognized.

[0129] Next, the registration unit 126 transmits the newly generated recognition module to the arithmetic device 200 via the first communication device 101 (step S308), and the series of steps ends.

[0130] In this way, when the extraction unit 123 has not extracted a recognition module candidate, or when the user determines that there is no matching recognition module, the registration unit 126 generates and registers a new recognition module. In this way, when the recognition module desired by the user does not exist, the search device 100 can enrich the database of recognition modules while providing the user with the recognition module desired by the user. In addition, the search device 100 can register various types of recognition modules corresponding to character strings, image features, or objects using a screen having a common user interface, thereby improving user convenience.

[0131] As described above in detail, the search device 100 stores a table of information on the recognition targets of a plurality of recognition modules, recognizes character strings, image features, and objects from an input image, and extracts (primary sorting) candidates for the corresponding recognition modules. This enables the search device 100 to effectively search for a usable recognition module from among existing recognition modules.

[0132] The search device 100 can effectively search for a usable recognition module, particularly a learning model, from among various types of recognition modules developed in various industries. Furthermore, users can easily search for a recognition module that matches their own applications, making it possible to improve the efficiency of system development. In particular, even if a user does not have specialized knowledge of image processing, the user can easily search for a recognition module that matches their own applications. Therefore, the search device 100 can improve the convenience for users.

[0133] FIG. 13 is a block diagram showing a schematic configuration of a first processing circuit 420 in a search device according to another embodiment.

[0134] The first processing circuit 420 is used in place of the first processing circuit 120, and executes search processing in place of the first processing circuit 120. The first processing circuit 420 includes an acquisition circuit 421, a recognition circuit 422, an extraction circuit 423, a verification circuit 424, an output control circuit 425, a registration circuit 426, and the like.

[0135] The acquisition circuit 421 is an example of an acquisition unit, and has the same function as the acquisition unit 121. The acquisition circuit 421 receives an input image, an area in the input image including a recognition target, a type of the recognition target, and the like from the first communication device 101, and outputs them to the recognition circuit 422.

[0136] The recognition circuit 422 is an example of a recognition unit, and has the same function as the recognition unit 122. The recognition circuit 422 recognizes a recognition target according to the input type within an input area of ​​the input image input from the acquisition circuit 421, and outputs the recognition result to the extraction circuit 423.

[0137] The extraction circuit 423 is an example of an extraction unit, and has the same function as the extraction unit 123. The extraction circuit 423 extracts recognition module candidates from the candidate table read from the first storage device 110 based on the recognition result input from the extraction circuit 423, and outputs the extraction result to the verification circuit 424.

[0138] The verification circuit 424 is an example of a verification unit, and has the same function as the verification unit 124. The verification circuit 424 receives the recognition module candidates extracted by the extraction circuit 423 from the database device 300 via the first communication device 101, verifies the received recognition module candidates, and outputs the verification result to the output control circuit 425.

[0139] The output control circuit 425 is an example of an output control unit, and has the same function as the output control unit 125. The output control circuit 425 generates verification screen display data based on the verification result input from the verification circuit 424, and transmits the data to the arithmetic device 200 via the first communication device 101.

[0140] The registration circuit 426 is an example of a registration unit, and has the same function as the registration unit 126. The registration circuit 426 generates a new recognition module in accordance with a generation request signal and a registration request signal received from the arithmetic device 200 via the first communication device 101. The registration circuit 426 transmits the generated recognition module to the database device 300 via the first communication device 101, and stores information about the generated recognition module in the first storage device 110.

[0141] As described above in detail, even when the first processing circuit 420 is used, the search device is able to effectively search for a usable recognition module from among existing recognition modules.

[0142] Although preferred embodiments have been described above, the search system is not limited to these embodiments. For example, the multiple recognition modules may be registered in the search device 100 instead of being registered in the database device 300. In this case, the search device 100 acquires the multiple recognition modules by reading them from the first storage device 110 instead of receiving them from the database device 300. Also, the search device 100 registers the multiple recognition modules by storing them in the first storage device 110 instead of transmitting them to the database device 300. Also, the calculation device 200 may receive the recognition modules directly from the database device 300 instead of receiving them from the search device 100. [Explanation of symbols]

[0143] 100 Search device, 101 First communication device, 110 First storage device, 121 Acquisition unit, 122 Recognition unit, 123 Extraction unit, 124 Verification unit, 125 Output control unit, 126 Registration unit

Claims

1. A memory unit that stores a table in which target information relating to a character string, image feature or object to be recognized by each of a character string recognition module that outputs information about a character string to be recognized when an image is input, an image feature recognition module that outputs information about an image feature of a target to be recognized when an image is input, and an object recognition module that outputs information about an object to be recognized when an image is input; An acquisition unit for acquiring an input image; a recognition unit that recognizes a character string, an image feature, or an object included in the input image; an extraction unit that extracts, from among the character string recognition modules, the image feature recognition modules, and the object recognition modules, a recognition module whose target information registered in the table corresponds to a character string, an image feature, or an object recognized by the recognition unit, as a candidate recognition module corresponding to the input image; an output unit that outputs information about the recognition module candidates; A search device comprising:

2. a verification unit that inputs the input image to the candidate recognition module extracted by the extraction unit and acquires information output from the candidate recognition module; The search device according to claim 1 , wherein the output unit outputs information output from the recognition module candidate as the information relating to the recognition module candidate.

3. The search device according to claim 1 , wherein the object recognition module includes a learning model that is trained to output information about a recognition target when an image is input.

4. The acquisition unit receives a designation of an area including a recognition target in the input image by a user, The search device according to claim 1 , wherein the recognition unit recognizes a character string, an image feature, or an object within the specified region of the input image.

5. The acquisition unit accepts a user's designation of a type of a recognition target included in the input image, The search device according to claim 1 , wherein the recognition unit recognizes a character string, an image feature, or an object corresponding to the specified type.

6. When the extraction unit has not extracted the recognition module candidate, the acquisition unit accepts a user's designation of an area in the input image that includes a recognition target; and The search device according to any one of claims 1 to 5, further comprising a registration unit that generates a new recognition module that outputs information about a recognition target included in the specified area when an image is input, and registers information about a character string, image feature, or object of the recognition target in the table as target information of the recognition module.

7. The storage unit further stores a plurality of learning images; The search device of claim 6, wherein when a predetermined number or more of learning images including an object included in the specified area are stored in the memory unit, the registration unit generates, as the object recognition module, a learning model that is trained to output information about the object when an image is input using the learning images including the object included in the specified area.

8. The computer storing a table in which target information relating to a character string, image feature or object to be recognized is registered for each of a character string recognition module which outputs information relating to a character string to be recognized when an image is input, an image feature recognition module which outputs information relating to an image feature of a recognition target when an image is input, and an object recognition module which outputs information relating to an object to be recognized when an image is input; Take an input image, Recognizing a character string, an image feature, or an object contained in the input image; extracting, from among the character string recognition module, the image feature recognition module, and the object recognition module, a recognition module whose target information registered in the table corresponds to the recognized character string, image feature, or object, as a candidate recognition module corresponding to the input image; outputting information about the candidate recognition modules; A search method comprising:

9. A control program for a computer having a storage unit, storing in the storage unit a table in which target information relating to a character string, image feature or object to be recognized by each of a character string recognition module which outputs information relating to a character string to be recognized when an image is input, an image feature recognition module which outputs information relating to an image feature of a recognition target when an image is input, and an object recognition module which outputs information relating to an object to be recognized when an image is input; Take an input image, Recognizing a character string, an image feature, or an object contained in the input image; extracting, from among the character string recognition module, the image feature recognition module, and the object recognition module, a recognition module whose target information registered in the table corresponds to the recognized character string, image feature, or object, as a candidate recognition module corresponding to the input image; outputting information about the candidate recognition modules; A control program for causing the computer to execute the above steps.

Images