Program, method, information processing device and system

Abstract

When searching for data using information about drawings, it can flexibly accommodate a variety of use cases. [Solution] A program to be executed by a computer having a processor and a memory. The program causes the processor to execute an output step of inputting drawing information related to a drawing, estimate information related to an estimate for an illustrated object illustrated in the drawing, and a first prompt instructing output of first information related to the illustrated object to a generative AI model and outputting the first information from the generative AI model, a construction step of storing the outputted first information in a database and constructing a search database, an input step of accepting input of second information related to a search target, and a presentation step of searching the search database for first information similar to the second information based on the accepted second information, as similar information, and presenting search information related to the search result.

Description

[Technical field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] Conventionally, a technique for searching for data similar to a search target using a database storing output data of a generative AI model has been known. For example, Patent Document 1 discloses a technique for calculating the similarity between a search image and an answer sentence to a question sentence about the search image (hereinafter, a first answer sentence) and a reference image and an answer sentence to a question sentence about the reference image (hereinafter, a second answer sentence) to search for a similar reference image similar to the search image with respect to the first answer sentence. The first answer sentence is obtained as an output result when the search image and a question sentence about the search image are input to a VQA (Visual Question Answering) model. The reference image and the second answer sentence are a search image and a first answer sentence used in a past search, and are stored in a reference case database in association with each other. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-39656 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 is a technology that is premised on being used in cases where humans and / or machines are the subject of image capture, such as disasters, accidents, breakdowns, and / or incidents, and is not premised on the subject of image capture, for example, blueprints and estimates. Therefore, with the technology disclosed in Patent Document 1, it is difficult to support searches using captured images of blueprints and captured images of estimates. Furthermore, it is extremely difficult to flexibly support various use cases, such as performing a search for blueprints while taking into account information on estimates, or performing a search for estimates while taking into account information on blueprints.

[0005] The objective of this disclosure is to flexibly respond to a variety of use cases in data searches using information related to drawings. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, a program according to one embodiment of the present disclosure is a program to be executed by a computer having a processor and a memory. The program causes the processor to execute an output step of inputting drawing information on a drawing, estimate information on an estimate of an illustrated object illustrated in the drawing, and a first prompt instructing an output of first information on the illustrated object to a generative AI model and outputting the first information from the generative AI model, a construction step of storing the outputted first information in a database and constructing a search database, an input step of accepting input of second information on a search target, and a presentation step of searching the search database for first information similar to the second information based on the accepted second information, and presenting search information related to the search result. Effect of the Invention

[0007] According to the present disclosure, data searches using drawing information can be flexibly adapted to a variety of use cases. [Brief description of the drawings]

[0008] [Figure 1]1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Diagram 2] 2 is a block diagram showing an example of the configuration of a terminal device 10 shown in FIG. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of a server 20 shown in FIG. 1. [Figure 4] 4 is a diagram showing a data structure of a user information table 2023 shown in FIG. [Diagram 5] FIG. 4 is a diagram showing a data structure of a search database 2024 shown in FIG. [Figure 6] 13 is a flowchart showing an example of the operation of the server 20 when presenting search information to a user. [Figure 7] 11 is a schematic diagram showing an example of an operation surface of a touch-sensitive device 131 when a user inputs second information. FIG. [Figure 8] 13 is a schematic diagram showing an example of a display screen of a display 141 when search information is presented to a user. FIG. [Figure 9] 13 is a schematic diagram showing another example of the display screen of the display 141 when searched information is presented to the user. FIG. [Figure 10] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiment, the same reference numerals are given to common components, and repeated explanations are omitted. Note that the following embodiment does not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiment are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0010] [1 Overall system configuration] Fig. 1 is a block diagram showing an example of the overall configuration of system 1. System 1 shown in Fig. 1 is a system for providing, for example, a drawing search service, a quotation search service, or a search service relating to a combination of these two (hereinafter collectively referred to as search services). System 1 includes, for example, a terminal device 10, a server 20, and a generation AI system 30. The terminal device 10, the server 20, and the generation AI system 30 are communicatively connected, for example, via a network 80.

[0011] FIG. 1 shows an example in which system 1 includes two terminal devices 10, but the number of terminal devices 10 included in system 1 may be less than three or more than three. FIG. 1 shows an example in which system 1 includes one generative AI system 30, but the number of generative AI systems 30 included in system 1 may be two or more. FIG. 1 shows an example in which generative AI system 30 is independent from server 20, but server 20 may include the functions of generative AI system 30.

[0012] In this embodiment, a collection of multiple devices may be one server. The method of allocating multiple functions required to realize the server 20 according to this embodiment to one or multiple pieces of hardware can be appropriately determined according to the processing capacity of each piece of hardware and / or the specifications required for the server 20.

[0013] 1 is, for example, an information processing device operated by a user (hereinafter, abbreviated as "user") who uses a search service. The terminal device 10 is realized by, for example, a mobile terminal such as a smartphone or a tablet. The terminal device 10 may also be realized by, for example, a stationary PC (Personal Computer), a laptop PC, or the like. In this embodiment, the terminal device 10 is assumed to be a tablet.

[0014] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving an input operation from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.). In this embodiment, the terminal device 10 includes a touch panel in which the input device 13 and the output device 14 are integrated.

[0015] The server 20 is, for example, an information processing device that develops a search service, and manages and processes various information used in the search service. The server 20 is realized by, for example, a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an interface of an input device for receiving an input operation from a user, and an output device for outputting information to the user.

[0016] The generative AI system 30 has, for example, a first generative AI model and a second generative AI model different from the first generative AI model. The first generative AI model and the second generative AI model are, for example, a large-scale language model or a multimodal generative AI model. The large-scale language model is an artificial intelligence system specialized in natural language processing, trained using a large amount of data set, and is constructed based on a neural network such as a transformer architecture. The multimodal generative AI model is an artificial intelligence system constructed by deep learning that collects two or more types of information such as text, audio, images, and videos, and processes them in an integrated manner. An example of a large-scale language model is "OpenAI ChatGPT GTP-3.5 (registered trademark)". An example of a multimodal generative AI model is "OpenAI ChatGPT GTP-4 (registered trademark)".

[0017] Each information processing device is configured by a computer 90 (see FIG. 10) equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer 90 and the basic functional configuration of the computer 90 realized by the basic hardware configuration will be described later. For the terminal device 10, the server 20, and the generation AI system 30, descriptions that overlap with the basic hardware configuration of the computer 90 and the basic functional configuration of the computer will be omitted.

[0018] <1.1 Terminal device configuration> Fig. 2 is a block diagram showing an example of the configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected to each other via a bus or the like. The terminal device 10 may include a voice processing unit, a microphone, a speaker, a camera, a position information sensor, or a combination of at least two of these.

[0019] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on a signal generated by the control unit 190 and transmits the signal to the outside (for example, the server 20). The communication unit 120 performs reception processing on a signal received from the outside and outputs the signal to the control unit 190.

[0020] The input device 13 is a device for inputting instructions or information by a user who operates the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 in which an instruction is input by touching an operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, a keyboard, a mouse, or the like. The input device 13 converts an instruction input by a user into an electrical signal and outputs it to the control unit 190. The input device 13 may include, for example, a receiving port that receives an electrical signal input from an external input device.

[0021] The output device 14 is a device for presenting information to a user who operates the terminal device 10. The output device 14 is realized, for example, by a display 141. The display 141 displays various information according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0022] The storage unit 180 is realized by, for example, the memory 15 and the storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, an app 181. The apps stored in the storage unit 180 are not limited to the app 181. The app 181 is an application for managing the use of a search service by a user. The app 181 is installed in the terminal device 10, and is a concept including a browser. The app 181 runs, for example, in the background of other apps installed in the terminal device 10, and monitors processes run by the user.

[0023] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating according to the program.

[0024] The operation reception unit 191 performs a process for receiving instructions or information input from the input device 13. Specifically, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131.

[0025] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, the transmitting / receiving unit 192 transmits instructions or information input by a user to the server 20. The transmitting / receiving unit 192 receives information transmitted from the server 20.

[0026] The presentation control unit 193 controls the output device 14 to present the information transmitted from the server 20 to the user. Specifically, the presentation control unit 194 causes the display 141 to display the search information received from the server 20. Details of the search information will be described later.

[0027] <1.2 Functional configuration of the server> Fig. 3 is a block diagram showing an example of a functional configuration of the server 20 shown in Fig. 1. As shown in Fig. 3, the server 20 performs the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0028] The communication unit 201 performs processing for the server 20 to communicate with an external device. The storage unit 202 stores, for example, drawing information 2021 and estimate information 2022.

[0029] The drawing information 2021 is information about drawings, and specifically, for example, is a data file containing information about drawings provided by a manufacturing supplier. The file format of the drawing information 2021 varies from supplier to supplier, and there are various formats. The drawing information 2021 may be, for example, image data or text information. The image data may or may not contain text information. The file format of the image data may be, for example, TIFF, JPEG, PDF (registered trademark), etc. Examples of text information that may be included in the drawing information 2021 include the drawing name, an overview of the product (or part), dimensional information (including dimensional tolerances), material information, information about the manufacturing method (processing method, molding method, presence or absence of welding, etc.), product name (or part name), product number, etc., if the illustrated object illustrated in the drawing is an industrial product (or its part).

[0030] In this embodiment, the illustrated object shown in the drawing is described as a part of an industrial product. However, there is no particular limitation on the type of the illustrated object. For example, the illustrated object may be the industrial product itself, that is, a finished product. Also, for example, the illustrated object may be a part of a part or a part of a finished product. Alternatively, the illustrated object may be a part or the whole of a product.

[0031] The quotation information 2022 is information on a quotation for a part illustrated in a drawing on which the drawing information 2021 is based, and specifically, for example, is a data file including information on a quotation provided by a supplier of the part. The quotation is, for example, a document received from a supplier in order to understand the appropriateness of the price of the part before ordering the part, and includes detailed details of the unit price (total), such as material cost, processing cost, and management cost. The file format of the quotation information 2022 varies from supplier to supplier, and there are various formats. The quotation information 2022 may be, for example, image data or text information. Furthermore, the image data may or may not include text information. The file format of the image data may be, for example, any of PNG, TIFF, JPEG, PDF, etc. Examples of text information that may be included in the quotation information 2022 include, for example, the above-mentioned material cost, processing cost, and management cost, as well as part name, part number, weight, dimensional information, standard number, construction method, supplier information, etc. It should be noted that the quotation to which the system 1 is applied is not limited to the quotation provided by the supplier. For example, the system 1 may be applied to a quotation prepared by a user (such as a company using a search service) himself / herself. For example, the quotation to which the system 1 is applied may also include, for example, a cost breakdown indicating the expenses and costs required to provide the parts.

[0032] The storage unit 202 stores, for example, a data set of drawing information 2021 and estimate information 2022 corresponding to a predetermined part (hereinafter, referred to as a basic information data set). In addition, from the viewpoint of simplifying the illustration, only one piece of drawing information 2021 and one piece of estimate information 2022 are stored in the storage unit 202 in the example of Fig. 3, but in reality, a basic information data set is stored for each type of part.

[0033] For example, in the case of competitive bids in which quotations are requested from multiple suppliers for one part, the basic information dataset may be configured such that multiple pieces of quotation information 2022 are associated with one piece of drawing information 2021. In addition, for example, in the case where a part illustrated in a drawing on which the drawing information 2021 is based is an assembly and is composed of multiple components, the basic information dataset may be configured such that one piece of quotation information 2022 is associated with information on multiple components included in one piece of drawing information 2021.

[0034] Regarding the upload of the drawing information 2021 and the quotation information 2022 to the server 20, for example, a plurality of basic information data sets uploaded to the server 20 in the past from an information processing device operated by the terminal device 10 or a user of the search service may be accumulated in the storage unit 202. Here, the "past" in "a plurality of basic information data sets uploaded to the server 20 in the past" specifically refers to, for example, the time when a designer of a part operates the terminal device 10 or the like to upload the drawing information 2021 of a drawing drawn by the designer himself to the server 20. The designer of the part is, for example, a designer of a company to which the user belongs, or a designer of a company to which the user of the search service belongs. In this case, the buyer of the part is, for example, a user or a user of the search service. In other words, when the user who is a buyer or a user of the search service receives a drawing from the designer, the drawing information 2021 has already been uploaded. The buyer receives a quotation from the supplier by providing the supplier with the drawing received from the designer.

[0035] In this embodiment, the drawing information 2021 is assumed to be image data of a drawing previously uploaded to the server 20. The quotation information 2022 is assumed to be text information relating to an estimate previously uploaded to the server 20. The text information relating to the estimate may be, for example, text information at the time of uploading to the server 20, or may be turned into text information by applying OCR processing or the like after the image data of the estimate is uploaded to the server 20. That is, in this embodiment, a data set of the image data of the drawing and the image data of the estimate becomes a basic information data set.

[0036] Furthermore, the storage unit 202 has, for example, a user information table 2023 and a search database 2024. The types of tables / databases that the storage unit 202 has are not limited to these. For example, the search database 2024 does not have to be stored in the storage unit 202. Specifically, the content of the search service may differ depending on the content of the first information stored in the search database 2024. Therefore, the search database 2024 may be stored for each server that manages search services with different contents. Also, for example, the search database 2024 may be stored in a server related to a search service for sharing information among a plurality of users.

[0037] The user information table 2023 is a table that stores various information related to users. The search database 2024 is a database that stores a plurality of first information. The user information table 2023 and the search database 2024 will be described in detail later.

[0038] The first information is information about an illustrated object illustrated in a drawing that is the basis of the drawing information 2021. In this embodiment, the first information is information about a part illustrated in a drawing that is the basis of the drawing information 2021. The first information is basically text information, but may also include image data.

[0039] In the case where the illustrated object is a part as in this embodiment, the text information as the first information is a sentence and text data explaining the components and functions of the part. The text information as the first information includes, for example, the product name, product number, material, dimensions, function, characteristics, construction method, installation method, maintenance method, etc. The image data as the first information is image data representing the appearance and structure of the part. The image data as the first information includes, for example, a captured image of the actual appearance of the part, a captured image of a blueprint or cross-section of the part, scan data of a blueprint or cross-section of the part, two-dimensional data representing a blueprint or cross-section of the part, three-dimensional data such as CAD data of the part, and a captured image of a manufacturing process of the part. The file format of the image data may be, for example, any of TIFF, JPEG, PDF (registered trademark), etc.

[0040] The first information is output from the first generation AI model by inputting drawing information 2021, estimate information 2022, and a first prompt that instructs the output of the first information into the first generation AI model (details will be described later).

[0041] Specifically, the first prompt plays a role of instructing the first generation AI model on what kind of first information to generate and output. For example, a first prompt such as "Please extract common information and highly related information between the part information of the part illustrated in the drawing on which the drawing information 2021 is based and the part information of the part to be estimated on which the quotation information 2022 is based, and use it as the first information" is assumed. In addition, a first prompt such as "Please output the first information in a text format and a list format" is assumed.

[0042] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the terminal device 10. The control unit 203 operates according to the program to fulfill the functions of a reception control module 2031, a transmission control module 2032, a database construction module 2033, a search module 2034, and a presentation control module 2035.

[0043] The reception control module 2031 controls the process in which the server 20 receives a signal from an external device in accordance with a communication protocol. The transmission control module 2032 controls the process in which the server 20 transmits a signal to an external device in accordance with a communication protocol.

[0044] The database construction module 2033, for example, stores the multiple pieces of first information output from the first generation AI model in a database held by the storage unit 202, and constructs the search database 2024. That is, the server 20 sequentially reads out multiple basic information data sets from the storage unit 202, inputs each basic information data set to the first generation AI model, and acquires the multiple pieces of first information from the first generation AI model.

[0045] Based on the second information, the search module 2034 searches the search database 2024 for first information similar to the second information as similar information. The second information is information related to a search target of the search service, and is transmitted from the terminal device 10, for example. The second information may be text information or image data. Alternatively, the second information may be a mixture of text information and image data. Details of the similarity judgment between the first information and the second information by the search module 2034 will be described later.

[0046] The search target of the search service is, for example, a drawing or an estimate. Hereinafter, the drawing to be searched is referred to as the searched drawing, and the estimate to be searched is referred to as the searched estimate. In other words, when the search target is the searched drawing, information on the parts illustrated in the searched drawing becomes the second information. Also, when the search target is the searched estimate, information on the parts that are the estimate target of the searched estimate becomes the second information. In this embodiment, the search target is the searched drawing, but as described above, the search target may be the searched estimate, or the search target may be both the searched drawing and the searched estimate.

[0047] When the search target is a search drawing, the text information as the second information is the same as the text information as the first information described above. Also, the image data as the second information is the same as the image data as the first information described above.

[0048] When the search target is a searched quotation, the text information as the second information is sentences and text data related to the price, conditions, etc. of the parts that are the subject of the searched quotation. The text information as the second information includes, for example, supplier information, customer (user) information, expiration date, part specifications, price, payment terms, warranty terms, etc. The image data as the second information is image data that displays the searched quotation. The image data as the second information includes a captured image of a screen that displays the searched quotation, scan data of the searched quotation, a captured image of the searched quotation, etc.

[0049] The presentation control module 2035 controls the process of presenting information to the user. For example, the presentation control module 2035 controls the process of presenting information on the search results of the search module 2034 to the user as search information.

[0050] <1.3 Configuration of generative AI system> The generative AI system 30 inputs, for example, the drawing information 2021, the estimate information 2022, and the first prompt received from the terminal device 10 to the first generative AI model, and causes the first generative AI model to output the first information. Specifically, for example, the first generative AI model includes a first multimodal generative AI model and a large-scale language model. The large-scale language model may be trained, for example, by a few-shot prompting technique.

[0051] In this embodiment, image data of a drawing previously uploaded to the server 20 is input to the first multimodal generative AI model together with a first prompt as drawing information 2021. The generative AI system 30 causes the first multimodal generative AI model to output first text information as the first information. The first text information is text information related to a part illustrated in a drawing previously uploaded to the server 20.

[0052] In this embodiment, text information related to an estimate previously uploaded to the server 20 is input to the large-scale language model as estimate information 2022 together with a first prompt. The generative AI system 30 causes the large-scale language model to output second text information as the first information. The second text information is text information related to the part that is the subject of the estimate on which the estimate information 2022 is based (in other words, the part depicted in the drawing on which the drawing information 2021 is based). The first text information and the second text information may be entirely the same or entirely different. Alternatively, the first text information and the second text information may be partially the same (in other words, partially different).

[0053] The generative AI system 30, for example, inputs the second information and the second prompt received from the terminal device 10 into the second generative AI model, and causes the second generative AI model to output the search information. The second prompt is text information instructing the output of the search information. In this embodiment, the second generative AI model is a second multimodal generative AI model different from the first multimodal generative AI model. Moreover, the image data of the search drawing and the text information related to the search drawing are the second information. In the following description, it is assumed that the search target is the search drawing. Moreover, the image data of the search drawing is referred to as the search image data, and the text information related to the search drawing is referred to as the search text information.

[0054] In addition, the first generative AI model may be, for example, either the first multimodal generative AI model or the large-scale language model. When the first generative AI model is the first multimodal generative AI model, the generative AI system 30 inputs the drawing information 2021, the estimate information 2022, and the first prompt to the first multimodal generative AI model as the first generative AI model. In this case, at least one of the drawing information 2021 and the estimate information 2022 may be image data or may be text information. In addition, the first information may also be image data or may be text information. When the first generative AI model is a large-scale language model, the generative AI system 30 inputs the drawing information 2021, the estimate information 2022, and the first prompt to the large-scale language model as the first generative AI model. In this case, the drawing information 2021, the estimate information 2022, and the first information are all text information.

[0055] The second generative AI model may be, for example, a large-scale language model. In this case, the generative AI system 30 inputs the second information and the second prompt into the large-scale language model as the second generative AI model. In this case, the second information is composed of only the search text information.

[0056] Furthermore, the generative AI system 30 may have, for example, a trained machine learning model such as a neural network model instead of the second generative AI model. In this case, the second information is input to the trained machine learning model, and the second prompt is not required. Alternatively, the generative AI system 30 may not have, for example, an AI model in general that uses the second information as input data. In this case, for example, the second information is sent to the search module 2034, and the search module 2034 generates search information.

[0057] The AI ​​model that the generative AI system 30 may have may be trained, for example, by supervised learning, or may be trained by both supervised learning and reinforcement learning. The AI ​​model may be a general-purpose model that can handle drawings and quotations provided by multiple suppliers, or may be an AI model dedicated to each supplier that has learned the contents of drawings and quotations for each supplier.

[0058] [2 Data Structure] 4 and 5 are diagrams showing the data structure of each table stored in the server 20. Note that Fig. 4 and Fig. 5 are merely examples and do not exclude data that is not listed. In addition, even if data is listed in the same table, it may be stored in separate storage areas in the storage unit 202.

[0059] Fig. 4 is a diagram showing the data structure of the user information table 2023. The user information table 2023 shown in Fig. 4 is a table having columns of name, age, sex, date of birth, and contact information, with a user ID as a key.

[0060] The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the user's name. The age is an item that stores the user's age. The gender is an item that stores the user's gender. The date of birth is an item that stores the user's date of birth. The contact information is an item that stores the contact information (e.g., a telephone number, an email address, etc.) of the terminal device 10 owned by the user.

[0061] Fig. 5 is a diagram showing the data structure of the search database 2024. The search database 2024 shown in Fig. 5 is in a data table format having columns of a drawing ID, a drawing URL, a quotation ID, a quotation URL, and part information, with a part ID as a key.

[0062] The part ID is an item for storing an identifier for uniquely identifying a part illustrated in a drawing on which the drawing information 2021 is based. The drawing ID is an item for storing an identifier for uniquely identifying a drawing on which the drawing information 2021 is based. The drawing URL is an item for storing image data of the drawing, that is, reference information (path) for the drawing information 2021. The quotation ID is an item for storing an identifier for uniquely identifying a quotation on which the quotation information 2022 is based. The quotation URL is an item for storing reference information (path) for image data of the quotation on which the quotation information 2022 is based. The part information is an item for storing information on a part illustrated in a drawing on which the drawing information 2021 is based, that is, first information.

[0063] The part information includes, for example, the product name, part number, quantity, category, shape, intersection, weight, dimensions (maximum diameter, minimum diameter), unit price, receipt date and time, component parts, materials, procurement classification, supplier information, and construction method. For example, when a part illustrated in a drawing on which the drawing information 2021 is based is composed of multiple component parts, information such as the product name and part number for each of the multiple component parts is included in the part information. For example, the material information includes the material name, material part number, input weight, yield, unit price, quantity, dimensions, etc. as the material content. For example, the part information includes the process name, equipment name, setup time, processing man-hours, total man-hours, total man-hour unit, man rate, machine rate, total wage rate, total wage rate unit, processing cost, etc. as the construction method content. Note that tokenized text information such as the product name included in the part information is preferably vectorized and stored in the search database 2024 so as to facilitate calculation of the similarity with the second information.

[0064] The part information may vary depending on the type of part. For example, if the part is a pipe or tube, the part information may include the outer diameter, inner diameter, wall thickness, etc. For example, if the part is a bolt, the part information may include the diameter, overall length, head shape, strength class, etc. Or, if the part is a bracket, the part information may include the function (support, attachment, fixing, etc.) etc.

[0065] [3 actions] The operation of the server 20 when presenting search information to the user will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the operation of the server 20 when presenting search information to the user. It is assumed that a plurality of pieces of drawing information 2021 and a plurality of pieces of quotation information 2022 transmitted from the terminal device 10 or the like have been uploaded to the server 20.

[0066] First, the user starts the terminal device 10. For example, the application 181 is automatically started by the OS of the terminal device 10 when the terminal device 10 starts. When the application 181 is started, the presentation control unit 193 may or may not present to the user that the application 181 is operating. In this embodiment, the presentation control unit 193 presents to the user that the application 181 is operating by displaying an input form screen (a predetermined input form) shown in FIG. 7 on the operation surface of the touch-sensitive device 131, in other words, on the display screen of the display 141. Details of the input form screen will be described later.

[0067] In step S11, the server 20 outputs the first information from the first generation AI model (output step). Specifically, for example, a person in charge of managing the system 1 or a person in charge of operating / managing the server 20 (hereinafter collectively referred to as a person in charge) operates an input device (not shown) provided in the server 20 to input a first prompt to the server 20. Note that the first prompt may be inputted not to the server 20 but to, for example, a PC (not shown) that provides a search service. The control unit 203 functions as an operation reception module (not shown) to receive the first prompt and transmit the first prompt to the transmission control module 2032.

[0068] The transmission control module 2032 sequentially reads out the basic information data set for each part type from the storage unit 202, and transmits it together with the first prompt to the generation AI system 30. At this time, the drawing information 2021 transmitted from the transmission control module 2032 is image data of the drawing. Also, the estimate information 2022 transmitted from the transmission control module 2032 is text information related to the estimate.

[0069] The generative AI system 30 inputs the drawing information 2021 (image data of the drawing) received from the transmission control module 2032 to the first multimodal generative AI model together with the first prompt. The first multimodal generative AI model that has received the drawing information 2021 and the first prompt outputs the first text information as the first information.

[0070] Furthermore, the generative AI system 30 inputs the quotation information 2022 (text information related to the quotation) received from the transmission control module 2032 into the large-scale language model together with the first prompt. The input process of the first prompt into the large-scale language model is, for example, similar to the input process into the first multimodal generative AI model described above. The large-scale language model to which the quotation information 2022 and the first prompt have been input outputs the second text information as the first information.

[0071] In this embodiment, the process content is that the drawing information 2021 and the estimate information 2022 stored in advance in the storage unit 202 are input to the first generation AI model, but is not limited to this process content. For example, the content of the first prompt input to the server 20 or a search service provider PC (not shown) may include at least one of the drawing information 2021 and the estimate information 2022. In other words, when the first prompt is input, a first prompt having dynamic content including at least one of the drawing information 2021 and the estimate information 2022 may be generated.

[0072] The generative AI system 30 transmits the first text information output from the first multimodal generative AI model and the second text information output from the large-scale language model to the server 20. The generative AI system 30 executes this series of processes for all of the basic information data sets sequentially transmitted from the transmission control module 2032 for each type of part.

[0073] In step S12, the server 20 constructs the search database 2024 (construction step). Specifically, for example, the reception control module 2031 receives the first text information and the second text information for each type of part sequentially transmitted from the generation AI system 30, and transmits them to the database construction module 2033. The database construction module 2033 stores the text information dataset in a database for storing first information stored in the storage unit 202, for example, every time it receives a dataset of the first text information and the second text information corresponding to a type of part (hereinafter, text information dataset).

[0074] At this time, if there is an item (e.g., product name, product number, etc.) included in the first text information that overlaps with an item included in the second text information, the database construction module 2033 selects, for example, the item with the higher information reliability from the first text information or the second text information. The database construction module 2033 deletes, for example, information corresponding to the item not selected. In this way, the text information dataset from which the specific text information has been deleted becomes the first information stored in the search database 2024.

[0075] The database construction module 2033 executes this series of processes for all the text information data sets. When the above-mentioned series of processes for all the text information data sets is completed, the database for storing the first information becomes the search database 2024. Note that the database construction module 2033 may, for example, once receive all the text information data sets, and then collectively store all the text information data sets in the database for storing the first information. Also, although there is no particular limitation on the state of the first information stored in the search database 2024, it is preferable that the first information in a vectorized state is stored in the search database 2024 from the viewpoint of facilitating a similarity judgment between the first information and the integrated text information (details will be described later). In this embodiment, it is assumed that the first information in a vectorized state is stored in the search database 2024. However, the search database 2024 may be, for example, a full-text search type database or a relational database.

[0076] In S13, the server 20 accepts the input of the second information (input step). Specifically, for example, the user operates the input device 13 to input the second information to the terminal device 10. The operation reception unit 191 accepts the second information input to the input device 13 and transmits it to the transmission / reception unit 192. The transmission / reception unit 192 transmits the second information received from the operation reception unit 191 to the server 20. The server 20 accepts the input of the second information by the user by the reception control module 2031 receiving the second information transmitted from the terminal device 10.

[0077] Hereinafter, the process of accepting the input of the second information will be described in detail with reference to Fig. 7. Fig. 7 is a schematic diagram showing an example of the operation surface of the touch-sensitive device 131 when the user inputs the second information. Here, in Figs. 7 and 8, the left side as viewed from the page is the upper side, and the right side as viewed from the page is the lower side. Also, in Figs. 7 and 8, the upper side as viewed from the page is the right side, and the lower side as viewed from the page is the left side.

[0078] As described above, when the application 181 is started, the presentation control unit 193 displays the drawing search screen on the operation surface (display screen of the display 141) of the touch-sensitive device 131. As shown in FIG. 7, the drawing search screen has, for example, an input form screen and a drawing display screen. In this embodiment, in the drawing search screen, the input form screen is arranged on the left side, and the drawing display screen is arranged on the right side. This arrangement can be changed arbitrarily.

[0079] The input form screen is provided with, for example, a usage scene button 1311, an important item field 1312, a refinement field 1313, an upload button 1314, a start button 1315, and a feedback field 1316.

[0080] When the usage scene button 1311 is tapped or the like, the presentation control unit 193 displays, for example, a drop-down menu (not shown) near the arrangement area of ​​the usage scene button 1311. The drop-down menu presents a list of usage scenes of the drawing search by the user.

[0081] In the important item field 1312, multiple types of item options 13121 are displayed. The items are indicators that may form part of the content of the second information, such as "dimensions," "tolerances," "materials," "construction method," "production volume," and "equipment." In addition, for example, check boxes 13122 are provided on the left side of each option as viewed from the screen. When a specific check box 13122 is tapped or the like, the operation receiving unit 191 receives the selection of the option 13121 corresponding to the specific check box 13122. As shown in FIG. 7, multiple options 13121 may be selected according to the content of the second information desired by the user.

[0082] By accepting this selection, the relevant information of the searched drawing corresponding to the type of the selected item is input to touch-sensitive device 131. For example, when the check box for the item "dimension" is tapped, information that "in drawing search, the dimensions of the parts illustrated in the searched drawing are used as the search criteria" is input to terminal device 10 as the relevant information of the searched drawing. When the relevant information of the searched drawing for all the types of selected items is input to touch-sensitive device 131, operation accepting unit 191 transmits all the relevant information input to touch-sensitive device 131 to transmitting / receiving unit 192.

[0083] The transmitting / receiving unit 192 generates search text information and a second prompt based on all the relevant information received from the operation receiving unit 191, for example. The transmitting / receiving unit 192 may generate the search text information and the second prompt together as one piece of text information, such as, for example, "I am looking for a drawing of a part that is manufactured in the same manufacturing facility as the part illustrated in this search drawing and employs a similar processing method. Can you search for a relevant drawing?" The transmitting / receiving unit 192 transmits the search text information and the second prompt to the server 20. Note that the operation receiving unit 191 may generate the search text information and the second prompt instead of the transmitting / receiving unit 192.

[0084] The presentation control unit 193 changes the type and number of items displayed in the important item field 1312 according to the type of use scene selected from the list of the drop-down menu, for example. This makes it possible to appropriately generate search text information according to the use scene of the drawing search desired by the user.

[0085] The refinement field 1313 is provided with a refinement item button 13131, a refinement item 13132, and a text box 13133. When the refinement item button 13131 is tapped or the like, the presentation control unit 193 displays, for example, a drop-down menu (not shown) near the arrangement area of ​​the refinement item button 13131. The drop-down menu presents a list of multiple types of items displayed in the important item field 1312. When the user selects a specific type of item from the list of the drop-down menu, the operation reception unit 191 accepts the selection and transmits information indicating the acceptance to the presentation control unit 193. The presentation control unit 193 switches the display of the refinement item 13132 to a specific type of item. In the example of FIG. 7, the display of the refinement item 13132 is switched to the item "tolerance".

[0086] Text box 13133 is composed of a text box for inputting an upper limit value and a text box for inputting a lower limit value. When the upper limit value and the lower limit value are inputted into text box 13133, transmitting / receiving unit 192 generates search text information that narrows down the search conditions so that the types of items displayed in narrowed-down items 13132 are within the numerical range from the lower limit value to the upper limit value. This makes it possible to generate search text information that includes the search conditions desired by the user.

[0087] In the example of FIG. 7, the transmitting / receiving unit 192 generates the search text information and the second prompt, but this is not limited to this case. For example, instead of the important item field 1312 and the refinement field 1313, a text box may be provided on the input form screen. In this case, the user inputs the search text information and the second prompt into the text box, and the operation receiving unit 191 receives the input of the search text information and the second prompt. The transmitting / receiving unit 192 transmits the search text information and the second prompt received from the operation receiving unit 191 to the server 20. The text box may be divided into a text box for inputting the search text information and a text box for inputting the second prompt.

[0088] When the upload button 1314 is tapped, the presentation control unit 193 displays, for example, a drop-down menu (not shown) near the arrangement area of ​​the upload button 1314. A list of searched image data is presented in the drop-down menu. In this embodiment, the searched image data presented in this list is image data of a drawing stored in the storage unit 202, that is, the drawing information 2021. Note that the searched image data presented in the list of the drop-down menu may be, for example, image data of one or more drawings stored in the terminal device 10 by the user. In this case, only the one or more image data may be presented in the list, or both the one or more image data and the multiple pieces of drawing information 2021 may be presented in the list.

[0089] When the user selects a specific search image data from the list in the drop-down menu, the operation acceptance unit 191 accepts the selection. The transmission / reception unit 192 transmits acceptance information to the server 20 indicating that the selection of the specific search image data has been accepted.

[0090] Here, the transmitting / receiving unit 192 transmits the search text information, the second prompt, and the reception information all at once to the server 20, but may transmit the search text information, the second prompt, the reception information, or a combination of two of them sequentially to the server 20. The server 20 receives the search text information, the second prompt, and the reception information transmitted from the terminal device 10 via the reception control module 2031, whereby the server 20 accepts input of the second information by the user.

[0091] When the start button 1315 is tapped or the like, the operation reception unit 191 receives it as an instruction to switch the input form screen from the initial screen to an inputtable screen, and transmits instruction information representing the instruction to the presentation control unit 193. The presentation control unit 193 that has received the instruction information switches the input form screen from the initial screen to an inputtable screen. Specifically, for example, on the input form screen in the initial state, nothing is displayed in each item of the important item field 1312 and the narrowing down items of the narrowing down field 1313 (not shown). On the other hand, for example, on the input form screen in the initial state, when the start button 1315 is tapped or the like after a usage scene is selected from the list of the drop-down menu of the usage scene button 1311, the input form screen is switched to the state shown in FIG. 7.

[0092] The search information is displayed in the feedback field 1316. Since Fig. 7 illustrates an example of the operation surface at the time when the second information is input, that is, before the drawing search is performed, nothing is displayed in the feedback field 1316. Note that the feedback field 1316 does not have to be provided on the drawing search screen, and for example, a feedback screen dedicated to displaying the search information may be displayed on the operation surface as a separate screen from the drawing search screen.

[0093] The drawing display screen is provided with, for example, a retrieved drawing display field 1317 and a similar drawing display field 1318. The retrieved image data ("retrieved drawing 1" in the figure) is displayed as the second information in the retrieved drawing display field 1317. When the presentation control module 2035 receives reception information indicating that the selection of the specific retrieved image data has been received from the operation reception unit 191, it reads out the specific retrieved image data (specific drawing information 2021) from the storage unit 202 and displays it in the retrieved drawing display field 1317.

[0094] 7 illustrates an example of the operation screen when the second information is input, that is, before the drawing search is performed, so nothing is displayed in the similar drawing display field 1318. Note that the screen configuration and display contents of the drawing search screen illustrated in FIG. 7 are merely examples, and various screen configurations and display contents can be adopted.

[0095] In S14, the server 20 searches for similar information from the search database 2024 based on the received second information (presentation step). Specifically, for example, the reception control module 2031 receives the search image data, the search text information, and the second prompt transmitted from the terminal device 10, and temporarily stores them in the storage unit 202. Note that the search image data, the search text information, and the second prompt may be temporarily stored in a storage unit (not shown) of the generation AI system 30.

[0096] The reception control module 2031, which has received the instruction information, reads out the search image data, the search text information, and the second prompt from the storage unit 202, and transmits them to the transmission control module 2032. The transmission control module 2032 transmits the received search image data, the search text information, and the second prompt to the generation AI system 30.

[0097] The generative AI system 30 inputs the received search image data, search text information, and second prompt into a second multimodal generative AI model, and causes the second multimodal generative AI model to output integrated text information. The integrated text information is text information related to the searched drawing, and is text information that integrates the content of the search text information and the content of the search image data. The generative AI system 30 transmits the integrated text information output from the second multimodal generative AI model to the server 20.

[0098] The reception control module 2031 receives the integrated text information sent from the generation AI system 30 and sends it to the search module 2034. The search module 2034, which has received the integrated text information, reads out the first information for each type of part from the search database 2024 and compares it with the integrated text information.

[0099] The search module 2034 performs the analogy judgment between the first information and the integrated text information, for example, using a method of cosine similarity. Cosine similarity is a method for calculating the similarity between two vectors in a vector space. The value of the cosine similarity ranges from -1 to 1. The closer the value of the cosine similarity is to 1, the more similar the two vectors are, and the closer it is to 0, the less similar they are. In addition, the closer the value of the cosine similarity is to -1, the more opposed the two vectors are. Since the cosine similarity depends on the direction and magnitude of the vector, in the analogy judgment, it is possible to focus only on the direction of the vector, regardless of the scale of the vector. Therefore, it is possible to easily perform the analogy judgment between two vectorized text information. The search module 2034 may perform the analogy judgment between the first information and the integrated text information using a method other than the cosine similarity.

[0100] Specifically, for example, the search module 2034 vectorizes the integrated text information received from the reception control module 2031. The search module 2034 calculates the cosine similarity between the already vectorized first information and the vectorized integrated text information using a general formula for cosine similarity. For example, if the calculated cosine similarity value is equal to or greater than a reference value (which can be set arbitrarily), the search module 2034 determines that the first information and the integrated text information are similar.

[0101] In S15, the server 20 causes the second generative AI model to output the search information (presentation step). Specifically, for example, when the search module 2034 can search for specific first information similar to the integrated text information from the search database 2024, the search module 2034 regards the specific first information as similar to the second information (search image data and search text information) and sets it as similar information. The search module 2034 transmits the searched similar information to the transmission control module 2032. The transmission control module 2032 transmits the received similar information to the generative AI system 30. The generative AI system 30 inputs the received similar information into the second multimodal generative AI model and causes the second multimodal generative AI model to output the search information.

[0102] Also, for example, when the search module 2034 is unable to search the search database 2024 for specific first information similar to the integrated text information, it generates unsearchable information indicating a search result indicating that the search was not possible. The search module 2034 transmits the unsearchable information to the transmission control module 2032. The transmission control module 2032 transmits the received unsearchable information to the generation AI system 30. The generation AI system 30 inputs the received unsearchable information into a second multimodal generative AI model, and causes the second multimodal generative AI model to output search information. Through this series of processes, the server 20 causes the second generation AI model to output search information.

[0103] In S16, the server 20 presents the search information to the user (presentation step). Specifically, for example, the generative AI system 30 transmits the search information output from the second generative AI model (second multimodal generative AI model) to the server 20. The reception control module 2031 transmits the received search information to the presentation control module 2035. The presentation control module 2035 transmits the received search information to the terminal device 10, thereby displaying the received search information on the display screen of the display 141 (the operation surface of the touch-sensitive device 131). Through this series of processes, the server 20 presents the search information to the user.

[0104] Hereinafter, an example of presenting search information to a user will be described with reference to FIG. 8. FIG. 8 is a schematic diagram showing an example of a display screen of the display 141 when presenting search information to a user. Since the display screen of the display 141 is also an operation surface of the touch-sensitive device 131, when the presentation control module 2035 transmits the search information to the terminal device 10, the drawing search screen transitions from the state shown in FIG. 7 to, for example, the state shown in FIG. 8. That is, the search information is displayed in the feedback field 1316 of the input form screen in the state shown in FIG. 7. Also, the search information is displayed in the similar drawing display field 1318 of the drawing display screen in the state shown in FIG. 7.

[0105] The search information may be information of any content and format as long as it is information related to the search result. In the example of FIG. 8, the feedback field 1316 displays search information in text format, such as "Similar drawings A to D were found under the above conditions." Also, in the example of FIG. 8, the feedback field 1316 displays advice information in text format, such as "If the tolerance is relaxed, drawings AA and BB correspond to similar drawings. If the item "Equipment" is unchecked, drawings CC and DD correspond to similar drawings." The advice information is information that presents the content of the second information that is preferably changed, etc., to facilitate the search for similar drawings (drawings including similar information) similar to the search drawing, according to the contents of the search drawing and the second information. The inclusion of advice information in the search information presented to the user makes it easier for the user to search for similar drawings.

[0106] Furthermore, the presentation control module 2035 displays, as search information, reference information (path) of image data of similar drawings stored in the search database 2024 in the similar drawing display field 1318. For example, when the user taps on the reference information displayed in the similar drawing display field 1318, the presentation control module 2035 displays image data of similar drawings ("Similar Drawing A" to "Similar Drawing D" in the figure) as shown in FIG. 8 in the similar drawing display field 1318.

[0107] [4 Screen Examples] The input screen for second information displayed on the operation surface of touch-sensitive device 131 and the presentation screen for searched information displayed on the display screen of display 141 may have various screen configurations other than the examples shown in Fig. 7 and Fig. 8. When input device 13 and output device 14 are touch panels as in this embodiment, for example, a chat-based input / presentation screen shown in Fig. 9 is assumed. Fig. 9 is a schematic diagram showing another example of the display screen of display 141 when searched information is presented to the user. The input / presentation screen shown in Figure 9 allows drawing searches to be performed by inputting only text information, for example, by outputting search information including advice information from the second generative AI model (second multimodal generative AI model) depending on the content of the text information (second information) that is the input data, and presenting the search information to the user.

[0108] Specifically, for example, a text input field 1411, an image display field 1412, a search information display field 1413, and a select box 1414 are displayed on the input / presentation screen shown in FIG.

[0109] Text information as the second information is input into the text input field 1411. In the example of Fig. 9, the text information "Could you please send me a drawing of the solenoid valve I requested from XX Corporation?...It is preferable that the material used is plastic." is input into the text input field 1411.

[0110] The terminal device 10 transmits the text information received by the operation reception unit 191 to the server 20. The server 20 transmits the text information received by the reception control module 2031 to, for example, the generation AI system 30. The generation AI system 30, for example, inputs the received text information into a second multimodal generative AI model and causes the second multimodal generative AI model to output search information including advice information. The generation AI system 30 transmits the search information output from the second multimodal generative AI model to the server 20. The server 20 transmits the operation information received by the reception control module 2031 to the presentation control module 2035. The presentation control module 2035 transmits the search information to the terminal device 10 and causes the search information to be displayed in the search information display field 1413.

[0111] Here, if one or more types of similar drawings are found, the presentation control module 2035 displays, for example, reference information (path) of image data of the similar drawings stored in the search database 2024 in the image display field 1412. For example, when the user taps on the reference information displayed in the image display field 1412, the presentation control module 2035 displays image data of the similar drawings in the image display field 1412 as shown in FIG.

[0112] In addition, when the search target of the system 1 is a searched estimate, in the example of FIG. 9, reference information (path) of image data of a similar estimate (an estimate including similar information) stored in the search database 2024 may be displayed in the image display field 1412. Also, for example, when the user taps on the reference information displayed in the image display field 1412, the presentation control module 2035 may display image data of the similar estimate in the image display field 1412. Furthermore, when the search target of the system 1 is both a searched estimate and a searched estimate, the presentation control module 2035 may display image data of a similar drawing and image data of a similar estimate in the image display field 1412.

[0113] As described above, by presenting to the user at least one of image data of a similar drawing and image data of a similar quotation, the user can easily determine whether or not the similar drawings, etc. searched for by server 20 match the drawings, etc. desired by the user.

[0114] In the example of Fig. 9, reference information for two types of similar drawings is displayed in image display field 1412 according to the content of the text information entered in text input field 1411, and image data of the two types of similar drawings is displayed in image display field 1412 by tapping on this reference information, etc. Also, in the example of Fig. 9, search information is displayed in search information display field 1413 as a text chat in response to the text information entered in text input field 1411, saying "There were 10 drawings that are close to the specified conditions. To narrow down the search further, please specify the date and time of receipt of the drawings and the equipment from the items below."

[0115] As in the example of Fig. 9, when the search information includes information indicating that it is preferable to narrow down a specific search condition (specific content included in the second information), the presentation control module 2035 displays, for example, a select box 1414 provided with options for narrowing down the specific search condition. In other words, the select box 1414 is not necessarily displayed on the input / presentation screen shown in Fig. 9, and may not be displayed depending on the display content of the search information display field 1413. In the example of Fig. 9, the select box 1414 provided with options for narrowing down the drawing receipt date and equipment is displayed based on the display content in the search information display field 1413 indicating that it is preferable to narrow down the drawing receipt date and equipment.

[0116] In the example of Figure 9, in response to the user selecting a particular option in the selection box 1414, new reference information (and image data of similar drawings obtained by tapping on the reference information, etc.) is displayed in the image display field 1412 as chat text, and search information such as "The above drawings have been found as candidates for similar drawings from the items you selected..." is displayed in the search information display field 1413.

[0117] 9, the presentation control module 2035 dynamically and flexibly displays search information and UI elements for narrowing down search conditions according to the contents of the text information entered in the text input field 1411. This allows the user to efficiently search for desired similar drawings with fewer UI operations.

[0118] [5 Summary] As described above, in this embodiment, the database construction module 2033 stores the first information obtained by inputting the drawing information 2021 and the estimate information 2022 into the first generation AI model in a database for storing first information, and constructs the search database 2024. The search module 2034 searches for similar information using the search database 2024 and causes the second generation AI model to output the search information. The presentation control module 2035 presents the search information to the user by displaying the search information on the display screen of the display 141. This allows drawing search to be performed taking into account the estimate information 2022, making it possible to flexibly respond to various use cases in drawing search using the drawing information 2021.

[0119] In this embodiment, the operation receiving unit 191 receives input of the second information by inputting corresponding information of the searched drawing into each of a plurality of items on an input form screen displayed on the operation surface of the touch-sensitive device 131. This eliminates the need for the user to input all the contents of the second information, improving the user's convenience in searching for drawings.

[0120] Furthermore, in this embodiment, the generative AI system 30 acquires the first information by inputting image data as drawing information 2021 into a first multimodal generative AI model and inputting text information as estimate information 2022 into a large-scale language model. The generative AI system 30 acquires search information by inputting search image data and search text information into a second multimodal generative AI model. This makes the first information and search information more accurate by taking into account the relevance of different types of information, image data and text information. This improves the accuracy of drawing search.

[0121] [6. Modifications] <6.1 First modified example> In the present embodiment, the system 1 has been described as a system for providing a search service related to a drawing search service, a quotation search service, or a combination of these two. However, in addition to these search services, the system 1 may also provide, for example, a search service for quotation assessment materials.

[0122] The quotation evaluation materials are materials used by the user's company to evaluate whether the contents of the quotation are appropriate, and examples of such materials include a request for approval, quotation comparison materials, etc. The request for approval is a material used to obtain approval / decision for ordering within the user's company for the parts that are the subject of the search quotation. The quotation comparison materials are materials related to the results of comparing a quotation with other quotations, and examples of such materials include similar parts quotation comparison materials, competitive quotation comparison materials, past quotation comparison materials for the same part number, past quotation comparison materials for the same supplier, etc.

[0123] Similar parts quotation comparison material is material relating to the results of comparing quotations for parts similar to the part that is the subject of quotation in the search quotation with the search quotation. Comparable quotation comparison material is material relating to the results of comparing quotations that are the subject of comparative quotations in the search quotation with the search quotation. Estimated past quotation comparison material for the same part number is material relating to the results of comparing quotations for parts with the same part number as the part that is the subject of quotation in the search quotation with the search quotation. Same supplier past quotation comparison material is material relating to the results of comparing quotations submitted by the same supplier as the supplier of the part that is the subject of quotation in the search quotation with the search quotation.

[0124] When the system 1 provides a search service for estimate assessment materials, for example, the server 20 may store information on the estimate assessment materials in addition to the drawing information 2021 and the estimate information 2022 in the memory unit 202. The transmission control module 2032 may read out the drawing information 2021, the estimate information 2022, and the information on the estimate assessment materials from the memory unit 202 and transmit them to the generative AI system 30. The generative AI system 30 may input these three types of received information together with the first prompt into the first generative AI model, and cause the first generative AI model to output the first information.

[0125] As a result, the first information that takes into account the information on the estimate assessment materials is output from the first generation AI model, and the accuracy of the first information stored in the search database 2024 is increased. Therefore, the accuracy of the search information is also increased, and the user can search for similar estimate assessment materials (estimate assessment materials including similar information) with high accuracy. Note that even when the system 1 does not provide, for example, a search service for estimate assessment materials, that is, when the system 1 provides only the search service according to this embodiment (a drawing search service, an estimate search service, or a search service relating to a combination of these two), the information on the estimate assessment materials may be used as input data to the first generation AI model.

[0126] In addition, when the system 1 provides a search service for estimate evaluation materials, the presentation control module 2035 may present, as search information, for example, text information related to similar estimate evaluation materials (estimate evaluation materials containing similar information) to the user. In this case, the presentation control module 2035 may present image data of the similar estimate evaluation materials together with the search information. By presenting this image data to the user, the user can easily determine whether the similar estimate evaluation materials searched by the server 20 match the estimate evaluation materials desired by the user.

[0127] In this embodiment and the first modified example, transaction communication information may be added to the input data to the first generation AI model. The transaction communication information is communication information related to a drawing on which the drawing information 2021 is based, a quotation on which the quotation information 2022 is based, an estimate assessment material on which the information on the estimate assessment material is based, or a combination of at least two of these. Specifically, for example, the transaction communication information is transaction information exchanged between a user and a supplier by wireless communication (or wired communication), and is stored in the storage unit 202.

[0128] The transaction communication information includes detailed information about the drawing and the quotation that does not appear in the drawing and the quotation itself. Therefore, by adding the transaction communication information to the input data to the first generation AI model, the first generation AI model outputs first information that takes into account the transaction communication information, thereby increasing the accuracy of the first information stored in the search database 2024. This also increases the accuracy of the search information, allowing the user to search for similar drawings, etc. with high accuracy.

[0129] The transaction communication information may include information regarding interactions leading up to the determination of the estimated amount described in the quotation on which the quotation information 2022 is based. By adding the transaction communication information including this information to the input data to the first generation AI model, the accuracy of the first information stored in the search database 2024 is further increased. In turn, the accuracy of the search information is further increased, allowing the user to search for similar drawings, etc. with higher accuracy.

[0130] <6.2 Second modified example> In this embodiment, the first information is composed of the first text information output from the first multimodal generative AI model and the second text information output from the large-scale language model. However, the generative AI system 30 may further input the first text information and the second text information to a third multimodal generative AI model different from both the first multimodal generative AI model and the second multimodal generative AI model, and output the first information from the third multimodal generative AI model. There is no particular limitation on the type of the third multimodal generative AI model, and any type of multimodal generative AI model may be used as the third multimodal generative AI model.

[0131] The first information output from the third multimodal generative AI model is third text information. The third text information is text information related to a part illustrated in a drawing previously uploaded to the server 20. The third text information may entirely match the first text information and the second text information, or may entirely differ from the first text information and the second text information. Alternatively, the third text information may partially match at least one of the first text information and the second text information.

[0132] Specifically, for example, the generative AI system 30 may have a third multimodal generative AI model in addition to the first generative AI model (first multimodal generative AI model and large-scale language model) and the second generative AI model (second multimodal generative AI model). The transmission control module 2032 may read image data of a drawing as the drawing information 2021 from the storage unit 202 and transmit the image data together with an explanatory text related to the image data to the generative AI system 30. For example, the reception control module 2031 may receive text information of the explanatory text inputted by the operation reception unit 191, and the transmission control module 2032 may transmit the text information to the generative AI system 30. Also, for example, the text information of the explanatory text may be associated with the image data of the drawing as the drawing information 2021 and stored in advance in the storage unit 202.

[0133] The generative AI system 30 may input the first text information output from the first multimodal generative AI model and the second text information output from the large-scale language model to a third multimodal generative AI model together with image data of the drawing as the drawing information 2021, an explanatory text, and the first prompt, and output the third text information from the third multimodal generative AI model. The search module 2034 may perform an analogy judgment between the third text information and the integrated text information using a cosine similarity method as in the present embodiment.

[0134] This allows the third text information, which more appropriately associates the image data of the drawing with the first text information and the second text information via the explanatory text, to be stored as the first information in the search database 2024. This increases the accuracy of the search information, allowing the user to search for similar drawings, etc. with high accuracy. Note that in the second modified example, it is not essential to input the explanatory text to the third multimodal generation AI model.

[0135] <6.3 Third Modification> In this embodiment, the second multimodal generative AI model has not been fine-tuned. However, for example, the generative AI system 30 may fine-tune the second multimodal generative AI model based on search information presented to a user. Here, fine-tuning is a process of adjusting parameters of an AI model to adapt a pre-trained AI model to a specific task or data set. The timing of the process fine-tuning is arbitrary, and may be performed monthly or daily.

[0136] For example, the generative AI system 30 may detect that it is time for fine tuning and read out a plurality of tuning datasets from a storage unit (not shown) and fine-tune the second multimodal generative AI model using the plurality of tuning datasets. The tuning dataset may be, for example, a dataset of past search information and past evaluation information, and may be stored in the storage unit 202. The past search information is, for example, search information that the presentation control module 2035 has previously displayed on the display screen (the operation surface of the touch-sensitive device 131) of the display 141. The past evaluation information is, for example, information on the evaluation of a user or the like on the past search information, and is associated with the past search information. The past evaluation information may be, for example, input from the input device 13 and transmitted to the generative AI system 30. Note that the plurality of tuning datasets may be stored in the storage unit 202. In this case, the transmission control module 2032 reads out the plurality of tuning datasets from the storage unit 202 and transmits them to the generative AI system 30.

[0137] Specifically, the fine-tuning of the second multimodal generative AI model by the generative AI system 30 may be performed as follows. That is, the generative AI system 30 fixes some layers of the second multimodal generative AI model and thaws other layers to make it learnable (model adjustment). Next, the generative AI system 30 sets / adjusts a learning rate and performs fine-tuning using multiple tuning datasets. Next, the generative AI system 30 evaluates the second multimodal generative AI model after fine-tuning. For example, recall, precision, F1 score, etc. are adopted as the evaluation method. If the evaluation result shows that the second multimodal generative AI model after fine-tuning does not perform sufficiently, the generative AI system 30 adjusts the model again and performs fine-tuning. If the evaluation result shows that the second multimodal generative AI model after fine-tuning performs sufficiently, the generative AI system 30 deploys the second multimodal generative AI model.

[0138] This makes it possible to construct a second multimodal generative AI model optimized for search services such as drawing search and quotation search, thereby increasing the accuracy of search information. This allows users to search for similar drawings, etc. with high accuracy. Note that if the second generative AI model is, for example, a large-scale language model or a neural network model (trained machine learning model) rather than the second multimodal generative AI model, any of these models may be fine-tuned.

[0139] [7 Basic Computer Hardware Configuration] 10 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921.

[0140] The processor 901 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0141] The main storage device 902 is a memory for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0142] The auxiliary storage device 903 is a storage for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0143] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

[0144] The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0145] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 90 and connecting them together via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0146] [8 Basic Functional Configuration of a Computer] A description will now be given of the functional configuration of a computer realized by the basic hardware configuration (FIG. 10) of a computer 90. The computer comprises at least the functional units of a control unit, a storage unit, and a communication unit.

[0147] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 connected to each other via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0148] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding the programs in the main storage device 902, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0149] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with a program. Furthermore, the control unit can cause the processor 901 to execute processes of adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0150] The term database refers to a relational database, which is used to manage sets of data called masters and tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table or master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated.

[0151] Usually, a column that serves as a primary key for uniquely identifying a record is set in each table and each master, but setting a primary key in a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in a specific table or master stored in the storage unit according to various programs.

[0152] Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0153] In addition, the database and master in this disclosure may include any data structure (such as a list, a dictionary, an associative array, or an object) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with a function, class, method, or the like written in any programming language.

[0154] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0155] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0156] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Java (registered trademark), and the like.

[0157] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

[0158] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the functions described. Processors include transistors and other circuits and are considered to be circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory.

[0159] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed in this specification or any hardware known to be programmed to realize or perform the described functions.

[0160] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0161] Although several embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are within the scope of the invention and its equivalents as described in the claims, as well as the scope and spirit of the invention.

[0162] [9 Notes] The matters described in the above embodiment and modified examples will be supplemented below.

[0163] <Appendix 1> A program to be executed by a computer having a processor and a memory, the program causing the processor to execute an output step of inputting drawing information regarding a drawing, estimate information regarding an estimate for an illustrated object illustrated in the drawing, and a first prompt that instructs the output of first information regarding the illustrated object into a first generative AI model, and outputting the first information from the first generative AI model, a construction step of storing a plurality of the outputted first information in a database to construct a search database, an input step of accepting input of second information related to a search target, and a presentation step of searching the search database for first information similar to the second information based on the accepted second information, and presenting search information related to the search result.

[0164] <Appendix 2> In the output step, communication information relating to a drawing, an estimate, a document relating to a comparison result when the estimate is compared with another estimate, or a combination of at least two of these is input to the first generating AI model (Supplementary Note 1).

[0165] <Appendix 3> The communication information includes information regarding the exchanges leading up to the determination of the estimated amount set forth in the quotation (Appendix 2).

[0166] <Appendix 4> A program described in any one of (Appendix 1) to (Appendix 3), in which in an input step, input of second information is accepted by inputting relevant information to be searched for into each of multiple types of items provided in a specified input form.

[0167] <Appendix 5> A program described in any one of (Appendix 1) to (Appendix 4), wherein in the presentation step, image data of a similar drawing including similarity information is presented.

[0168] <Appendix 6> A program described in any one of (Appendix 1) to (Appendix 4), in which, in the presentation step, image data of a similar estimate including similar information is presented.

[0169] <Appendix 7> A program described in any one of (Appendix 1) to (Appendix 4), in which, in the presentation step, image data of materials relating to the comparison results when a similar estimate containing similar information is compared with another estimate is presented.

[0170] <Appendix 8> A program described in any of (Appendix 1) to (Appendix 7), in which, in the presentation step, a second prompt instructing the output of the second information and the search information is input to a second generative AI model different from the first generative AI model, and the search information output from the second generative AI model is presented.

[0171] <Appendix 9> The search information includes an explanatory text explaining the search results (Appendix 8).

[0172] <Appendix 10> In the output step, the first generative AI model includes a first multimodal generative AI model and a large-scale language model, and image data of the drawing as drawing information is input to the first multimodal generative AI model together with a first prompt, and first text information regarding the illustrated object is output from the first multimodal generative AI model as the first information, and text information regarding an estimate as quotation information is input to the large-scale language model together with the first prompt, and second text information regarding the illustrated object is output from the large-scale language model as the first information, and in the input step, input of image data of the search target and text information regarding the search target is accepted as input of the second information, and in the presentation step, the accepted image data of the search target, the accepted text information regarding the search target, and a second prompt instructing output of the search information are input to a second multimodal generative AI model different from the first multimodal generative AI model, and the search information output from the second multimodal generative AI model is presented (the program described in Appendix 1).

[0173] <Appendix 11> In an output step, the first text information, the second text information, image data of the drawing, an explanatory text relating to the image data of the drawing, and the first prompt are input to a third multimodal generation AI model different from both the first multimodal generation AI model and the second multimodal generation AI model, and third text information relating to the illustrated object is output from the third multimodal generation AI model as the first information (Appendix 10).

[0174] <Appendix 12> The program described in (Appendix 10) or (Appendix 11) causes a processor to execute an adjustment step of fine-tuning the second multimodal generative AI model based on the presented search information.

[0175] <Appendix 13> A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 12).

[0176] <Appendix 14> An information processing device comprising a control unit and a memory unit, wherein the control unit executes all of the steps executed in the invention according to any one of (Appendix 1) to (Appendix 12).

[0177] <Appendix 15> A system comprising means for executing all the steps performed in any of the inventions according to (Appendix 1) to (Appendix 12). [Explanation of symbols]

[0178] 1. System 10...Terminal device 120…Communications Department 13...Input device 14...Output device 15…Memory 16…Storage 19…Processor 20…Server 22…Communication IF 23... Input / Output IF 25…Memory 26…Storage 29…Processor 30…Generative AI system

Claims

1. A program stored in a memory of a computer having a processor and a memory, the program causing the processor to: an output step of inputting drawing information on a drawing, estimate information on an estimate for an illustrated object illustrated in the drawing, and a first prompt instructing output of first information, which is text information on the illustrated object, into a first generating AI model, and causing the first generating AI model to output the first information; a construction step of storing the plurality of pieces of first information outputted for the plurality of drawings in a database to construct a search database; an input step of accepting input of image data to be searched and text information related to the search target as input of second information; A program that executes a presentation step of inputting the received second information and a second prompt that instructs the output of integrated text information that integrates the content of the image data of the search target and text information related to the search target into a second generation AI model different from the first generation AI model, searching for the first information similar to the integrated text information from the search database as similar information based on the integrated text information output from the second generation AI model, inputting the searched similar information and a third prompt that instructs the output of search information into the second generation AI model, and presenting the search information output from the second generation AI model.

2. The program according to claim 1, wherein in the output step, communication information relating to the drawing, the estimate, materials relating to a comparison result when the estimate is compared with other estimates, or a combination of at least two of these is further input to the first generating AI model.

3. The program according to claim 2 , wherein the communication information includes information regarding exchanges leading up to the determination of the estimated amount stated in the quotation.

4. 2. The program according to claim 1, wherein said input step accepts input of said second information by inputting the relevant information to be searched into each of a plurality of items provided in a predetermined input form.

5. The program according to claim 1 , wherein the search information includes image data of similar drawings that include the similarity information.

6. The program according to claim 1 , wherein the search information includes image data of similar quotations that include the similar information.

7. The program according to claim 1 , wherein the search information includes image data of materials related to a comparison result when a similar estimate including the similar information is compared with another estimate.

8. The program according to claim 1 , wherein the search information includes an explanatory text explaining the similar information.

9. In the output step, The first generative AI model includes a first multi-modal generative AI model and a large-scale language model; the second generative AI model includes a second multi-modal generative AI model; inputting image data of the drawing as the drawing information into the first multimodal generative AI model together with the first prompt, and outputting first text information relating to the illustrated object from the first multimodal generative AI model; inputting text information relating to the quotation as the quotation information into the large-scale language model together with the first prompt, and outputting second text information relating to the illustrated object from the large-scale language model; The program according to claim 1 , wherein the first text information and the second text information are the first information to be output from the first generative AI model.

10. The program of claim 9, wherein in the output step, the first text information, the second text information, the image data of the drawing, an explanatory text relating to the image data of the drawing, and the first prompt are input to a third multimodal generation AI model different from both the first multimodal generation AI model and the second multimodal generation AI model, and third text information relating to the illustrated object is output from the third multimodal generation AI model as the first information.

11. The program of claim 9 , further comprising causing the processor to execute an adjustment step of fine-tuning the second multi-modal generative AI model based on presented search information.

12. A method implemented on a computer having a processor and a memory, the processor executing all of the steps performed in the invention according to any one of claims 1 to 11.

13. An information processing device including a computer having a processor and a memory, wherein the processor executes all of the steps executed in the invention relating to any one of claims 1 to 11.

14. A system including a computer having a processor and a memory, wherein the processor executes all of the steps performed in an invention according to any one of claims 1 to 11.

Images