Program, method, information processing apparatus, and system

Abstract

To flexibly cope with various use cases in data retrieval using information about a drawing.SOLUTION: A program for causing a computer including a processor and a memory to execute: The program causes the processor to execute an output step of inputting, to a generation AI model, drawing information related to a drawing, estimate information related to an estimate of an illustrated object illustrated in the drawing, and a first prompt instructing output of first information related to the illustrated object, and outputting the first information from the generation AI model, a construction step of constructing a search database by storing the plurality of pieces of output first information in the database, an input step of receiving input of second information related to a search target, and a search step of searching the search database for the first information similar to the second information as similar information based on the received second information; And a presentation step of presenting search information regarding the search result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] Conventionally, a technique for searching for data similar to a search target using a database storing output data of a generative AI model has been known. For example, Patent Document 1 discloses a technique for calculating the similarity between a search image and an answer sentence to a question sentence about the search image (hereinafter referred to as a first answer sentence) and a reference image and an answer sentence to a question sentence about the reference image (hereinafter referred to as a second answer sentence), thereby searching for a similar reference image that is similar to the search image with respect to the first answer sentence. The first answer sentence is obtained as an output result when the search image and a question sentence about the search image are input into a VQA (Visual Question Answering) model. The reference image and the second answer sentence are search images and first answer sentences used in past searches, and are stored in a reference case database in association with each other. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-39656 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 is intended to be used in cases where humans and / or machines are the subject of image capture, such as disasters, accidents, breakdowns, and / or incidents, and is not intended to be used to capture images of, for example, blueprints and estimates. Therefore, the technology disclosed in Patent Document 1 has difficulty in supporting searches using captured images of blueprints and estimates. Furthermore, it is extremely difficult to flexibly support a variety of use cases, such as searching for blueprints while taking into account information about estimates, or searching for estimates while taking into account information about blueprints.

[0005] The purpose of this disclosure is to flexibly respond to a variety of use cases in data searches using information related to drawings. [Means for solving the problem]

[0006] To solve the above-described problems, a program according to one embodiment of the present disclosure is a program to be executed by a computer including a processor and a memory. The program causes the processor to execute an output step of inputting drawing information related to a drawing, estimate information related to an estimate for an illustrated object illustrated in the drawing, and a first prompt instructing the processor to output first information related to the illustrated object to a generative AI model and causing the generative AI model to output the first information, a construction step of storing the outputted first information in a database and constructing a search database, an input step of accepting input of second information related to the search target, and a presentation step of searching the search database for first information similar to the second information based on the accepted second information and presenting search information related to the search result. [Effects of the Invention]

[0007] According to the present disclosure, data searches using drawing information can be flexibly adapted to a variety of use cases. [Brief explanation of the drawings]

[0008] [Figure 1]1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Figure 2] 2 is a block diagram showing an example of the configuration of a terminal device 10 shown in FIG. 1. FIG. [Figure 3] 2 is a block diagram showing an example of a functional configuration of a server 20 shown in FIG. 1. FIG. [Figure 4] 4 is a diagram showing the data structure of a user information table 2023 shown in FIG. 3. FIG. [Figure 5] FIG. 4 is a diagram showing the data structure of a search database 2024 shown in FIG. 3. [Figure 6] 10 is a flowchart showing an example of the operation of the server 20 when presenting search information to a user. [Figure 7] FIG. 10 is a schematic diagram showing an example of the operation surface of the touch-sensitive device 131 when the user inputs second information. [Figure 8] FIG. 10 is a schematic diagram showing an example of a display screen of a display 141 when search information is presented to a user. [Figure 9] FIG. 10 is a schematic diagram showing another example of the display screen of the display 141 when search information is presented to the user. [Figure 10] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0010] [1. Overall system configuration] FIG. 1 is a block diagram showing an example of the overall configuration of system 1. System 1 shown in FIG. 1 is a system for providing, for example, a drawing search service, an estimate search service, or a search service relating to a combination of these two (hereinafter collectively referred to as search services). System 1 includes, for example, a terminal device 10, a server 20, and a generation AI system 30. The terminal device 10, the server 20, and the generation AI system 30 are communicatively connected, for example, via a network 80.

[0011] While Figure 1 shows an example in which system 1 includes two terminal devices 10, the number of terminal devices 10 included in system 1 may be less than three or more than three. While Figure 1 shows an example in which system 1 includes one generative AI system 30, the number of generative AI systems 30 included in system 1 may be two or more. Although Figure 1 shows an example in which generative AI system 30 is independent from server 20, server 20 may include the functions of generative AI system 30.

[0012] In this embodiment, a collection of multiple devices may be considered as one server. The way in which the multiple functions required to realize the server 20 according to this embodiment are allocated to one or more pieces of hardware can be determined appropriately depending on the processing capacity of each piece of hardware and / or the specifications required for the server 20.

[0013] The terminal device 10 shown in Fig. 1 is, for example, an information processing device operated by a user (hereinafter abbreviated as "user") who uses a search service. The terminal device 10 is realized by, for example, a mobile terminal such as a smartphone or a tablet. The terminal device 10 may also be realized by, for example, a stationary PC (Personal Computer), a laptop PC, or the like. In this embodiment, the terminal device 10 is assumed to be a tablet.

[0014] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving input operations from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.). In this embodiment, the terminal device 10 is assumed to include a touch panel in which the input device 13 and the output device 14 are integrated.

[0015] The server 20 is, for example, an information processing device that provides a search service, and manages and processes various types of information used in the search service. The server 20 is realized, for example, by a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an input device for receiving input operations from a user, and as an interface for an output device for outputting information to the user.

[0016] The generative AI system 30 has, for example, a first generative AI model and a second generative AI model different from the first generative AI model. The first generative AI model and the second generative AI model are, for example, large-scale language models or multimodal generative AI models. The large-scale language model is an artificial intelligence system specialized for natural language processing, trained using a large amount of data set, and constructed based on a neural network such as a Transformer architecture. The multimodal generative AI model is, for example, an artificial intelligence system constructed using deep learning that collects two or more types of information, such as text, audio, images, and video, and processes them in an integrated manner. An example of a large-scale language model is "OpenAI ChatGPT GTP-3.5 (registered trademark)." An example of a multimodal generative AI model is "OpenAI ChatGPT GTP-4 (registered trademark)."

[0017] Each information processing device is configured by a computer 90 (see FIG. 10) equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer 90 and the basic functional configuration of the computer 90 realized by the basic hardware configuration will be described later. For the terminal device 10, the server 20, and the generation AI system 30, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer 90 will be omitted.

[0018] <1.1 Terminal device configuration> Fig. 2 is a block diagram showing an example of the configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected by, for example, a bus. The terminal device 10 may also include an audio processing unit, a microphone, a speaker, a camera, a position information sensor, or a combination of at least two of these.

[0019] The communication unit 120 performs processing such as modulation and demodulation for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on signals generated by the control unit 190 and transmits the signals to the outside (for example, the server 20). The communication unit 120 performs reception processing on signals received from the outside and outputs the signals to the control unit 190.

[0020] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 is realized, for example, by a touch-sensitive device 131 that inputs instructions by touching the operation surface. If the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, or the like. The input device 13 converts instructions input by the user into electrical signals and outputs them to the control unit 190. The input device 13 may include, for example, a receiving port that receives electrical signals input from an external input device.

[0021] The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized, for example, by a display 141. The display 141 displays various information according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0022] The storage unit 180 is realized by, for example, the memory 15 and the storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, an app 181. The apps stored in the storage unit 180 are not limited to the app 181. The app 181 is an application for managing the use of a search service by a user. The app 181 is installed in the terminal device 10, and is a concept that includes a browser. The app 181 runs, for example, in the background of other apps installed in the terminal device 10, and monitors processes executed by the user.

[0023] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating in accordance with the program.

[0024] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131.

[0025] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, the transmitting / receiving unit 192 transmits instructions or information input by the user to the server 20. The transmitting / receiving unit 192 receives information transmitted from the server 20.

[0026] The presentation control unit 193 controls the output device 14 to present the information transmitted from the server 20 to the user. Specifically, the presentation control unit 194 causes the display 141 to display the search information received from the server 20. Details of the search information will be described later.

[0027] <1.2 Functional configuration of the server> Fig. 3 is a block diagram showing an example of the functional configuration of the server 20 shown in Fig. 1. As shown in Fig. 3, the server 20 performs the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0028] The communication unit 201 performs processing for communication between the server 20 and external devices. The storage unit 202 stores, for example, drawing information 2021 and estimate information 2022.

[0029] The drawing information 2021 is information about drawings, and specifically, for example, is a data file containing information about drawings provided by a manufacturing supplier. The file format of the drawing information 2021 varies depending on the supplier, for example, and there are various formats. The drawing information 2021 may be, for example, image data or text information. Furthermore, the image data may or may not contain text information. The file format of the image data may be, for example, TIFF, JPEG, PDF (registered trademark), etc. Examples of text information that may be included in the drawing information 2021 include, if the object illustrated in the drawing is an industrial product (or its component), the drawing name, an overview of the product (or component), dimensional information (including dimensional tolerances), material information, information about the manufacturing method (processing method, molding method, whether welding is performed, etc.), the product name (or component name), and product number.

[0030] In this embodiment, the illustrated object shown in the drawing is described as a part of an industrial product. However, there is no particular limitation on the type of illustrated object. For example, the illustrated object may be the industrial product itself, i.e., a finished product. Also, for example, the illustrated object may be a part of a part or a part of a finished product. Alternatively, the illustrated object may be a part or all of a product.

[0031] The quotation information 2022 is information about a quotation for a part depicted in the drawing on which the drawing information 2021 is based. Specifically, the quotation is a data file containing information about a quotation provided by a parts supplier. The quotation is a document received from a supplier to determine the appropriateness of the price of the part before ordering, and includes detailed breakdowns of the unit price (total), such as material costs, processing costs, and management costs. The file format of the quotation information 2022 varies depending on the supplier. The quotation information 2022 may be in various formats, for example, depending on the supplier. The quotation information 2022 may be, for example, image data or text information. Furthermore, the image data may or may not contain text information. The image data file format may be, for example, PNG, TIFF, JPEG, PDF, etc. Examples of text information that may be included in the quotation information 2022 include the material costs, processing costs, and management costs mentioned above, as well as the part name, part number, weight, dimensional information, base number, construction method, supplier information, etc. Note that quotations to which the system 1 is applicable are not limited to quotations provided by suppliers. For example, the system 1 may be applied to quotations prepared by users (such as companies using the search service) themselves. Also, for example, quotations that are applied to the system 1 may include cost details indicating the expenses and costs required to provide the parts.

[0032] The storage unit 202 stores, for example, a data set of drawing information 2021 and estimate information 2022 corresponding to a predetermined part (hereinafter referred to as a basic information data set). In addition, from the viewpoint of simplifying the illustration, only one piece of drawing information 2021 and one piece of estimate information 2022 are stored in the storage unit 202 in the example of Fig. 3, but in reality, a basic information data set is stored for each type of part.

[0033] For example, in the case of competitive bidding in which quotations are requested from multiple suppliers for one part, the basic information dataset may be configured such that multiple pieces of quotation information 2022 are associated with one piece of drawing information 2021. Furthermore, for example, if the part illustrated in the drawing on which the drawing information 2021 is based is an assembly and is composed of multiple components, the basic information dataset may be configured such that one piece of quotation information 2022 is associated with information on multiple components included in one piece of drawing information 2021.

[0034] The uploading of the drawing information 2021 and the quotation information 2022 to the server 20 may be, for example, a plurality of basic information data sets previously uploaded to the server 20 from an information processing device operated by the terminal device 10 or a user of the search service, and stored in the storage unit 202. Here, the "past" in "a plurality of basic information data sets previously uploaded to the server 20" specifically refers to, for example, the time when a component designer operated the terminal device 10 or the like to upload the drawing information 2021 of a drawing he or she created to the server 20. The component designer may be, for example, a designer at the company to which the user belongs, or a designer at the company to which the search service user belongs. In this case, the buyer of the component may be, for example, the user or the user of the search service. In other words, the drawing information 2021 has already been uploaded by the time the buyer user or the user of the search service receives the drawing from the designer. The buyer receives a quotation from the supplier by providing the supplier with the drawing received from the designer.

[0035] In this embodiment, the drawing information 2021 is assumed to be image data of a drawing that was previously uploaded to the server 20. Furthermore, the estimate information 2022 is assumed to be text information relating to an estimate that was previously uploaded to the server 20. The text information relating to the estimate may, for example, be in the form of text information at the time of uploading to the server 20, or may be converted to text information by undergoing OCR processing or the like after the image data of the estimate is uploaded to the server 20. In other words, in this embodiment, a data set of the image data of the drawing and the image data of the estimate constitutes a basic information data set.

[0036] Furthermore, the storage unit 202 has, for example, a user information table 2023 and a search database 2024. The types of tables / databases that the storage unit 202 has are not limited to these. For example, the search database 2024 does not have to be stored in the storage unit 202. Specifically, the content of a search service may differ depending on the content of the first information stored in the search database 2024. Therefore, the search database 2024 may be stored for each server that manages search services with different contents. Also, for example, the search database 2024 may be stored in a server related to a search service for sharing information among multiple users.

[0037] The user information table 2023 is a table that stores various types of information related to users. The search database 2024 is a database that stores multiple pieces of first information. The user information table 2023 and the search database 2024 will be described in detail later.

[0038] The first information is information relating to an object shown in a drawing that is the basis of the drawing information 2021. In this embodiment, the first information is information relating to a part shown in a drawing that is the basis of the drawing information 2021. The first information is basically text information, but may also include image data.

[0039] When the illustrated object is a part as in this embodiment, the text information as the first information is sentences and text data explaining the components and functions of the part. The text information as the first information includes, for example, the product name, product number, material, dimensions, function, characteristics, construction method, installation method, maintenance method, etc. The image data as the first information is image data representing the appearance and structure of the part. The image data as the first information includes, for example, a captured image of the actual appearance of the part, a captured image of a blueprint or cross-section of the part, scan data of a blueprint or cross-section of the part, 2D data representing a blueprint or cross-section of the part, 3D data such as CAD data of the part, and a captured image of the manufacturing process of the part. The file format of the image data may be, for example, TIFF, JPEG, PDF (registered trademark), etc.

[0040] The first information is output from the first generation AI model (details of which will be described later) by inputting drawing information 2021, estimate information 2022, and a first prompt that instructs the output of the first information into the first generation AI model.

[0041] Specifically, the first prompt serves to instruct the first generation AI model on what kind of first information to generate and output. For example, a first prompt such as "Please extract common information and highly related information from the part information of the part illustrated in the drawing that forms the basis of the drawing information 2021 and the part information of the part to be estimated that forms the basis of the estimate information 2022, and use this as the first information" is conceivable. Another example of a first prompt is "Please output the first information in text format and list format."

[0042] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the terminal device 10. The control unit 203 operates in accordance with the program to fulfill the functions of a reception control module 2031, a transmission control module 2032, a database construction module 2033, a search module 2034, and a presentation control module 2035.

[0043] The reception control module 2031 controls the process of the server 20 receiving signals from external devices in accordance with a communication protocol. The transmission control module 2032 controls the process of the server 20 transmitting signals to external devices in accordance with a communication protocol.

[0044] The database construction module 2033, for example, stores the plurality of pieces of first information output from the first generation AI model in a database held by the storage unit 202, and constructs the search database 2024. That is, the server 20 sequentially reads out the plurality of basic information data sets from the storage unit 202, inputs each basic information data set into the first generation AI model, and acquires the plurality of pieces of first information from the first generation AI model.

[0045] Based on the second information, the search module 2034 searches the search database 2024 for first information similar to the second information as similar information. The second information is information related to the search target of the search service, and is transmitted from, for example, the terminal device 10. The second information may be text information or image data. Alternatively, the second information may be a mixture of text information and image data. Details of the similarity determination between the first information and the second information by the search module 2034 will be described later.

[0046] The search target of the search service is, for example, a drawing or an estimate. Hereinafter, the drawing to be searched is referred to as the searched drawing, and the estimate to be searched is referred to as the searched estimate. In other words, when the search target is a searched drawing, information about the parts depicted in the searched drawing becomes the second information. Also, when the search target is a searched estimate, information about the parts that are the subject of the estimate in the searched estimate becomes the second information. In this embodiment, the search target is assumed to be a searched drawing, but as described above, the search target may also be a searched estimate, or the search target may be both a searched drawing and a searched estimate.

[0047] When the search target is a search drawing, the text information as the second information is the same as the text information as the first information described above. Also, the image data as the second information is the same as the image data as the first information described above.

[0048] When the search target is a searched quotation, the text information as the second information is sentences and text data related to the price, conditions, etc. of the part that is the subject of the quotation in the searched quotation. The text information as the second information includes, for example, supplier information, customer (user) information, expiration date, part specifications, price, payment terms, warranty terms, etc. The image data as the second information is image data that displays the searched quotation. The image data as the second information includes a captured image of the screen displaying the searched quotation, scanned data of the searched quotation, a captured image of the searched quotation, etc.

[0049] The presentation control module 2035 controls the process of presenting information to the user. For example, the presentation control module 2035 controls the process of presenting information related to the search results of the search module 2034 to the user as search information.

[0050] <1.3 Configuration of generative AI system> The generative AI system 30 inputs, for example, drawing information 2021, estimate information 2022, and a first prompt received from the terminal device 10 into a first generative AI model and causes the first generative AI model to output the first information. Specifically, for example, the first generative AI model includes a first multimodal generative AI model and a large-scale language model. The large-scale language model may be trained using, for example, a few-shot prompting technique.

[0051] In this embodiment, image data of a drawing previously uploaded to the server 20 is input to the first multimodal generative AI model together with a first prompt as drawing information 2021. The generative AI system 30 causes the first multimodal generative AI model to output first text information as the first information. The first text information is text information about a component depicted in a drawing previously uploaded to the server 20.

[0052] In this embodiment, text information related to an estimate previously uploaded to the server 20 is input as estimate information 2022 to the large-scale language model together with a first prompt. The generative AI system 30 causes the large-scale language model to output second text information as the first information. The second text information is text information related to the parts that are the subject of the estimate on which the estimate information 2022 is based (in other words, the parts depicted in the drawing on which the drawing information 2021 is based). The first text information and the second text information may be entirely identical or entirely different. Alternatively, the first text information and the second text information may be partially identical (in other words, partially different).

[0053] The generation AI system 30, for example, inputs the second information and second prompt received from the terminal device 10 into the second generation AI model, causing the second generation AI model to output search information. The second prompt is text information instructing the output of search information. In this embodiment, the second generation AI model is a second multimodal generation AI model different from the first multimodal generation AI model. Furthermore, the second information is the image data of the search drawing and the text information related to the search drawing. In the following description, it is assumed that the search target is the search drawing. Furthermore, the image data of the search drawing is referred to as search image data, and the text information related to the search drawing is referred to as search text information.

[0054] The first generative AI model may be, for example, either a first multimodal generative AI model or a large-scale language model. When the first generative AI model is the first multimodal generative AI model, the generative AI system 30 inputs the drawing information 2021, the estimate information 2022, and the first prompt to the first multimodal generative AI model as the first generative AI model. In this case, at least one of the drawing information 2021 and the estimate information 2022 may be image data or text information. The first information may also be image data or text information. When the first generative AI model is a large-scale language model, the generative AI system 30 inputs the drawing information 2021, the estimate information 2022, and the first prompt to the large-scale language model as the first generative AI model. In this case, the drawing information 2021, the estimate information 2022, and the first information are all text information.

[0055] The second generative AI model may also be, for example, a large-scale language model. In this case, the generative AI system 30 inputs the second information and the second prompt into the large-scale language model as the second generative AI model. In this case, the second information consists of only the search text information.

[0056] Furthermore, the generative AI system 30 may have, for example, a trained machine learning model such as a neural network model instead of the second generative AI model. In this case, the second information is input to the trained machine learning model, and the second prompt is not required. Alternatively, the generative AI system 30 may not have an AI model in general that uses the second information as input data. In this case, for example, the second information is sent to the search module 2034, and the search module 2034 generates search information.

[0057] The AI ​​model that can be possessed by generative AI system 30 may be trained by, for example, supervised learning, or by both supervised learning and reinforcement learning. Furthermore, the AI ​​model may be a general-purpose model that can handle drawings and quotations provided by multiple suppliers, or may be an AI model dedicated to an individual supplier that has been trained with the contents of drawings and quotations for each supplier.

[0058] [2 Data Structure] 4 and 5 are diagrams showing the data structure of each table stored in the server 20. Note that Figures 4 and 5 are merely examples and do not exclude data that is not listed. Furthermore, even if data is listed in the same table, it may be stored in separate storage areas in the storage unit 202.

[0059] Fig. 4 is a diagram showing the data structure of the user information table 2023. The user information table 2023 shown in Fig. 4 is a table having columns of name, age, sex, date of birth, and contact information, with the user ID as a key.

[0060] The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the user's name. The age is an item that stores the user's age. The gender is an item that stores the user's gender. The date of birth is an item that stores the user's date of birth. The contact information is an item that stores the contact information (e.g., telephone number, email address, etc.) of the terminal device 10 that the user has.

[0061] Fig. 5 is a diagram showing the data structure of the search database 2024. The search database 2024 shown in Fig. 5 is in a data table format having columns of a drawing ID, a drawing URL, an estimate ID, an estimate URL, and part information, with a part ID as a key.

[0062] The part ID is an item that stores an identifier for uniquely identifying a part depicted in a drawing that is the basis of the drawing information 2021. The drawing ID is an item that stores an identifier for uniquely identifying a drawing that is the basis of the drawing information 2021. The drawing URL is an item that stores image data of the drawing, that is, reference information (path) for the drawing information 2021. The quotation ID is an item that stores an identifier for uniquely identifying a quotation that is the basis of the quotation information 2022. The quotation URL is an item that stores reference information (path) for image data of the quotation that is the basis of the quotation information 2022. The part information is an item that stores information about a part depicted in a drawing that is the basis of the drawing information 2021, that is, first information.

[0063] The part information includes, for example, the product name, part number, quantity, category, shape, tolerance, weight, dimensions (maximum diameter, minimum diameter), unit price, receipt date and time, component parts, materials, procurement classification, supplier information, and construction method. For example, if a part illustrated in a drawing that forms the basis of the drawing information 2021 is made up of multiple component parts, the part information includes information such as the product name and part number for each of the multiple component parts. For example, the part information includes the material name, material part number, input weight, yield, unit price, quantity, and dimensions as material details. For example, the part information includes the process name, equipment name, setup time, processing man-hours, total man-hours, total man-hour unit, man rate, machine rate, total wage rate, total wage rate unit, and processing cost as construction details. Note that tokenized text information such as the product name included in the part information is preferably vectorized and stored in the search database 2024 to facilitate calculation of similarity with the second information.

[0064] The content of the part information may vary depending on the type of part. For example, if the part is a pipe or tube, the part information may include the outer diameter, inner diameter, wall thickness, etc. For example, if the part is a bolt, the part information may include the diameter, overall length, head shape, strength class, etc. Or, if the part is a bracket, the part information may include the function (support, attachment, fixing, etc.), etc.

[0065] [3 actions] The operation of the server 20 when presenting search information to the user will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the operation of the server 20 when presenting search information to the user. It is assumed that a plurality of pieces of drawing information 2021 and a plurality of pieces of estimate information 2022 transmitted from the terminal device 10 or the like have been uploaded to the server 20.

[0066] First, the user starts up the terminal device 10. For example, the application 181 is automatically started by the OS of the terminal device 10 when the terminal device 10 starts up. When the application 181 is started up, the presentation control unit 193 may or may not present to the user that the application 181 is running. In this embodiment, the presentation control unit 193 presents to the user that the application 181 is running by displaying an input form screen (predetermined input form) shown in FIG. 7 on the operation surface of the touch-sensitive device 131, in other words, on the display screen of the display 141. Details of the input form screen will be described later.

[0067] In step S11, server 20 outputs first information from the first generation AI model (output step). Specifically, for example, a person in charge of managing system 1 or a person in charge of operating / managing server 20 (hereinafter collectively referred to as a person in charge) operates an input device (not shown) provided in server 20 to input a first prompt to server 20. Note that the first prompt may not be input to server 20, but may instead be input to, for example, a PC (not shown) of a search service provider. Control unit 203 functions as an operation reception module (not shown) to receive the first prompt and transmit the first prompt to transmission control module 2032.

[0068] The transmission control module 2032 sequentially reads out the basic information data set for each part type from the storage unit 202 and transmits it together with the first prompt to the generation AI system 30. At this time, the drawing information 2021 transmitted from the transmission control module 2032 is image data of the drawing. Furthermore, the estimate information 2022 transmitted from the transmission control module 2032 is text information related to the estimate.

[0069] The generative AI system 30 inputs the drawing information 2021 (image data of the drawing) received from the transmission control module 2032, together with the first prompt, into the first multimodal generative AI model. The first multimodal generative AI model, which has received the drawing information 2021 and the first prompt, outputs the first text information as the first information.

[0070] Furthermore, the generative AI system 30 inputs the quotation information 2022 (text information related to the quotation) received from the transmission control module 2032 into the large-scale language model together with the first prompt. The input process of the first prompt into the large-scale language model is, for example, similar to the input process into the first multimodal generative AI model described above. The large-scale language model that has received the quotation information 2022 and the first prompt outputs the second text information as the first information.

[0071] In the present embodiment, the process involves inputting the drawing information 2021 and the estimate information 2022 stored in advance in the storage unit 202 into the first generated AI model, but the process is not limited to this. For example, the content of the first prompt input into the server 20 or a search service provider PC (not shown) may include at least one of the drawing information 2021 and the estimate information 2022. In other words, when the first prompt is input, a first prompt with dynamic content including at least one of the drawing information 2021 and the estimate information 2022 may be generated.

[0072] The generative AI system 30 transmits the first text information output from the first multimodal generative AI model and the second text information output from the large-scale language model to the server 20. The generative AI system 30 performs this series of processes for all of the basic information data sets sequentially transmitted for each part type from the transmission control module 2032.

[0073] In step S12, the server 20 constructs the search database 2024 (construction step). Specifically, for example, the reception control module 2031 receives the first text information and second text information for each type of part sequentially transmitted from the generation AI system 30, and transmits them to the database construction module 2033. For example, every time the database construction module 2033 receives a data set of the first text information and second text information corresponding to a type of part (hereinafter referred to as a text information data set), the database construction module 2033 stores the text information data set in a database for storing first information stored in the storage unit 202.

[0074] At this time, if there are overlapping items between the items included in the first text information (e.g., product name, product number, etc.) and the items included in the second text information, the database construction module 2033 selects, for example, the item with the higher information reliability from the first text information or the second text information. The database construction module 2033, for example, deletes information corresponding to the items not selected. In this way, the text information dataset from which the specific text information has been deleted becomes the first information stored in the search database 2024.

[0075] The database construction module 2033 executes this series of processes for all text information datasets. When the series of processes described above is completed for all text information datasets, the database for storing first information becomes the search database 2024. Note that the database construction module 2033 may, for example, once receive all text information datasets and then collectively store all of the text information datasets in the database for storing first information. Furthermore, although there are no particular limitations on the state of the first information stored in the search database 2024, it is preferable that the first information be stored in a vectorized state in the search database 2024, from the viewpoint of facilitating a similarity determination between the first information and the integrated text information (details will be described later). In this embodiment, it is assumed that the first information is stored in a vectorized state in the search database 2024. However, the search database 2024 may be, for example, a full-text search database or a relational database.

[0076] In S13, the server 20 accepts input of second information (input step). Specifically, for example, the user operates the input device 13 to input the second information to the terminal device 10. The operation acceptance unit 191 accepts the second information input to the input device 13 and transmits it to the transmission / reception unit 192. The transmission / reception unit 192 transmits the second information received from the operation acceptance unit 191 to the server 20. The server 20 accepts input of the second information by the user when the reception control module 2031 receives the second information transmitted from the terminal device 10.

[0077] Hereinafter, the process of accepting input of the second information will be described in detail with reference to Fig. 7. Fig. 7 is a schematic diagram showing an example of the operation surface of touch-sensitive device 131 when the user inputs the second information. Here, in Figs. 7 and 8, the left side as you face the paper is the upper side, and the right side as you face the paper is the lower side. Also, in Figs. 7 and 8, the upper side as you face the paper is the right side, and the lower side as you face the paper is the left side.

[0078] As described above, when the application 181 is launched, the presentation control unit 193 displays a drawing search screen on the operation surface (display screen of the display 141) of the touch-sensitive device 131. As shown in FIG. 7, the drawing search screen has, for example, an input form screen and a drawing display screen. In this embodiment, on the drawing search screen, the input form screen is arranged on the left side, and the drawing display screen is arranged on the right side. Note that this arrangement can be changed as desired.

[0079] The input form screen is provided with, for example, a usage scenario button 1311, an important item field 1312, a refinement field 1313, an upload button 1314, a start button 1315, and a feedback field 1316.

[0080] When the usage scene button 1311 is tapped or the like, the presentation control unit 193 displays, for example, a drop-down menu (not shown) near the arrangement area of ​​the usage scene button 1311. The drop-down menu presents a list of usage scenes for the user's drawing search.

[0081] The important item field 1312 displays multiple types of item options 13121. The items are indicators that may form part of the content of the second information, such as "dimensions," "tolerances," "materials," "construction methods," "production volume," and "equipment." Also, for example, check boxes 13122 are provided on the left side of each option as viewed from the screen. When a specific check box 13122 is tapped or the like, the operation receiving unit 191 receives the selection of the option 13121 corresponding to the specific check box 13122. As shown in FIG. 7, multiple options 13121 may be selected depending on the content of the second information desired by the user.

[0082] By accepting this selection, relevant information of the searched drawing corresponding to the type of selected item is input to touch-sensitive device 131. For example, when the checkbox for the item "dimensions" is tapped, information that "in drawing searches, the dimensions of the parts depicted in the searched drawing are used as search criteria" is input to terminal device 10 as relevant information of the searched drawing. When relevant information of the searched drawing for all the selected item types is input to touch-sensitive device 131, operation accepting unit 191 transmits all relevant information input to touch-sensitive device 131 to transmitting / receiving unit 192.

[0083] The transmitting / receiving unit 192 generates search text information and a second prompt based on all relevant information received from the operation receiving unit 191, for example. The transmitting / receiving unit 192 may generate the search text information and the second prompt together as one piece of text information, such as, for example, "I'm looking for a drawing of a part that is manufactured in the same manufacturing facility as the part depicted in this search drawing and employs a similar processing method. Can you search for relevant drawings?" The transmitting / receiving unit 192 transmits the search text information and the second prompt to the server 20. Note that the operation receiving unit 191 may generate the search text information and the second prompt instead of the transmitting / receiving unit 192.

[0084] The presentation control unit 193 changes the type and number of items displayed in the important item field 1312 depending on the type of usage scenario selected from the drop-down menu list, for example. This allows appropriate generation of search text information depending on the usage scenario of the drawing search desired by the user.

[0085] The refinement field 1313 is provided with a refinement item button 13131, a refinement item 13132, and a text box 13133. When the refinement item button 13131 is tapped or the like, the presentation control unit 193 displays, for example, a drop-down menu (not shown) near the arrangement area of ​​the refinement item button 13131. The drop-down menu presents a list of multiple types of items displayed in the important item field 1312. When the user selects a specific type of item from the list in the drop-down menu, the operation acceptance unit 191 accepts the selection and transmits information indicating the acceptance to the presentation control unit 193. The presentation control unit 193 switches the display of the refinement item 13132 to the specific type of item. In the example of FIG. 7, the display of the refinement item 13132 is switched to the item "tolerance."

[0086] Text box 13133 is composed of a text box for inputting an upper limit value and a text box for inputting a lower limit value. When the upper limit value and the lower limit value are input into text box 13133, transmitting / receiving unit 192 generates search text information that narrows the search conditions so that the types of items displayed in narrowed-down items 13132 are within the numerical range from the lower limit value to the upper limit value. This makes it possible to generate search text information that incorporates the search conditions desired by the user.

[0087] 7, the transmitting / receiving unit 192 generates search text information and the second prompt, but this is not limited to this. For example, instead of the important item field 1312 and the narrowing field 1313, text boxes may be provided on the input form screen. In this case, the user inputs the search text information and the second prompt into the text boxes, and the operation accepting unit 191 accepts the input of the search text information and the second prompt. The transmitting / receiving unit 192 transmits the search text information and the second prompt received from the operation accepting unit 191 to the server 20. The text boxes may be divided into a text box for inputting the search text information and a text box for inputting the second prompt.

[0088] When the upload button 1314 is tapped or the like, the presentation control unit 193 displays, for example, a drop-down menu (not shown) near the arrangement area of ​​the upload button 1314. A list of searched image data is presented in the drop-down menu. In this embodiment, the searched image data presented in this list is image data of drawings stored in the storage unit 202, that is, the drawing information 2021. Note that the searched image data presented in the list of the drop-down menu may be, for example, image data of one or more drawings stored in the terminal device 10 by the user. In this case, only the one or more image data may be presented in the list, or both the one or more image data and the plurality of pieces of drawing information 2021 may be presented in the list.

[0089] When the user selects specific search image data from the list in the drop-down menu, the operation acceptance unit 191 accepts the selection. The transmission / reception unit 192 transmits acceptance information to the server 20 indicating that the selection of the specific search image data has been accepted.

[0090] Here, the transmitting / receiving unit 192 transmits the search text information, the second prompt, and the acceptance information all at once to the server 20, but may transmit the search text information, the second prompt, the acceptance information, or a combination of two of them sequentially to the server 20. The server 20 receives the search text information, the second prompt, and the acceptance information transmitted from the terminal device 10 via the reception control module 2031, and thereby accepts input of the second information by the user.

[0091] When the start button 1315 is tapped or the like, the operation receiving unit 191 receives this as an instruction to switch the input form screen from the initial screen to an inputtable screen, and transmits instruction information representing this instruction to the presentation control unit 193. The presentation control unit 193, which has received the instruction information, switches the input form screen from the initial screen to an inputtable screen. Specifically, for example, on the input form screen in its initial state, nothing is displayed in the items in the important item field 1312 and the narrowed-down items in the narrowed-down field 1313 (not shown). On the other hand, for example, on the input form screen in its initial state, when the start button 1315 is tapped or the like after a usage scenario is selected from the list of the drop-down menu of the usage scenario button 1311, the input form screen switches to the state shown in FIG. 7.

[0092] The search information is displayed in the feedback field 1316. Since Fig. 7 illustrates an example of the operation screen at the time when the second information is input, that is, before the drawing search is performed, nothing is displayed in the feedback field 1316. Note that the feedback field 1316 does not have to be provided on the drawing search screen, and for example, a feedback screen dedicated to displaying the search information may be displayed on the operation screen as a separate screen from the drawing search screen.

[0093] The drawing display screen is provided with, for example, a searched drawing display field 1317 and a similar drawing display field 1318. The searched image data ("searched drawing 1" in the drawing) is displayed as second information in the searched drawing display field 1317. When the presentation control module 2035 receives reception information from the operation reception unit 191 indicating that the selection of specific searched image data has been received, it reads out the specific searched image data (specific drawing information 2021) from the storage unit 202 and displays it in the searched drawing display field 1317.

[0094] 7 illustrates an example of the operation screen at the time when the second information is input, i.e., before the drawing search is performed, so nothing is displayed in the similar drawing display field 1318. Note that the screen configuration and display contents of the drawing search screen shown in FIG. 7 are merely examples, and various screen configurations and display contents can be adopted.

[0095] In S14, the server 20 searches for similar information from the search database 2024 based on the received second information (presentation step). Specifically, for example, the reception control module 2031 receives the search image data, search text information, and second prompt transmitted from the terminal device 10, and temporarily stores them in the storage unit 202. Note that the search image data, search text information, and second prompt may be temporarily stored in a storage unit (not shown) of the generation AI system 30.

[0096] Upon receiving the instruction information, the reception control module 2031 reads the search image data, search text information, and second prompt from the storage unit 202 and transmits them to the transmission control module 2032. The transmission control module 2032 transmits the received search image data, search text information, and second prompt to the generation AI system 30.

[0097] The generative AI system 30 inputs the received search image data, search text information, and second prompt into a second multimodal generative AI model, causing the second multimodal generative AI model to output integrated text information. The integrated text information is text information related to the searched drawing, and is text information that integrates the content of the search text information and the content of the searched image data. The generative AI system 30 transmits the integrated text information output from the second multimodal generative AI model to the server 20.

[0098] The reception control module 2031 receives the integrated text information sent from the generation AI system 30 and sends it to the search module 2034. Upon receiving the integrated text information, the search module 2034 reads out the first information for each type of part from the search database 2024 and compares it with the integrated text information.

[0099] The search module 2034 performs a similarity determination between the first information and the integrated text information using, for example, a cosine similarity method. Cosine similarity is a method for calculating the similarity between two vectors in a vector space. The cosine similarity value ranges from -1 to 1. The closer the cosine similarity value is to 1, the more similar the two vectors are, and the closer it is to 0, the less similar they are. Furthermore, the closer the cosine similarity value is to -1, the more opposed the two vectors are. Because cosine similarity depends on the direction and magnitude of the vectors, the similarity determination can be performed by focusing only on the direction of the vectors, regardless of the scale of the vectors. This makes it easy to determine the similarity between two vectorized pieces of text information. The search module 2034 may also perform a similarity determination between the first information and the integrated text information using a method other than cosine similarity.

[0100] Specifically, for example, the search module 2034 vectorizes the integrated text information received from the reception control module 2031. The search module 2034 calculates the cosine similarity between the already vectorized first information and the vectorized integrated text information using a general formula for cosine similarity. For example, if the calculated cosine similarity value is equal to or greater than a reference value (which can be set arbitrarily), the search module 2034 determines that the first information and the integrated text information are similar.

[0101] In S15, the server 20 causes the second generation AI model to output the search information (presentation step). Specifically, for example, if the search module 2034 is able to search the search database 2024 for specific first information similar to the integrated text information, the search module 2034 considers the specific first information to be similar to the second information (the search image data and the search text information) and sets it as similar information. The search module 2034 transmits the searched similar information to the transmission control module 2032. The transmission control module 2032 transmits the received similar information to the generation AI system 30. The generation AI system 30 inputs the received similar information into the second multimodal generation AI model and causes the second multimodal generation AI model to output the search information.

[0102] Also, for example, if the search module 2034 is unable to search the search database 2024 for specific first information similar to the integrated text information, it generates unsearchable information indicating that the search was unsuccessful. The search module 2034 transmits the unsearchable information to the transmission control module 2032. The transmission control module 2032 transmits the received unsearchable information to the generation AI system 30. The generation AI system 30 inputs the received unsearchable information into a second multimodal generation AI model and causes the second multimodal generation AI model to output search information. Through this series of processes, the server 20 causes the second generation AI model to output search information.

[0103] In S16, the server 20 presents the search information to the user (presentation step). Specifically, for example, the generation AI system 30 transmits the search information output from the second generation AI model (second multimodal generation AI model) to the server 20. The reception control module 2031 transmits the received search information to the presentation control module 2035. The presentation control module 2035 transmits the received search information to the terminal device 10, thereby displaying the received search information on the display screen of the display 141 (the operation surface of the touch-sensitive device 131). Through this series of processes, the server 20 presents the search information to the user.

[0104] An example of presenting search information to a user will be described below with reference to FIG. 8. FIG. 8 is a schematic diagram showing an example of the display screen of the display 141 when presenting search information to a user. Since the display screen of the display 141 is also the operation surface of the touch-sensitive device 131, when the presentation control module 2035 transmits the search information to the terminal device 10, the drawing search screen transitions from the state shown in FIG. 7 to, for example, the state shown in FIG. 8. That is, the search information is displayed in the feedback field 1316 of the input form screen in the state shown in FIG. 7. The search information is also displayed in the similar drawing display field 1318 of the drawing display screen in the state shown in FIG. 7.

[0105] The search information may be information of any content and format as long as it is information related to the search results. In the example of FIG. 8, the feedback field 1316 displays search information in text format, such as "Similar drawings A to D were found under the above conditions." Also, in the example of FIG. 8, the feedback field 1316 displays advice information in text format, such as "If the tolerance is relaxed, drawings AA and BB will be similar drawings. If the item "Equipment" is unchecked, drawings CC and DD will be similar drawings." The advice information presents the content of the second information that should be modified, etc., to facilitate the search for similar drawings (drawings containing similar information) that are similar to the search drawing, depending on the content of the search drawing and the second information. The inclusion of advice information in the search information presented to the user makes it easier for the user to search for similar drawings.

[0106] Furthermore, the presentation control module 2035 displays, as search information, reference information (path) of image data of similar drawings stored in the search database 2024 in the similar drawing display field 1318. For example, when the user taps on the reference information displayed in the similar drawing display field 1318, the presentation control module 2035 displays image data of similar drawings ("Similar Drawing A" to "Similar Drawing D" in the figure) as shown in FIG. 8 in the similar drawing display field 1318.

[0107] [4 Screen Examples] The input screen for second information displayed on the operation surface of touch-sensitive device 131 and the presentation screen for searched information displayed on the display screen of display 141 can have various screen configurations other than the examples shown in Figures 7 and 8. When input device 13 and output device 14 are touch panels as in this embodiment, for example, a chat-based input / presentation screen shown in Figure 9 is conceivable. Figure 9 is a schematic diagram showing another example of the display screen of display 141 when searched information is presented to the user. The input / presentation screen shown in Figure 9 allows drawing searches to be performed simply by inputting text information, for example, by outputting search information including advice information from the second generation AI model (second multimodal generation AI model) according to the content of the text information (second information) that is the input data, and presenting this information to the user.

[0108] Specifically, for example, a text input field 1411, an image display field 1412, a search information display field 1413, and a select box 1414 are displayed on the input / presentation screen shown in FIG.

[0109] Text information as the second information is input into the text input field 1411. In the example of Fig. 9, the text information "Could you please send me the drawing of the solenoid valve I requested from XX Co., Ltd.? Preferably, it should be made of plastic." is input into the text input field 1411.

[0110] The terminal device 10 transmits the text information received by the operation reception unit 191 to the server 20. The server 20 transmits the text information received by the reception control module 2031 to, for example, the generation AI system 30. The generation AI system 30, for example, inputs the received text information into a second multimodal generation AI model and causes the second multimodal generation AI model to output search information including advice information. The generation AI system 30 transmits the search information output from the second multimodal generation AI model to the server 20. The server 20 transmits the operation information received by the reception control module 2031 to the presentation control module 2035. The presentation control module 2035 transmits the search information to the terminal device 10 and displays the search information in the search information display field 1413.

[0111] Here, if one or more types of similar drawings are found, the presentation control module 2035 displays, for example, reference information (path) of image data of the similar drawings stored in the search database 2024 in the image display field 1412. For example, when the user taps on the reference information displayed in the image display field 1412, the presentation control module 2035 displays image data of the similar drawings in the image display field 1412 as shown in FIG.

[0112] 9, when the search target of the system 1 is a searched estimate, reference information (path) of image data of a similar estimate (an estimate including similar information) stored in the search database 2024 may be displayed in the image display field 1412. Also, for example, when the user taps on the reference information displayed in the image display field 1412, the presentation control module 2035 may display image data of the similar estimate in the image display field 1412. Furthermore, when the search target of the system 1 is both a searched estimate and a searched estimate, the presentation control module 2035 may display image data of the similar drawing and image data of the similar estimate in the image display field 1412.

[0113] As described above, by presenting the user with at least one of the image data of a similar drawing and the image data of a similar quotation, the user can easily determine whether the similar drawing, etc. searched by the server 20 matches the drawing, etc. desired by the user.

[0114] In the example of Fig. 9, reference information for two types of similar drawings is displayed in image display field 1412 according to the content of the text information entered in text input field 1411, and by tapping on this reference information, image data of the two types of similar drawings is displayed in image display field 1412. Also, in the example of Fig. 9, search information is displayed in search information display field 1413 as a text chat in response to the text information entered in text input field 1411, saying, "There were 10 drawings that matched the specified conditions. To narrow down the search further, please specify the date and time the drawings were received and the equipment from the items below."

[0115] 9, when the search information includes information indicating that it is preferable to narrow down specific search criteria (specific content included in the second information), the presentation control module 2035 displays, for example, a select box 1414 provided with options for narrowing down the specific search criteria. In other words, the select box 1414 is not necessarily displayed on the input / presentation screen shown in Fig. 9, and may not be displayed depending on the display content of the search information display field 1413. In the example of Fig. 9, the select box 1414 provided with options for narrowing down the drawing receipt date and equipment is displayed based on the display content in the search information display field 1413 indicating that it is preferable to narrow down the drawing receipt date and equipment.

[0116] In the example of Figure 9, when a user selects a specific option in the select box 1414, new reference information (and image data of similar drawings obtained by tapping on the reference information, etc.) is displayed in the image display field 1412 as chat text, and search information such as "The above drawing has appeared as a candidate for a similar drawing based on the items you selected..." is displayed in the search information display field 1413.

[0117] 9, the presentation control module 2035 dynamically and flexibly displays search information and UI elements for narrowing down search conditions in accordance with the content of the text information entered in the text input field 1411. This allows the user to efficiently search for desired similar drawings with minimal UI operations.

[0118] [5 Summary] As described above, in this embodiment, the database construction module 2033 stores the first information obtained by inputting the drawing information 2021 and the estimate information 2022 into the first generation AI model in a database for storing first information, and constructs the search database 2024. The search module 2034 searches for similar information using the search database 2024 and causes the second generation AI model to output the searched information. The presentation control module 2035 presents the searched information to the user by displaying it on the display screen of the display 141. This enables drawing searches to be performed taking into account the estimate information 2022, making it possible to flexibly respond to a variety of use cases in drawing searches using the drawing information 2021.

[0119] In addition, in this embodiment, the operation accepting unit 191 accepts input of the second information by inputting relevant information of the searched drawing into each of a plurality of items on an input form screen displayed on the operation surface of the touch-sensitive device 131. This eliminates the need for the user to input all of the content of the second information, thereby improving the user's convenience in drawing searches.

[0120] Furthermore, in this embodiment, the generative AI system 30 acquires first information by inputting image data as drawing information 2021 into a first multimodal generative AI model and inputting text information as estimate information 2022 into a large-scale language model. The generative AI system 30 acquires search information by inputting search image data and search text information into a second multimodal generative AI model. This makes the first information and search information more accurate, taking into account the relevance of different types of information, image data and text information. This improves the accuracy of drawing searches.

[0121] [6. Modifications] <6.1 First Modification> In this embodiment, the system 1 has been described as a system for providing a search service relating to a drawing search service, an estimate search service, or a combination of these two. However, in addition to these search services, the system 1 may also provide, for example, a search service for estimate assessment materials.

[0122] Quotation evaluation materials are materials used within the user's company to evaluate whether the contents of a quotation are appropriate, and examples include approval documents and quotation comparison documents. Approval documents are materials used within the user's company to obtain approval / decision for ordering the parts that are the subject of the search quotation. Quotation comparison materials are materials related to the results of comparing a quotation with other quotations, and examples include similar part quotation comparison documents, competitive quotation comparison documents, past quotation comparison documents for the same part number, and past quotation comparison documents for the same supplier.

[0123] Similar parts quotation comparison documents are documents showing the results of comparing quotations for parts similar to the part that is the subject of a quotation in a search quotation with a search quotation. Competitive quotation comparison documents are documents showing the results of comparing quotations that are the subject of a competitive quotation in a search quotation with a search quotation. Estimated past quotation comparison documents for the same part number are documents showing the results of comparing quotations for parts with the same part number as the part that is the subject of a quotation in a search quotation with a search quotation. Same supplier past quotation comparison documents are documents showing the results of comparing quotations submitted by the same supplier as the supplier of the part that is the subject of a quotation in a search quotation with a search quotation.

[0124] When the system 1 provides a search service for estimate assessment materials, for example, the server 20 may store information about the estimate assessment materials in addition to the drawing information 2021 and the estimate information 2022 in the memory unit 202. The transmission control module 2032 may read out the drawing information 2021, the estimate information 2022, and the information about the estimate assessment materials from the memory unit 202 and transmit it to the generative AI system 30. The generative AI system 30 may input these received three types of information together with a first prompt into a first generative AI model, and cause the first generative AI model to output the first information.

[0125] As a result, the first generation AI model outputs first information that takes into account information about the estimate assessment materials, thereby increasing the accuracy of the first information stored in the search database 2024. This also increases the accuracy of the search information, allowing the user to search for similar estimate assessment materials (estimate assessment materials that include similar information) with high accuracy. Note that even if the system 1 does not provide, for example, a search service for estimate assessment materials, that is, even if the system 1 provides only the search service according to this embodiment (a drawing search service, an estimate search service, or a search service relating to a combination of these two), information about the estimate assessment materials may be used as input data to the first generation AI model.

[0126] Furthermore, when the system 1 provides a search service for estimate evaluation materials, the presentation control module 2035 may present the user with, for example, text information about similar estimate evaluation materials (estimate evaluation materials containing similar information) as search information. In this case, the presentation control module 2035 may present image data of the similar estimate evaluation materials together with the search information. By presenting this image data to the user, the user can easily determine whether the similar estimate evaluation materials searched by the server 20 match the estimate evaluation materials desired by the user.

[0127] In this embodiment and the first modified example, transaction communication information may be added to the input data to the first generation AI model. The transaction communication information is communication information related to a drawing on which the drawing information 2021 is based, an estimate on which the estimate information 2022 is based, an estimate assessment document on which information related to the estimate assessment document is based, or a combination of at least two of these. Specifically, for example, the transaction communication information is transaction information exchanged between a user and a supplier via wireless communication (or wired communication), and is stored in the memory unit 202.

[0128] The transaction communication information includes detailed information about the drawing and the quotation that is not included in the drawing and the quotation itself. Therefore, by adding the transaction communication information to the input data to the first generation AI model, the first generation AI model outputs first information that takes into account the transaction communication information, thereby increasing the accuracy of the first information stored in the search database 2024. This also increases the accuracy of the search information, allowing users to search for similar drawings, etc. with high accuracy.

[0129] The transaction communication information may also include information regarding the exchanges that occurred up to the determination of the estimated amount stated in the quotation that is the basis for the quotation information 2022. Adding transaction communication information that includes this information to the input data for the first generation AI model further increases the accuracy of the first information stored in the search database 2024. This in turn further increases the accuracy of the search information, allowing the user to search for similar drawings, etc. with greater precision.

[0130] <6.2 Second Modification> In this embodiment, the first information is composed of the first text information output from the first multimodal generative AI model and the second text information output from the large-scale language model. However, the generative AI system 30 may further input the first text information and the second text information to a third multimodal generative AI model different from both the first and second multimodal generative AI models, and output the first information from the third multimodal generative AI model. There are no particular limitations on the type of the third multimodal generative AI model, and any type of multimodal generative AI model may be used as the third multimodal generative AI model.

[0131] The first information output from the third multimodal generative AI model is third text information. The third text information is text information about a component depicted in a drawing previously uploaded to the server 20. The third text information may be entirely identical to the first text information and the second text information, or may be entirely different from the first text information and the second text information. Alternatively, the third text information may be partially identical to at least one of the first text information and the second text information.

[0132] Specifically, for example, the generative AI system 30 may have a third multimodal generative AI model in addition to the first generative AI model (first multimodal generative AI model and large-scale language model) and the second generative AI model (second multimodal generative AI model). The transmission control module 2032 may read image data of a drawing as drawing information 2021 from the storage unit 202 and transmit the image data together with an explanatory text related to the image data to the generative AI system 30. With regard to the explanatory text, for example, the reception control module 2031 may receive text information of the explanatory text input by the operation reception unit 191, and the transmission control module 2032 may transmit the text information to the generative AI system 30. Also, for example, the text information of the explanatory text may be associated with the image data of the drawing as drawing information 2021 and stored in advance in the storage unit 202.

[0133] The generative AI system 30 may input the first text information output from the first multimodal generative AI model and the second text information output from the large-scale language model, along with image data of the drawing, an explanatory text, and the first prompt as drawing information 2021, to a third multimodal generative AI model, and cause the third multimodal generative AI model to output the third text information. The search module 2034 may determine whether the third text information and the integrated text information are similar to each other using a cosine similarity method, as in the present embodiment.

[0134] This allows the third text information, which more appropriately associates the image data of the drawing with the first text information and the second text information via the explanatory text, to be stored as the first information in the search database 2024. This increases the accuracy of the search information, allowing the user to search for similar drawings, etc. with high precision. Note that in the second modified example, it is not essential to input the explanatory text into the third multimodal generation AI model.

[0135] <6.3 Third Modification> In this embodiment, the second multimodal generative AI model has not been fine-tuned. However, for example, the generative AI system 30 may fine-tune the second multimodal generative AI model based on search information presented to a user. Here, fine-tuning is a process of adjusting the parameters of a pre-trained AI model to adapt it to a specific task or dataset. The timing of the fine-tuning process is arbitrary and may be performed monthly or daily.

[0136] For example, the generative AI system 30 may detect that the time for fine tuning has arrived, read multiple tuning datasets from a storage unit (not shown), and fine-tune the second multimodal generative AI model using the multiple tuning datasets. The tuning datasets may be, for example, datasets of past search information and past evaluation information, and may be stored in the storage unit 202. The past search information may be, for example, search information that the presentation control module 2035 previously displayed on the display screen of the display 141 (the operation surface of the touch-sensitive device 131). The past evaluation information may be, for example, information regarding user evaluations of the past search information, and may be associated with the past search information. The past evaluation information may be, for example, input from the input device 13 and transmitted to the generative AI system 30. The multiple tuning datasets may be stored in the storage unit 202. In this case, the transmission control module 2032 reads the multiple tuning datasets from the storage unit 202 and transmits them to the generative AI system 30.

[0137] Specifically, fine-tuning of the second multimodal generative AI model by the generative AI system 30 may be performed as follows. That is, the generative AI system 30 fixes some layers of the second multimodal generative AI model and decompresses other layers to enable learning (model adjustment). Next, the generative AI system 30 sets / adjusts a learning rate and performs fine-tuning using multiple tuning datasets. Next, the generative AI system 30 evaluates the second multimodal generative AI model after fine-tuning. As an evaluation method, for example, recall, precision, F1 score, or the like is adopted. If the evaluation result indicates that the second multimodal generative AI model after fine-tuning does not exhibit sufficient performance, the generative AI system 30 adjusts the model again and performs fine-tuning. If the evaluation result indicates that the second multimodal generative AI model after fine-tuning exhibits sufficient performance, the generative AI system 30 deploys the second multimodal generative AI model.

[0138] This makes it possible to construct a second multimodal generative AI model optimized for search services such as drawing searches and estimate searches, thereby increasing the accuracy of search information. This allows users to search for similar drawings, etc. with high precision. Note that if the second generative AI model is, for example, a large-scale language model or a neural network model (trained machine learning model) instead of the second multimodal generative AI model, either of these models may be fine-tuned.

[0139] [7 Basic Computer Hardware Configuration] 10 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus 921.

[0140] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0141] The main storage device 902 is a memory for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0142] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0143] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards.

[0144] The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0145] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0146] [8 Basic Functional Configuration of a Computer] The following describes the functional configuration of a computer realized by the basic hardware configuration (FIG. 10) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0147] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0148] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding them in the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0149] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. Furthermore, the control unit can cause the processor 901 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0150] A database refers to a relational database, which manages data sets called masters and tables in a tabular format structurally defined by rows and columns, by relating them to each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated.

[0151] Typically, each table and each master has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs.

[0152] Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0153] Note that the databases and masters in this disclosure may include any data structure in which information is structurally defined (such as a list, dictionary, associative array, or object). The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0154] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0155] Furthermore, some or all of the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0156] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, and Java (registered trademark).

[0157] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or the storage medium.

[0158] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory.

[0159] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0160] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0161] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0162] [9. Appendix] The matters explained in the above embodiment and modified examples will be supplemented below.

[0163] <Appendix 1> A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the following steps: inputting drawing information regarding a drawing, estimate information regarding an estimate for an illustrated object depicted in the drawing, and a first prompt instructing the output of first information regarding the illustrated object into a first generation AI model, and causing the first generation AI model to output the first information; a construction step storing the outputted first information in a database and constructing a search database; an input step accepting input of second information regarding the search target; and a presentation step searching the search database for first information similar to the second information based on the received second information, and presenting search information regarding the search results.

[0164] <Appendix 2> In the output step, communication information relating to a drawing, an estimate, materials relating to the comparison results when the estimate is compared with other estimates, or a combination of at least two of these, is input into the first generation AI model (Supplementary Note 1).

[0165] <Appendix 3> The communication information includes information regarding the exchanges leading up to the determination of the estimated amount set forth in the quotation (Appendix 2).

[0166] <Appendix 4> A program described in any one of (Appendix 1) to (Appendix 3), in which, in an input step, the input of second information is accepted by inputting the relevant information to be searched into each of multiple types of items provided in a specified input form.

[0167] <Appendix 5> The program according to any one of (Supplementary Note 1) to (Supplementary Note 4), wherein the presentation step presents image data of similar drawings containing similarity information.

[0168] <Appendix 6> A program described in any one of (Appendix 1) to (Appendix 4), wherein in the presentation step, image data of a similar estimate containing similar information is presented.

[0169] <Appendix 7> A program described in any one of (Appendix 1) to (Appendix 4), in which, in the presentation step, image data of materials relating to the comparison results when a similar estimate containing similar information is compared with another estimate is presented.

[0170] <Appendix 8> A program described in any of (Appendix 1) to (Appendix 7), in which, in the presentation step, a second prompt instructing the output of the second information and the search information is input to a second generation AI model different from the first generation AI model, and the search information output from the second generation AI model is presented.

[0171] <Appendix 9> The search information includes an explanatory text that explains the search results (Supplementary Note 8).

[0172] <Appendix 10> In the output step, the first generation AI model includes a first multimodal generation AI model and a large-scale language model, and image data of the drawing as drawing information is input to the first multimodal generation AI model together with a first prompt, causing the first multimodal generation AI model to output first text information regarding the illustrated object as the first information, and text information regarding an estimate as estimate information is input to the large-scale language model together with the first prompt, causing the large-scale language model to output second text information regarding the illustrated object as the first information, and in the input step, input of image data of the search target and text information regarding the search target is accepted as input of the second information, and in the presentation step, the accepted image data of the search target, the accepted text information regarding the search target, and a second prompt instructing output of the search information are input to a second multimodal generation AI model different from the first multimodal generation AI model, and the search information output from the second multimodal generation AI model is presented (the program described in Appendix 1).

[0173] <Appendix 11> In the output step, the first text information, the second text information, the image data of the drawing, the explanatory text relating to the image data of the drawing, and the first prompt are input into a third multimodal generation AI model different from both the first multimodal generation AI model and the second multimodal generation AI model, and the third multimodal generation AI model is caused to output third text information relating to the illustrated object as the first information (Appendix 10).

[0174] <Appendix 12> A program as described in (Appendix 10) or (Appendix 11), which causes a processor to execute an adjustment step of fine-tuning the second multimodal generation AI model based on the presented search information.

[0175] <Appendix 13> A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 12).

[0176] <Appendix 14> An information processing device comprising a control unit and a storage unit, wherein the control unit executes all of the steps executed in the invention according to any one of (Appendix 1) to (Appendix 12).

[0177] <Appendix 15> A system comprising means for executing all steps performed in any of the inventions according to (Appendix 1) to (Appendix 12). [Explanation of symbols]

[0178] 1. System 10...Terminal device 120…Communications Department 13...Input device 14...Output device 15...Memory 16…Storage 19...Processor 20...Server 22...Communication IF 23...Input / output interface 25…Memory 26…Storage 29...Processor 30...Generative AI system

Claims

1. A program to be executed by a computer including a processor and a memory, the program causing the processor to: an output step of inputting drawing information relating to a drawing, estimate information relating to an estimate of a depicted object depicted in the drawing, and a first prompt instructing the output of first information relating to the depicted object into a first generated AI model, and causing the first generated AI model to output the first information; a construction step of storing the plurality of pieces of first information that have been output in a database to construct a search database; an input step of accepting input of second information related to the search target; a presentation step of searching a search database for the first information similar to the second information as similar information based on the received second information, and presenting search information related to the search result; A program that executes the following.

2. The program according to claim 1, wherein in the output step, communication information relating to the drawing, the estimate, materials relating to the comparison results when the estimate is compared with other estimates, or a combination of at least two of these, is input to the first generative AI model.

3. The program according to claim 2 , wherein the communication information includes information about exchanges leading up to the determination of the estimated amount stated in the estimate.

4. 2. The program according to claim 1, wherein the input step accepts input of the second information by inputting the relevant information to be searched into each of a plurality of types of items provided in a predetermined input form.

5. 2. The program according to claim 1, wherein the presentation step comprises presenting image data of similar drawings containing the similarity information.

6. The program according to claim 1 , wherein the presenting step presents image data of similar quotations containing the similar information.

7. 2. The program according to claim 1, wherein the presentation step presents image data of materials relating to a comparison result when the similar estimate containing the similar information is compared with another estimate.

8. The program described in claim 1, wherein in the presentation step, a second prompt instructing the output of the second information and the search information is input to a second generation AI model different from the first generation AI model, and the search information output from the second generation AI model is presented.

9. The program according to claim 8 , wherein the search information includes an explanation of the search results.

10. In the output step, the first generative AI model includes a first multi-modal generative AI model and a large-scale language model; inputting image data of the drawing as the drawing information together with the first prompt into the first multimodal generation AI model, and outputting first text information regarding the illustrated object as the first information from the first multimodal generation AI model; inputting text information relating to the quotation as the quotation information into the large-scale language model together with the first prompt, and outputting second text information relating to the illustrated object as the first information from the large-scale language model; In the input step, input of the image data of the search target and text information related to the search target is accepted as input of the second information; In the presentation step, The program of claim 1, wherein the received image data of the search target, the received text information related to the search target, and a second prompt instructing the output of the search information are input to a second multimodal generation AI model different from the first multimodal generation AI model, and the search information output from the second multimodal generation AI model is presented.

11. The program of claim 10, wherein in the output step, the first text information, the second text information, the image data of the drawing, an explanatory text relating to the image data of the drawing, and the first prompt are input to a third multimodal generation AI model different from both the first multimodal generation AI model and the second multimodal generation AI model, and third text information relating to the illustrated object is output from the third multimodal generation AI model as the first information.

12. and fine-tuning the second multimodal generative AI model based on the presented search information. The program according to claim 10, which causes the processor to execute the steps of:

13. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 12.

14. 13. An information processing device comprising a control unit and a storage unit, wherein the control unit executes all of the steps executed in the invention according to any one of claims 1 to 12.

15. A system comprising means for executing all steps performed in the invention according to any one of claims 1 to 12.

Images