Information processing system, information processing method, and information processing program

Abstract

The appearance of an object is evaluated using at least one of a single type of data and multiple types of data. [Solution] The information processing system inputs appearance data representing the appearance of an object into a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the appearance base data, the information processing system obtains related data associated with the appearance base data from multiple base data stored in a database. The information processing system inputs the appearance base data and related data into a second base model, which is a trained model, thereby obtaining evaluation data representing the evaluation of the object's appearance, and outputs the evaluation data.

Description

Technical Field

[0004] , , , , , , , , , , ,

[0005] ,

[0003]

[0001] The disclosed technology relates to an information processing system, an information processing method, and an information processing program. <000000�>

Background Art

[0002] Patent Document 1 discloses an inspection system and an inspection method capable of improving the efficiency of appearance inspection. The inspection system disclosed in Patent Document 1 includes a question generation unit that generates a first question based on reference information indicating the number and arrangement of each article in a reference product having a plurality of articles, and a learned model configured to answer a natural language question about the content of an image. A question answering unit that generates a first answer to the first question for an image of an inspection target product, and a determination unit that determines whether the inspection target product satisfies the reference indicated by the reference information based on the first answer (for example, summary).

[0003] Patent Document 2 discloses a system capable of corresponding to various products and manufacturing processes by efficiently utilizing a knowledge base using RAG (Retrieval-Augmented Generation) (for example, paragraph

[0044] ).

[0004] Patent Document 3 discloses a means for digitizing quality management information including at least any one of inspection items, points of interest, points to note, ideal states, assumed quality risks, preventive measures against risks, and corrective measures for each section based on a QC flowchart, and based on the digitized quality management information. A system including a means for creating a tool capable of announcing the quality management information using a generation AI model and a means for providing the tool is disclosed (for example, summary).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] Incidentally, when evaluating the appearance of an object, image data obtained by photographing the object's exterior is often used. However, in some cases, it is preferable to evaluate the appearance of an object using other types of data rather than image data.

[0007] The disclosed technology was made in view of the above circumstances and provides an information processing system, method, and program that can perform an appearance evaluation of an object using at least one of a single type of data and multiple types of data. [Means for solving the problem]

[0008] To achieve the above objective, a first aspect of this disclosure is an information processing system equipped with a processor, wherein the processor inputs appearance data representing the appearance of an object to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby acquiring appearance base data output from the first base model, and based on the appearance base data, acquires related data associated with the appearance base data from a plurality of base data stored in a database, and inputs the appearance base data and the related data to a second base model, which is a trained model, thereby acquiring evaluation data representing an evaluation of the appearance of the object, and outputs the evaluation data.

[0009] A second aspect of this disclosure is an information processing method in which a computer performs processing, which involves inputting appearance data representing the appearance of an object into a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model; obtaining related data associated with the appearance base data from a plurality of base data stored in a database based on the appearance base data; inputting the appearance base data and the related data into a second base model, which is a trained model, thereby obtaining evaluation data representing an evaluation of the appearance of the object, and outputting the evaluation data.

[0010] A third aspect of this disclosure is an information processing program for causing a computer to perform processing, which involves inputting appearance data representing the appearance of an object into a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby acquiring appearance base data output from the first base model; acquiring related data associated with the appearance base data from a plurality of base data stored in a database based on the appearance base data; inputting the appearance base data and the related data into a second base model, which is a trained model, thereby acquiring evaluation data representing an evaluation of the appearance of the object, and outputting the evaluation data. [Effects of the Invention]

[0011] According to the disclosed technology, the effect is that the appearance evaluation of an object can be performed using at least one of a single type of data and multiple types of data. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of a schematic configuration of an information processing system according to an embodiment. [Figure 2] This is a diagram illustrating an information processing system of an embodiment. [Figure 3] This figure shows an example of data stored in the first database. [Figure 4] This figure shows an example of data stored in the second database. [Figure 5] This figure shows an example of a computer that makes up each device. [Figure 6] This figure shows an example of information processing performed by the server of the embodiment. [Modes for carrying out the invention]

[0013] Embodiments of the disclosed technology will be described in detail below with reference to the drawings.

[0014] <Information Processing System of an Embodiment> Figure 1 is a schematic diagram of the information processing system 10 according to the embodiment. As shown in Figure 1, the information processing system 10 of the embodiment comprises a plurality of user terminals 12A, 12B, 12C, ..., a server 14 which is an example of an information processing system, a first database server 16, and a second database server 18. Each device is connected to communicate via a network 20 such as the Internet. Hereinafter, any one of the plurality of user terminals 12A, 12B, 12C, ... will simply be referred to as user terminal 12.

[0015] Figure 2 is a diagram illustrating an information processing system according to an embodiment. In Figure 2, 16A represents processing performed by the first database server 16, 18A represents processing performed by the second database server 18, and processing different from 16A and 18A represents processing performed by server 14. The base model shown in Figure 2 is a trained model, for example, a so-called large-scale language model. The base model shown in Figure 2 may also be a language base model, non-language base model, or multimodal base model capable of image processing and speech processing.

[0016] The language-based model processes text data as input and generates a token sequence by arranging the tokens with the highest probability from the vocabulary learned by the language-based model (specifically, an ID number identified by an integer corresponding to a certain combination of characters). The language-based model converts this token sequence into a tensor (multi-dimensional array) and generates a token sequence corresponding to the input text data. Then, the language-based model can generate natural language or a specified non-language data format, such as JSON, csv, vector, or definable data including tensors (hereinafter simply referred to as "base data") by generating a combination of characters corresponding to each token.

[0017] Also, the non-language-based model, which is a machine learning model different from the language-based model, is good at processing non-language data such as image data, audio data, sensor data, or graph data (e.g., the structure of a protein) rather than text data. The output of the non-language-based model varies widely, including tensors, masks, vectors, three-dimensional structure coordinates, grid data, waveforms, or spectrograms (corresponding to the "base data" described above). Therefore, the base data generated by the non-language-based model may be difficult for humans to understand, and it is also possible to convert the base data output by the non-language-based model into base data in a natural language format that is easier for humans to understand by converting it using the language-based model.

[0018] In the present embodiment, as shown in FIG. 2, appearance data representing the appearance of an object to be evaluated is converted into base data using the first base model A and the first base model B. As will be described later, the appearance data in the present embodiment is single-type data (single-modal data) or multiple types of data (multi-modal data). In the present embodiment, by converting the appearance data into base data, the appearance of the object is evaluated. Specifically, in the present embodiment, by converting the appearance data into base data, it becomes possible to uniformly handle the appearance data even if it is multiple types of data (multi-modal data) or single-type data (single-modal data).

[0019] Also, in this embodiment, as shown in 16A of FIG. 2, a third base model is pre-trained by reinforcement learning based on normal appearance data representing the normal appearance of an object and abnormal appearance data representing the abnormal appearance of the object. Then, a first database 164 is constructed by a plurality of base data output from the third base model. Specifically, by inputting each of the plurality of normal appearance data into the third base model, base data regarding the normal appearance data is output from the third base model. Also, by inputting each of the plurality of abnormal appearance data into the third base model, base data regarding the abnormal appearance data is output from the third base model. The first database 164 is pre-constructed by these base data. As will be described later, since the first database 164 is used for the RAG technology (Retrieval-Augmented Generation), it can be said that it is a database for retrieval expansion. Note that the third base model is pre-trained by RLHF (Reinforcement Learning from Human Feedback).

[0020] Also, in this embodiment, as shown in 18A of FIG. 2, by inputting various data for each company (for example, in-house articles, manuals, customer / supplier information, and market data, etc.) prepared in advance for each user into the fourth base model, the various data is converted into base data. Then, a second database 184 is generated by a plurality of base data output from the fourth base model. Since the second database 184 is also used for the RAG technology (Retrieval-Augmented Generation), it can be said that it is a database for retrieval expansion. Note that the various data may be prepared for each craftsman who performs the appearance inspection of the object instead of for each company. In this case, since the craftsman's unique know-how is reflected in the various data, it becomes possible to accurately perform the appearance inspection of the object.

[0021] The appearance data of this embodiment includes at least one of the following: numerical data related to the appearance of an object, image data obtained by photographing the appearance of an object, video data obtained by photographing the appearance of an object, 3D data representing the appearance of an object, point cloud data representing the appearance of an object, infrared data representing the appearance of an object, X-ray data representing the appearance of an object, ultrasonic data representing the appearance of an object, spectral, polarization, or spectral data representing the appearance of an object, acoustic or vibration data representing the appearance of an object, temperature data representing the appearance of an object, friction data representing the appearance of an object, physical property data representing the appearance of an object, hardness data representing the appearance of an object, structural data representing the appearance of an object, graph structure data representing the appearance of an object, chemical composition data representing the appearance of an object, charge or electronic density of states data representing the appearance of an object, synthesis recipe data representing the appearance of an object, knowledge data representing the appearance of an object, language data representing the appearance of an object, and sound data related to the appearance of an object. Knowledge data of an object refers to, for example, textual data representing knowledge about an object. Therefore, in this embodiment, single-modal processing or multimodal processing is performed on appearance data containing multiple types of data, and the appearance data is converted into data that can be input into the base model.

[0022] Specifically, as shown in Figure 2, Server 14 inputs appearance data into the first base model A, thereby obtaining base data output from the first base model A. Furthermore, for appearance data that is difficult for the first base model A to process, Server 14 preprocesses the appearance data by inputting that type of appearance data into a predetermined machine learning model. Then, Server 14 inputs the preprocessed appearance data into the first base model B, thereby obtaining base data output from the first base model B. Hereinafter, at least one of the base data output from the first base model A and the base data output from the first base model B will also be referred to as appearance base data.

[0023] In this embodiment, as shown in Figure 2, a known RAG (Retrieval-Augmented Generation) technique is used to obtain base data related to the appearance base data from the search extension database. Then, as shown in Figure 2, the server 14 integrates the appearance base data and the base data related to the appearance base data by performing a data integration and standardization process. Specifically, the server prompts the appearance base data and the base data related to the appearance base data by inputting them into a pre-prepared prompt template. The server 14 then inputs the prompted data into the second base model to obtain evaluation data representing the evaluation of the object's appearance output from the second base model. The server 14 outputs the evaluation data for the object's appearance.

[0024] In this embodiment, appearance base data is obtained by integrating appearance data of an object, which is composed of multiple types of data. Furthermore, in this embodiment, instead of using a third base model that has been pre-trained based on normal appearance data and abnormal appearance data, a first database 164 is constructed using multiple base data output from the third base model. In addition, in this embodiment, a second database 184 is constructed by integrating various data specific to each company. In this embodiment, appearance evaluation or appearance inspection of an object is performed using this base data.

[0025] Within trained machine learning models, including language models, a function (or probability distribution) is represented that minimizes the error function between the inference data output by the machine learning model and the ground truth data. However, the range of information that the function represented within the machine learning model can output is infinite. For example, the number of language combinations in the language data output by a language model is nearly infinite. However, the languages ​​used in reality are finite. Therefore, if a language model, which is a machine learning model, is used as is, there is a possibility that the language model will output inappropriate language data.

[0026] Therefore, in this embodiment, instead of using the language model as is, a search extension database is pre-constructed using the underlying data output from the language and non-language models, and the appearance data of an object is evaluated using this search extension database. Since the underlying data stored in the search extension database is finite, the possibility of inappropriate underlying data being included is low, making it possible to appropriately evaluate the appearance data of an object. If inappropriate data is included in the underlying data output from the underlying model, such inappropriate underlying data can be excluded and the search extension database can be pre-constructed. Alternatively, even if inappropriate underlying data is included in the search extension database, such inappropriate underlying data is determined to have a low degree of relevance to the appearance underlying data by RAG technology, so it is unlikely to be used as input data when evaluating the appearance of an object. For this reason, according to this embodiment, the appearance evaluation of an object can be appropriately performed using the underlying data.

[0027] Furthermore, in this embodiment, the appearance evaluation of an object can be performed without operating the third and fourth base models each time appearance data is input. When processing is performed by a large-scale language model, a huge amount of power is consumed. In particular, since the third base model in this embodiment can be used by multiple users, if the third base model is operated each time appearance data is input by multiple users, a huge amount of power will be consumed. In contrast, as in this embodiment, instead of operating the third and fourth base models, a search extension database is built in advance using the language data output from the third and fourth base models, and the appearance data of an object is evaluated using the search extension database. This makes it possible to reduce the amount of power consumed even if the number of users using this system increases.

[0028] Furthermore, in this embodiment, as shown in Figure 2, the appearance of an object can be appropriately evaluated using the appearance inspection base model and the individual company base model. Specifically, by utilizing the base data output from the third base model, which is pre-trained using training data independent of individual companies / craftsmen, and the base data obtained by inputting various data representing information about individual companies / craftsmen into the fourth base model, it becomes possible to accurately evaluate the appearance of an object while also considering the unique information of individual companies / craftsmen.

[0029] Therefore, according to this embodiment, it becomes possible to appropriately evaluate the appearance data of the object to be evaluated. A more detailed explanation follows below.

[0030] (User terminal 12) The user terminal 12 is operated by a user who intends to evaluate the appearance of an object. The user is a person belonging to a specified company. The user terminal 12 is, for example, a personal computer or portable terminal owned by the user. Alternatively, the user terminal 12 is a terminal that is incorporated into a specified device and can be operated via the device's display unit (not shown). As will be described later, the user terminal 12 is implemented by a computer.

[0031] For example, the user transmits appearance data representing the appearance of the object to be evaluated to the server 14 via the user terminal 12. As described above, the appearance data in this embodiment includes at least one of the following: numerical data related to the appearance of an object, image data obtained by photographing the appearance of an object, video data obtained by photographing the appearance of an object, 3D data representing the appearance of an object, point cloud data representing the appearance of an object, infrared data representing the appearance of an object, X-ray data representing the appearance of an object, ultrasonic data representing the appearance of an object, spectral, polarization, or spectral data representing the appearance of an object, acoustic or vibration data representing the appearance of an object, temperature data representing the appearance of an object, friction data representing the appearance of an object, physical property data representing the appearance of an object, hardness data representing the appearance of an object, structural data representing the appearance of an object, graph structure data representing the appearance of an object, chemical composition data representing the appearance of an object, charge or electronic density of states data representing the appearance of an object, synthesis recipe data representing the appearance of an object, knowledge data representing the appearance of an object, language data representing the appearance of an object, and sound data related to the appearance of an object. For this reason, the appearance data in this embodiment is not limited to what is visible to a human eye.

[0032] (Server 14) Server 14 evaluates the appearance of an object based on appearance data transmitted from the user terminal 12. As shown in Figure 1, server 14 functionally consists of a model storage unit 140, a control unit 142, and an output unit 144. As will be described later, server 14 is implemented by a computer.

[0033] The model storage unit 140 stores the first base model A, the first base model B, a machine learning model, and the second base model. As described above, the first base model A, the first base model B, and the second base model are used to convert appearance data into appearance base data. Also, as described above, the machine learning model performs preprocessing on types of data that are difficult to convert into appearance base data using the first base model A and the first base model B.

[0034] The control unit 142 acquires appearance data representing the appearance of the object to be evaluated, transmitted from the user terminal 12. The control unit 142 inputs the appearance data into the first base models A and B, thereby acquiring appearance base data output from the first base models A and B. As mentioned above, if the appearance data contains types of data that are difficult for the first base models A and B to process directly, the control unit 142 performs preprocessing using a machine learning model stored in the model storage unit 140 and inputs the preprocessed appearance data into the first base model B. When inputting the appearance data into the first base models A and B, the control unit 142 inputs a prompt to the first base models A and B instructing them to convert the appearance data into base data. This converts the appearance data into base data. In this case, the appearance base data would be, for example, if the base model is a language base model, data such as "'abnormality': True, 'abnormality_reason': ''There is a minute scratch of about YY [mm] at XX location on the object...'.

[0035] The control unit 142 retrieves first related data associated with the appearance base data from multiple base data stored in the first database 164 of the first database server 16, based on the appearance base data. The control unit 142 also retrieves second related data associated with the appearance base data from multiple base data stored in the second database 184 of the second database server 18, based on the appearance base data. The control unit 142 retrieves base data related to the appearance base data from the first database 164 and the second database 184 using, for example, a known RAG (Retrieval-Augmented Generation) technique.

[0036] Next, the control unit 142 inputs appearance base data, first related data, second related data, and a prompt instructing the creation of an evaluation of the object's appearance to the second base model stored in the model storage unit 140, thereby obtaining evaluation data representing the evaluation of the object's appearance output from the second base model. In this case, if the base model is a language base model, the evaluation data would be, for example, "'evaluation': 'Passing requirement 015', 'evaluation_reason': 'Although there are minute scratches of approximately YY [mm] at XX locations on the object, they are within the range that can be considered normal...'".

[0037] Furthermore, the control unit 142 inputs the appearance base data, the first related data, the second related data, and a prompt instructing the creation of an improvement plan for the appearance evaluation to the second base model stored in the model storage unit 140, thereby obtaining plan data representing the improvement plan output from the second base model. In this case, the plan data would be, for example, if the base model is a language base model, data such as "In order to eliminate minute scratches of approximately YY [mm] at XX locations on the object..."

[0038] The output unit 144 outputs the evaluation data and planning data acquired by the control unit 142 to the user terminal 12. The user operating the user terminal 12 checks the evaluation data to confirm, for example, whether the appearance of the object being evaluated is normal or abnormal. The user also checks the planning data.

[0039] (First database server 16) As shown in Figure 1, the first database server 16 functionally comprises a base model storage unit 160, a first database control unit 162, and a first database 164. As will be described later, the first database server 16 is implemented by a computer.

[0040] The base model storage unit 160 stores the third base model described above.

[0041] The first database control unit 162 pre-trains the third base model using RLHF (Reinforcement Learning from Human Feedback). Specifically, the first database control unit 162 pre-trains the third base model based on normal appearance data representing the normal appearance of an object and abnormal appearance data representing the abnormal appearance of an object. By using RLHF, the third base model is learned in response to direct feedback from humans, and the base data regarding the appearance of objects output from the third base model becomes appropriate.

[0042] The first database control unit 162 inputs each of the multiple normal appearance data and each of the multiple abnormal appearance data into the third base model, thereby obtaining base data representing each of the multiple normal appearance data and base data representing each of the multiple abnormal appearance data. The first database control unit 162 then stores the base data representing each of the multiple normal appearance data and base data representing each of the multiple abnormal appearance data into the first database 164. The base data stored in the first database 164 is used by RAG technology as described above.

[0043] Similar to server 14, if the appearance data includes types of data that are difficult to process using the third-tier infrastructure model, preprocessing will be performed using other types of machine learning models.

[0044] The first database 164 stores base data representing each of multiple normal appearance data and base data representing each of multiple abnormal appearance data. Figure 3 shows an example of data stored in the first database 164. For example, in the case of language data, as shown in Figure 3, the language data generated from the appearance data and the data ID, which is a chunk ID consisting of tens to hundreds of token IDs, are stored in association with a vector (tensor) and metadata. The language data, vector (tensor), and metadata are examples of base data in this disclosure. The base data is data generated by the base model.

[0045] (Second database server 18) As shown in Figure 1, the second database server 18 functionally consists of a base model storage unit 180, a second database control unit 182, and a second database 184. The second database server 18 is pre-configured for each user. As will be described later, the second database server 18 is implemented by a computer.

[0046] The base model storage unit 180 stores the fourth base model described above.

[0047] The second database control unit 182 inputs various company-specific data (e.g., internal documents, manuals, customer / business partner information, and market data, etc.) that are prepared in advance for each user into the fourth base model, thereby converting the various data into base data. The second database control unit 182 then stores the multiple base data output from the fourth base model into the second database 184. The base data stored in the second database 184 is used by RAG technology, as described above.

[0048] Similar to server 14, if the appearance data includes types of data that are difficult to process with the fourth infrastructure model, preprocessing will be performed using other types of machine learning models.

[0049] The second database 184 stores foundational data representing various information about the user. Figure 4 shows an example of the data stored in the second database 184 when it is language data. For example, as shown in Figure 4, chunk IDs and language data generated from the manual are stored in association with vectors (tensors) and metadata. Language data, vectors (tensors), and metadata are examples of foundational data in this disclosure.

[0050] Furthermore, when generating evaluation data or planning data for appearance data input by a user (for example, a company that is the first user), the system may be configured to retrieve related second-party data associated with the appearance base data from a second database for a second user (a company different from the first user) (for example, "Connectable to other companies' base models" in Figure 2). In this case, when generating evaluation data or planning data for the first user, it becomes possible to use the base data of the second user, who is someone else. This makes it possible to generate highly accurate evaluation data or planning data by, for example, using the base data of another user when generating evaluation data or planning data for a user. Also, as mentioned above, if a base model is prepared for each craftsman who performs the appearance inspection of an object, rather than for each individual company, it is possible to provide a craftsman base model that reflects the craftsman's unique know-how to a third party.

[0051] The user terminal 12, server 14, first database server 16, and second database server 18 can be implemented, for example, by the computer 50 shown in Figure 5. The computer 50 includes a CPU 51, memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output interface (I / F) 54 to which external devices and output devices (for example, the display unit of the user terminal 12) are connected, and a read / write (R / W) unit 55 that controls the reading and writing of data to the recording medium. The computer 50 also includes a network I / F 56 that connects to a network such as the Internet. The CPU 51, memory 52, storage unit 53, input / output I / F 54, R / W unit 55, and network I / F 56 are connected to each other via a bus 57.

[0052] The storage unit 53 can be implemented using a Hard Disk Drive (HDD), Solid State Drive (SSD), flash memory, etc. The storage unit 53, as a storage medium, stores a program that allows the computer 50 to function. The CPU 51 reads the program from the storage unit 53, loads it into memory 52, and sequentially executes the processes contained in the program.

[0053] [Operation of Server 14 in the Embodiment] Next, the specific operation of the server 14 in this embodiment will be described. The server 14 performs the information processing shown in Figure 6.

[0054] First, in step S100, the control unit 142 acquires appearance data representing the appearance of the object to be evaluated, which is transmitted from the user terminal 12.

[0055] Next, in step S102, the control unit 142 inputs the appearance data acquired in step S100 to the first board models A and B, thereby acquiring the appearance board data output from the first board models A and B.

[0056] In step S104, the control unit 142 retrieves first related data associated with the appearance base data from a plurality of base data stored in the first database 164 of the first database server 16, based on the appearance base data acquired in step S102. Also in step S104, the control unit 142 retrieves second related data associated with the appearance base data from a plurality of base data stored in the second database 184 of the second database server 18, based on the appearance base data acquired in step S102.

[0057] In step S106, the control unit 142 inputs the appearance base data acquired in step S102, the first related data acquired in step S104, the second related data acquired in step S104, and a prompt instructing the creation of an evaluation of the object's appearance to the second base model, thereby acquiring evaluation data representing the evaluation of the object's appearance output from the second base model. Also in step S106, the control unit 142 inputs the appearance base data acquired in step S102, the first related data acquired in step S104, the second related data acquired in step S104, and a prompt instructing the creation of an improvement plan for the appearance evaluation to the second base model stored in the model storage unit 140, thereby acquiring plan data representing the improvement plan output from the second base model.

[0058] In step S108, the output unit 144 outputs the evaluation data and planning data acquired in step S106 to the user terminal 12.

[0059] As described above, the server in this embodiment acquires appearance base data output from the first base model by inputting appearance data representing the appearance of an object into the first base model. Based on the appearance base data, the server acquires related data associated with the appearance base data from multiple base data stored in the database. Then, the server acquires evaluation data representing the evaluation of the appearance of an object by inputting the appearance base data and related data into the second base model. This makes it possible to perform an evaluation of the appearance of an object using at least one of a single type of data and multiple types of data. Specifically, the evaluation of the appearance of an object can be performed by combining either linguistic and non-linguistic data, or both.

[0060] Traditionally, visual inspection of objects often involved evaluating abnormalities in the appearance of the object from images. However, automating visual inspection of objects required homogenizing the shooting environment and introducing special shooting and analysis techniques. Furthermore, it was necessary to develop a separate inspection system for each type of abnormality being targeted.

[0061] In contrast, this embodiment can also convert appearance data related to the normality and abnormality of an object's appearance into natural language using a language model. Furthermore, this embodiment includes an appearance inspection base model (corresponding to the first database server 16 described above) that can generate all kinds of abnormalities and quality in natural language by performing reinforcement learning (RLHF) with human feedback on the text generated by the language model. When evaluating unknown appearance data, the appearance data is converted into appearance base data by the base model, and search extension is performed by obtaining data related to the appearance base data from the appearance inspection base model. This makes it possible to evaluate all appearance inspections in natural language. In addition, company-specific data and manuals are made into company / craftsman base models for search extension, and by including these company / craftsman base models in the search extension, it is possible to generate appearance evaluations, quality evaluations, quality improvement plans, and procurement / sales plans. It is also possible to use some or all of the components of the information processing system according to the above embodiment as an AI agent.

[0062] Furthermore, this embodiment can be used for visual inspection and quality evaluation of product components, parts, or the entire product; for inspection and evaluation of the exterior walls and hazards of structures; or for assessing the freshness and quality of fruits, vegetables, and fresh foods.

[0063] Furthermore, this embodiment enables the evaluation of any appearance data, not limited to image data, in natural language using a language model. It achieves versatility and diversity in anomaly detection, which was not possible with conventional appearance inspections, and makes it possible to reflect the unique know-how and data held by each company / craftsman in appearance inspections. In addition, it is possible to use some or all of the components of the information processing system according to the above embodiment as an AI agent to create improvement plans, etc.

[0064] Furthermore, this embodiment enables visual inspection without requiring special image data or image analysis techniques. Therefore, it is possible to train language and non-language models to evaluate any abnormality or quality, rather than being limited to specific abnormality checks as in conventional visual inspections.

[0065] Furthermore, the technology disclosed herein is not limited to the embodiments described above, and various modifications and applications are possible without departing from the spirit of this invention.

[0066] For example, although the present specification describes an embodiment in which the program is pre-installed, it is also possible to provide the program by storing it on a computer-readable recording medium.

[0067] Furthermore, each process performed in the above embodiment may be executed by any of the user terminal 12, server 14, first database server 16, and second database server 18.

[0068] Furthermore, although the above embodiment describes an example where a base model or machine learning model is stored in the memory of each device, it is not limited to this. Each device may utilize language models, non-language models, or machine learning models provided by an external device. For example, each device may utilize various models provided by an external device via an API (Application Programming Interface).

[0069] Furthermore, although the above embodiment was described on the premise that the first base model A, the first base model B, the second base model, the third base model, and the fourth base model are all different base models, the embodiment is not limited to this. For example, at least some of these multiple base models may be the same base model. For example, the first base model A and the second base model may be the same base model. Or, the first base model A, the second base model, and the fourth base model may be the same base model. Also, although the above embodiment was described using the first base model A, the first base model B, and a machine learning model as an example when generating appearance base data, appearance base data may also be generated using a single trained model that has the functions of the first base model A, the first base model B, and a machine learning model.

[0070] In addition, the processing that the CPU reads and executes in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processing, such as ASICs (Application Specific Integrated Circuits). Alternatively, GPGPUs (General-purpose graphics processing units) or TPUs (Tensor Processing Units) may be used as processors. Furthermore, each processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0071] Furthermore, while the above embodiments describe a configuration in which the program is pre-stored (installed) on storage, the invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from an external device via a network.

[0072] Furthermore, each process in this embodiment may be configured by a computer or server equipped with a general-purpose processing unit and storage device, and each process may be executed by a program. This program is stored in the storage device and can be recorded on a recording medium such as a magnetic disk, optical disk, or semiconductor memory, or provided over a network. Of course, none of the other components have to be implemented by a single computer or server; they may be implemented in a distributed manner across multiple computers connected by a network.

[0073] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0074] (Note) The following is an addendum regarding the nature of this disclosure.

[0075] (Note 1) An information processing system equipped with a processor, The aforementioned processor, Appearance data representing the appearance of an object is input to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the aforementioned appearance base data, related data associated with the appearance base data is obtained from multiple base data stored in the database. By inputting the aforementioned appearance base data and the aforementioned related data into a second base model, which is a trained model, evaluation data representing the evaluation of the appearance of the object is obtained. Outputting the aforementioned evaluation data, Information processing system. (Note 2) The appearance data includes at least a portion of the following: numerical data related to the appearance of the object, image data obtained by photographing the appearance of the object, video data obtained by photographing the appearance of the object, three-dimensional data representing the appearance of the object, point cloud data representing the appearance of the object, infrared data representing the appearance of the object, X-ray data representing the appearance of the object, ultrasonic data representing the appearance of the object, spectral, polarization, or spectral data representing the appearance of the object, acoustic or vibration data representing the appearance of the object, temperature data representing the appearance of the object, friction data representing the appearance of the object, physical property data representing the appearance of the object, hardness data representing the appearance of the object, structural data representing the appearance of the object, graph structure data representing the appearance of the object, chemical composition data representing the appearance of the object, charge or electronic density of states representing the appearance of the object, synthesis recipe data representing the appearance of the object, knowledge data representing the appearance of the object, language data representing the appearance of the object, and sound data related to the appearance of the object. The information processing system described in Appendix 1. (Note 3) The multiple data stored in the aforementioned database are data output from a third-tier base model, which is a pre-trained model that has been trained based on normal appearance data representing the normal appearance of an object and abnormal appearance data representing the abnormal appearance of an object. The information processing system described in Appendix 1 or Appendix 2. (Note 4) The aforementioned third base model is a base model that has been pre-trained using RLHF (Reinforcement Learning from Human Feedback). The information processing system described in Appendix 3. (Note 5) The aforementioned database is the first database, The aforementioned related data is the first related data, The aforementioned processor, From multiple base data stored in a second database prepared in advance for each user, second related data related to the aforementioned appearance base data is obtained. By inputting the aforementioned appearance base data, the first related data, and the second related data into the second base model, evaluation data representing the evaluation of the appearance of the object is obtained. The information processing system described in Appendix 3. (Note 6) The aforementioned processor, When generating the evaluation data of the appearance data input by the first user, the second related data related to the appearance base data is obtained from the second database for a second user different from the first user. The information processing system described in Appendix 5. (Note 7) The aforementioned processor is When acquiring the first related data or the second related data, RAG (Retrieval-Augmented Generation) is used to acquire the first related data or the second related data. The information processing system described in Appendix 5. (Note 8) The aforementioned processor, By inputting the aforementioned appearance base data, the aforementioned related data, and a prompt instructing the creation of an appearance evaluation into the second base model, the evaluation data is obtained. By inputting the aforementioned appearance base data, the aforementioned related data, and a prompt instructing the creation of an improvement plan for the evaluation of the appearance into the second base model, plan data representing the improvement plan is obtained. Output the evaluation data and the planning data. An information processing system described in any one of the items in Appendix 1 to Appendix 7. (Note 9) Appearance data representing the appearance of an object is input to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the aforementioned appearance base data, related data associated with the appearance base data is obtained from multiple base data stored in the database. By inputting the aforementioned appearance base data and the aforementioned related data into a second base model, which is a trained model, evaluation data representing the evaluation of the appearance of the object is obtained. Outputting the aforementioned evaluation data, An information processing method in which a computer performs the processing. (Note 10) Appearance data representing the appearance of an object is input to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the aforementioned appearance base data, related data associated with the appearance base data is obtained from multiple base data stored in the database. By inputting the aforementioned appearance base data and the aforementioned related data into a second base model, which is a trained model, evaluation data representing the evaluation of the appearance of the object is obtained. Outputting the aforementioned evaluation data, An information processing program that causes a computer to perform a task. [Explanation of Symbols]

[0076] 10. Information Processing Systems 12 User terminals 14 Servers 16. First Database Server 18. Second Database Server 20 Networks 140 Model Memory Unit 142 Control Unit Output section of 144 164 First Database 184 Second Database

Claims

1. An information processing system equipped with a processor, The aforementioned processor, Appearance data representing the appearance of an object is input to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the aforementioned appearance base data, related data associated with the appearance base data is obtained from multiple base data stored in the database. By inputting the aforementioned appearance base data and the aforementioned related data into a second base model, which is a trained model, evaluation data representing the evaluation of the appearance of the object is obtained. Outputting the aforementioned evaluation data, Information processing system.

2. The appearance data includes at least a portion of the following: numerical data related to the appearance of the object, image data obtained by photographing the appearance of the object, video data obtained by photographing the appearance of the object, three-dimensional data representing the appearance of the object, point cloud data representing the appearance of the object, infrared data representing the appearance of the object, X-ray data representing the appearance of the object, ultrasonic data representing the appearance of the object, spectral, polarization, or spectral data representing the appearance of the object, acoustic or vibration data representing the appearance of the object, temperature data representing the appearance of the object, friction data representing the appearance of the object, physical property data representing the appearance of the object, hardness data representing the appearance of the object, structural data representing the appearance of the object, graph structure data representing the appearance of the object, chemical composition data representing the appearance of the object, charge or electronic density of states representing the appearance of the object, synthesis recipe data representing the appearance of the object, knowledge data representing the appearance of the object, language data representing the appearance of the object, and sound data related to the appearance of the object. The information processing system according to claim 1.

3. The multiple data stored in the aforementioned database are data output from a third base model, which is a pre-trained model that has been trained based on normal appearance data representing the normal appearance of an object and abnormal appearance data representing the abnormal appearance of an object. The information processing system according to claim 1 or claim 2.

4. The third base model is a language model pre-trained using RLHF (Reinforcement Learning from Human Feedback). The information processing system according to claim 3.

5. The aforementioned database is the first database, The aforementioned related data is the first related data, The aforementioned processor, From multiple base data stored in a second database prepared in advance for each user, second related data related to the aforementioned appearance base data is obtained. By inputting the aforementioned appearance base data, the first related data, and the second related data into the second base model, evaluation data representing the evaluation of the appearance of the object is obtained. The information processing system according to claim 3.

6. The aforementioned processor, When generating the evaluation data of the appearance data input by the first user, the second related data related to the appearance base data is obtained from the second database for a second user different from the first user. The information processing system according to claim 5.

7. The aforementioned processor, When acquiring the first related data or the second related data, RAG (Retrieval-Augmented Generation) is used to acquire the first related data or the second related data. The information processing system according to claim 5.

8. The aforementioned processor, By inputting the aforementioned appearance base data, the aforementioned related data, and a prompt instructing the creation of an appearance evaluation into the second base model, the evaluation data is obtained. By inputting the aforementioned appearance base data, the aforementioned related data, and a prompt instructing the creation of an improvement plan for the evaluation of the appearance into the second base model, plan data representing the improvement plan is obtained. Output the evaluation data and the planning data. The information processing system according to claim 1 or claim 2.

9. Appearance data representing the appearance of an object is input to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the aforementioned appearance base data, related data associated with the appearance base data is obtained from multiple base data stored in the database. By inputting the aforementioned appearance base data and the aforementioned related data into a second base model, which is a trained model, evaluation data representing the evaluation of the appearance of the object is obtained. Outputting the aforementioned evaluation data, An information processing method in which a computer performs the processing.

10. Appearance data representing the appearance of an object is input to a first base model, which is a trained model capable of processing at least one of a single type of data and multiple types of data, thereby obtaining appearance base data output from the first base model. Based on the aforementioned appearance base data, related data associated with the appearance base data is obtained from multiple base data stored in the database. By inputting the aforementioned appearance base data and the aforementioned related data into a second base model, which is a trained model, evaluation data representing the evaluation of the appearance of the object is obtained. Outputting the aforementioned evaluation data, An information processing program that causes a computer to perform a task.

Images