Information processing system and information processing method

Abstract

Provided is an information processing system capable of adding an appropriate hatching pattern to a blank drawing or image. This information processing system has first and second components and a large language model. The first component has a function for receiving first to third information and then transferring the information to the second component. The second component has a function for generating a first instruction sentence after receiving the first to third information, and entering the first instruction sentence into the large language model. The large language model has a function for determining whether there is a problem with the first to third information on the basis of the first instruction sentence. The second component has a function for generating a second instruction sentence on the basis of the first to third information and then entering the second instruction sentence into the large language model. The large language model has a function for generating fourth information on the basis of the second instruction sentence. The first component has a function for generating a first image after receiving the fourth information.

Description

Information processing system and information processing method

[0001] One aspect of the present invention relates to an information processing system and an information processing method.

[0002] Note that one embodiment of the present invention is not limited to the above technical field. The technical field of one embodiment of the invention disclosed in this specification and the like relates to an object, a method, or a manufacturing method. Alternatively, one embodiment of the present invention relates to a process, a machine, manufacture, or a composition of matter. Therefore, more specifically, examples of the technical field of one embodiment of the present invention disclosed in this specification include a semiconductor device, a display device, a light-emitting device, a power storage device, a memory device, a driving method thereof, or a manufacturing method thereof.

[0003] In recent years, the development of language models using neural networks has been actively pursued, with large-scale language models (LLMs) attracting particular attention. A large-scale language model is a natural language processing model trained using a large amount of data. A large-scale language model can realize, for example, a dialogue model that responds to user instructions. Non-Patent Document 1 discloses GPT-4 (Generative Pre-trained Transformer 4) (registered trademark) as a large-scale language model, and ChatGPT as a dialogue model.

[0004] Summary of ChatGPT / GPT-4 Research and Perspective Towards the Future of Large Language Models, Yiheng Liu et al. (Submitted on 4 Apr 2023, [online], Internet <URL: https: / / arxiv.org / abs / 2304.01852>

[0005] Large-scale language models can be applied to a variety of applications. For example, if a user works in intellectual property-related fields, the user can utilize large-scale language models to create new specifications for application processing. For example, by inputting instructions into the large-scale language model that include the purpose of the invention and the structure of the specification that the user wants to create, the large-scale language model can automatically create a specification that meets the user's needs.

[0006] In addition, when processing an application, users must prepare drawings (patent drawings) in addition to the specification. These drawings must be prepared in a format compliant with the patent office of the country of application. For example, color is not permitted in patent drawings. Instead, plain drawings (i.e., drawings without coloring or patterns, showing only the overall outline and boundaries of each area) must be devised to distinguish each area within the drawing by adding a black-and-white hatching pattern. The type of hatching pattern chosen varies from user to user, resulting in differences in the readability of the resulting patent drawings. In order to prepare a complete set of excellent application documents, it is important to not only prepare high-quality specifications, but also to create easy-to-read patent drawings. It is also desirable for anyone to be able to easily create easy-to-read patent drawings, regardless of their user experience or skill level.

[0007] In view of the above, an object of one embodiment of the present invention is to provide an information processing system that can apply an appropriate hatching pattern to a plain drawing or image. Another object of one embodiment of the present invention is to provide a novel information processing system that is highly convenient, useful, or reliable. Another object of one embodiment of the present invention is to provide an information processing method that can apply an appropriate hatching pattern to a plain drawing or image. Another object of one embodiment of the present invention is to provide a novel information processing method that is highly convenient, useful, or reliable.

[0008] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily need to solve all of these problems. Note that problems other than these will become apparent from the description in the specification, drawings, claims, etc., and it is possible to extract other problems from the description in the specification, drawings, claims, etc.

[0009] One aspect of the present invention is a system having a first component, a second component, and a third component, the third component having a function of performing processing using a large-scale language model, the first component having a function of receiving first information to third information and a function of transferring the first information to third information to the second component, the first information being a list of hatching patterns, the second information being a first image without hatching patterns, and the third information being a description of each area of ​​the first image, the second component having a function of receiving the first information to third information and a function of inputting the first information to third information and a first instruction statement to the third component, and the third component performing processing based on the first instruction statement. and a function to generate fourth information, the fourth information including a solution when there is a problem with the first information to the third information, the first component having a function to accept the fourth information, the second component having a function to input the first information to the third information and the second instruction statement to the third component, the third component having a function to generate fifth information based on the first information to the third information and the second instruction statement, the fifth information being a list of ideas for hatching patterns to be applied to each region of the first image, and the first component having a function to accept the fifth information and a function to generate a second image based on the fifth information.

[0010] Furthermore, in the above, it is preferable that the first instruction sentence has a first sentence and a second sentence, the first sentence being a sentence asking whether there is any problem with the first information to the third information, and the second sentence being a sentence asking for a solution if there is a problem with one or more of the first information to the third information.

[0011] Furthermore, in the above, it is preferable that the second instruction sentence has a third sentence, which is a sentence instructing to generate a list of hatching patterns corresponding to each area of ​​the first image, and that the second image is an image in which a hatching pattern is applied to each of the areas.

[0012] In the above, it is preferable that the second component has a function of generating a first instruction sentence using the first information to the third information, and a function of generating a second instruction sentence based on the first information to the third information.

[0013] In the above, the third component preferably has a function of performing processing using a graph neural network.

[0014] In the above, it is preferable that the first image and the second image are vector images.

[0015] Furthermore, one aspect of the present invention has first to sixth steps, in which in the first step, the first component receives the first to third information and then passes the first to third information to the second component, in the second step, the second component receives the first to third information and passes the first to third information and the first instruction statement to the third component, and in the third step, the third component determines whether there is a problem with the first to third information based on the first instruction statement, and determines that there is a problem with one or more of the first to third information. If it is determined that there is no problem with the first information to the third information, in a fourth step, the second component passes the first information to the third information and the second instruction sentence to the third component; in a fifth step, the third component generates fifth information based on the second instruction sentence and passes the fifth information to the first component; and in a sixth step, the first component receives the fifth information and generates a first image based on the fifth information.

[0016] Furthermore, in the above, it is preferable that the first information is a list of hatching patterns, the second information is a second image without hatching patterns, the third information is an explanation of each area of ​​the second image, the first instruction sentence has a first sentence and a second sentence, the first sentence is a sentence asking whether there is a problem with the first information to the third information, the second sentence is a sentence asking for a solution if there is a problem with one or more of the first information to the third information, and the fourth information is a sentence including the solution.

[0017] Furthermore, in the above, it is preferable that the second instruction sentence has a third sentence, the third sentence being a sentence instructing to generate a list of hatching patterns corresponding to each area of ​​the second image, the fifth information being a list of hatching patterns, and the first image being an image in which a hatching pattern is applied to each of the areas.

[0018] In the above, it is preferable that the first image and the second image are vector images.

[0019] According to one embodiment of the present invention, an information processing system capable of applying an appropriate hatching pattern to a plain drawing or image can be provided. Alternatively, according to one embodiment of the present invention, a novel information processing system with excellent convenience, usefulness, or reliability can be provided. Alternatively, according to one embodiment of the present invention, an information processing method with excellent convenience, usefulness, or reliability can be provided. Alternatively, according to one embodiment of the present invention, a novel information processing method with excellent convenience, usefulness, or reliability can be provided.

[0020] Note that the description of these effects does not preclude the existence of other effects. Note that one embodiment of the present invention does not necessarily have all of these effects. Note that effects other than these will become apparent from the description in the specification, drawings, claims, etc., and it is possible to extract other effects from the description in the specification, drawings, claims, etc.

[0021] FIG. 1 is a diagram illustrating the configuration of an information processing system. FIG. 2 is a block diagram illustrating the configuration of the information processing system. FIG. 3 is a block diagram illustrating the configuration of an information processing device used in the information processing system. FIG. 4 is a flowchart illustrating an information processing method. FIG. 5 is a diagram illustrating information input to the information processing system. FIGS. 6A to 6C are diagrams illustrating information input to the information processing system. FIG. 7 is a diagram illustrating information input to the information processing system. FIG. 8 is a diagram illustrating information input to the information processing system. FIG. 9 is a diagram illustrating an instruction sentence input to a large-scale language model. FIG. 10 is a diagram illustrating information generated by a large-scale language model. FIG. 11 is a diagram illustrating an instruction sentence input to a large-scale language model. FIG. 12 is a diagram illustrating information generated by a large-scale language model. FIGS. 13A to 13C are diagrams illustrating images generated by the information processing system.

[0022] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be readily understood by those skilled in the art that various changes in form and details can be made without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below. In the configuration of the invention described below, the same parts or parts having similar functions will be denoted by the same reference numerals in different drawings, and repeated description thereof will be omitted.

[0023] In this specification, the ordinal numbers such as "first" and "second" are used for convenience and do not limit the number of components or the order of the components (for example, the order of processes or the order of stacking). Furthermore, the ordinal numbers assigned to components in one part of this specification may not match the ordinal numbers assigned to the same components in other parts of this specification or in the claims.

[0024] (Embodiment 1) An information processing system according to one embodiment of the present invention has a function of automatically generating fourth information (a list of hatching patterns that are appropriate to be applied to each of the regions of the first image) using a large-scale language model based on first information (a list of hatching patterns), second information (a plain drawing or image (first image) without any hatching patterns) input by a user, and third information (a sentence explaining each region of the first image).

[0025] In addition, an information processing system according to one embodiment of the present invention has the function of automatically generating a second image (an image in which an appropriate hatching pattern is applied to each area of ​​the first image) based on fourth information generated by the large-scale language model, and presenting the image to a user.

[0026] Because the information processing system of one embodiment of the present invention has the above-mentioned functions, anyone can easily create attractive images (for example, patent drawings in which each area is clearly identified with an appropriate hatching pattern) simply by providing the necessary information to the information processing system, regardless of the user's sense, level of expertise, etc.

[0027] An information processing system according to one embodiment of the present invention will be described below with reference to FIGS. 1 to 3 and FIGS. 5 to 13. FIG.

[0028] 1 illustrates an example of the configuration of components of an information processing system according to one embodiment of the present invention and a network connecting these components. It also illustrates an example of the flow of data exchanged between the components of the information processing system.

[0029] FIG. 2 shows a more detailed example of the flow of data exchanged between the components of the information processing system shown in FIG.

[0030] In Figures 1 and 2, the components are classified by function and shown as independent components or blocks, but in reality it is difficult to completely separate the components by function, and one component may be involved in multiple functions.

[0031] 1, an information processing system according to one aspect of the present invention includes a component 10, a component 20, and a component 30. In the information processing system, predetermined data is exchanged between the components via a network 50.

[0032] Hereinafter, functions of an information processing system according to one embodiment of the present invention will be described for each component constituting the information processing system with reference to Figures 1, 2, and 5 to 13. Note that some of the descriptions regarding data exchange between the components may be repeated.

[0033] <<Configuration Example of Component 10>> As the component 10, for example, a desktop computer can be used.

[0034] The component 10 can accept data input by a user, and can provide data output by the component 20 to the user.

[0035] For example, dedicated application software or a web browser may be operated. A user can access the information processing system via either of these, thereby enjoying services using the information processing system according to one aspect of the present invention.

[0036] The component 10 has a function of accepting information input by the user (information IN1, information IN2, and information IN3 shown in FIG. 1) and transferring it to the component 20 in a process T1 indicated by an arrow in FIG.

[0037] Of the information IN1 to information IN3, information IN1 and information IN2 are image data, and information IN3 is text data.

[0038] In the following, we will explain a case in which an information processing system of one embodiment of the present invention is used for the purpose of adding appropriate hatching patterns to plain patent drawings, but this is not limited to this, and the information processing system can be used in a variety of cases.

[0039] Information IN1 is image data displaying a list of hatching patterns. Fig. 5 shows, as an example of information IN1, image data displaying a list of 15 types of hatching patterns (hatching patterns A to O).

[0040] Information IN2 is image data to which a hatching pattern selected from information IN1 ( FIG. 5 ) is to be applied. As an example of information IN2, FIGS. 6A to 6C show plain patent drawings with no hatching pattern, but with only leading lines and symbols indicating each region. FIGS. 6A to 6C are schematic diagrams showing the configuration of a vertical transistor (transistor 200) in which drain current flows in a direction (Z direction, vertical direction) approximately perpendicular to the substrate surface (XY plane). FIG. 6A is a cross-sectional view of transistor 200 in the XZ plane, FIG. 6B is a cross-sectional view of transistor 200 in the YZ plane, and FIG. 6C is a perspective view of transistor 200.

[0041] Note that, as information IN2, vector-format image data (vector image) or raster-format image (raster image) can be used. It is preferable to use a vector image as information IN2. While a raster image is expressed as a collection of multiple "pixels," a vector image is expressed as multiple "points" and "lines" connecting them. Therefore, by using a vector image as information IN2, it is possible to perform processing using artificial intelligence (AI) that incorporates, for example, a graph neural network (GNN).

[0042] Information IN3 is text data describing each region of the image ( FIGS. 6A to 6C ) input as information IN2. For example, if information IN2 is a patent drawing, information IN3 is text data describing the reference numerals assigned to each region or component in the drawing. The text data may be words, such as the names of each region or component, or may be sentences. FIG. 7 shows, as an example of information IN3, a list of component names (insulating layer, conductive layer, oxide semiconductor layer, etc.) describing the reference numerals assigned to each region of the transistor 200 shown in FIGS. 6A to 6C . FIG. 8 shows, as an example of information IN3, sections of the specification (such as paragraphs AAAA, BBBB, CCCC, and DDDD) that describe the components of the transistor 200 shown in FIGS. 6A to 6C . When information IN3 is sentences, only sections related to the components of the transistor 200 may be input, or the entire specification may be input.

[0043] Furthermore, for example, the user can perform processing equivalent to inputting information IN3 by inputting the name of the relevant specification or the storage location or path of the specification (text file) into component 10. Furthermore, for example, the user can perform processing equivalent to inputting information IN3 by selecting the relevant specification file from the display screen of component 10.

[0044] Furthermore, the component 10 has a function of receiving information generated by the component 30 (information SG shown in FIG. 1) and providing it to the user in process T4 indicated by an arrow in FIG. 2. Details of the information SG will be described below in "Configuration Example of Component 30."

[0045] 2, the component 10 has a function of receiving information (information OUT shown in FIG. 1) generated by the component 30. Details of the information OUT will be described below in "Configuration Example of Component 20."

[0046] The component 10 also has a function of generating an image (image IMG) based on the information OUT and providing the image IMG to the user.

[0047] Image IMG is image data generated based on information IN1 to information IN3. As an example of image IMG, FIGS. 13A to 13C show patent drawings in which hatching patterns are applied to each region. FIGS. 13A to 13C are schematic diagrams of transistor 200 corresponding to FIGS. 6A to 6C, respectively. In image IMG, regions (components of transistor 200) indicated by leading lines and symbols applied to the images ( FIGS. 6A to 6C ) input as information IN2 are applied with one of the hatching patterns selected from information IN1 ( FIG. 5 ).

[0048] For example, by specifying in advance the image editing application (such as Vectorworks (registered trademark)) that the user will use to create the patent drawing, an image in a format that can be edited by that image editing application (a vector image in the case of using Vectorworks) is generated as image IMG. The image IMG shown in Figures 13A to 13C is an image in which appropriate hatching patterns have been applied to each area of ​​the plain patent drawing shown as information IN2 in Figures 6A to 6C in accordance with information OUT.

[0049] <<Configuration Example of Component 20>> The component 20 may be, for example, a workstation, a server computer, or a supercomputer.

[0050] Furthermore, the component 20 preferably has a function as a parallel computer. By using the component 20 as a parallel computer, it is possible to perform large-scale calculations required for AI learning and inference, for example.

[0051] Furthermore, the component 20 can perform processing using a natural language processing model that uses AI.

[0052] For example, processing can be performed using natural language models such as BERT (Bidirectional Encoder Representations from Transformers), T5 (Text-to-Text Transfer Transformer), GPT-3, GPT-3.5, GPT-4, LaMDA (Language Model for Dialogue Applications), PaLM (Pathways Language Model), and Llama2.

[0053] The component 20 has a function of receiving information IN1 to IN3 passed from the component 10 in a process T1 indicated by an arrow in FIG.

[0054] As mentioned above, it is preferable that the image data received by the component 20 as the information IN2 be in vector format. Therefore, when a user inputs a raster image to the component 10, the component 20 has a function of converting the raster image into a vector image as the information IN2.

[0055] Component 20 also has the function of passing a directive PT1 to component 30. Directive PT1 is a statement to be passed to component 30, including instructions using information IN1 to IN3 passed from component 10.

[0056] The directive PT1 describes in natural language the specific processing to be executed by the component 30. Figure 9 shows an example of the directive PT1, which states, "From the hatching patterns shown in information IN1, I would like to select hatching patterns that fit each area shown in information IN3 in the vector image shown in information IN2. Please tell me whether information IN1, IN2, and IN3 are sufficient to execute the above processing." This is followed by a second directive, "If they are not sufficient, please tell me how to solve the problem."

[0057] 9 shows an example in which the directive PT1 is composed of the two directives described above, but this is not a limitation. The directive PT1 only needs to include at least the first directive. The second directive is a statement written as an additional condition associated with the first directive. The additional conditions can be two or more, and the number of statements written in the directive PT1 can be increased depending on the number of conditions desired by the user.

[0058] The component 20 may be configured to have a function to automatically generate the instruction statement PT1 in accordance with the contents of the information IN1 to IN3. Alternatively, a fixed phrase such as that shown in Fig. 9 may be prepared in advance, and the fixed phrase may be used as the instruction statement PT1 by inserting the information IN1 to IN3 into the fixed phrase.

[0059] 2, component 20 has a function of transferring information IN1 to information IN3 and directive PT1 to component 30 and causing component 30 to execute the instructions described in directive PT1. Specifically, component 20 has a function of causing component 30 to determine whether information IN1 to information IN3 are sufficient as user input in accordance with the first instruction in directive PT1 shown in Fig. 9, which reads, "From the hatching patterns shown in information IN1, I would like to select a hatching pattern that matches each area shown in information IN3 in the vector image shown in information IN2. Please tell me whether information IN1, IN2, and IN3 are sufficient to execute the above process."

[0060] Furthermore, if component 30 determines that the first instruction content is "sufficient," component 20 has the function of accepting information IN1 to IN3 passed from component 30 in process T3 indicated by the arrow in Figure 2.

[0061] As described above, component 20 has already received information IN1 to IN3 from component 10 in process T1. Therefore, component 20 may have a function to save information IN1 to IN3 at this time. In this case, component 20 does not need to receive information IN1 to IN3 from component 30 again in process T3.

[0062] Component 20 also has the function of passing a directive PT2 to component 30. Directive PT2 is a statement to be passed to component 30, including instruction content using information IN1 to IN3 passed from component 30 or component 10.

[0063] The instruction PT2 describes in natural language the specific processing content to be executed by the component 30. In Fig. 11, an example of the instruction PT2 is "Tell me which hatching patterns from the hatching patterns shown in information IN1 fit each area shown in information IN3 in the vector image shown in information IN2," along with two points to note: "adjacent areas should be easily distinguishable," and "compared to information IN3, they should suit human perception."

[0064] In addition to the information IN1 to IN3, the instruction statement PT2 may also include information describing whether or not the areas in the image indicated by the information IN2 are adjacent to each other.

[0065] The component 20 may be configured to have a function to automatically generate the instruction statement PT2 in accordance with the contents of the information IN1 to IN3. Alternatively, the component 20 may be configured to have a function to prepare a fixed phrase as shown in Fig. 11 in advance, and use the fixed phrase by inserting the information IN1 to IN3 into the fixed phrase as the instruction statement PT2.

[0066] 2, component 20 has a function of transferring information IN1 to information IN3 and directive PT2 to component 30 and causing component 30 to execute the instructions described in directive PT2. Specifically, component 20 has a function of causing component 30 to select hatching patterns from information IN1 that fit each region indicated by information IN3 and generate a list of the selection results (a list of hatching pattern ideas, information OUT shown in FIG. 1). FIG. 12 shows, as an example of information OUT, a list that associates and displays symbols indicating each region (component) of transistor 200 indicated in information IN2 (FIGS. 6A to 6C) with the corresponding hatching patterns selected from information IN1 (FIG. 5).

[0067] The information OUT shown in Figure 12 presents that, of the regions of transistor 200, hatching pattern J is assigned to the region corresponding to insulating layer 210, hatching pattern A is assigned to the region corresponding to conductive layer 220_1, hatching pattern B is assigned to the region corresponding to conductive layer 220_2, hatching pattern C is assigned to the region corresponding to oxide semiconductor layer 230, hatching pattern D is assigned to the region corresponding to conductive layer 240, hatching pattern E is assigned to the region corresponding to insulating layer 250, hatching pattern F is assigned to the region corresponding to conductive layer 260, and hatching pattern J is assigned to the region corresponding to insulating layer 280.

[0068] 2 and the like, the component 20 can also have a function of receiving the information OUT generated by the component 30. Furthermore, the component 20 can also have a function of generating a script that is suitable for an image editing application such as the above-mentioned Vectorworks based on the information OUT and transferring the script to the component 10.

[0069] <<Configuration Example of Component 30>> A large computer such as a server computer or a supercomputer can be used as the component 30. Note that the component 30 is larger in scale and has higher computing power than the component 20.

[0070] Furthermore, the component 30 preferably has a function as a parallel computer, which enables the component 30 to perform large-scale calculations required for AI learning and inference, for example.

[0071] Furthermore, the component 30 can perform processing using a natural language processing model that uses AI, particularly a general-purpose language processing model that can perform various natural language processing tasks.

[0072] For example, processing can be performed using natural language models such as BERT, T5, GPT-3, GPT-3.5, GPT-4, LaMDA, PaLM, Llama2, ALBERT, and XLNet. In particular, processing using GPT-4 is preferable. This allows for more natural sentence generation or dialogue.

[0073] The component 30 can also have a function of performing processing using AI incorporating GNN. By using AI incorporating GNN, it is possible to perform learning, inference, etc. on image data such as patent drawings. Representative models using GNN include, for example, GCN and GraphSAGE3.

[0074] Note that a person who provides a service using an information processing system according to one embodiment of the present invention does not necessarily have to own the component 30. For example, a service provider can use part of a service provided by another business or the like as the component 30.

[0075] The component 30 has a function of receiving the information IN1 to IN3 and the directive PT1 passed from the component 20 in the process T2 indicated by the arrow in FIG.

[0076] Component 30 also has a function of executing the instruction contents described in instruction statement PT1. Specifically, component 30 has a function of determining whether information IN1 to IN3 are sufficient as user input contents in accordance with the first instruction content of instruction statement PT1 shown in Fig. 9, which reads, "I want to select hatching patterns from the hatching patterns shown in information IN1 that fit each area shown in information IN3 in the vector image shown in information IN2. Please tell me whether information IN1, IN2, and IN3 are sufficient to execute the above process."

[0077] If the first instruction content is judged to be "sufficient," the component 30 has a function of transferring the information IN1 to IN3 to the component 20 in process T3 indicated by an arrow in FIG.

[0078] As described above, if component 20 has the function of saving information IN1 to IN3 received from component 10 in process T1, component 30 does not need to transfer information IN1 to IN3 to component 20 again in process T3.

[0079] Furthermore, if the component 30 determines that the first instruction is "insufficient," the component 30 has the function of generating information SG including a solution in accordance with the second instruction written in the instruction sentence PT1, which is "If it is insufficient, please tell me the solution."

[0080] The information SG is written in natural language. In Fig. 10, it is assumed that the number of hatching patterns indicated by the information IN1 is smaller than the number of areas indicated by the information IN3, and is judged to be "insufficient." As an example of the information SG, the description content is shown as "Increase the number of hatching patterns in the information IN1 to be greater than the number of areas in the information IN3."

[0081] In addition, if the second instruction content is not described in the instruction sentence PT1, if it is determined that information IN1 to information IN3 are "insufficient," it can also have the function of generating a message as information SG urging the user to re-input information IN1 to information IN3.

[0082] 2, component 30 has a function of transferring information SG to component 10. As described above, information SG transferred to component 10 is provided to the user. After checking information SG provided by component 10, the user can take necessary measures (for example, by setting the number of hatching patterns in information IN1 to be greater than the number of areas in information IN3 and re-entering the information into component 10).

[0083] 2. The component 30 also has a function of receiving the information IN1 to IN3 and directive statement PT2 passed from the component 20 in a process T5 indicated by an arrow in FIG.

[0084] The component 30 also has a function of executing the instruction content written in the instruction statement PT2. Specifically, the component 30 has a function of generating information OUT based on the instruction statement PT2. Details of the instruction statement PT2 and the information OUT are as described above with reference to FIGS. 11 and 12, respectively.

[0085] Component 30 executes the instruction content described in instruction statement PT2 based on information IN1 to information IN3. As described above, information IN3 can be a list of the names of each area or each component of the image data corresponding to information IN2 (see FIG. 7 ), or a list of sentences describing each area or each component. It can also be the entire contents of a document (text file) describing information IN2 (see FIG. 8 ). By performing processing using a natural language processing model, component 30 can generate appropriate information OUT from information IN1 to information IN3 based on instruction statement PT2, regardless of whether information IN3 corresponds to any of the above cases.

[0086] Furthermore, by performing processing using a model that uses a GNN, component 30 can convert image data corresponding to information IN2 into graph data and perform tasks related to the vertices (also called nodes) and edges (also called edges or links) that make up the image, as well as the entire graph. For example, processing such as classification, regression, and clustering can be performed. By performing processing using a model that uses a GNN, the adjacent relationships between each region of the image data can be quantified, so component 30 can recognize, for example, the instruction content written in instruction statement PT2 (see FIG. 11) that "adjacent regions should be easy to distinguish."

[0087] In this way, in the information processing system according to one aspect of the present invention, the component 30 can perform processing using at least one or both of a model using natural language processing and a model using GNN. By performing processing using both a model using natural language processing and a model using GNN, the component 30 can execute the instruction content of the instruction statement PT2 with higher accuracy.

[0088] Component 30 also has a function of transferring information OUT to component 10 in process T6 indicated by an arrow in FIG.

[0089] Component 30 has the function of performing processing using a large-scale language model (separate from the large-scale language model that component 20 may have). Note that this large-scale language model has already learned a dataset. As a result, as described above, component 30 can determine whether information IN1 to IN3 are sufficient as input content based on information IN1 to IN3 and instruction statement PT1 passed from component 20. If the information is insufficient, component 30 can generate a specific solution (information SG). Furthermore, based on information IN1 to IN3 and instruction statement PT2 passed from component 20, component 30 can infer appropriate hatching patterns to be applied to the input image (information IN2) from the information IN1 by estimating them based on the descriptions (information IN3) of each region of the input image, and generate a list of the selection results (a list of hatching pattern ideas) as information OUT.

[0090] The information processing system according to one embodiment of the present invention has the above-described various functions, and therefore, regardless of the user's sense, skill level, etc., anyone can easily create attractive images (e.g., patent drawings in which each area is clearly identified with an appropriate hatching pattern) simply by providing the necessary information to the information processing system. Therefore, a novel information processing system that is highly convenient, useful, and reliable can be provided.

[0091] An information processing system according to one aspect of the present invention includes an information processing device that performs the functions of the above-described components.

[0092] For example, an information processing system according to an embodiment of the present invention can be configured with an information processing device that performs the functions of component 10, an information processing device that performs the functions of component 20, and an information processing device that performs the functions of component 30. Note that the number of information processing devices that make up the information processing system according to an embodiment of the present invention is one or more. Furthermore, for example, the information processing system according to an embodiment of the present invention can be configured by connecting a plurality of information processing devices using a network 50.

[0093] By configuring an information processing system according to one embodiment of the present invention using a plurality of information processing devices, the load related to information processing can be distributed.

[0094] Below, a configuration example of an information processing device that can be used in an information processing system of one embodiment of the present invention will be described in detail.

[0095] <<Configuration Example of Information Processing Device>> An information processing device (here, an information processing device that performs the functions of the component 20) that can be used in an information processing system of one embodiment of the present invention includes, for example, an input unit 110, a storage unit 120, a processing unit 130, an output unit 140, and a transmission path 150 (see FIG. 3 ).

[0096] In the drawings accompanying this specification, the components are classified by function and shown as independent blocks in the block diagrams. However, in reality, it is difficult to completely separate the components by function, and one component may be involved in multiple functions. For example, part of the processing unit 130 may function as the input unit 110. Also, one function may be involved in multiple components. For example, the processing performed by the processing unit 130 may be executed by different servers depending on the processing.

[0097] [Input Unit 110] The input unit 110 can receive data from outside the information processing device. For example, the input unit 110 receives data via the network 50.

[0098] The input unit 110 supplies the received data to one or both of the storage unit 120 and the processing unit 130 via the transmission path 150 .

[0099] [Storage Unit 120] The storage unit 120 has a function of storing a program executed by the processing unit 130. The storage unit 120 can also have a function of storing data generated by the processing unit 130 (e.g., calculation results, analysis results, inference results), data accepted by the input unit 110, etc.

[0100] The storage unit 120 may have a database. Furthermore, the information processing device may have a database separate from the storage unit 120. The information processing device may have a function to retrieve data from a database that exists outside the storage unit 120, outside the information processing device, or outside the information processing system. Furthermore, the information processing device may have a function to retrieve data from both its own database and an external database.

[0101] Either or both of a storage and a file server can be used as the memory unit 120. Also, a database that records paths of files stored in a file server can be used as the memory unit 120.

[0102] The storage unit 120 includes at least one of a volatile memory and a non-volatile memory. Examples of the volatile memory include a dynamic random access memory (DRAM) and a static random access memory (SRAM). Examples of the non-volatile memory include a resistive random access memory (ReRAM), a phase change random access memory (PRAM), a ferroelectric random access memory (FeRAM), a magnetoresistive random access memory (MRAM), and a flash memory. The storage unit 120 may include at least one of NOSRAM (registered trademark) and DOSRAM (registered trademark). The storage unit 120 may include a recording media drive. Examples of the recording media drive include a hard disk drive (HDD) and a solid state drive (SSD).

[0103] NOSRAM is an abbreviation for "Nonvolatile Oxide Semiconductor Random Access Memory (RAM)." NOSRAM refers to a memory in which memory cells are two-transistor (2T) or three-transistor (3T) gain cells and transistors (also referred to as OS transistors) that use metal oxide in their channel formation regions. OS transistors have an extremely small leakage current, i.e., a current that flows between the source and drain in an off state. NOSRAM can be used as a nonvolatile memory by retaining a charge corresponding to data in the memory cell using its extremely small leakage current characteristic. In particular, NOSRAM can read stored data without destroying it (nondestructive read), making it suitable for arithmetic processing in which only data read operations are repeated a large number of times. NOSRAM can increase its data capacity by stacking layers, and therefore can be used as a large-scale cache memory, main memory, or storage memory to improve the performance of semiconductor devices.

[0104] DOSRAM is an abbreviation for "Dynamic Oxide Semiconductor RAM" and refers to a RAM having 1T (transistor) 1C (capacitor) type memory cells. DOSRAM is a DRAM formed using OS transistors, and is a memory that temporarily stores information sent from an external device. DOSRAM is a memory that takes advantage of the low off-state current of OS transistors.

[0105] In this specification and the like, a metal oxide refers to an oxide of a metal in a broad sense. Metal oxides are classified into oxide insulators, oxide conductors (including transparent oxide conductors), oxide semiconductors (also referred to as oxide semiconductors or simply as OSs), and the like. For example, when a metal oxide is used for a semiconductor layer of a transistor, the metal oxide may be referred to as an oxide semiconductor.

[0106] The metal oxide included in the channel formation region preferably contains indium (In). When the metal oxide included in the channel formation region contains indium, the carrier mobility (electron mobility) of the OS transistor is increased. Furthermore, the metal oxide included in the channel formation region is preferably an oxide semiconductor containing element M. The element M is preferably at least one of aluminum (Al), gallium (Ga), and tin (Sn). Other elements applicable to element M include boron (B), silicon (Si), titanium (Ti), iron (Fe), nickel (Ni), germanium (Ge), yttrium (Y), zirconium (Zr), molybdenum (Mo), lanthanum (La), cerium (Ce), neodymium (Nd), hafnium (Hf), tantalum (Ta), and tungsten (W). However, a combination of two or more of the above elements may be used as element M. The element M is, for example, an element having a high bond energy with oxygen. For example, it is an element having a higher bond energy with oxygen than indium. The metal oxide contained in the channel formation region is preferably a metal oxide containing zinc (Zn), since zinc-containing metal oxides may be easily crystallized.

[0107] The metal oxide contained in the channel formation region is not limited to a metal oxide containing indium, but may be, for example, a metal oxide containing zinc but not indium, such as zinc tin oxide or gallium tin oxide, a metal oxide containing gallium, or a metal oxide containing tin.

[0108] [Processing Unit 130] The processing unit 130 has a function of performing processes such as calculation, analysis, and inference using data supplied from one or both of the input unit 110 and the storage unit 120. The processing unit 130 can supply generated data (e.g., calculation results, analysis results, and inference results) to one or both of the storage unit 120 and the output unit 140.

[0109] The processing unit 130 has a function of acquiring data from the storage unit 120. The processing unit 130 can also have a function of recording or registering data in the storage unit 120.

[0110] The processing unit 130 may include, for example, an arithmetic circuit, a central processing unit (CPU), and a graphics processing unit (GPU).

[0111] The processing unit 130 may include a microprocessor such as a DSP (Digital Signal Processor). The microprocessor may be implemented by a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an FPAA (Field Programmable Analog Array). The processing unit 130 may also include a quantum processor. The processing unit 130 can perform various data processing and program control by interpreting and executing instructions from various programs using the processor. Programs that can be executed by the processor are stored in at least one of the memory area of ​​the processor and the storage unit 120.

[0112] The processing unit 130 may include a main memory. The main memory may include at least one of a volatile memory such as a RAM and a non-volatile memory such as a ROM (Read Only Memory). The main memory may also include at least one of the above-mentioned NOSRAM and DOSRAM.

[0113] The RAM may be, for example, a DRAM or an SRAM, and a virtual memory space is allocated and used as a working space for the processing unit 130. The operating system, application programs, program modules, program data, lookup tables, etc. stored in the storage unit 120 are loaded into the RAM for execution. The data, programs, and program modules loaded into the RAM are each directly accessed and operated by the processing unit 130.

[0114] The ROM can store BIOS (Basic Input / Output System), firmware, etc., which do not require rewriting. Examples of ROM include mask ROM, OTPROM (One Time Programmable Read Only Memory), and EPROM (Erasable Programmable Read Only Memory). Examples of EPROMs include UV-EPROMs (Ultra-Violet Erasable Programmable Read Only Memories), which allow stored data to be erased by exposure to ultraviolet light, EEPROMs (Electrically Erasable Programmable Read Only Memories), and flash memories.

[0115] The processing section 130 can include one or both of an OS transistor and a transistor having silicon in a channel formation region (Si transistor).

[0116] The processing unit 130 preferably includes an OS transistor. Because an OS transistor has an extremely small off-state current, using the OS transistor as a switch for retaining charge (data) flowing into a capacitor functioning as a memory element can ensure a long data retention period. By using this characteristic in at least one of the register and cache memory of the processing unit, the processing unit can be operated only when necessary, and can be turned off in other cases by saving information from the previous processing in the memory element. In other words, normally-off computing is possible, and the power consumption of the information processing system can be reduced.

[0117] It is preferable that the information processing device uses AI for at least some of its processing.

[0118] It is particularly preferable that the information processing device uses an artificial neural network (ANN, hereinafter also simply referred to as a neural network). A neural network is realized by a circuit (hardware) or a program (software).

[0119] In this specification, a neural network refers to a general model that mimics the neural circuit network of a living organism, determines the connection strength between neurons through learning, and has problem-solving capabilities. A neural network has an input layer, an intermediate layer (hidden layer), and an output layer.

[0120] In this specification and the like, when discussing neural networks, determining the connection strengths (also called weighting coefficients) between neurons from existing information may be referred to as "learning."

[0121] In this specification and the like, the act of constructing a neural network using connection strengths obtained by learning and deriving a new conclusion from it may be referred to as "inference."

[0122] [Output Unit 140] The output unit 140 can output at least one of the calculation result, analysis result, and inference result in the processing unit 130 to the outside of the information processing device. For example, the output unit 140 can transmit data via the network 50.

[0123] [Transmission Path 150] The transmission path 150 has a function of transmitting data. Data can be transmitted and received between the input unit 110, the storage unit 120, the processing unit 130, and the output unit 140 via the transmission path 150.

[0124] <<Configuration Example of Network 50>> The network 50 that can be used in the information processing system of one embodiment of the present invention can connect multiple information processing devices. This allows the connected multiple information processing devices to transmit and receive data to and from each other. Furthermore, the load related to information processing can be distributed.

[0125] When wireless communication is performed, communication standards such as the fourth generation mobile communication system (4G), fifth generation mobile communication system (5G), and sixth generation mobile communication system (6G), or specifications standardized by the IEEE such as Wi-Fi (registered trademark) and Bluetooth (registered trademark), can be used as communication protocols or communication technologies.

[0126] For example, a local network can be used for the network 50. Also, an intranet or an extranet can be used for the network 50. Also, a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a global area network (GAN), etc. can be used for the network 50.

[0127] Furthermore, for example, a global network can be used for the network 50. Specifically, the Internet, which is the foundation of the World Wide Web (WWW), can be used.

[0128] Furthermore, a person who provides a service using the information processing system according to one aspect of the present invention can provide the service using the information processing method according to one aspect of the present invention via the network 50, for example.

[0129] When the information processing system according to an embodiment of the present invention is built within a local network, the possibility of confidential information leaking can be reduced, for example, compared to when the Internet is used.

[0130] This embodiment mode can be appropriately combined with other embodiment modes described in this specification.

[0131] Embodiment 2 In this embodiment, an information processing method according to one embodiment of the present invention will be described with reference to FIG.

[0132] FIG. 4 is a diagram illustrating an information processing method according to one embodiment of the present invention.

[0133] In the following, detailed explanations may be omitted for the specific contents of information IN1 to information IN3, instruction statement PT1, instruction statement PT2, information SG, information OUT, image IMG, etc., and the specific functions of component 10, component 20, and component 30, as reference may be made to the description in embodiment 1.

[0134] Furthermore, with regard to the symbols used in the following description, the contents explained in embodiment 1 can be applied to symbols that are the same as those used in embodiment 1. Therefore, detailed explanations of the meanings and definitions of symbols may be omitted below.

[0135] <Example of Information Processing Method> An information processing method according to one embodiment of the present invention includes steps S1 to S7.

[0136] [Step S1] In step S1, the component 10 receives information IN1 to IN3 (see FIGS. 5 to 8) input by the user, and then passes the information IN1 to IN3 to the component 20. The process in step S1 corresponds to process T1 shown in FIG.

[0137] [Step S2] After receiving the information IN1 to IN3 passed from the component 10, the component 20 creates a directive PT1 (see FIG. 9) using the information IN1 to IN3 in step S2, and passes the information IN1 to IN3 and the directive PT1 to the component 30. The process in step S2 corresponds to process T2 shown in FIG. 2.

[0138] [Step S3] In step S3, the component 30 determines whether the information IN1 to IN3 are sufficient as user input, based on the first instruction content (see FIG. 9) written in the instruction statement PT1.

[0139] If the determination in step S3 is "Yes," the component 30 performs a process of transferring the information IN1 to information IN3 to the component 20. This process corresponds to the process T3 shown in FIG.

[0140] As described in the first embodiment, if the component 20 has the function of saving the information IN1 to IN3 received from the component 10 in the process T1 (corresponding to step S1) shown in FIG. 2, when the answer in step S3 is determined to be "Yes," the component 30 does not need to transfer the information IN1 to IN3 to the component 20 again.

[0141] If the determination in step S3 is "No," the component 30 generates information SG (see FIG. 10) based on the second instruction content (see FIG. 9) described in the instruction statement PT1, and performs processing to transfer information SG to the component 10. This processing corresponds to processing T4 shown in FIG. 2.

[0142] After receiving the information SG from the component 30, the component 10 presents the information SG to the user. The user checks the information SG and takes necessary measures (for example, by setting the number of hatching patterns in the information IN1 to be greater than the number of areas in the information IN3 and re-entering the information into the component 10).

[0143] [Step S4] After receiving the information IN1 to IN3 passed from the component 30, the component 20 creates a directive PT2 (see FIG. 11) using the information IN1 to IN3 in step S4, and passes the information IN1 to IN3 and the directive PT2 to the component 30. The process in step S4 corresponds to process T5 shown in FIG. 2.

[0144] [Step S5] In step S5, the component 30 generates information OUT (see FIG. 12) based on the instruction content (see FIG. 11) described in the instruction statement PT2, and performs processing to transfer the information OUT to the component 10. This processing corresponds to processing T6 shown in FIG. 2.

[0145] [Step S6] In step S6, the component 10 receives the information OUT generated by the component 30 in step S5, and then generates an image IMG (see FIGS. 13A to 13C) based on the information OUT.

[0146] The information processing method for generating the image IMG is not limited to the content described in step S6 above. For example, the information processing method may be one in which the component 20 receives the information OUT generated by the component 30 in step S5. In this case, the component 20 generates a script suitable for an image editing application, such as the above-mentioned Vectorworks, based on the information OUT and passes the script to the component 10. This allows the component 10 to generate the image IMG based on the script in a later step.

[0147] [Step S7] In step S7, the component 10 presents the image IMG to the user. If the user checks the image IMG and determines that there is no problem, the process ends. If the user determines that there is a problem with the image IMG (for example, if the user wants to generate an image with a hatching pattern different from the information OUT), the user may take measures such as changing the information IN1 and executing the processes related to steps S1 to S7 described above again.

[0148] By applying the above-described information processing method, the information processing system according to one embodiment of the present invention allows anyone to easily create attractive images (e.g., patent drawings in which each area is clearly identified with an appropriate hatching pattern) simply by providing the necessary information to the information processing system, regardless of the user's sense, skill, etc. Therefore, a novel information processing system that is highly convenient, useful, and reliable can be provided.

[0149] This embodiment mode can be appropriately combined with other embodiment modes described in this specification.

[0150] 10: component, 20: component, 30: component, 50: network, 110: input unit, 120: memory unit, 130: processing unit, 140: output unit, 150: transmission path, 200: transistor, 210: insulating layer, 220_1: conductive layer, 220_2: conductive layer, 230: oxide semiconductor layer, 240: conductive layer, 250: insulating layer, 260: conductive layer, 280: insulating layer

Claims

a first component, a second component, and a third component; the third component has a function of performing processing using a large-scale language model; the first component has a function of receiving first information to third information and a function of transferring the first information to the third information to the second component; the first information is a list of hatching patterns, the second information is a first image without the hatching pattern; the third information is a description of each area of ​​the first image, the second component has a function of receiving the first information to the third information and a function of inputting the first information to the third information and a first instruction statement to the third component; the third component has a function of determining whether there is a problem with the first information to the third information based on the first instruction sentence, and a function of generating fourth information; the fourth information is information including a solution when there is a problem with the first information to the third information, the first component has a function of receiving the fourth information; the second component has a function of inputting the first information to the third information and a second instruction sentence to the third component; the third component has a function of generating fifth information based on the first information to the third information and the second instruction statement; the fifth information is a list of hatching pattern ideas to be applied to each region of the first image, the first component has a function of receiving the fifth information and a function of generating a second image based on the fifth information. Information processing system. 　 In claim 1, the first directive comprises a first sentence and a second sentence; the first sentence is a sentence asking whether there is any problem with the first information to the third information, the second sentence is a sentence asking for a solution when there is a problem with one or more of the first information to the third information; Information processing system. 　 In claim 1 or claim 2, the second directive has a third sentence, the third statement is a statement instructing to generate a list of hatching patterns corresponding to each region of the first image, the second image is an image in which the hatching pattern is applied to each of the regions; Information processing system. 　 In claim 1, the second component has a function of generating the first instruction sentence using the first information to the third information, and a function of generating the second instruction sentence based on the first information to the third information. Information processing system. 　 In claim 1, The third component has a function of performing processing using a graph neural network. Information processing system. 　 In claim 5, the first image and the second image are each a vector image. Information processing system. 　 The method includes first to sixth steps, In the first step, the first component receives the first information to the third information, and then transfers the first information to the third information to the second component; In the second step, the second component receives the first information to the third information and transfers the first information to the third information and a first instruction statement to a third component; In the third step, the third component determines whether there is a problem with the first information to the third information based on the first instruction sentence; when it is determined that there is a problem with one or more of the first information to the third information, the third component generates fourth information and then transfers the fourth information to the first component; If it is determined that there is no problem with the first information to the third information, in the fourth step, the second component passes the first information to the third information and the second instruction statement to the third component; In the fifth step, the third component generates fifth information based on the second instruction statement, and then passes the fifth information to the first component; In the sixth step, the first component receives the fifth information and then generates a first image based on the fifth information. Information processing methods. 　 In claim 7, the first information is a list of hatching patterns, the second information is a second image without the hatching pattern; the third information is a description of each area of ​​the second image, the first directive comprises a first sentence and a second sentence; the first sentence is a sentence asking whether there is any problem with the first information to the third information, the second sentence is a sentence asking for a solution when there is a problem with one or more of the first information to the third information, the fourth information is a sentence containing the solution; Information processing methods. 　 In claim 8, the second directive has a third sentence, the third statement is a statement instructing to generate a list of hatching patterns corresponding to each region of the second image, the fifth information is a list of the hatching patterns, the first image is an image in which the hatching pattern is applied to each of the regions; Information processing methods. 　 In claim 9, the first image and the second image are each a vector image. Information processing methods.

Images