Design support device, design support method, and program

Abstract

Provided is a design support device for designing a non-text processing model for processing non-text information. The design support device displays a plurality of candidate regions in which symbols representing candidates for a model constituting the non-text processing model are arranged. One or more first symbols representing a first model for receiving inputs of non-text information and outputting text information corresponding to the input non-text information are arranged in a first candidate region. One or more second symbols representing a second model for receiving inputs of text information and converting the input text information into text information corresponding to a predetermined purpose are arranged in a second candidate region. One or more third symbols representing a third model for receiving inputs of text information and outputting non-text information corresponding to the input text information are arranged in a third candidate region. The plurality of candidate regions include at least two candidate regions among the first candidate region, the second candidate region, and the third candidate region.

Description

Design support device, design support method and program

[0001] The present invention relates to a design support device, a design support method, and a program.

[0002] There is growing interest in technologies that use mathematical models, such as generative AI.

[0003] “Newly developed next-generation media processing AI, “MediaGnosis™,” with human-like information processing mechanisms – Realizing applications that utilize multimedia in an integrated manner with a single AI,” [online], November 1, 2021, Nippon Telegraph and Telephone Corporation, [Retrieved May 15, 2024], Internet <URL: https: / / group.ntt / jp / newsrelease / 2021 / 11 / 01 / 211101b.html>

[0004] However, obtaining a mathematical model is not always easy, and for example, it may be difficult for someone unfamiliar with programming to design a desired mathematical model.

[0005] In view of the above circumstances, an object of the present invention is to facilitate the design of a mathematical model.

[0006] One aspect of the present invention is a design support device for designing a non-text processing model that processes non-text information, which displays a plurality of candidate areas in which symbols representing candidate models that constitute the non-text processing model are placed, the plurality of candidate areas including at least two of the following candidate areas: a first candidate area in which one or more first symbols representing a first model that accepts input of non-text information and outputs text information corresponding to the input non-text information are placed; a second candidate area in which one or more second symbols representing a second model that accepts input of text information and converts the input text information into text information according to a predetermined purpose are placed; and a third candidate area in which one or more third symbols representing a third model that accepts input of text information and outputs non-text information corresponding to the input text information are placed.

[0007] One aspect of the present invention is a design support device for designing a combination model composed of multiple models, which accepts an instruction to combine a first model, which is one of the multiple models that make up the combination model, with input information to be input to the first model, and, if the input information does not satisfy an acceptability condition for the first model, displays a message that the acceptability condition is not satisfied.

[0008] One aspect of the present invention is a design support device for designing a combination model composed of multiple models, the combination model including a first model and a second model that receives output data of the first model, and the design support device displays options for accepting a selection of whether to use, as input data for the second model, internal feature values ​​of the first model or label data obtained by converting the internal feature values.

[0009] One aspect of the present invention is a design support method for designing a non-text processing model that outputs non-text information in response to input of non-text information, wherein a computer displays a plurality of candidate areas in which symbols representing candidate models that constitute the non-text processing model are placed, the plurality of candidate areas including at least two of the following: a first candidate area in which one or more first symbols representing a first model that accepts input of non-text information and outputs text information corresponding to the input non-text information are placed; a second candidate area in which one or more second symbols representing a second model that accepts input of text information and converts the input text information into text information according to a predetermined purpose are placed; and a third candidate area in which one or more third symbols representing a third model that accepts input of text information and outputs non-text information corresponding to the input text information are placed.

[0010] One aspect of the present invention is a program for causing a computer to function as the design support device described above.

[0011] The present invention makes it possible to facilitate the design of a mathematical model.

[0012] 1 is an explanatory diagram illustrating a design support system according to a first embodiment. FIG. 1 is a diagram illustrating the relationship between a candidate model, an element model, and a combination model according to the first embodiment. FIG. 2 is a diagram illustrating an example of a display screen in an initial state according to the first embodiment. FIG. 3 is a diagram illustrating an example of a display screen when an input sample is set according to the first embodiment. FIG. 4 is a diagram illustrating an example of a display screen when an element model is arranged according to the first embodiment. FIG. 5 is a diagram illustrating an example of a display screen when an element model is arranged according to the first embodiment. FIG. 6 is a diagram illustrating an example of a display screen when an element is associated with another element according to the first embodiment. FIG. 7 is a flowchart illustrating an association process starting from an input tab according to the design support system according to the first embodiment. FIG. 8 is a flowchart illustrating an association process starting from an output tab according to the design support system according to the first embodiment. FIG. 9 is a diagram illustrating an example of a display screen when a combination model is completed according to the first embodiment. FIG. 10 is a diagram illustrating an example of a display screen when a prompt is set by the design support device according to the first embodiment. FIG. 11 is a flowchart illustrating an operation check process for a combination model according to the first embodiment. FIG. 12 is a first diagram illustrating an example of a display screen when an operation check of a combination model is performed according to the first embodiment. FIG. 13 is a second diagram illustrating an example of a display screen when an operation check of a combination model is performed according to the first embodiment. FIG. 14 is a diagram illustrating an example of a display screen when an output data rewrite operation is performed according to the first embodiment. FIG. 15 is a diagram illustrating an example of a display screen according to the second embodiment. FIG. 16 is a diagram illustrating an example of a display screen according to the third embodiment. FIG. 17 is a diagram illustrating an example of a display screen according to the fourth embodiment. FIG. 18 is a diagram illustrating an example of a display screen according to the sixth embodiment. FIG. 19 is a diagram illustrating an example of a display screen according to a seventh embodiment. FIG. 20 is a diagram illustrating an example of a screen displayed by a predetermined display device in a modified example. FIG. 21 is a diagram illustrating an example of a hardware configuration of a design support apparatus according to at least one embodiment.

[0013] First Embodiment FIG. 1 is an explanatory diagram illustrating a design support system 100 according to a first embodiment. The design support system 100 supports the design of a mathematical model. The design support system 100 according to the first embodiment generates a new mathematical model by combining multiple mathematical models. Hereinafter, a mathematical model formed by combining multiple mathematical models will also be referred to as a "combined model." Furthermore, the individual mathematical models that make up the combined model will also be referred to as "element models." The design support system 100 includes a design support device 1, a display device 2, and an input device 3.

[0014] The display device 2 is, for example, a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The display device 2 is an example of a predetermined display destination. FIG. 1 shows an example of a design support system 100 in which the predetermined display destination is the display device 2, but the predetermined display destination does not necessarily have to be a device different from the design support device 1 and may be provided in the design support device 1, for example. The input device 3 is, for example, an input device such as a mouse, a keyboard, or a touch panel. Note that the input device 3 may have the functions of the display device 2, such as a touch panel.

[0015] The design support device 1 includes an input unit 11 , a model information storage unit 12 , a model design unit 13 , a model calculation unit 14 , a data acquisition unit 15 , a display control unit 16 , and an output unit 17 .

[0016] The input unit 11 receives operation input from the user via the input device 3 .

[0017] 2 is a diagram showing the relationship between a candidate model C, an element model E, and a combination model M according to the first embodiment. The model information storage unit 12 stores information about a candidate model C that is a candidate for the element model E that constitutes the combination model M. The model information storage unit 12 stores a model ID that identifies the candidate model C, a symbol that represents the candidate model C, the type of the candidate model C, the input specifications of the candidate model C, the output specifications of the candidate model C, and the location of the candidate model C in association with each other.

[0018] Examples of symbols include still images such as icons, character strings such as names, animated images, and symbols.

[0019] The type of candidate model C can be any of the following: NonText-to-Text, which generates text information from non-text information; Text-to-Text, which generates text information from text information; and Text-to-Nontext, which generates non-text information from text information. Text information refers to linguistic information such as character strings, labels, numerical values, and Boolean values. Text information is not limited to text data such as plain text, but also includes data that expresses linguistic information such as character strings as vectors, and marked-up character strings such as JSON. Text information may also be information in which vector sequences are replaced with labels. Non-text information refers to information containing non-linguistic information such as still images, moving images, audio, and sensor data. For example, audio may contain linguistic information, but it also includes non-linguistic information such as the volume, pitch, and speed of the voice, and therefore falls under the category of non-text information. Audio is not limited to human speech, but may also include audio data recorded from environmental sounds. Non-linguistic information that has been verbalized by classification, recognition, etc., and expressed as labels, scores, etc., is included in text information. In the following description, non-linguistic information that has not been verbalized is not included in text information.

[0020] Examples of non-text-to-text type models include a mathematical model that accepts input of voice data and outputs text data, a mathematical model that accepts input of video data and outputs text data, a mathematical model that accepts a predetermined amount of input such as heart rate measurement results and outputs text data, and a mathematical model that accepts a predetermined amount of time-series input such as a time series of heart rate measurement results and outputs text data. Examples of non-text-to-text type mathematical models include neural network models that have undergone supervised learning in advance, GMM-HMM, and DNN-HMM. Hereinafter, non-text-to-text type mathematical models are also referred to as first models. The text information output by a non-text-to-text type mathematical model represents the semantic content of linguistic or non-linguistic information contained in the input data.

[0021] An example of a text-to-text type model is a candidate model that accepts input text information and outputs text data based on the text information. Examples of text-to-text type mathematical models include so-called large language models (LLMs), models that are LLMs specialized for specific tasks, rule-based dialogue / question-answering models, and statistical-based machine translation (SMT). Models such as LLMs may be models trained to predict the next word from an input word sequence using text data as training data. Hereinafter, text-to-text type mathematical models are also referred to as second models. In particular, input data for large-scale language models is also referred to as prompts. In a narrow sense, a large-scale language model may refer to a decoder-only model, but is not limited to this and may also include an encoder-decoder type model. In a narrow sense, a large-scale language model may refer to a general-purpose language model, but is not limited to this and may also be a model specialized for a specific task (translation, summarization, reading comprehension, dialogue). Furthermore, the large-scale language model is not necessarily limited to one trained using a "large" language corpus, and the number of internal parameters is not limited. For example, the large-scale language model may be one trained using a corpus of a specific field or language. In the following description, an example will be given in which the LLM is an application incorporating a pre-trained neural network model that accepts input of a natural language prompt and outputs text data in response to the prompt.

[0022] The LLM can change its function depending on additional information provided as a prompt. For example, if the prompt is "Please translate the following English sentence into Japanese. {input text}," the LLM will estimate the input text as "the following English sentence" in the prompt. Note that the {} indicates a variable, and the value of the input text is substituted for {input text} in the prompt. In this case, the LLM estimates the Japanese translation of the input text and outputs the estimated result.

[0023] An example of a Text-to-NonText type model is a mathematical model that synthesizes speech from input text information and outputs speech data. However, the target generated from text information may also be acoustic information or an image other than speech. An example of a Text-to-NonText type mathematical model is a neural network model that has undergone supervised learning in advance. Hereinafter, the Text-to-NonText type mathematical model will also be referred to as the third model.

[0024] The input specifications of the candidate model C are the specifications of data that can be input to the candidate model C. The output specifications of the candidate model C are the specifications of data that the candidate model C outputs. The specifications are expressed, for example, by the type of data (character string, still image, moving image, audio, etc.) and the data format. Specific examples of the data format include the file format, the number of dimensions of vectors and tensors, the number of bits, and encoding. The location of the candidate model C may be the storage of the design support device 1 or an external server. The location of the candidate model C may be expressed by an address such as a uniform resource locator (URL) or a directory path, or by an access means such as an application programming interface (API) for accessing the candidate model C.

[0025] The model design unit 13 generates design data D for the combination model M using multiple candidate models C in accordance with user operations received by the input unit 11. As shown in Fig. 2, the design data D stores, for each element model E constituting the combination model M, an element ID that identifies the element model E, a model ID that indicates a candidate model implemented in the element model, and a source element ID that indicates the source of data input to the element model. The element model E can also be considered an instance of a class that calls the candidate model C.

[0026] Here, the design data D will be described using an example of a combination model M that inputs voice data and outputs voice data, as shown in FIG. 2 . The combination model M shown in FIG. 2 is composed of element models E1, E2, and E3. The element model E1 implements a candidate model C02 that converts input voice data into a character string. The element model E2 implements a candidate model C01 that generates a new character string from an input character string. The element model E3 implements a candidate model C03 that synthesizes an input character string into voice and outputs voice data. The input data of the combination model M is input to the element model E1. The output data of the element model E1 is input to the element model E2. The output data of the element model E2 is input to the element model E3. The output data of the element model E3 is treated as the output data of the combination model M. The design data D representing such a combination model M can be represented by the element ID, model ID, and source element of each element model, as shown in FIG. 2 . In the design data D, the input element Ei and the output element Eo represent the input and output of the combination model M, and do not implement a candidate model. In the design data D shown in FIG. 2, the model ID "C01" and the source element "E1" are stored in association with each other for the element E2. This indicates that the element E2 implements the candidate model C01 that generates a character string from a character string, and that the output data of the element model E1 is input.

[0027] The model calculation unit 14 inputs data into the candidate model C and performs calculations to obtain output data. The data acquisition unit 15 acquires input data for the combination model M. The input data for the combination model M may be provided by being uploaded by a user, for example.

[0028] The display control unit 16 causes the display device 2 to display the processing results of the model design unit 13 and the model calculation unit 14. The output unit 17 outputs the design data D generated by the model design unit 13 or the combination model M constructed based on the design data D. For example, the output unit 17 may generate and output a program capable of executing calculations using the combination model M constructed based on the design data D. Output includes, for example, writing to a storage medium, transmitting via a network, etc.

[0029] <Operation of Design Support Device> The design support device 1 supports the design of the combination model M in the following procedure: First, the display control unit 16 creates a display screen to be displayed on the display device 2 based on the information of the candidate model C stored in the model information storage unit 12 (step S1).

[0030] 3 is a diagram showing an example of a display screen in an initial state according to the first embodiment. The display screen displayed on the display device 2 by the design support device 1 includes a candidate pane P1 for presenting candidate models and a design pane P2 for designing the combination model M. The candidate pane P1 is located on the left side of the display screen, and the design pane P2 is located on the right side of the display screen. Note that the candidate pane P1 does not necessarily have to be displayed on the left side of the screen. The candidate pane P1 may be displayed on the right side, upper side, or lower side of the screen, for example.

[0031] The candidate pane P1 is provided with candidate areas P11-P13 for each model type. Specifically, the candidate pane P1 is provided with a first candidate area P11 displaying a nontext-to-text type candidate model C, a second candidate area P12 displaying a text-to-text type candidate model C, and a third candidate area P13 displaying a text-to-nontext type candidate model C. The candidate areas are arranged in the candidate pane P1 from top to bottom in the order of the first candidate area P11, the second candidate area P12, and the third candidate area P13. A symbol representing the corresponding type of candidate model C is arranged in each candidate area. That is, a text-to-text type candidate model C that generates text information from text information is arranged below a nontext-to-text type candidate model C that generates text information from non-text information. A text-to-nontext type candidate model C that generates non-text information from text information is arranged below a text-to-text type candidate model C that generates text information from text information. This allows the design support device 1 to display the candidate models C in an arrangement such that the output data of the candidate model C located above can be used as input data for the candidate model C located directly below. The display control unit 16 determines the area in which to place the symbol of the candidate model C from among the three candidate areas P11 to P13, based on the type of each candidate model C stored in the model information storage unit 12.

[0032] In the initial state, the design pane P2 is provided with an input setting button B1 for setting input data for the combination model M and an output setting button B2 for setting output data for the combination model M. The input setting button B is a button for setting the data format of the input data for the combination model M and accepting the upload of an input sample, which is a sample of the input data, when pressed. Note that the input setting button B1 and the output setting button B2 do not necessarily have to be provided from the initial state. For example, the input setting button B1 and the output setting button B2 may be provided by a user operation. In another embodiment, the input setting button B1 may be represented as a balloon En3 (described later) in the input tab En1 of the first model. In another embodiment, the output setting button B2 may be represented as a balloon En3 (described later) in the output tab En2 of the third model.

[0033] Next, the design support device 1 accepts a setting operation of input data, a setting operation of output data, or a setting operation of the element models E that constitute the combination model M from the user.

[0034] The input unit 11 of the design support device 1 accepts input data setting operations by accepting pressing of the input setting button B1 via the input device 3. When the input setting button B1 is pressed, the input unit 11 accepts input of input data specifications. That is, the input unit 11 accepts input of the type of input data (e.g., character string, still image, moving image, audio). When the input unit 11 accepts the input data specifications, the model design unit 13 records the input value as the specifications of the input element Ei in the design data D. Next, the input unit 11 can accept setting of an input sample, which is a sample of the input data. The input unit 11 may accept input of a path indicating the location of the input sample from the input device 3. This enables the design support device 1 to acquire the input sample. Note that, in another embodiment, the input unit 11 may determine the input data specifications and set the input sample by, for example, dragging and dropping a symbol representing the input sample onto the design pane P2. Note that the input data does not have to be a single piece of data. For example, the design support device 1 may accept a combination of multiple data items as input data, such as a combination of audio data and a still image.

[0035] FIG. 4 is a diagram illustrating an example of a display screen when an input sample is set according to the first embodiment. Once the input sample is set, the display control unit 16 displays the input setting button B1 in a manner capable of representing the input sample. Specifically, a play button B11 is displayed on the input setting button B1 to which the input sample has been set, and the input sample is represented by pressing the play button B11. Examples of representations of the input sample include displaying still image data, playing video data or audio data, rendering a three-dimensional model, and graphically displaying heart rate data. Furthermore, once the specifications of the input data are set, the display control unit 16 displays an output tab B12 below the input setting button B1. The output tab B12 is used to set a connection relationship with an element model E. The output tab B12 is connected to the element model E that is the input destination of the input data. Note that if the input sample is image data, the input sample may be displayed as the texture of the input setting button B1, rather than being displayed by pressing the play button B11.

[0036] The input unit 11 of the design support device 1 accepts an output data setting operation by accepting pressing of the output setting button B2 via the input device 3. When the output setting button B2 is pressed, the input unit 11 accepts input of the output data specifications. In other words, the input unit 11 accepts input of the type of output data. When the input unit 11 accepts the output data specifications, the model design unit 13 records the input value as the specification of the output element Eo in the design data D. Note that the design support device 1 according to another embodiment does not need to limit the output data specifications. When the output data specifications are set, the display control unit 16 displays an input tab B21 above the output setting button B2. The input tab B21 is used to set a connection relationship with the element model E. The input tab B21 is connected to the element model E that outputs the output data.

[0037] The input unit 11 of the design support device 1 accepts an element model setting operation by accepting an operation of dragging and dropping a symbol arranged in the candidate pane P1 to the design pane P2 via the input device 3. Note that the element model setting operation may also be accepted by other operation methods for selecting a symbol, such as double-clicking the symbol. Upon accepting the element model setting operation, the model design unit 13 adds a new element model En (n is a natural number) to the design data D and records the model ID of the selected candidate model C as the model ID of the element model En.

[0038] FIG. 5 is a diagram showing an example of a display screen when an element model E1 is arranged according to the first embodiment. When a setting operation for the element model E1 is accepted, the display control unit 16 arranges a symbol E10 representing the element model E1 in the design pane P2. An input tab E11 is displayed above the symbol E10, and an output tab E12 is displayed below it. Multiple element models E1 can be arranged in the design pane P2. FIG. 6 is a diagram showing an example of a display screen when multiple element models E1, E2, and E3 are arranged according to the first embodiment. Similarly, when a setting operation for multiple element models En is accepted, the display control unit 16 arranges symbols En0 representing the multiple set element models En in the design pane P2. An input tab En1 is displayed above the symbol En0, and an output tab En2 is displayed below it.

[0039] When the element model En is placed in the design pane P2, the input unit 11 accepts the setting of the element model En and the operation of associating elements with each other.

[0040] The input unit 11 of the design support device 1 accepts a setting operation for an element model En by clicking an input tab En1 of the element model En via the input device 3. In the setting operation for the element model En, parameters used for executing the element model En are set. Examples of parameters include a pre-emphasis filter coefficient when the element model En is an automatic speech recognition model (ASR), and a prompt with variables when the element model En is a large-scale language model (LLM). A prompt with variables is a prompt incorporating variables for embedding input data.

[0041] The input unit 11 of the design support device 1 accepts an association operation by a drag operation of an input tab (B21 or En1) or an output tab (B12 or En2) via the input device 3. Fig. 7 is a diagram showing an example of a display screen when an association operation between elements is performed according to the first embodiment.

[0042] 8 is a flowchart showing an association process starting from an input tab performed by the design support system 100 according to the first embodiment. When the input unit 11 accepts an association operation starting from an input tab, the model design unit 13 identifies input specifications for the element (element model En or output element Eo) related to the starting input tab (step S101). If the element is an element model En, the model design unit 13 identifies the input specifications using the following procedure. First, the model design unit 13 identifies the model ID of a candidate model C implemented by the element model En from the design data D. Next, the model design unit 13 reads the input specifications of the candidate model C related to the identified model ID from the model information storage unit 12. If the element is an output element Eo, the model design unit 13 reads the specifications of the output element Eo from the design data D.

[0043] Next, the model design unit 13 identifies the output specifications of each of the multiple elements recorded in the design data D (step S102). If the element is an element model En, the output specifications are identified using the following procedure. First, the model design unit 13 identifies, from the design data D, the model ID of a candidate model C implemented by the element model En. Next, the model design unit 13 reads, from the model information storage unit 12, the output specifications of the candidate model C associated with the identified model ID. If the element is an input element Ei, the model design unit 13 reads, from the design data D, the specifications of the input element Ei.

[0044] The model design unit 13 identifies, from the multiple elements, an element whose output specification identified in step S102 matches the input element of the starting element identified in step S101 (step S103). The display control unit 16 causes the display device 2 to display a connection candidate line L2 connecting the input tab of the starting element and the output tab of the element identified in step S103 (step S104). That is, on the display screen of the display device 2, connectable elements are connected by the connection candidate line L2. The connection candidate line L2 is an example of a second connection image indicating that the type of data generated by the model represented by the upstream symbol of the two symbols is a type that can be input to the model represented by the downstream symbol of the two symbols.

[0045] The user specifies the data input source by dragging from the input tab that serves as the starting point and then dropping onto the output tab of any element. The model design unit 13 determines whether the output specifications of the element related to the end point of the association operation match the input specifications of the element identified in step S101 (step S105). This is equivalent to determining whether the output tab of the end point is connected by a connection candidate line L2.

[0046] If the output specifications match the input specifications (step S105: YES), the model design unit 13 records the element ID of the end element as the source element of the start element in the design data D (step S106). The display control unit 16 causes the display device 2 to display a connection line L1 connecting the input tab of the start point and the output tab of the end point (step S107). The connection line L1 and the connection candidate line L2 are displayed in different modes. For example, the connection line L1 may be a solid line, and the connection candidate line L2 may be a dashed line. Furthermore, the connection line L1 may be a thicker line than the connection candidate line L2.

[0047] If the output specifications do not match the input specifications (step S105: NO), the model design unit 13 does not update the source element of the starting element in the design data D. At this time, the display control unit 16 causes the display device 2 to display a warning that the two selected elements cannot be connected because the input specifications and output specifications do not match (step S108).

[0048] 9 is a flowchart showing an association process starting from an output tab performed by the design support system 100 according to the first embodiment. When the input unit 11 accepts an association operation starting from an output tab, the model design unit 13 identifies output specifications for the element (element model En or output element Eo) associated with the starting output tab (step S201). Next, the model design unit 13 identifies input specifications for each of the elements recorded in the design data D (step S202).

[0049] The model design unit 13 identifies, from among the multiple elements, an element whose input specifications identified in step S202 match those of the starting element identified in step S201 (step S203). The display control unit 16 causes the display device 2 to display a connection candidate line L2 connecting the output tab of the starting element and the input tab of the element identified in step S203 (step S204). That is, on the display screen of the display device 2, connectable elements are connected by the connection candidate line L2.

[0050] The user specifies the output destination of the data by dragging from the output tab that is the starting point and then dropping onto the input tab of any element. The model design unit 13 determines whether the input specifications of the element related to the end point of the association operation match the output specifications of the element that is the starting point identified in step S201 (step S205).

[0051] If the output specifications match the input specifications (step S205: YES), the model design unit 13 records the element ID of the starting element as the source element of the ending element in the design data D (step S206). The display control unit 16 displays a connection line L1 connecting the output tab of the starting point and the input tab of the ending point on the display device 2 (step S207).

[0052] If the output specifications do not match the input specifications (step S205: NO), the model design unit 13 does not update the source element of the end element in the design data D. At this time, the display control unit 16 causes the display device 2 to display a warning that the two selected elements cannot be connected because the input specifications and output specifications do not match (step S208).

[0053] By repeating the above-described operations, the user can associate a plurality of element models E and design a combination model M for obtaining output data from input data. Fig. 10 is a diagram showing an example of a display screen when the combination model M is completed, according to the first embodiment.

[0054] <Setting a Large-Scale Language Model> As described above, the Text-to-Text type candidate model C includes a large-scale language model. The type of information output by the large-scale language model varies depending on the given prompt. Therefore, the design support device 1 accepts editing operations for the given prompt for the element model E in which the large-scale language model is implemented. The prompt includes a variable into which data output by the upstream element model E is assigned.

[0055] For example, the input unit 11 accepts a prompt editing operation by accepting a click on the input tab or output tab of an element model E in which a large-scale language model is implemented. That is, upon accepting a click on an element model E, the input unit 11 identifies the model ID of a candidate model C that implements the element model, and accepts the prompt editing operation if the candidate model C associated with the identified model ID is a large-scale language model. Furthermore, for example, the input unit 11 accepts the prompt editing operation if an association operation is performed to associate the input tab of an element model E in which a large-scale language model is implemented with the output tab of another element model E, and if the prompt does not include a variable into which output data of the other element model E is assigned. That is, the design support device 1 prompts the user to edit the prompt of the large-scale language model when the model involved in the association operation is a large-scale language model and the input data does not satisfy the acceptability condition.

[0056] FIG. 11 is a diagram showing an example of a display screen when setting a prompt in the design support device 1 according to the first embodiment. When the input unit 11 accepts a prompt editing operation, the display control unit 16 displays a prompt editing window W near the element model E in which the large-scale language model is implemented. The prompt editing window displays the prompt set for the element model E. The prompt set for the element model E is recorded in the design data D. Editing of the prompt is accepted from the input device 3 via the input unit 11. The prompt can include a variable to which output data of the upstream element model E is assigned. In the example shown in FIG. 11, the variable is represented by {}. When the prompt editing window W is closed, the model design unit 13 updates the prompt of the design data D to the input value. At this time, the model design unit 13 determines whether variables corresponding to all element models E connected upstream of the element model E related to the large-scale language model are described in the prompt. If some variables are not described in the prompt, the model design unit 13 may delete the element model related to the variable from the source element of the element model E related to the large-scale language model. If the variable corresponding to the upstream connected element model E does not match the variable described in the prompt, the display control unit 16 may cause the display device 2 to display a warning indicating that the variables do not match.

[0057] <Confirming the operation of the combination model M> When an input sample is set in the input setting button B1 and the input setting button B1 is connected to at least one element model E, the design support system 100 can use the input sample to confirm the operation of the combination model M being designed.

[0058] 12 is a flowchart showing the operation confirmation process for the combination model M according to the first embodiment. The model calculation unit 14 refers to the design data D generated by the model design unit 13 and identifies an element related to the start point of the calculation (step S301). The element related to the start point of the calculation may be, for example, an input element Ei or an arbitrary element model E designated by the user. The element related to the start point of the calculation may be designated by, for example, clicking the play button B11 or a symbol on the display screen.

[0059] The model calculation unit 14 identifies an element model En that uses the element related to the starting point as a source element (step S302). The model calculation unit 14 identifies, from the model information storage unit 12, the location of a candidate model C that implements the identified element model En, and inputs the output data of the element related to the starting point to the candidate model C (step S303). If the element related to the starting point is an input element Ei, the output data of the element related to the starting point is the input sample set in the input setting button B1. If the element related to the starting point is an element model En, the output data of the element related to the starting point is the output data obtained by executing the element model En.

[0060] The execution of the calculation by the candidate model C may be performed by the design support device 1 or another device. For example, the model calculation unit 14 may download the main body of the candidate model C from a specified location and perform calculation processing by inputting input samples into the candidate model C. Furthermore, for example, if the candidate model C is a model that runs on an external server and the main body cannot be obtained, the input samples may be sent to the external server and output data as the calculation result may be received from the external server.

[0061] The display control unit 16 places a balloon En3 displaying the data obtained in step S303 in association with the symbol En0 of the element model En identified in step S302 (step S304). If the output data is text information, a character string representing the text information is displayed in the balloon En3. If the output data is non-text information, an area for expressing the non-text information (for example, an area where a play button is placed) is displayed in the balloon En3.

[0062] 13 is a first diagram showing an example of a display screen when checking the operation of a combination model M according to the first embodiment. In the example shown in FIG. 13, the element related to the starting point is input element Ei, and the elements that use input element Ei as a source element are element model E1, element model E2, and element model E3. Below symbol E10 representing element model E1, a balloon E13 displaying output data of element model E1 is placed. Below symbol E20 representing element model E2, a balloon E23 displaying output data of element model E2 is placed. Below symbol E30 representing element model E3, a balloon E33 displaying output data of element model E3 is placed.

[0063] Next, the model calculation unit 14 refers to the design data D and determines whether or not there is another element that uses the element model En identified in step S302 as a source element (step S305). If there is another element that uses the element model En identified in step S302 as a source element, it determines whether or not that element is an output element Eo (step S306).

[0064] If there is another element model En that uses the element model En identified in step S302 as a source element (step S306: NO), the process returns to step S303, where calculations are performed on the other element model En using the data obtained in step S303. This executes calculations on each element downstream from the element related to the starting point, and the output data is displayed. FIG. 14 is a second diagram showing an example of a display screen when checking the operation of the combination model M according to the first embodiment. In the example shown in FIG. 14, a balloon E43 is placed for element model E4, which is an element downstream of element models E1, E2, and E3. In FIG. 14, a play button B22 is displayed on the output setting button B2. Clicking the play button B22 displays the output data of element model E5, which is the final output of the combination model M.

[0065] If there are no other elements that use the element model En identified in step S302 as a source element (step S305: NO), or if the other element that uses the element model En identified in step S302 as a source element is an output element Eo (step S306: YES), the design support device 1 terminates the operation confirmation process.

[0066] The user may be able to set whether or not to display the acquired execution results in association with the element model from which the execution results were obtained. Such setting may be performed in any manner. The user may perform such setting by operating the keyboard, the mouse, or the like.

[0067] For example, such settings may be configured in a program so that they can be set in a tab that appears on the screen when, for example, a user right-clicks on a mouse, or such settings may be configured in a program so that they can be set in a tab that appears on the screen when, for example, a user right-clicks on a predetermined image displayed on the screen.

[0068] Furthermore, the design support device 1 can accept a user's operation to rewrite output data of some elements. FIG. 15 is a diagram showing an example of a display screen during an operation to rewrite output data according to the first embodiment. In the example shown in FIG. 15, the user rewrites the output data of element model E1. When the output data is text information, the input unit 11 can accept a rewrite of the output data by, for example, clicking on a balloon E13 and accepting character input via a keyboard. When the output data is non-text information, the input unit 11 can rewrite the output data with data indicated by a file icon by, for example, dragging and dropping the file icon onto the balloon E13.

[0069] At this time, the design support device 1 performs the operation check process according to the procedure shown in FIG. 12 , starting from the element whose output data has been rewritten. This allows the user to check how the output of the combination model M changes when the output data is changed. As shown in FIG. 15 , output data is recalculated for element model E4, which is downstream of element model E1, and the output data is displayed in balloon E43. Furthermore, output data is recalculated for element model E5, which is downstream of element model E4, and the output data can be displayed by the play button B22 of the output setting button B2. Depending on the type of mathematical model, it is unclear how the mathematical model will behave, and tuning to obtain the desired output is not easy. Therefore, according to the design support device 1 according to the first embodiment, by allowing the output of the mathematical model to be arbitrarily changed based on the input sample, it is possible to check the change in the output of the entire combination model M without tuning.

[0070] In addition, the design support system 1 can reduce the amount of calculation by starting from the element whose output data has been rewritten and not recalculating the upstream elements and parallel elements. As shown in Figure 15, the values ​​of balloons E23 and E33 are unchanged from the example in Figure 14.

[0071] Each display screen according to the first embodiment will be described in detail below.

[0072] <Regarding the first candidate area P11>

[0073] In the example of FIG. 3, three symbols C111 to C113 are displayed in the first candidate area P11.

[0074] In the example of Figure 3, symbol C111 is a candidate symbol representing a mathematical model of speech recognition named "ASR." The mathematical model of speech recognition represented by symbol C111 is a mathematical model that accepts speech data as input and converts the speech represented by the accepted speech data into text. Therefore, the mathematical model "ASR" is an example of a non-text-to-text type mathematical model.

[0075] Therefore, the mathematical model "ASR" is an example of a first model. Because the symbol C111 indicates such a mathematical model "ASR," the symbol C111 is an example of a first candidate symbol. A "candidate symbol" refers to a symbol that represents a candidate model. Furthermore, because the symbol C111 indicates the mathematical model "ASR," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "ASR" is one of the candidate models.

[0076] In the example of Fig. 3, symbol C112 is a candidate symbol representing a mathematical model of emotion estimation named "Sent." The mathematical model "Sent" is a mathematical model that accepts voice data as input, infers the emotion of the speaker of the voice based on the voice represented by the accepted voice data, and outputs text data representing text indicating the estimated emotion. Therefore, the mathematical model "Sent" is an example of a non-text-to-text type mathematical model.

[0077] Therefore, the mathematical model "Sent" is an example of a first model. Because the symbol C112 indicates such a mathematical model "Sent," the symbol C112 is an example of a first candidate symbol. Also, because the symbol C112 indicates the mathematical model "Sent," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "Sent" is one of the candidate models.

[0078] In the example of FIG. 1 , icon C113 is a candidate icon representing a mathematical model for age estimation named "Age." The mathematical model "Age" is a mathematical model that receives voice data as input, estimates the age of the speaker of the voice based on the voice represented by the received voice data, and outputs the estimated age. A numerical value representing age is an example of text information because it does not include non-verbal information (non-verbal information is verbalized). Therefore, the mathematical model "Age" is an example of a non-text-to-text type mathematical model.

[0079] Therefore, the mathematical model "Age" is an example of a first model. Since icon C113 indicates such a mathematical model "Age," icon C113 is an example of a first candidate icon. Furthermore, since icon C113 indicates the mathematical model "Age," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "Age" is one of the candidate models. Note that the first models displayed in the example of FIG. 1 all differ in the semantic content of the information estimated from the input data. Specifically, one estimates text data, which is linguistic information contained in the voice data; one estimates emotion, which is non-linguistic information expressed in the voice data; and one estimates age, which is non-linguistic information expressed in the voice data.

[0080] On the other hand, multiple different first models may all estimate the same type of information. For example, multiple mathematical models for estimating emotions, manufactured by different manufacturers, may all be first models. This also applies to other mathematical models, such as the second model and the third model.

[0081] Although the mathematical model for speech recognition, the mathematical model for emotion estimation, and the mathematical model for age estimation have been exemplified as examples of the first model, the first model is not necessarily limited to these. For example, the first model may be a mathematical model that estimates the gender of the speaker and outputs text data indicating the estimation result. For example, the first model may be a mathematical model that estimates the nationality of the speaker and outputs text data indicating the estimation result.

[0082] That is, the first model may be a mathematical model that estimates a predetermined type of attribute for the speaker and outputs text data indicating the estimation result. In this way, the first model may be any mathematical model as long as it outputs text data indicating the predetermined type of information estimated based on input information.

[0083] <Regarding Second Candidate Area P12> In the example of FIG. 1, three icons C121 to C123 are displayed in area A102.

[0084] In the example of FIG. 1 , icon C121 is a candidate icon representing a mathematical model named "large." The mathematical model "large" is a candidate model that estimates a response corresponding to the text indicated by the text data output by the first model based on the text data. Therefore, the mathematical model "large" is an example of a second model. Therefore, icon C121 is an example of a second candidate icon.

[0085] Since the icon C121 indicates the mathematical model "large," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "large" is present as one of the candidate models.

[0086] The mathematical model "large" may be, for example, a large-scale language model (LLM) that estimates a response based on the text indicated by the input text data.

[0087] In the example of FIG. 1 , icon C122 is an icon representing a mathematical model named "middle." The mathematical model "middle" is a mathematical model different from the mathematical model "large," and is a candidate model that estimates a response corresponding to the text indicated by the text data output by the first model based on the text data. Therefore, the mathematical model "middle" is an example of a second model. Therefore, icon C122 is an example of a second candidate icon.

[0088] The difference between the "large" mathematical model and the "middle" mathematical model may be any difference, such as a difference in the size of the mathematical model or a difference in the manufacturer. Note that, for example, if the mathematical model is configured as a neural network, the difference in the size of the mathematical model is a difference in the number of nodes included in the neural network.

[0089] Since the icon C122 indicates the mathematical model "middle," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "middle" is present as one of the candidate models.

[0090] In the example of FIG. 1 , icon C123 is an icon representing a mathematical model named "small." The mathematical model "small" is a mathematical model different from the mathematical model "large" and the mathematical model "middle," and is a candidate model that estimates a response corresponding to the text indicated by the text data output by the first model based on the text data. Therefore, the mathematical model "small" is an example of a second model. Therefore, icon C123 is an example of a second candidate icon.

[0091] The difference between the mathematical model "small" and the mathematical model "large" and the mathematical model "middle" may be any difference, for example, a difference in the size of the mathematical model, or a difference in manufacturer.

[0092] Since the icon C123 indicates the mathematical model "small," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "small" is present as one of the candidate models.

[0093] <Regarding Third Candidate Region P13> In the example of FIG. 1, icons C131 and C132 are displayed in third candidate region P13.

[0094] In the example of Figure 1, icon C131 is a candidate icon indicating a mathematical model named "TTS." The mathematical model "TTS" is a candidate model (Text-to-Speech) that accepts input of text information and converts the text information into voice data, which is non-text information. Therefore, the mathematical model "TTS" is an example of a third model. Therefore, icon C131 is an example of a third candidate icon.

[0095] Since the icon C131 indicates the mathematical model "TTS," it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "TTS" is one of the candidate models.

[0096] In the example of FIG. 1 , symbol C132 is a candidate symbol representing a mathematical model named "TTI." The mathematical model "TTI" is a candidate model (Text-to-Image) that accepts input of text information and converts the text information into image data, which is non-text information. Therefore, the mathematical model "TTI" is an example of a candidate model. Therefore, symbol C132 is an example of a third symbol.

[0097] Since the symbol C132 indicates the mathematical model "TTI", it can be said that the candidate symbol in the example of FIG. 1 indicates that the mathematical model "TTI" exists as one of the candidate models.

[0098] In the example of Fig. 1, the first candidate region P11, the second candidate region P12, and the third candidate region P13 are all demarcated by frame lines, but the frame lines are shown with different line types. Therefore, in the example of Fig. 1, the first candidate region P11, the second candidate region P12, and the third candidate region P13 are displayed so that they can be distinguished from one another at least by the difference in line type.

[0099] <Effect of displaying candidate symbols in specified display locations> The first model, second model, and third model are each the mathematical models defined above, and the first model is displayed in the first candidate area P11, the second model is displayed in the second candidate area P12, and the third model is displayed in the third candidate area P13.

[0100] Therefore, the candidate area can be said to be an image that displays each candidate model in a categorized manner. Displaying multiple candidate models in a categorized manner reduces the burden on a user who constructs a combinatorial mathematical model to acquire a combinatorial mathematical model, compared to displaying multiple candidate models without categorization. Therefore, the design support device 1, which displays candidate symbols that have such effects in a predetermined display location, can reduce the burden on a user who constructs a mathematical model to acquire a mathematical model. Furthermore, the first candidate area P11, in which the candidate model C that outputs text information is located, is adjacent to the second candidate area P12, in which the candidate model C that inputs text information is located. The second candidate area P12, in which the candidate model C that outputs text information is located, is adjacent to the third candidate area P13, in which the candidate model C that inputs text information is located. This allows the user to intuitively recognize the input / output relationship of the candidate model C from the arrangement of the candidate areas on the display screen generated by the design support device 1.

[0101] Second Embodiment In the design support system 100 according to the first embodiment, a user places a symbol in the design pane P2 by moving a symbol (icon) displayed in the candidate pane P1 to any position in the design pane P2 by a drag-and-drop operation, etc. In contrast, in the design support system 100 according to the second embodiment, an area in the design pane P2 in which the symbol can be placed is determined in advance depending on the type of candidate model C represented by the symbol.

[0102] FIG. 16 is a diagram showing an example of a display screen according to the second embodiment. As shown in FIG. 16, the design support device 1 causes the display device 2 to display a display screen in which multiple design areas P21-P23 are set in the design pane P2. The design areas P21-P23 may be color-coded according to the type of component model E that can be placed. For example, the design areas P21-P23 may show designs according to the types of component models E that can be placed, thereby indicating differences in the types of component models that can be placed depending on the design. The display screen shown in FIG. 16 displays a first design area P21, a second design area P22, and a third design area P23.

[0103] An element model implementing the NonText-to-Text type candidate model C can be placed in the first design area P21. That is, when a user performs an operation to place a symbol placed in the first candidate area P11 in the design pane P2, the symbol is placed in the first design area P21 of the design pane P2. Note that, if a user performs an operation to place a symbol in the second candidate area P12 or the third candidate area P13 in the first design area P21, the model design unit 13 may not accept the operation. In this case, the display control unit 16 may display an error message on the display screen indicating that the candidate model cannot be placed. The error message indicating that the symbol cannot be placed is an example of information indicating that the symbol cannot be installed. On the other hand, in another embodiment, when a user performs an operation to place a symbol in the second candidate area P12 or the third candidate area P13 in the first design area P21, the model design unit 13 may automatically place the symbol in the corresponding design area. The element symbol placed in the first design area P21 indicates that the candidate model indicated by the first candidate symbol has been set as an element model. An element symbol is a symbol representing an element model. Therefore, the first design area P21 is an image showing an area in which element symbols indicating the mathematical model indicated by the first candidate symbol as the first candidate model can be arranged.

[0104] An element model implementing the Text-to-Text type candidate model C can be placed in the second design area P22. That is, when a user performs an operation to place a symbol placed in the second candidate area P12 in the design pane P2, the symbol is placed in the second design area P22 of the design pane P2. Note that if a user attempts to place a symbol in the first candidate area P11 or the third candidate area P13 in the second design area P22, the model design unit 13 may not accept the operation. In this case, the display control unit 16 may display an error message on the display screen indicating that the candidate model cannot be placed. The element symbol placed in the second design area P22 indicates that the candidate model indicated by the second candidate symbol has been set as the element model. Therefore, the second design area P22 is an image showing an area in which an element symbol representing the mathematical model indicated by the second candidate symbol as the second model can be placed.

[0105] An element model implementing the Text-to-NonText type candidate model C can be placed in the third design area P23. That is, when a user performs an operation to place a symbol placed in the third candidate area P13 in the design pane P2, the symbol is placed in the third design area P23 in the design pane P2. Note that if a user attempts to place a symbol in the first candidate area P11 or the second candidate area P12 in the third design area P23, the model design unit 13 may not accept the operation. In this case, the display control unit 16 may display an error message on the display screen indicating that the candidate model cannot be placed. The element symbol placed in the third design area P23 indicates that the candidate model indicated by the third candidate symbol has been set as the element model. Therefore, the third design area P23 is an image showing the possible area for an element symbol representing the mathematical model indicated by the third candidate symbol as the third model.

[0106] As described above, when the candidate regions P11-P13 are color-coded according to the type of candidate model, the design regions P21-P23 may be colored the same color as the corresponding candidate regions P11-P13.

[0107] As described above, according to the design support system 100 of the second embodiment, possible placement positions are predetermined and displayed according to the type of candidate model, so that after placing an element symbol, the user can easily understand the role of the element model indicated by the element symbol. Furthermore, the user can easily recognize that an element symbol placed in a certain candidate area can be connected to an element symbol placed in another candidate area adjacent to the candidate area, i.e., that the input / output conditions are met. Furthermore, since the user can design the combination model M while checking the role of the mathematical model indicated by the element symbol, the design support system 100 can reduce the design burden on the user.

[0108] Third Embodiment The design support system 100 according to the first embodiment creates a combination model M by combining mathematical models whose information is recorded in advance in the model information storage unit 12. In contrast, the design support system 100 according to the third embodiment incorporates an arbitrary model into a candidate model C in response to a user operation.

[0109] 17 is a diagram showing an example of a display screen according to the third embodiment. Model addition buttons P111-P131 are arranged in each of the candidate areas P11-P13 on the display screen according to the third embodiment.

[0110] The add model button P111 is located in the first candidate area P11. When the user selects the add model button P111 by operating the keyboard, mouse, or the like, the display control unit 16 displays a list of mathematical models that can be set as the first model in a predetermined display location. The list may be displayed, for example, as a pop-up. Operating the add model button P111 in the first candidate area P11 displays a list of mathematical models that can be set as the first model. This allows the user to easily select a candidate model that can be added to the first candidate area P11, even if they are not familiar with the processing of mathematical models.

[0111] When the user selects some or all of the mathematical models displayed in the list, the data acquisition unit 15 records information about the selected mathematical models in the model information storage unit 12. Furthermore, the display control unit 16 displays a first candidate symbol indicating the added first model in the first candidate area P11 in accordance with the information stored in the model information storage unit 12. Note that the user's selection is performed by a user operation such as keyboard operation or mouse operation.

[0112] The add model button P121 is arranged in the second candidate area P12. When the user selects the add model button P121, the display control unit 16 displays a list of mathematical models that can be set as the second model on a predetermined display destination.

[0113] When the user selects some or all of the mathematical models displayed in the list, the data acquisition unit 15 records information about the selected mathematical models in the model information storage unit 12. Furthermore, the display control unit 16 displays a second candidate symbol indicating the added second model in the second candidate area P12 in accordance with the information stored in the model information storage unit 12.

[0114] The add model button P131 is arranged in the third candidate area P13. When the user selects the add model button P131, the display control unit 16 displays a list of mathematical models that can be set as the third model on a predetermined display destination.

[0115] When the user selects some or all of the mathematical models displayed in the list, the data acquisition unit 15 records information about the selected mathematical models in the model information storage unit 12. In addition, the display control unit 16 displays a third candidate symbol indicating the added third model in the third candidate area P13.

[0116] In the third embodiment, when the Add Model button P121 is pressed, a list of configurable mathematical models is displayed, and the user is allowed to select a candidate model to add from the list. However, this is not limited to this. For example, a design support device 1 according to another embodiment may display a file selection dialog when the Add Model button P121 is pressed, and allow the user to input the path of the program file of the candidate model to add. The path of the program file is not limited to a path on the local system, and may be represented by a URL. This allows the design support device 1 to add candidate models with a high degree of freedom when the user is familiar with processing mathematical models.

[0117] Fourth Embodiment In the design support device 1 according to the first to third embodiments, each element model En handles character strings (text) as text information. On the other hand, a mathematical model that handles text information can handle not only character strings but also intermediate feature quantities such as vectors that represent text information. The design support device 1 according to the fourth embodiment has a function to switch the text information handled by the mathematical model between character strings and intermediate feature quantities.

[0118] In order to handle text information as intermediate features between two mathematical models, the dimensions and meanings of the intermediate features handled by the two mathematical models must match. Therefore, the candidate model C in the design support device 1 according to the fourth embodiment is designed so that the dimensions of all intermediate features and the types of data (character strings, still images, moving images, audio, etc.) assumed to be expressed by the intermediate features are common to other connectable candidate models C. It is desirable that the two mathematical models that input and output intermediate features are trained to interpret similar intermediate features in the same way.

[0119] 18 is a diagram showing an example of a display screen according to the fourth embodiment. The display control unit 16 according to the fourth embodiment places a switch En4 on the display screen, associated with an element model En, for switching between using a character string or an intermediate feature as text information used in calculation. The switch En4 can be placed at least one of between the first symbol and the second symbol and between the second symbol and the third symbol. The input unit 11 accepts a user's operation to switch the switch En4. The design data D according to the fourth embodiment holds the value of the switch En4, i.e., information indicating whether to use a character string or an intermediate feature as text information.

[0120] If the value of switch En4 indicates that a character string is to be used, the model calculation unit 14 outputs the character string to the candidate model C implemented by the upstream element model En, and inputs the character string to the candidate model C implemented by the downstream element model En.

[0121] On the other hand, when the value of switch En4 indicates that an intermediate feature is to be used, the model calculation unit 14 causes the candidate model C implemented by the upstream element model En to output the intermediate feature and input the intermediate feature to the candidate model C implemented by the downstream element model En. At this time, the model calculation unit 14 converts the intermediate feature output from the element model En into a character string using a decoder or the like corresponding to the candidate model C. The display control unit 16 displays the character string converted from the intermediate feature in the balloon En3 of the element model En. The character string converted from the intermediate feature does not necessarily match the meaning expressed by the intermediate feature, but is in a format that allows the user to understand its meaning. Thus, the design support device 1 according to the fourth embodiment can allow the user to understand the content of the processing of each element model En during operation check, even when intermediate features are used as input / output data between element models En.

[0122] Fifth Embodiment The design support device 1 according to the fourth embodiment connects element models E using intermediate features by standardizing the dimensions of intermediate features of all candidate models C and the types of data assumed to be represented by the intermediate features. In contrast, the design support device 1 according to the fifth embodiment enables designing a combination model M using intermediate features by using candidate models C that do not share common intermediate features. The design support device 1 according to the fifth embodiment stores, in the model information storage unit 12, information on a pre-designed combination model consisting of multiple element models En, in addition to information on the candidate models C. The information on the pre-designed combination model includes, for example, the model IDs of each of the multiple element models constituting the combination model. The pre-designed combination model is designed so that data exchange between element models En is performed using intermediate features rather than character strings. The pre-designed combination model is a model trained to have the same input / output relationship as a combination model that uses text information for input / output between element models E. For example, the pre-designed combination model may be a model obtained by distillation learning using a combination model that uses text information for input / output between element models E as a teacher model. The designed combination model is a model equivalent to a combination model that uses text information for input and output between element models E.

[0123] 18, the design support device 1 according to the fifth embodiment generates a display screen including a switch E44, as in the fourth embodiment. However, the display control unit 16 allows the switch E44 to be operated only when the combination model M configured in the design pane P2 matches a combination model that has already been designed and is recorded in the model information storage unit 12. When the combination model M configured in the design pane P2 does not match any of the combination models that have already been designed and are recorded in the model information storage unit 12, the display control unit 16 fixes the value of the switch E44 to a character string and does not accept any operation.

[0124] When the value of switch E44 is an intermediate feature, the model calculation unit 14 obtains output data by inputting an input sample to a designed combination model recorded in the model information storage unit 12, instead of the combination model M configured in the design pane P2. The model calculation unit 14 acquires intermediate features output by each element model En of the designed combination model and converts the intermediate features into character strings using an encoder or the like. The display control unit 16 displays the character string converted from the intermediate features in the balloon En3 of the element model En, as shown in FIG. 18 . In this way, the design support device 1 according to the fifth embodiment can enable the user to understand the content of the processing of each element model En during operation check, even when intermediate features are used as input / output data between element models En.

[0125] (Other Embodiments) Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and also includes designs within the scope that do not deviate from the gist of the present invention.

[0126] For example, the design support device 1 according to the above-described embodiment displays the first candidate region P11, the second candidate region P12, and the third candidate region P13 in the candidate pane P1, but this is not limited to this. For example, if the combination model M designed by the design support device 1 according to another embodiment is a model that generates text information from non-text information, the design support device 1 may not need to display the third candidate region P13. Similarly, if the combination model M designed by the design support device 1 according to another embodiment is a model that generates non-text information from text information, the design support device 1 may not need to display the first candidate region P11. Furthermore, if only a specific model is available as a Text-to-Text model, the design support device 1 may initially place an element icon representing the Text-to-Text model in the design area, and may not need to display the second candidate region P12.

[0127] Furthermore, for example, a design support device 1 according to another embodiment may display candidate areas other than the first candidate area P11, the second candidate area P12, and the third candidate area P13 in the candidate pane P1. FIG. 19 illustrates an example of a display screen according to a sixth embodiment. For example, the design support device 1 according to the sixth embodiment may display a post-processing candidate area P14 for displaying a post-processing model that performs post-processing on text information. Since the post-processing model is an example of a second model that generates text information from text information, the post-processing candidate area P14 can also be considered an example of a second candidate area. In FIG. 19 , a candidate symbol C141 representing an “error correction” model that receives text information obtained by speech recognition and outputs text information with speech recognition errors corrected is placed in the post-processing candidate area P14. An element symbol E60 representing an element model E6 that implements the “error correction” model is placed in the design pane P2. A balloon E63 associated with the element symbol E60 displays corrected text information output by an element model E1 that implements an “ASR” model connected upstream. In another embodiment, the post-processing candidate region P14 may be located between the second candidate region P12 and the third candidate region P13. Also, in another embodiment, the candidate symbol C141 representing the "error correction" model, which is a text-to-text model, may be located in the second candidate region P12.

[0128] The candidate symbols displayed in the post-processing candidate area P14 may be added by the user. Examples of post-processing models, which are mathematical models indicated by the candidate symbols displayed in the post-processing candidate area P14, include a mathematical model that checks the appropriateness of output based on a rule, a mathematical model that performs normalization processing, and a mathematical model for grounding.

[0129] Furthermore, the design support device 1 according to another embodiment may display a preprocessing candidate area for displaying a preprocessing model that performs preprocessing of non-text information. The preprocessing model is a model that generates non-text information from non-text information. The preprocessing candidate area may be located above the first candidate area P11, for example. Examples of the preprocessing model include a voice conversion model that converts voice into a predetermined voice, and a voice activity detection model that detects and extracts voice activity from input voice.

[0130] As shown in FIG. 15 , the design support device 1 according to the above-described embodiment accepts rewriting of the output data of the element model En, enabling operation confirmation through the rewriting. However, the design support device 1 according to other embodiments is not limited to this. For example, the design support device 1 according to the seventh embodiment may include, as the candidate model C, a “stub” that does not accept data input but supplies fixed output data. FIG. 20 is a diagram illustrating an example of a display screen according to the seventh embodiment. For example, the design support device 1 according to the seventh embodiment places a candidate model C141 related to the “stub” in the first candidate area P11 on the display screen. In the example shown in FIG. 20 , the user places an element model E6 that implements the “stub” in the design pane P2 and inputs the output data of the element model E6 to the element model E4 instead of the output data of the element model E1. This allows the user to check the operation of the design support device 1 by rewriting the output data of the “stub” element model E6. Note that, although the “stub” is used as a temporary element for operation confirmation in the example shown in FIG. 20 , the “stub” may also be used as the final element of the combination model M. For example, a "stub" may be incorporated into the combination model M as a component that outputs a fixed value such as a setting value.

[0131] In the examples of FIGS. 2 to 7, the position at which each symbol is displayed may be designated by the user, or may be determined according to a predetermined rule.

[0132] In the above-described embodiment, the output of a first model does not become the input of another first model. On the other hand, in other embodiments, the output of a first model may become the input of another first model. For example, when a first model accepts input of non-text information (e.g., voice data) and text information, or when a first model outputs non-text information and text information, the output data of the first model can be input to another first model. Specifically, when a first model that processes voice accepts text information as reference information in addition to voice data, the first model can accept the text information output by the other first model as input data.

[0133] In other embodiments, the output of a second model may be the input of another second model. The output of a third model may be the input of another third model. For example, if a third model accepts inputs of text information and non-text information (e.g., audio data) or outputs text information and non-text information, the output data of the third model may be input to another third model.

[0134] In another embodiment, the candidate symbols and symbols representing element models may be changeable by the user. Changing the symbols may involve, for example, changing the color of the symbols or modifying their shape. Furthermore, the color, line type, and shape of the frame representing the area image may also be changeable by the user.

[0135] <Template Technology> In the design support device 1 according to another embodiment, the placement of element models does not necessarily have to start from the state of screen H2 in Fig. 2 or screen H3 in Fig. 3. For example, in another embodiment, a template in which one or more element symbols have already been placed may be recorded in advance in the model information storage unit 12, etc. In such a case, the user can easily design a desired combinatorial mathematical model by changing the template.

[0136] Such a template may be created by the user, and the template created by the user may be recorded in the model information storage unit 12 or the like.

[0137] The design support device 1 according to the embodiment described above displays candidate symbols in a plurality of candidate areas in the candidate pane P1, but is not limited to this. For example, the design support device 1 according to another embodiment can reduce the burden required for designing a mathematical model by performing the connection possibility determination process shown in FIG. 8 or 9.

[0138] 21 is a diagram showing an example of a screen displayed by a predetermined display destination in a modified example. An image G103 is displayed on the display screen as an example of a candidate symbol. The image G103 is displayed when an image G405 labeled "Add Condition" is clicked. Note that the image G103 may be displayed at all times. The image G103 corresponds to the candidate pane P1 according to the embodiment described above.

[0139] 21, five first candidate symbols, "ASR", "Sent", "Age", "Gen", and "Whis", are displayed in the candidate area. In screen H14, three second candidate symbols, "large", "middle", and "small", are displayed in the candidate area.

[0140] The display screen shown in FIG. 21 has areas A301 and A303. Areas A301 and A303 are shown in different colors. Area A301 is an area where setting information for the NonText-to-Text model is displayed. In other words, area A301 corresponds to the first design area P21 in the embodiment described above. An icon C401 is displayed in area A301. Like the input setting button B1 in the embodiment described above, icon C401 accepts uploading of sample data to be input to the combinatorial mathematical model when pressed.

[0141] Also displayed in area A301 is image G401. Image G401 is a text box for displaying text estimated by the first model "ASR" set as the element model based on the voice data indicated by symbol C401. In the example of Fig. 21, the text is "I had a fight with a friend today. What kind of music should I listen to at a time like this?"

[0142] Also displayed in area A301 is image G402. Image G402 shows text indicating speaker attributes estimated by the first model "Gen" set as the component model based on the speech data indicated by icon C401. In the example of FIG. 21 , the text is "child."

[0143] Also displayed in area A301 is image G403. Image G403 shows text indicating the speaker's emotion estimated by the first model "Sent" set as the element model based on the speech data indicated by icon C401. In the example of FIG. 21 , the text is "sadness." Image G404 shows an example of an arbitrary candidate model selected and added from image G103. In this way, arbitrary candidate models can be added to area A301.

[0144] Images G401, G402, G403, and G404 all correspond to symbol En0, which represents an element model that implements the first model according to the embodiment described above. In other words, a text box for displaying text information that represents the analysis results of audio data, which is non-text information, can be said to be a symbol that represents the first model. Furthermore, images G401, G402, G403, and G404 all correspond to balloon En3, which displays the output data of the first model according to the embodiment described above.

[0145] Also displayed in area A301 is an image G405 labeled "Add Condition." When the user selects this image G405 by operating the mouse or the like, image G103 pops up, and the user can add a first model that performs estimation based on the voice data indicated by icon C401. The added first model performs estimation based on the voice data indicated by icon C401. Text indicating the results of the estimation is displayed in area A301.

[0146] At the boundary between area A301 and area A303, image G406, which reads "Have LLM generate a response," is displayed. Image G406 indicates that a second model set as the component model performs estimation based on the results displayed in area A301. In the example of FIG. 21 , as shown in image G406, the second model set as the component model is an LLM (large-scale language model). Note that in this modification, an initial model is selected in advance, and the model calculation unit 14 executes processing using this initial model. In this modification, since there is only one LLM as the second model, it is obvious that all outputs of the first model are input to the LLM, and all outputs of the LLM are input to a single third model (TTS). Therefore, image G406 representing the second model does not need to be connected by connection line L1 to images G401, G402, G403, and G404 representing the first model, and image G407 representing the third model. When image G406 is pressed, image G103 pops up, allowing the user to select the type of LLM. By selecting an LLM in image G103, the user can change the type of LLM to be incorporated into the combinatorial mathematical model.

[0147] Image G407 is displayed in area A303. Image G407 shows, in text, the estimated response of the large-scale language model set as the component model.

[0148] Image G408 is displayed in area A303. Image G408 is an image showing the tone of voice when converting a response estimated by a large-scale language model into speech. Image G409 is an image showing the speaker when converting a response estimated by a large-scale language model into speech. For example, if the response estimated by a large-scale language model is output in a female voice, image G409 displays the text "female." Parameters other than the response, such as tone of voice and speaker, to be passed to the third model may be generated by the second model, which is a large-scale language model, or may be directly input by the user. Parameters directly input by the user may be displayed in area A303 as, for example, a text box representing a stub according to the above-described embodiment. When estimating parameters using the second model, it is necessary to set a prompt in advance to output the parameter value.

[0149] Also displayed in area A303 is an image G410 labeled "Add Condition." The user can add a condition to be included in the prompt for the large-scale language model by selecting this image G410 using a mouse or the like. In other words, the images (images G408, G409, etc.) added by selecting image G410 correspond to the prompt edit window W according to the above-described embodiment. Parameters such as the condition may be determined by the second model, which is the large-scale language model, or may be input directly by the user. The parameters input directly by the user may be displayed in area A303 as, for example, a text box representing a stub according to the above-described embodiment.

[0150] An icon C402 is displayed in area A303. Icon C402 indicates a third model set as an element model. In other words, icon C402 corresponds to symbol En0, which represents an element model implementing the third model according to the above-described embodiment. Note that in this modified example, the Text-to-NonText model is a predetermined TTS model, and therefore the design support system 1 does not accept the addition of a third model using image G103. Icon C402 can be selected by the user using a mouse or the like, and when icon C402 is selected, audio data that is the execution result of the element model indicated by icon C402 is displayed. In other words, icon C402 corresponds to play button B22 according to the above-described embodiment.

[0151] When the element model indicated by icon C402 is executed, audio data of a voice speaking the text indicated by image G407 is generated by the element model indicated by symbol 402 under the conditions displayed in area A303, such as images G408 and G409. The generated audio data may be output via a speaker or the like. In this modification, as shown in FIG. 21 , the display (design area) related to the model that converts text information into audio data is omitted. In this way, when there is only one candidate model for each of the first model, second model, and third model, the design support device 1 may omit displaying the candidate model.

[0152] <Examples of Candidate Models> The first model may be, for example, a mathematical model of language identification. Voice data is input to the mathematical model of language identification. The first model may be a mathematical model of comprehension estimation. For example, data on spoken voice and data on a face image of the speaker are input to the mathematical model of comprehension estimation. The mathematical model of comprehension estimation outputs a level of comprehension for the response information.

[0153] The first model may be a mathematical model for speaker identification. For example, speech data and face image data of the speaker are input to the mathematical model for speaker identification. The mathematical model for speaker identification outputs a speaker label.

[0154] The first model may be a mathematical model for estimating a physical condition. The mathematical model for estimating a physical condition may include, for example, input data indicating the content of an utterance, data indicating the rate of utterance, data indicating the tone of voice of the utterance, information on the speaker's cough and facial color, and information on the speaker's movements. The mathematical model for estimating a physical condition outputs text data indicating the physical condition.

[0155] The first model may be a mathematical model for noise estimation. For example, audio data, image data, and location information are input to the mathematical model for noise estimation. The mathematical model for noise estimation outputs a noise label (e.g., “STATION”, “STREET”, etc.) and enhanced audio data (having the same size and format as the input audio data).

[0156] The second model may be, for example, a mathematical model for generating a summary. The mathematical model for generating a summary receives text data output by, for example, an ASR (Automatic Speech Recognition) model. The mathematical model for generating a summary outputs text data representing a summary of the input text information.

[0157] The second model may be, for example, a mathematical model of test execution, which receives text data output by an ASR (audio recognition) model, for example, and outputs a unit test (UT) item and an execution result.

[0158] The second model may be, for example, a mathematical model of translation. The mathematical model of translation receives, for example, text data output by an ASR (audio recognition) model and information indicating the language into which the text is to be translated. The mathematical model of translation may also receive, for example, information indicating the language into which the text is to be translated. The mathematical model of translation outputs text data indicating the text of the translation result.

[0159] Note that these second models can also receive text data output by another second model (for example, an LLM (Large Scale Language Model)).

[0160] The third model may be, for example, a mathematical model that accepts input of text information and generates sounds such as music or natural sounds from the input text information, a mathematical model that generates images or videos from the input text information, or a mathematical model that generates spaces / objects from the input text information.

[0161] Note that estimation by each element model may use pre-set personal information of the user such as age and gender, past history, information on the user's pathology, information on the user's hobbies and interests, information indicating the user's situation, etc. These can be realized, for example, by using the stub according to the seventh embodiment shown in FIG.

[0162] The design support system 100 and the design support device 1 may each be implemented using a plurality of information processing devices communicably connected via a network. In this case, each process executed by the design support device 1 may be distributed and executed by the plurality of information processing devices.

[0163] <Example of Hardware Configuration> Fig. 22 is a diagram illustrating an example of the hardware configuration of the design support device 1 according to at least one embodiment. The design support device 1 includes a control unit 51, an interface unit 52, and a storage unit 53. The control unit 51 includes a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus. In the design support device 1, the processor 91 reads a program stored in the storage unit 53 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the design support device 1 to function as a device including the processing units illustrated in Fig. 1. The processor 91 also allocates a storage area for the model information storage unit 12 in the storage unit 53.

[0164] The control unit 51 controls, for example, the operation of each functional unit included in the design support device 1. The control unit 51 acquires, for example, information stored in the storage unit 53. Specifically, the process of acquiring information stored in the storage unit 53 is reading. The control unit 51 controls, for example, a predetermined display destination.

[0165] The interface unit 52 includes a communication interface for connecting the design support device 1 to an external device. The interface unit 52 communicates with the external device via a wired or wireless connection. The external device is, for example, a predetermined display destination. In such a case, the control unit 51 controls the predetermined display destination via the communication interface provided in the interface unit 52.

[0166] The interface unit 52 may be configured to include an input device 3 such as a mouse, keyboard, or touch panel. The interface unit 52 may be configured as an interface that connects these input devices 3 to the design support apparatus 1. In this way, the input device of the interface unit 52 accepts input of various information to the design support apparatus 1 via a wired or wireless connection. Note that signals or information do not necessarily need to be input to the communication interface of the interface unit 52, and may also be input to the input device 3 of the interface unit 52.

[0167] The interface unit 52 may be configured to include, for example, a display device 2 such as a CRT display, a liquid crystal display, or an organic EL display, and an output device such as a speaker that outputs various signals or information. The interface unit 52 may be configured as an interface that connects these output devices to the design support device 1. Therefore, the interface unit 52 may output, for example, information input to the input device 3 of the interface unit 52 as an image or sound. When the interface unit 52 includes the display device 2, the predetermined display destination may be the display device 2 included in the interface unit 52.

[0168] The storage unit 53 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 53 stores various information related to the design support device 1. The storage unit 53 stores various information generated by the operation of the control unit 51, for example. The storage unit 53 may exist on a cloud, for example.

[0169] Note that all or part of the functions of the design support system 100 and the design support device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line. The program may be for realizing part of the functions to be performed by the control unit 51. For example, the program may be combined with other programs already stored in the storage unit 53 or other programs implemented in other devices to perform the functions. The control unit 51 may be virtualized on one or more computers.

[0170] The following additional notes are provided regarding the above-described embodiments.

[0171] (Supplementary Item 1) An apparatus comprising at least one processor, the at least one processor being configured to cause a display device to display a plurality of candidate areas in which a plurality of symbols are placed, the plurality of symbols representing candidate models capable of constituting a non-text processing model, the plurality of candidate areas including at least two of the following candidate areas: a first candidate area in which one or more first symbols representing a first model that accepts input of non-text information and outputs text information corresponding to the input non-text information are placed; a second candidate area in which one or more second symbols representing a second model that accepts input of text information and converts the input text information into text information according to a predetermined purpose are placed; and a third candidate area in which one or more third symbols representing a third model that accepts input of text information and outputs non-text information corresponding to the input text information are placed. (Supplementary Item 2) The device according to Supplementary Item 1, wherein the at least one processor is further configured to cause the display device to display a plurality of design areas in which symbols related to a model selected by a user can be placed, and the plurality of installation areas include at least two of: a first design area in which a first symbol selected by the user from among the one or more first symbols can be placed, a second design area in which a second symbol selected by the user from among the one or more second symbols can be placed, and a third design area in which a third symbol selected by the user from among the one or more third symbols can be placed. (Supplementary Item 3) The device according to Supplementary Item 1, wherein the first candidate area and the second candidate area are placed adjacent to each other, or the second candidate area and the third candidate area are placed adjacent to each other. (Supplementary Item 4) In the device described in Supplementary Item 2, the first candidate area or the first symbol selected by the user and the first design area are displayed in a manner that clearly indicates that they are related to each other, the second candidate area or the second symbol selected by the user and the second design area are displayed in a manner that clearly indicates that they are related to each other, or the third candidate area or the third symbol selected by the user and the third design area are displayed in a manner that clearly indicates that they are related to each other.(Supplementary Item 5) The device described in Supplementary Item 2, wherein the at least one processor is configured as follows: accept a placement instruction to place a symbol selected by the user in a placement destination area that is one of the plurality of design areas, place the symbol selected by the user in the placement destination area if the placement destination area is a design area in which the symbol selected by the user can be placed, and display on the display device information indicating that the symbol cannot be placed in the placement destination area if the placement destination area is not a design area in which the symbol selected by the user can be placed. (Supplementary Item 6) The device described in Supplementary Item 5, wherein the at least one processor is configured to cause the display device to display a connection image connecting two symbols placed in the plurality of design areas. (Supplementary Item 7) In the device described in Supplementary Item 6, the connection image connecting two symbols is either a first connection image indicating that data generated by the model indicated by the upstream symbol of the two symbols is input to the model indicated by the downstream symbol of the two symbols, or a second connection image indicating that the type of data generated by the model indicated by the upstream symbol of the two symbols is a type that can be input to the model indicated by the downstream symbol of the two symbols, and the first connection image and the second connection image are displayed in different ways.(Supplementary Item 8) The device according to Supplementary Item 5, wherein the at least one processor is configured to: generate first output data by inputting an input sample that is non-text information into the first model represented by the first symbol placed in the first design area, generate second output data by inputting the first output data into the second model represented by the second symbol placed in the second design area, generate third output data by inputting the second output data into the third model represented by the third symbol placed in the third design area, display the first output data on the display device in association with the first symbol, display the second output data on the display device in association with the second symbol, and display on the display device an expression area for representing the third output data in association with the third symbol. (Supplementary Item 9) The device according to Supplementary Item 8, wherein the at least one processor is configured to, when the first output data or the second output data is a feature representing text, cause the display device to display a character string representing the first output data or the second output data. (Supplementary Item 10) The device according to Supplementary Item 8, wherein the at least one processor is configured as follows: to accept an operation specifying a symbol to be placed in any of the plurality of design areas, output data associated with the symbol, or a connection relationship between symbols to be placed in the plurality of design areas, to cause the model indicated by a symbol downstream of the target of the operation to generate output data, and to update and display the output data associated with the symbol downstream of the target of the operation. (Supplementary Item 11) An device, comprising at least one processor, wherein the at least one processor is configured as follows: to accept an instruction to combine a first model that is one of a plurality of models constituting the combination model with input information to be input to the first model, and, when the input information does not satisfy an acceptability condition for the first model, to display on a display device that the acceptability condition is not satisfied.(Supplementary Item 12) The device according to Supplementary Item 11, wherein the input information is output data of a second model that is one of a plurality of models that constitute the combination model, and the instruction is an instruction that specifies the first model and the second model. (Supplementary Item 13) The device according to Supplementary Item 11, wherein the plurality of models include a large-scale language model that generates text information based on a prompt that is text information, and the at least one processor is configured to prompt editing of the prompt of the large-scale language model when the model related to the instruction or a model that generated the input information related to the instruction is the large-scale language model, and the input information does not satisfy the acceptance condition. (Supplementary Item 14) An device for designing a combination model composed of a plurality of models, comprising at least one processor, wherein the combination model includes a first model and a second model that inputs output data of the first model, and the at least one processor is configured to cause a display device to display options that accept a selection of whether the input data of the second model is internal features of the first model or label data obtained by converting the internal features. (Supplementary Item 15) The device according to Supplementary Item 14, wherein the at least one processor is configured as follows: when an option that sets the input data of the second model as label data is selected, acquire internal features by inputting input samples to the first model, convert the internal features into label data, acquire first output data that is output data of the second model by inputting the label data to the second model, and display the label data and the first output data on the display device; when an option that sets the input data of the second model as internal features is selected, acquire internal features by inputting input samples to the first model, acquire second output data that is output data of the second model by inputting the internal features to the second model, convert the internal features into label data, and display the label data and the second output data on the display device.(Supplementary Item 16) The device according to Supplementary Item 14, wherein the at least one processor is configured as follows: when an option that uses the input data of the second model as label data is selected, acquire internal features by inputting input samples to the first model, convert the internal features into label data, acquire first output data that is output data of the second model by inputting the label data to the second model, and display the label data and the first output data on the display device; when an option that uses the input data of the second model as internal features is selected, acquire internal features by inputting input samples to the first model, convert the internal features into label data, acquire third output data that is output data by inputting the input samples to a third model that has functions equivalent to a combination of the first model and the second model, and display the label data and the third output data on the display device. (Supplementary Item 17) A method, comprising: a computer, in accordance with a program, displaying a plurality of candidate areas in which symbols representing candidate models constituting the non-text processing model are arranged, the plurality of candidate areas including at least two of: a first candidate area in which one or more first symbols representing a first model that accepts input of non-text information and outputs text information corresponding to the input non-text information are arranged, a second candidate area in which one or more second symbols representing a second model that accepts input of text information and converts the input text information into text information according to a predetermined purpose are arranged, and a third candidate area in which one or more third symbols representing a third model that accepts input of text information and outputs non-text information corresponding to the input text information. (Supplementary Item 18) A non-transitory recording medium having recorded thereon a program for causing a computer to function as the device according to any one of Supplementary Items 1 to 16.

[0172] 100... design support system, 1... design support device, 51... control unit, 52... interface unit, 53... storage unit, 91... processor, 92... memory

Claims

1. A design support device for designing a combination model composed of multiple models, the design support device displaying multiple candidate areas in which symbols representing candidate models that constitute the combination model are placed, the multiple candidate areas including at least two of the following candidate areas: a first candidate area in which one or more first symbols representing a first model that accepts input of non-text information and outputs text information corresponding to the input non-text information are placed; a second candidate area in which one or more second symbols representing a second model that accepts input of text information and converts the input text information into text information suitable for a predetermined purpose are placed; and a third candidate area in which one or more third symbols representing a third model that accepts input of text information and outputs non-text information corresponding to the input text information are placed.

2. A design support device as described in claim 1, which displays a plurality of design areas in which symbols related to a model selected by a user can be placed, said plurality of design areas including at least two of: a first design area in which a first symbol selected by the user from said one or more first symbols can be placed; a second design area in which a second symbol selected by the user from said one or more second symbols can be placed; and a third design area in which a third symbol selected by the user from said one or more tertiary symbols can be placed.

3. The design support device according to claim 1, wherein the first candidate area and the second candidate area are arranged adjacent to each other, or the second candidate area and the third candidate area are arranged adjacent to each other.

4. The design support device according to claim 2, wherein the first candidate area or the first symbol selected by the user and the first design area are displayed in a manner that clearly indicates that they are related to each other; the second candidate area or the second symbol selected by the user and the second design area are displayed in a manner that clearly indicates that they are related to each other; or the third candidate area or the third symbol selected by the user and the third design area are displayed in a manner that clearly indicates that they are related to each other.

5. The design support device according to claim 2, wherein the device receives a placement instruction to place the symbol selected by the user in a placement destination area which is one of the plurality of design areas, places the symbol in the placement destination area if the placement destination area is a design area in which the symbol selected by the user can be placed, and displays information indicating that the symbol cannot be placed in the placement destination area if the placement destination area is not a design area in which the symbol selected by the user can be placed.

6. The design support device according to claim 5, wherein a connection image is displayed that connects two symbols arranged in the plurality of design areas.

7. The design support device according to claim 6, wherein it is determined whether two symbols arranged in said plurality of design areas are connectable to each other, and if connectable, it displays said connection image.

8. The design support device according to claim 6, wherein the connection image connecting two symbols is either a first connection image indicating that data generated by a model indicated by the upstream symbol of the two symbols is input to a model indicated by the downstream symbol of the two symbols, or a second connection image indicating that the type of data generated by a model indicated by the upstream symbol of the two symbols is a type that can be input to a model indicated by the downstream symbol of the two symbols, and wherein the first connection image and the second connection image are displayed in different manners.

9. The design support device according to claim 5, further comprising: generating first output data by inputting an input sample, which is non-text information, into the first model represented by the first symbol placed in the first design area; generating second output data by inputting the first output data into the second model represented by the second symbol placed in the second design area; generating third output data by inputting the second output data into the third model represented by the third symbol placed in the third design area; displaying the first output data in association with the first symbol; displaying the second output data in association with the second symbol; and displaying an expression area representing the third output data in association with the third symbol.

10. The design support device according to claim 9, wherein, when the first output data or the second output data is a feature representing text, a character string representing the first output data or the second output data is displayed.

11. A design support device according to claim 9, which accepts an operation targeting at least one of a symbol placed in any of the plurality of design areas, output data associated with the symbol, and a connection relationship between symbols placed in the plurality of design areas, causes the model indicated by a symbol downstream from the target of the operation to generate output data, and updates and displays the output data associated with the symbol downstream from the target of the operation.

12. A design support device for designing a combination model composed of multiple models, the design support device receiving an instruction to combine a first model, which is one of the multiple models that make up the combination model, with input information to be input to the first model, and when the input information does not satisfy an acceptability condition for the first model, displaying that the acceptability condition is not satisfied.

13. The design support device according to claim 12, wherein the input information is output data of a second model, which is one of a plurality of models that make up the combination model, and the instruction is an instruction that specifies the first model and the second model.

14. The design support device according to claim 12, wherein the plurality of models include a large-scale language model that generates text information based on a prompt that is text information, and when the model related to the instruction or the model from which the input information related to the instruction was generated is the large-scale language model and the input information does not satisfy the acceptance conditions, the device prompts the user to edit the prompt of the large-scale language model.

15. A design support device for designing a combination model composed of multiple models, wherein the combination model includes a first model and a second model that inputs output data of the first model, and the design support device displays options for accepting a selection of whether the input data of the second model should be internal feature values ​​of the first model or label data converted from the internal feature values.

16. The design support device according to claim 15, wherein, when an option is selected that sets the input data of the second model as label data, an input sample is input into the first model to obtain label data, the label data is input into the second model to obtain first output data that is output data of the second model, and the label data and the first output data are displayed; and when an option is selected that sets the input data of the second model as internal features, an input sample is input into the first model to obtain internal features, the internal features are input into the second model to obtain second output data that is output data of the second model, the internal features are converted into label data, and the label data and the second output data are displayed.

17. The design support device according to claim 15, wherein, when an option is selected that sets the input data of the second model as label data, an input sample is input into the first model to obtain label data, the label data is input into the second model to obtain first output data that is output data of the second model, and the label data and the first output data are displayed; and when an option is selected that sets the input data of the second model as an internal feature, an input sample is input into the first model to obtain label data, the input sample is input into a third model that has functions equivalent to a combination of the first model and the second model to obtain third output data that is output data, and the label data and the third output data are displayed.

18. A design support method for designing a combination model composed of multiple models, wherein a computer displays multiple candidate areas in which symbols representing candidate models that constitute the combination model are placed, and the multiple candidate areas include at least two of the following: a first candidate area in which one or more first symbols representing a first model that accepts input of non-text information and outputs text information corresponding to the input non-text information are placed; a second candidate area in which one or more second symbols representing a second model that accepts input of text information and converts the input text information into text information suitable for a predetermined purpose are placed; and a third candidate area in which one or more third symbols representing a third model that accepts input of text information and outputs non-text information corresponding to the input text information are placed.

19. A program for causing a computer to function as the design support device according to any one of claims 1 to 17.

Images