Information processing apparatus and information processing program

Abstract

To provide the details of a question reflecting a user's idea.SOLUTION: An information processing apparatus of an embodiment receives image information indicating an image related to a target of a user's question, executes image analysis processing on the image information to acquire image analysis information indicating a result of image analysis, from question history information indicating a plurality of question histories in which the details of questions are described, question history information indicating a question history in which the details of a question related to the result of image analysis indicated in the image analysis information is described, generates question candidate information indicating a question candidate in which the details of a question to the target of the user's question is described, on the basis of the question history information acquired by a question history information acquisition unit, and provides the question candidate information to the user in a selectable state, inputs the question candidate information selected by the user to a generation model functioned to output an answer sentence to the details of the question to acquire an answer sentence output from the generation model, and provides the answer sentence to the user.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and an information processing program. [Background technology]

[0002] In recent years, technologies have been proposed for automatically creating sentences using AI models (also called generative AI) such as large-scale language models (LLMs). Large-scale language models are AI models in the field of natural language processing that have been pre-trained with large-scale corpora, and are designed to generate and output answer sentences related to the question in the specified format when, for example, text describing a question is input along with instruction information (also called a prompt) that specifies the output format.

[0003] In addition, it is conceivable that the service using the above AI model could be used as an automatic response service for customers who visit a store. For example, by using the above AI model in a store such as a supermarket, it is conceivable that a service that automatically responds to questions from customers could be provided.

[0004] It is known that the accuracy of the answers generated by the AI ​​model can be improved by inputting the question content appropriately. Therefore, if the question content is not input appropriately, the user may not be able to obtain the answer text that he or she intended. For example, in the above-mentioned store environment, where some users are unfamiliar with machine operation, there is a possibility that the user may not be able to input the question content that he or she intended into the AI ​​model, and therefore, there is a need for technology that can support user operation. Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide an information processing device and an information processing program that can provide questions that reflect the user's thoughts. [Means for solving the problem]

[0006] an image analysis information acquisition unit that executes an image analysis process on the image information and acquires image analysis information that indicates the image analysis results; a question history information acquisition unit that acquires, from question history information that indicates each of a plurality of question histories in which question contents are described, question history information that indicates question histories in which question contents related to the image analysis results indicated in the image analysis information; a question candidate information generation unit that generates question candidate information that indicates question candidates that describe questions regarding the question target, based on the question history information acquired by the question history information acquisition unit; a first provision unit that provides the question candidate information to the user in a selectable state; an answer sentence acquisition unit that inputs the question candidate information selected by the user to a generation model that is functionally configured to output an answer sentence to the question content and acquires the answer sentence output by the generation model; and a second provision unit that provides the answer sentence to the user. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of an information processing system according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. [Figure 3] FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. [Figure 4] FIG. 4 is a schematic diagram showing an example of image information according to this embodiment. [Figure 5] FIG. 5 is a schematic diagram showing an example of question candidate information according to this embodiment. [Figure 6] FIG. 6 is a schematic diagram showing an example of response information according to the present embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of control processing of the information processing device according to this embodiment. [Figure 8]FIG. 8 is a flowchart showing an example of control processing of the information processing device according to this embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of a functional configuration of an information processing device according to the second modification. [Figure 10] FIG. 10 is a schematic diagram showing an example of image information according to the second modified example. [Figure 11] FIG. 11 is a schematic diagram showing an example of question candidate information according to the second modified example. [Figure 12] FIG. 12 is a schematic diagram showing an example of image information according to the second modified example. [Figure 13] FIG. 13 is a schematic diagram showing an example of question candidate information according to the second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of an information processing system will be described with reference to the drawings. In the embodiment shown below, an example will be described in which the system is applied to a store such as a supermarket. Note that the present invention is not limited to the following embodiment, and the components in the following embodiment include those that would be easily conceived by a person skilled in the art, those that are substantially the same, and those that are within the scope of what is called equivalent. Furthermore, various omissions, substitutions, modifications, and combinations of the components can be made without departing from the spirit of the following embodiment.

[0009] 1 is a schematic diagram showing a general configuration of an information processing system 10 according to an embodiment. The information processing system 10 is a system including an input / output device 1 and an information processing device 2.

[0010] The input / output device 1 is, for example, a portable terminal device (hereinafter also referred to as a mobile terminal) such as a smartphone. The input / output device 1 is, for example, a mobile terminal owned by a user. The input / output device 1 may be a tablet-type terminal device. Furthermore, the input / output device 1 may be a mobile terminal rented to a user by a store.

[0011] The input / output device 1, for example, acquires an image input by a user and outputs the acquired image to the information processing device 2. The input / output device 1 also acquires, for example, question candidate information corresponding to the image output from the information processing device 2 and provides the acquired question candidate information to the user. The input / output device 1 also acquires, for example, an answer sentence or answer information corresponding to the question candidate output from the information processing device 2 and provides the acquired answer sentence or answer information to the user. Details of the question candidate information, the answer sentence, and the answer information will be described later.

[0012] The information processing device 2 is, for example, a store server or a server device managed by a company that operates a restaurant. In this embodiment, the information processing device 2 is, for example, a computer equipped with a generation AI that automatically generates text, such as a large-scale language model (LLM). The large-scale language model generates a response sentence according to the content of a question, for example, in response to a question about a product. The large-scale language model also generates a response sentence according to the content of a question, for example, in response to a question about how to use a device used by a store clerk, such as a POS terminal.

[0013] The information processing device 2 is composed of one or more computers. The information processing device 2 supports sales promotion of products sold in stores based on questions acquired via the input / output device 1. The information processing device 2 is connected to the input / output device 1 via a communication line such as a mobile network.

[0014] Next, the hardware configuration of the information processing device 2 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the information processing device 2 according to an embodiment. As shown in Fig. 2, the information processing device 2 includes a CPU (Central Processing Unit) 21, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 23, an auxiliary storage device 24, and a communication I / F (interface) 25.

[0015] The CPU 21 is the main controller. The RAM 22 expands programs and various data. The ROM 23 stores various programs. The auxiliary storage device 24 stores various programs and data. The communication I / F 25 is an interface for data communication with the input / output device 1.

[0016] The CPU 21, RAM 22, ROM 23, auxiliary storage device 24, and communication I / F 25 are connected to one another via a bus 26. The CPU 21, RAM 22, and ROM 23 constitute a control unit. That is, the control unit executes control processing of the information processing device 2, which will be described later, by the CPU 21 operating in accordance with a control program stored in the ROM 23 or auxiliary storage device 24 and loaded in the RAM 22.

[0017] The auxiliary storage device 24 is configured with a non-volatile memory such as an HDD (Hard Disc Drive) or flash memory that retains stored information even when the power is turned off. The auxiliary storage device 24 is an example of a storage unit. The auxiliary storage device 24 stores image information and question information received by the information processing device 2, image analysis information and question history information acquired by the information processing device 2, answers, product information, and question candidate information and answer information generated by the information processing device 2. Details of the image information, question information, image analysis information, question history information, answers, product information, question candidate information, and answer information will be described later.

[0018] The information processing device 2 is installed in a secure environment. A secure environment is a configuration in which the information processing device 2 is placed in a local environment for use. With this configuration, since the text information input for processing a large-scale language model provided in the information processing device 2, which will be described later, may include customer information of customers, etc., the information processing device 2 can contribute to the protection of personal information by being placed in a local environment for use.

[0019] Fig. 3 is a block diagram showing an example of a functional configuration of the information processing device 2 according to the embodiment. As shown in Fig. 3, the information processing device 2 includes a receiving unit 211, an image analysis information acquiring unit 212, a question history information acquiring unit 213, a question candidate information generating unit 214, a first providing unit 215, an answer sentence acquiring unit 216, a product information acquiring unit 217, an answer information generating unit 218, a second providing unit 219, and a storage unit 220. Note that the functional configuration of the information processing device 2 is not limited to this.

[0020] The receiving unit 211 receives image information indicating an image related to the user's question. Specifically, the receiving unit 211 receives image information indicating an image related to the user's question, which is output by the input / output device 1. The receiving unit 211 also receives image information indicating an image of a product as image information related to the user's question. Specifically, the receiving unit 211 receives image information indicating an image of a product as image information related to the user's question, which is output by the input / output device 1.

[0021] Here, image information will be described with reference to FIG. 4. FIG. 4 is a schematic diagram showing an example of image information according to the embodiment. Image information 41 shown in FIG. 4 is, for example, an image of a product sold in a store or the like, and shows an image with the product name "AA" attached. Furthermore, image information 41 may include personal information (customer information) such as the name, telephone number, and email address of the user who sent image information 41. The customer information includes information about the store used by the user. Furthermore, receiving unit 211 stores the received image information 41 in auxiliary storage device 24.

[0022] The image analysis information acquisition unit 212 performs image analysis processing on the image information and acquires image analysis information indicating the image analysis results. Specifically, the image analysis information acquisition unit 212 performs image analysis processing on the image information received by the receiving unit 211 and acquires image analysis information indicating the image analysis results. For example, when image analysis processing is performed on the image information 41 shown in FIG. 4, the image analysis information acquisition unit 212 acquires image analysis information including the result of the image analysis being the product name "AA." The image analysis information acquisition unit 212 stores the acquired image analysis information in the auxiliary storage device 24.

[0023] The image analysis method is not limited, but for example, the product shown in the image can be identified by using known object recognition or character recognition techniques. Object recognition is also called generic object recognition.

[0024] 3 , the question history information acquisition unit 213 acquires, from the question history information indicating each of a plurality of question histories in which question content is described, question history information indicating a question history in which question content related to the image analysis result indicated in the image analysis information. Specifically, the question history information acquisition unit 213 acquires, from the question history information indicating each of a plurality of question histories in which question content is described and which is stored in the auxiliary storage device 24, question history information indicating a question history in which question content related to the image analysis result indicated in the image analysis information acquired by the image analysis information acquisition unit 212 is described. For example, if the result of the image analysis is the product name "AA," the question history information acquisition unit 213 extracts question history information related to the product name "AA" from the auxiliary storage device 24.

[0025] The question candidate information generation unit 214 generates question candidate information indicating question candidates each describing a question to be asked to the question target, based on the question history information. Specifically, the question candidate information generation unit 214 generates question candidate information indicating question candidates each describing a question to be asked to the question target, based on the question history information acquired by the question history information acquisition unit 213. Here, the question candidate information will be described with reference to FIG. 5. FIG. 5 is a schematic diagram showing an example of question candidate information according to the embodiment. Question candidate information 51 shown in FIG. 5 displays a plurality of selectable question candidates 52. FIG. 5 also shows a pull-down menu 53 so that the user can select one question candidate from the plurality of question candidates 52.

[0026] For example, if the question history related to the product name "AA" includes a question ascertaining whether the product name "AA" is sold in a store, the question candidate information generation unit 214 generates a question such as "Does store B sell AA?". Also, if the question history related to the product name "AA" includes a question ascertaining the selling price of the product name "AA" in a store, the question candidate information generation unit 214 generates a question such as "Please tell me the price of AA at store B." Then, the question candidate information generation unit 214 generates question candidate information 51 by treating each generated question as a question candidate 52.

[0027] The store name "Store B" may be obtained from the question history information, or the name of a store that is included in the customer information and frequently visited by the user may be extracted and added to the question. The order in which the question candidates are displayed may be determined by obtaining total information such as the frequency of appearance of the question candidates from the auxiliary storage device 24, and displaying the question candidates in descending order of frequency of appearance.

[0028] 3, first providing unit 215 provides the question candidate information to the user in a selectable state. Specifically, first providing unit 215 provides the question candidate information generated by question candidate information generation unit 214 to the user in a selectable state. For example, first providing unit 215 outputs question candidate information 51 to input / output device 1 in a selectable state and provides it to the user.

[0029] The receiving unit 211 receives question information including a question candidate selected by the user. Specifically, the receiving unit 211 receives question information indicating the question candidate selected by the user, output by the input / output device 1. The question information may include personal information such as the name, telephone number, and email address of the user who asked the question. The receiving unit 211 also stores the received question information in the auxiliary storage device 24.

[0030] The answer sentence acquisition unit 216 inputs the question candidate information selected by the user to a generative model that is functionally configured to output an answer sentence to the question content, and acquires the answer sentence output by the generative model. Specifically, the answer sentence acquisition unit 216 inputs the question candidate information received by the receiving unit 211 to a generative model that is functionally configured to output an answer sentence to the question content, and acquires the answer sentence output by the generative model.

[0031] For example, the answer sentence acquisition unit 216 inputs a question candidate and instruction information (also called a prompt) that indicates an answer to the question candidate into a large-scale language model using a known method, thereby causing the large-scale language model to generate an answer sentence that answers the question sentence, and acquires the answer sentence output by the large-scale language model. The answer sentence includes at least product information related to the question candidate and a sentence related to sales promotion of the product. The product information is, for example, the product genre, product name, etc. The question sentence may be included in the prompt.

[0032] The answer sentence acquisition unit 216 generates a prompt that instructs (describes) the addition of product information related to the candidate question, the addition of a promotional statement for the product, etc., in order to answer a question from a user. The prompt may be one prepared in advance in the auxiliary storage device 24, or may be dynamically generated by the answer sentence acquisition unit 216. The prompt may also include a statement indicating an information source that stores information about the candidate product. Furthermore, if the image information includes customer information, the customer information may be added to the prompt. This allows the answer sentence to reflect the customer information. The answer sentence acquisition unit 216 stores the acquired answer sentence in the auxiliary storage device 24.

[0033] The product information acquisition unit 217 acquires product information about the product indicated in the reply. Specifically, the product information acquisition unit 217 acquires product information about the product indicated in the reply from the auxiliary storage device 24. The product information includes product information about the product being promoted, such as the product genre, product code, product name, and price. The promotional product may be a product that is subject to a discount, a product with a large sales volume, a product with a large inventory among the products owned by the store, a product that is the subject of a campaign, or the like.

[0034] The answer information generation unit 218 generates answer information including an answer sentence and product information. Specifically, the answer information generation unit 218 generates answer information including the answer sentence acquired by the answer sentence acquisition unit 216 and the product information acquired by the product information acquisition unit 217. Here, the answer information will be described with reference to FIG. 6. FIG. 6 is a schematic diagram showing an example of answer information according to the embodiment. Answer information 61 shown in FIG. 6 shows question candidates 62 selected by the user and answer sentences 63.

[0035] For example, if the question candidate 62 is "Is AA sold at store B?", the answer sentence 63 will be "AA is sold at store B." Note that although the answer information 61 does not include information related to product information, for example, the price of the product name "AA" may be included in the contents of the answer sentence 63.

[0036] 3, the second providing unit 219 provides the answer sentence to the user. Specifically, the second providing unit 219 provides the answer sentence acquired by the answer sentence acquisition unit 216 to the user. For example, the second providing unit 219 outputs the answer sentence to the input / output device 1 to provide it to the user.

[0037] Furthermore, the second providing unit 219 provides the user with the answer information 61. Specifically, the second providing unit 219 provides the user with the answer information 61 generated by the answer information generating unit 218. For example, the second providing unit 219 outputs the answer information 61 to the input / output device 1 to provide it to the user.

[0038] The storage unit 220 stores the question candidates selected by the user as question history information in the storage unit 220. Specifically, the storage unit 220 stores the question candidates selected by the user received by the receiving unit 211 in the auxiliary storage device 24 as question history information.

[0039] The storage unit 220 also stores the answer information as question history information in the storage unit. Specifically, the storage unit 220 stores the answer information generated by the answer information generation unit 218 in the auxiliary storage device 24 as question history information.

[0040] Next, the control process of the information processing device 2 will be described with reference to Fig. 7 and Fig. 8. Fig. 7 and Fig. 8 are flowcharts showing an example of the control process of the information processing device 2 according to the embodiment. Fig. 7 describes the control process of providing question candidates to the user.

[0041] The receiving unit 211 receives image information indicating an image related to the user's question, output by the input / output device 1 (step S71). Subsequently, the image analysis information acquiring unit 212 performs image analysis processing on the image information received by the receiving unit 211, and acquires image analysis information indicating the image analysis results (step S72). Subsequently, the question history information acquiring unit 213 acquires, from the question history information indicating each of a plurality of question histories in which question contents are described and which are stored in the auxiliary storage device 24, question history information indicating a question history in which question contents related to the image analysis results indicated in the image analysis information acquired by the image analysis information acquiring unit 212 are described (step S73).

[0042] Next, question candidate information generation unit 214 generates question candidate information indicating question candidates describing questions to the question target based on the question history information acquired by question history information acquisition unit 213 (step S74). Next, first provision unit 215 provides the question candidate information generated by question candidate information generation unit 214 to the user in a selectable state (step S75). When this process ends, the process of information processing device 2 ends.

[0043] This allows the user to check the question candidates related to the image. Furthermore, the user can select a question candidate that reflects the user's thoughts from the checked question candidates. Therefore, the information processing device 2 can provide information that reflects the user's thoughts from the image input by the user.

[0044] 8 illustrates a control process for providing a user with an answer to a question candidate. The receiving unit 211 receives question information indicating a question candidate selected by the user, output by the input / output device 1 (step S81). Next, the answer sentence acquiring unit 216 inputs the question candidate information received by the receiving unit 211 into a generative model that is functionally configured to output an answer sentence to the question content, and acquires the answer sentence output by the generative model (step S82). Next, the product information acquiring unit 217 acquires product information regarding the product indicated in the answer sentence from the auxiliary storage device 24 (step S83).

[0045] Next, the answer information generation unit 218 generates answer information including the answer sentence acquired by the answer sentence acquisition unit 216 and the product information acquired by the product information acquisition unit 217 (step S84). Next, the second providing unit 219 provides the user with the answer sentence acquired by the answer sentence acquisition unit 216. In addition, the second providing unit 219 provides the user with the answer information 61 generated by the answer information generation unit 218 (step S85).

[0046] Next, the storage unit 220 stores the question candidate selected by the user received by the receiving unit 211 as question history information in the auxiliary storage device 24. The storage unit 220 also stores the answer information generated by the answer information generating unit 218 as question history information in the auxiliary storage device 24 (step S86). When this process ends, the process of the information processing device 2 ends. This allows the user to check the answer sentence corresponding to the question candidate.

[0047] As described above, the information processing device 2 of this embodiment receives image information indicating an image related to a user's question, performs image analysis processing on the image information, obtains image analysis information indicating an image analysis of the image information, obtains question history information indicating a question history related to the image analysis indicated in the image analysis information, generates question candidate information indicating question candidates for the question based on the question history information, and provides the question candidate information to the user.The information processing device 2 then inputs the generated question candidate information into a large-scale language model that is configured to output an answer sentence to the question candidate selected by the user, obtains the answer sentence output by the large-scale language model, and provides the question candidate and the answer sentence to the user.

[0048] As a result, the information processing device 2 can provide the user with question candidate information that reflects the user's thoughts.The information processing device 2 can provide the user with answer information that corresponds to the question candidate selected by the user.Therefore, the information processing device 2 can provide the user with question content that reflects the user's thoughts from the image.

[0049] The above-described embodiment can be modified as needed by partially changing the configuration or functions of each of the above-described devices. Therefore, several modifications of the above-described embodiment will be described below as other embodiments. The following mainly focuses on differences from the above-described embodiment, and detailed descriptions of commonalities with the content already described will be omitted. The modifications described below may be implemented individually or in appropriate combination.

[0050] (First Modification) For example, the image information may include coordinate information indicating a specific position or area designated by the user within the image. When multiple products are displayed in the same image, the image information includes coordinate information. The coordinate information is two-dimensional coordinates on the image, and is the coordinate of a position designated by the user by performing a touch operation or the like on the displayed image. The coordinate information is set, for example, when the user performs a touch operation on a specific product among the multiple products. At this time, the image analysis information acquisition unit 212 of the information processing device 2 executes image analysis processing on the image indicated in the coordinate information as image analysis information, and acquires image analysis information indicating the image analysis results for the image of the coordinate information.

[0051] Then, the question history information acquisition unit 213 acquires question history information indicating a question history in which questions related to the image analysis results are described, the question candidate information generation unit 214 generates question candidate information based on the question history information, and the first provision unit 215 provides the user with the question candidate information in a selectable state. This enables the information processing device 2 to provide the user with questions related to the product specified by the user.

[0052] (Second Modification) In the above-described embodiment, the image information has been described as images relating to products sold in stores, etc. In the second modified example, the image information will be described as images relating to devices used in stores, etc.

[0053] FIG. 9 is a block diagram showing an example of a functional configuration of an information processing device 2 according to the second modification. As shown in Figure 9, the information processing device 2 includes a receiving unit 211, an image analysis information acquisition unit 212, a question history information acquisition unit 213, a question candidate information generation unit 214, a first providing unit 215, an answer sentence acquisition unit 216, a product information acquisition unit 217, an answer information generation unit 218, a second providing unit 219, a storage unit 220, and a device information acquisition unit 221.

[0054] The receiving unit 211 receives image information indicating an image related to a user's question, output by the input / output device 1. In the second modified example, the input / output device 1 is a device used in a store, such as a POS terminal. Here, the image information will be described with reference to FIG. 10. FIG. 1 is a schematic diagram showing an example of image information according to the second modified example.

[0055] 10 is an image relating to the display screen of a POS terminal used in a store, and includes product registration information 102 showing a list of products registered in the POS terminal, product individual information 103 relating to the quantity and unit price of the product "KKK" displayed in the product registration information 102, and price information 104 showing the total price obtained by multiplying the quantity and unit price of each product displayed in the product registration information 102. The image information 101 may include information such as the identifier of the POS terminal that sent the image information 101 and the identifier of the user using the POS terminal.

[0056] 9, image analysis information acquisition unit 212 performs image analysis processing on image information 101 received by receiving unit 211, and acquires image analysis information indicating the image analysis results for the image information. Furthermore, question history information acquisition unit 213 acquires question history information indicating a question history related to the image analysis results indicated in the image analysis information acquired by image analysis information acquisition unit 212, from auxiliary storage device 24. Furthermore, question candidate information generation unit 214 generates question candidate information indicating question candidates for questions, based on the question history information acquired by question history information acquisition unit 213.

[0057] For example, if the image analysis result is a screen of a POS terminal, the question history information acquisition unit 213 acquires the question history related to the POS terminal from the auxiliary storage device 24. Then, the question candidate information generation unit 214 generates question candidate information from the question history related to the POS terminal so that the candidate questions are displayed in descending order of frequency of appearance.

[0058] Here, the question candidate information will be explained using Fig. 11. Fig. 11 is a schematic diagram showing an example of question candidate information according to the second modified example. Question candidate information 111 shown in Fig. 11 shows a plurality of question candidates 112. Fig. 11 also shows a pull-down menu 113 so that the user can select one question candidate from the plurality of question candidates 112.

[0059] The question candidates 112 shown in Figure 11 show multiple question candidates regarding the operation of a POS terminal, such as "Please tell me how to operate the POS," "What should I do if I want to cancel an item?", and "What should I do if an item has a discount sticker on it?"

[0060] 9 , the first providing unit 215 outputs the question candidate information 111 to the input / output device 1 to provide it to the user. At this time, the user may find that the question candidate included in the question candidate information 111 output to the input / output device 1 does not reflect the user's thoughts. In this case, the user again specifies an image related to the user's question, and transmits image information including two-dimensional coordinates of the specified image specified by the user from the input / output device 1 to the information processing device 2.

[0061] Here, two-dimensional coordinates on an image designated by a user will be described with reference to Fig. 12. Fig. 12 is a schematic diagram showing an example of image information according to a second modified example. Fig. 12 is a schematic diagram in which a designated image 105 designated by a user is added to the image information 101 shown in Fig. 10. The designated image 105 shown in Fig. 12 is, for example, an image related to an indicator of a POS terminal.

[0062] 9, receiving unit 211 again receives image information 101 including coordinate information corresponding to the image, and image analysis information acquisition unit 212 performs image analysis processing on the image indicated in the coordinate information as image analysis information for image information 101 received by receiving unit 211, and acquires image analysis information indicating an image analysis of the image indicated in the coordinate information. Furthermore, question history information acquisition unit 213 acquires question history information indicating a question history related to the image analysis indicated in the image analysis information acquired by image analysis information acquisition unit 212 from auxiliary storage device 24. Furthermore, question candidate information generation unit 214 generates question candidate information indicating question candidates for the question, based on the question history information acquired by question history information acquisition unit 213.

[0063] The regenerated question candidate information will now be described with reference to Fig. 13. Fig. 13 is a schematic diagram showing an example of question candidate information according to the second modified example. Question candidate information 131 shown in Fig. 13 shows a plurality of question candidates 132. Fig. 13 also shows a pull-down menu 133 so that the user can select one question candidate from the plurality of question candidates 132.

[0064] The candidate questions 132 shown in Figure 13 show multiple candidate questions regarding the indicator of a POS terminal, such as "When does the indicator appear?", "Please tell me a list of icons that appear on the indicator", and "Please tell me what to do when an icon appears on the indicator".

[0065] 9, the first providing unit 215 outputs the question candidate information 131 to the input / output device 1 to provide it to the user. Furthermore, the receiving unit 211 receives the question information indicating the question candidate selected by the user output by the input / output device 1, and the answer sentence acquiring unit 216 inputs the question candidate received by the receiving unit 211 to a large-scale language model that is functionally configured to output an answer sentence to the question candidate, and acquires the answer sentence output by the large-scale language model.

[0066] The device information acquisition unit 221 acquires device information about the device indicated in the reply. Specifically, the device information acquisition unit 221 acquires device information about the device indicated in the reply from the auxiliary storage device 24. The device information includes, for example, an operation manual about operating the device and an installation procedure manual about installing the device.

[0067] The answer information generation unit 218 generates answer information including question candidates, answer sentences, and device information. Specifically, the answer information generation unit 218 generates answer information including the question candidates received by the receiving unit 211, the answer sentences acquired by the answer sentence acquisition unit 216, and the device information acquired by the device information acquisition unit 221. Furthermore, the second providing unit 219 provides the answer information including the question candidates, answer sentences, and device information generated by the answer information generation unit 218 to the user.

[0068] The storage unit 220 then associates the question candidate selected by the user received by the receiving unit 211, the answer sentence acquired by the answer sentence acquisition unit 216, and the device information acquired by the device information acquisition unit 221, and stores them as question history information in the auxiliary storage device 24. As a result, even if the information processing device 2 provides a question candidate that does not reflect the user's thoughts, the user can specify an image that reflects the user's thoughts again to be provided with question candidate information that reflects the user's thoughts. Note that the user may specify an image that reflects the user's thoughts at any timing and transmit image information including two-dimensional coordinates of the specified image specified by the user from the input / output device 1 to the information processing device 2.

[0069] (Third Modification) In the above-described embodiment, the information processing device 2 equipped with a large-scale language model is configured to be deployed and used in a local environment, but this is not limiting, and the information processing device 2 equipped with a large-scale language model may also be configured to be deployed and used in a highly secure cloud environment. The highly secure cloud environment is not disclosed on the Internet, but is connected to an in-house server environment via a highly confidential communication line.

[0070] The program executed by the information processing device 2 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the information processing device 2 of the embodiment and the modified example may be provided or distributed via a network such as the Internet.

[0071] The programs executed by each device in the above-described embodiments are provided in a state where they are pre-installed in a ROM, a storage unit, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0072] Furthermore, the programs executed by each device in the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device in the above-described embodiments may be provided or distributed via a network such as the Internet.

[0073] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and their modifications can be embodied in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0074] 1 Input / Output Device 2. Information processing equipment 211 Receiving unit 212 Image analysis information acquisition unit 213 Question History Information Acquisition Unit 214 Question candidate information generation unit 215 1st provision part 216 Answer sentence acquisition part 217 Product Information Acquisition Department 218 Answer information generation section 219 2nd provision part 220 Storage Unit 221 Device information acquisition unit [Prior art documents] [Patent documents]

[0075] [Patent Document 1] Japanese Patent Publication No. 2022-138750

Claims

1. a receiving unit that receives image information indicating an image related to a question target of a user; an image analysis information acquisition unit that executes image analysis processing on the image information and acquires image analysis information indicating the image analysis result; a question history information acquiring unit that acquires, from among a plurality of question history information indicating each of the question histories in which question content is described, question history information indicating a question history in which question content related to the image analysis result indicated in the image analysis information; a question candidate information generating unit that generates question candidate information indicating question candidates each describing a question to be asked to the question target based on the question history information acquired by the question history information acquiring unit; a first providing unit that provides the question candidate information to the user in a selectable state; an answer sentence acquisition unit that inputs the question candidate information selected by the user into a generative model that is configured to output an answer sentence to the question content, and acquires the answer sentence output by the generative model; a second providing unit that provides the answer sentence to the user; An information processing device comprising:

2. the receiving unit receives image information showing an image of a product as image information related to the question target, a product information acquisition unit that acquires product information about the product indicated in the reply; an answer information generation unit that generates answer information including the answer sentence and the product information, the second providing unit provides the answer information to the user. The information processing device according to claim 1 .

3. the image information includes coordinate information indicating a specific position or area designated by the user within the image; The image analysis information acquisition unit executes the image analysis process on the image indicated by the coordinate information, and acquires image analysis information indicating an image analysis result for the image indicated by the coordinate information, as the image analysis information. The information processing device according to claim 1 .

4. a device information acquisition unit that acquires device information about the device indicated in the response; an answer information generation unit that generates answer information including the question candidate, the answer sentence, and the device information; the device information includes an operation manual for operating the device and an installation procedure for installing the device; the second providing unit provides the answer information to the user. The information processing device according to claim 3 .

5. a storage unit that associates the question candidate selected by the user with the answer sentence and stores the association information in a storage unit as the question history information; The information processing device according to claim 1 .

6. The computer of the information processing device, a receiving unit that receives image information indicating an image related to a question target of a user; an image analysis information acquisition unit that executes image analysis processing on the image information and acquires image analysis information indicating the image analysis result; a question history information acquiring unit that acquires, from among a plurality of question history information indicating each of the question histories in which question content is described, question history information indicating a question history in which question content related to the image analysis result indicated in the image analysis information; a question candidate information generating unit that generates question candidate information indicating question candidates each describing a question to be asked to the question target based on the question history information acquired by the question history information acquiring unit; a first providing unit that provides the question candidate information to the user in a selectable state; an answer sentence acquisition unit that inputs the question candidate information selected by the user into a generative model that is configured to output an answer sentence to the question content, and acquires the answer sentence output by the generative model; a second providing unit that provides the answer sentence to the user; An information processing program that makes it function as such.

Images