Information processing device, information processing method, and program

Abstract

This information processing device comprises a receiving unit that receives metadata generated on the basis of the results of sensing, and an answer acquiring unit that generates a prompt to be input to a large language model and inputs the prompt to the large language model to obtain an answer, wherein the answer acquiring unit executes: first prompt generation processing for generating a prompt for instructing analysis of the metadata and identification of a problem; second prompt generation processing for generating a prompt for instructing a search for solutions that solve the identified problem; third prompt generation processing for generating a prompt for instructing a requirements definition of an application for realizing a solution selected by a user from among the solutions obtained by the search; and fourth prompt generation processing for generating a prompt for instructing generation of an application based on the requirements definition.

Description

Information Processing Apparatus, Information Processing Method, Program

[0001] This technology relates to an information processing apparatus, an information processing method, and a program for solving problems.

[0002] There is a need for technologies that solve the problems faced by users, and technologies that can solve even the problems that users are not aware of are particularly in high demand. For example, in Patent Document 1 below, a technology for extracting potential needs of users and recommending applications that meet those needs is disclosed.<LID=

[0003] Japanese Patent Application Laid-Open No. 2018-205986

[0004] However, the technology in Patent Document 1 is limited to extracting needs related to the movement route, and it is difficult to say that it has a configuration for solving problems that users are not truly aware of.

[0005] This technology has been made in view of such problems, and aims to solve problems that users are not aware of.

[0006] The information processing apparatus according to this technology includes a receiving unit that receives metadata generated based on the result of sensing, and an answer acquisition unit that generates a prompt to be input to a large language model (LLM: Large Language Model) and obtains an answer by inputting the prompt to the large language model. The answer acquisition unit executes a first prompt generation process for generating a prompt for instructing analysis of the metadata and identification of problems, a second prompt generation process for generating a prompt for instructing search for solutions to the identified problems, a third prompt generation process for generating a prompt for instructing definition of requirements for an application that realizes the solution selected by the user from the solutions obtained by the search, and a fourth prompt generation process for generating a prompt for instructing generation of an application based on the requirement definition. The answer acquisition unit can obtain, by using a large language model, problems in the space where the sensing target is located as a result of analyzing events and phenomena occurring in the space from the metadata.

[0007] The information processing method of this technology involves an information processing device executing the following: receiving metadata generated based on the sensing results; a first prompt generation process that generates a prompt instructing the analysis of the metadata and the identification of a problem; a second prompt generation process that generates a prompt instructing the search for solutions to resolve the identified problem; a third prompt generation process that generates a prompt instructing the definition of requirements for an application that implements a solution selected by the user from among the solutions obtained through the search; and a fourth prompt generation process that generates a prompt instructing the creation of an application based on the requirements definition.

[0008] The program of this technology causes the information processing device to execute the following: a process for receiving metadata generated based on the sensing results; a first prompt generation process for generating a prompt instructing the analysis of the metadata and identification of a problem; a second prompt generation process for generating a prompt instructing the search for solutions to resolve the identified problem; a third prompt generation process for generating a prompt instructing the user to define the requirements for an application that implements the solution selected by the user from among the solutions obtained through the search; and a fourth prompt generation process for generating a prompt instructing the user to generate an application based on the requirements definition.

[0009] This is a block diagram showing the schematic configuration of the information processing system in this embodiment. This is a block diagram showing another example configuration of the information processing system. This is a block diagram showing the hardware configuration of a computer device such as an information processing device. This is a block diagram showing the hardware configuration of an imaging device. This is a block diagram showing the functional configuration of an information processing device. This is a diagram showing the process up to the creation of a requirements definition document. This is a diagram showing an example of a screen for presenting the issues identified by the issue identification process to the user. This is a diagram showing the process up to presenting the application to the user. This is a diagram showing an example of a presentation screen presented to the user during the application generation process. This is a diagram showing an example of a presentation screen presented to the user after the generation of the adjusted application.

[0010] The embodiments of the information processing device relating to this technology will be described below in the following order, with reference to the attached drawings. <1. Configuration of the Information Processing System> <2. Hardware Configuration of Each Device> <3. Hardware Configuration of the Imaging Device> <4. Functions of the Information Processing Device> <5. Implementation of Functions> <6. Processing Flow> <7. Variations of Implementation> <7-1. Metadata> <7-2. User Information> <7-3. Identified Issues and Solutions> <8. Database> <9. Others> <10. Summary> <11. This Technology>

[0011] <1. Configuration of the Information Processing System> The overall configuration of the information processing system S, including the information processing device 1 of this technology, will be explained with reference to Figure 1, etc.

[0012] The information processing system S comprises an information processing device 1, a user terminal 2, and an imaging device 3. The information processing device 1, user terminal 2, and imaging device 3 are capable of data communication via a predetermined communication network, such as the Internet (Network NW).

[0013] The information processing device 1, user terminal 2, and imaging device 3 may each be provided in multiple quantities. For example, the information processing system S shown in Figure 1 is configured to have multiple imaging devices 3.

[0014] The information processing device 1 is, for example, a server device that provides predetermined functions to a user terminal 2 used by a user.

[0015] User terminal 2 is a terminal device used by the user and is used to utilize the functions provided by information processing device 1.

[0016] The imaging device 3 is a device that is placed in a predetermined space SP managed by a user using a user terminal 2, and performs imaging of the space SP as the sensing target.

[0017] In this embodiment, as an example, the user terminal 2 is a device used by a user acting as a store manager, and the imaging device 3 is a surveillance camera placed at a predetermined location in the store. The space SP is the sales floor space within the store.

[0018] Furthermore, multiple imaging devices 3 may be installed at approximately the same location in the spatial SP. For example, if two imaging devices 3 are installed at approximately the same location, one imaging device 3 may perform a process to generate metadata about the subject, which is obtained from the image of the spatial SP that is the target of sensing, while the other imaging device 3 may perform a self-positioning process to estimate the positions of the two imaging devices 3.

[0019] For example, if the spatial SP is a large space such as a shopping mall, it is conceivable that numerous imaging devices 3 would be installed. In this case, by estimating the self-position of each imaging device 3 within the spatial SP, it becomes possible to perform metadata analysis at a more advanced level.

[0020] The information processing device 1 receives metadata output from the imaging device 3, analyzes the metadata, and provides the user terminal 2 with an application AP for solving problems related to spatial SP.

[0021] Furthermore, the information processing device 1 utilizes a Large Language Model (LLM) M when presenting an application AP for solving the problem.

[0022] As is well known, the large-scale language model M generates response information according to the content of an input prompt, and is an AI (Artificial Intelligence) model capable of performing various natural language processing tasks. For example, when a question is input as a prompt, the large-scale language model M generates answer information to the question by performing data retrieval processing according to the content of the question and sentence generation processing according to the search results. In addition to tasks such as generating answer information to questions, the large-scale language model M is also capable of performing various tasks that generate response information according to the content of an input prompt, such as creating a computer program that implements the processing specified by the prompt, or generating images or music that satisfy the conditions specified by the prompt.

[0023] Examples of large-scale language models M that can be used by the information processing device 1 in this embodiment include the following:

[0024] ・GPT (Generative Pre-trained Transformer) ・BERT (Bidirectional Encoder Representations from Transformers) ・T5 (Text-To-Text Transfer Transformer) ・XLNet ・ERNIE (Enhanced Representation through Knowledge Integration) ・ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) ・BLOOM (Big Science Large Open-science Open-access Multilingual Language Model) ・Mistral ・OPT (Open Pre-trained Transformer) ・Topher ・LaMDA (Language Model for Dialogue Applications) ・Turing-NLG (Turing Natural Language Generation) ・PaLM (Pathways Language Mode) ・Llama (Language Large Models Meta AI)

[0025] Furthermore, new large-scale language models M are emerging daily, and the large-scale language models M usable in this embodiment are not limited to these models.

[0026] The information processing device 1 performs the process of generating various prompts to be input to the large-scale language model M. Furthermore, the information processing device 1 inputs the generated prompts to the large-scale language model M and obtains response information from the large-scale language model M.

[0027] The information processing device 1 provides the response information obtained from the large-scale language model M to the user terminal 2 either as is or after processing. The response information may be provided by transmitting the information to the user terminal 2, or by indicating the location where the response information is stored, for example, a URL (Uniform Resource Locator).

[0028] Various types of information can be provided by the information processing device 1 to the user terminal 2. For example, the information processing device 1 may present multiple problems identified with respect to the spatial SP, present an application AP to solve a selected problem, or present instructions on how to use the application AP.

[0029] Thus, in the information processing system S of this embodiment, based on metadata obtained from the captured image of the spatial SP to be sensed by the imaging device 3, problems related to the spatial SP are identified, an application AP to solve the problems is generated, and the generated application AP is presented.

[0030] There are various ways in which the information processing device 1 can utilize the large-scale language model M. For example, as shown in Figure 1, the information processing device 1 may use the large-scale language model M stored in a memory area provided inside the information processing device 1. Alternatively, as shown in Figure 2, the information processing device 1 may use the large-scale language model M stored in a memory area of ​​another server device 4.

[0031] <2. Hardware Configuration of Each Device> Figure 3 shows an example of the hardware configuration of the information processing device 1, user terminal 2, and other server device 4. Here, if the information processing device 1, user terminal 2, and other server device 4 are not distinguished, they will be referred to as computer device Com.

[0032] The computer device Com comprises a processing circuit 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53.

[0033] The processing circuit 51, ROM 52, and RAM 53 are capable of communicating data with each other via the bus 54.

[0034] An input / output interface (I / F) 55 is further connected to bus 54.

[0035] The input / output interface 55 is connected to an input device 56, an output device 57, a storage unit 58, a communication interface 59, and a drive 60, respectively.

[0036] The processing circuit 51 is, for example, a circuit that performs various calculations, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The processing circuit 51 may also consist of multiple circuits, such as one that includes both a CPU and a GPU.

[0037] The processing circuit 51 includes arithmetic circuits that perform calculations, control circuits that control these arithmetic circuits, storage circuits such as registers and caches, and an internal bus used as a data transmission path.

[0038] The processing circuit 51 executes various processes according to the program stored in the ROM 52 or the program loaded into the RAM 53 from the storage unit 58. The RAM 53 also stores data necessary for the processing circuit 51 to execute various processes as appropriate.

[0039] The input device 56 could be, for example, a pointing device 56a such as a mouse, a keyboard 56b, a camera 56c, a microphone 56d, or various other controls and operating devices such as keys, dials, touch panels, touchpads, or remote controllers. The input device 56 detects user operations and transmits a signal corresponding to the input operation to the processing circuit 51. The processing circuit 51 interprets the signal and executes the corresponding processing.

[0040] The output device 57 may be, for example, a speaker 57a or a display device 57b consisting of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel.

[0041] The display device 57b is used for various information displays, and is composed of, for example, a device provided on the housing of the computer device Com, or a separate device connected to the computer device Com.

[0042] The display device 57b executes displays such as images for various image processes and moving images of processing targets on the display screen based on the instructions of the processing circuit 51. Further, the display device 57b performs displays such as various operation menus, icons, messages, etc., that is, displays as a GUI (Graphical User Interface) based on the instructions of the processing circuit 51.

[0043] The storage unit 58 is composed of an HDD (Hard Disc Drive), a solid-state memory, etc. The large-scale language model M is stored in the storage unit 58 of the information processing device 1 shown in FIG. 1 and the other server device 4 shown in FIG. 2.

[0044] Further, in the storage unit 58 in which the large-scale language model M is stored, information used by the large-scale language model M and information about the application AP used for past problem solving, for example, requirement specifications, metadata, and the program of the application AP are associated and stored.

[0045] The communication interface 59 serves as a transmission unit and a reception unit related to wired or wireless communication performed between the information processing device 1 and other computer devices Com. The communication interface 59 is composed of an interface circuit for performing communication using various communication standards such as USB (Universal Serial Bus), Bluetooth (registered trademark), and Wi-Fi (registered trademark). Further, the communication interface 59 may be configured as a circuit possessed by a NIC (Network Interface Card), a modem, a Wi-Fi adapter, etc.

[0046] The drive 60 is a device on which a removable recording medium 61 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.

[0047] The drive 60 can read data files such as programs used for each process from the attached removable recording medium 61. The read data files are stored in the storage unit 58, the images included in the data files are displayed on the display device 57b, or the sound is output from the speaker 57a. Also, computer programs and the like read from the removable recording medium 61 by the drive 60 are installed in the storage unit 58 as necessary.

[0048] In the computer device Com having the hardware configuration as described above, for example, software for the processing of the present embodiment can be installed via network communication by the communication interface 59 or via the removable recording medium 61. Alternatively, the software may be stored in the ROM 52, the storage unit 58, etc. in advance. In the computer device Com, the processing circuit 51 performs processing operations based on various programs, thereby executing necessary information processing and communication processing as the information processing device 1 and other server devices 4.

[0049] Note that the computer device Com such as the information processing device 1 is not limited to being configured by a single device as shown in FIG. 3, and a plurality of computer devices may be configured as a system. The plurality of computer devices may be configured as a system by a LAN (Local Area Network) or the like, or may be arranged remotely by a VPN (Virtual Private Network) or the like using the Internet or the like. The plurality of computer devices may include computer devices as a server group (cloud) available by cloud computing services.

[0050] <3. Hardware Configuration of Imaging Device> The hardware configuration of the imaging device 3 will be described with reference to FIG. 4.

[0051] The imaging device 3 includes an imaging optical system 71, a driver 72, an image sensor 73, an in-camera processing circuit 74, a camera storage unit 75, a camera communication interface (I / F) 76, and an operation unit 77.

[0052] The image sensor 73, the camera's internal processing circuit 74, the camera's memory unit 75, the camera's communication interface 76, and the operation unit 77 are connected via a bus 78, enabling them to communicate data with each other.

[0053] The imaging optical system 71 includes lenses such as a cover lens, zoom lens, and focus lens, as well as an aperture (iris) mechanism. The imaging optical system 71 guides light (incident light) from the subject and focuses it onto the light-receiving surface of the image sensor 73.

[0054] The driver 72 comprehensively represents the drive unit for the zoom lens, focus lens, and aperture mechanism of the imaging optical system 71. Specifically, the driver 72 includes actuators for driving the zoom lens, focus lens, and aperture mechanism, and drive circuits for the actuators.

[0055] The camera's internal processing circuit 74 includes, for example, a microcomputer having a CPU, ROM, and RAM, and is configured as a post-processing chip that performs predetermined processing on the signal output from the image sensor 73. The CPU of the camera's internal processing circuit 74 performs overall control of the imaging device 3 by executing various processes according to a program stored in the ROM or a program loaded into the RAM.

[0056] Furthermore, the camera's internal processing circuit 74 issues drive instructions to the driver 72 for the zoom lens, focus lens, aperture mechanism, etc. In response to these drive instructions, the driver 72 will perform actions such as moving the focus lens and zoom lens, and opening and closing the aperture blades of the aperture mechanism.

[0057] Furthermore, the camera's internal processing circuit 74 controls the writing and reading of various data to and from the camera's memory unit 75.

[0058] The camera memory unit 75 is, for example, a non-volatile memory device such as an HDD (Hard Disk Drive) or a flash memory device, and is used as a storage or recording location for image data output from the image sensor 73.

[0059] Furthermore, the camera's internal processing circuit 74 performs various data communications with external devices via the camera communication interface 76. In this example, the camera communication interface 76 is configured to enable data communication with at least the information processing device 1 shown in Figures 1 and 2.

[0060] The camera communication interface 76 is comprised of interface circuits for performing communication using various communication standards such as USB, Bluetooth, and Wi-Fi.

[0061] The operation unit 77 is provided as an operator on the housing of the imaging device 3, and is, for example, a button or a switch. If the imaging device 3 is equipped with a touch panel display, the display is also considered part of the operation unit 77.

[0062] The image sensor 73 is a sensor device provided by the imaging device 3, and is configured as an image sensor such as a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type.

[0063] The image sensor 73, which is a sensor device provided by the imaging device 3, includes not only sensors that generate color or monochrome images by performing photoelectric conversion based on incident visible light, but also a variety of other sensors such as distance measuring sensors that generate distance images, EVS (Event Based Vision Sensor) that generates images consisting of event data, thermosensors that generate heatmap images, spectral sensors that generate spectral images, and polarization sensors that generate polarization images.

[0064] In other words, the image sensor 73 in the following explanation is merely an example of a sensor device.

[0065] The image sensor 73 comprises an imaging unit 91, an image signal processing circuit 92, an in-sensor processing circuit 93, and a sensor memory unit 94, each of which is capable of communicating data with one another via an in-sensor bus 95.

[0066] Furthermore, the image sensor 73 is capable of communicating with external camera processing circuits 74 and the like via sensor buses 95 and 78 through an interface (not shown). Specifically, the image sensor 73 is capable of communicating to output image data output from the image signal processing circuit 92 to the camera processing circuit 74 and camera memory unit 75, and to receiving control information from the camera processing circuit 74 to control the behavior of the image sensor 73.

[0067] The imaging unit 91 includes a pixel array unit 91a in which pixels having photoelectric conversion elements such as photodiodes as light-receiving units are arranged in two dimensions, and a readout circuit that reads out electrical signals obtained by photoelectric conversion from each pixel of the pixel array unit 91a, and is capable of outputting these electrical signals as image data such as RAW image data.

[0068] Furthermore, the image data output from the pixel array unit 91a may be any two-dimensional data containing pixel-specific values, such as distance images, EVS data, heatmap images, spectral images, or polarization images, not just color images or monochrome images. As an example of image data in the following explanation, we will give an example of image data such as a color image.

[0069] The readout circuit in the pixel array unit 91a performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectric conversion, and further performs A / D (Analog / Digital) conversion.

[0070] The image signal processing circuit 92 performs development processing on the RAW image data (imaging image signal) as digital data after A / D conversion processing. Development processing includes, for example, preprocessing, syncing processing, and YC generation processing, and may also include resolution conversion processing, codec processing, etc., as needed.

[0071] In the preprocessing stage, clamping is performed on the image data to clamp the R, G, and B black levels to predetermined levels, and correction processing is performed between the R, G, and B color channels. In the simulcasting stage, color separation is performed so that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaicing is performed as the color separation process. In the YC generation stage, luminance (Y) signals and color (C) signals are generated (separated) from the R, G, and B image data. In the resolution conversion stage, resolution conversion is performed on the image data that has undergone various signal processing.

[0072] In codec processing, the image data that has undergone the various processing steps described above is subjected to encoding for recording or communication purposes, and file generation. In codec processing, video files may be generated in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) or H.264. Still image files may also be generated in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or GIF (Graphics Interchange Format).

[0073] The sensor's internal processing circuit 93 issues instructions to the imaging unit 91 to control the execution of the imaging operation. Similarly, the sensor's internal processing circuit 93 also controls the execution of processing by the image signal processing circuit 92.

[0074] The sensor processing circuit 93 performs processing to generate metadata related to the image data output from the imaging unit 91. The metadata generated by the sensor processing circuit 93 differs depending on the installation location of the imaging device 3.

[0075] For example, when the imaging device 3 is installed in a retail store, the metadata may include the location, age (age group), gender, gaze information, and posture information of the customer included in the image data.

[0076] Furthermore, when the imaging device 3 is located in a space SP within a factory where work is performed in the manufacturing industry, the metadata may include the position and posture information of workers, the position of furniture, etc., which are included in the image data.

[0077] Furthermore, when the imaging device 3 is placed in the space SP within a restaurant, the metadata may include the location, age (age group), gender, gaze information, and posture information of customers included in the image data, the location, gaze information, and posture information of staff, the location of tables, counters, and chairs, and the location and number of dishes, etc.

[0078] These metadata examples are merely illustrations, and various types can be considered depending on the spatial SP in which the imaging device 3 is installed. For example, the spatial SP in which the imaging device 3 is installed can be a variety of places such as a school, intersection, parking lot, airport, train station, commercial facility, elevator, detached house, apartment building, sports facility, concert hall, farm, livestock barn, etc., and the metadata generated in each of these places may differ.

[0079] Furthermore, this metadata, when combined with user information described later, is used in analysis to identify various issues related to the spatial SP where the imaging device 3 is installed. These issues include those that the user operating the imaging device 3 is unaware of.

[0080] The metadata generation process may be performed in the camera's internal processing circuit 74 instead of the sensor's internal processing circuit 93, or it may be performed by the cooperation of the sensor's internal processing circuit 93 and the camera's internal processing circuit 74.

[0081] The sensor memory section 94 comprehensively includes ROM, RAM, SRAM (Static RAM), etc., and a portion of it may be located inside the image signal processing circuit 92 or the sensor internal processing circuit 93.

[0082] The sensor memory unit 94 has functions such as a frame memory and a storage function for storing image data.

[0083] Furthermore, the metadata generation process may be implemented using an AI model. For example, in the image sensor 73 of the imaging device 3, an AI model may be stored in the sensor memory unit 94, and the in-sensor processing circuit 93 may acquire metadata generated by inputting image data to the AI ​​model stored in the sensor memory unit 94. When the in-camera processing circuit 74 performs the metadata generation process, the AI ​​model used for the metadata generation process may be stored in the camera storage unit 75.

[0084] Furthermore, multiple AI models may be used in the metadata generation process. For example, the AI ​​model that generates metadata indicating the attributes of the subject included in the image data may be different from the AI ​​model that generates metadata indicating the subject's gaze and posture.

[0085] The sensor processing circuit 93 comprehensively represents the processing circuit within the image sensor. For example, the sensor processing circuit 93 may be configured such that a processing circuit for image processing of RAW image data and a processing circuit for image processing using an AI model are each provided on separate chips.

[0086] The imaging device 3 provides metadata about the image data to the information processing device 1 by adopting the various configurations described above. Furthermore, by enabling the imaging device 3 to output only metadata without outputting the image data externally, communication volume can be reduced and privacy can be protected.

[0087] <4. Functions of the Information Processing Device> The functions realized by the execution of a program by the processing circuit 51 of the information processing device 1 will be explained with reference to Figure 5.

[0088] The processing circuit 51 functions as a receiving unit F1, a response acquisition unit F2, and a presentation processing unit F3 by executing a predetermined program.

[0089] The receiving unit F1 receives metadata from the imaging device 3, which is metadata obtained as a result of sensing the spatial SP and generated by the imaging device 3.

[0090] The response acquisition unit F2 generates prompts to be input to the large-scale language model M and performs the process of acquiring response information output from the large-scale language model M based on the input of the prompts.

[0091] There are several possible prompts that the response acquisition unit F2 generates.

[0092] The first prompt is one that instructs the analysis of metadata and the identification of issues. This prompt will be called the "first prompt," and the process of generating the first prompt will be called the "first prompt generation process."

[0093] The second prompt is a prompt that instructs the system to search for solutions to the identified problem. This prompt will be called the "second prompt," and the process of generating the second prompt will be called the "second prompt generation process."

[0094] The third prompt is a prompt that instructs the user to define the requirements for the application AP that will implement the solution selected by the user from among the solutions obtained through the search. This prompt will be called the "third prompt," and the process of generating the third prompt will be called the "third prompt generation process."

[0095] The fourth prompt is a prompt that instructs the generation of an application AP based on the requirements definition. This prompt will be called the "fourth prompt," and the process of generating the fourth prompt will be called the "fourth prompt generation process." The application AP generated here is considered a pre-test application and a pre-adjustment application. The pre-adjustment application will be called "pre-adjustment application AP1."

[0096] The fifth prompt is a prompt that instructs the testing of the generated, unadjusted application AP1. This prompt will be referred to as the "fifth prompt," and the process of generating the fifth prompt will be referred to as the "fifth prompt generation process." The fifth prompt generation process may also generate a fifth prompt that instructs the generation of test data and the execution of the tests. Depending on the test instructions, tests corresponding to so-called CI / CD (Continuous Integration / Continuous Delivery) may be performed.

[0097] The sixth prompt is a prompt that instructs the application to be adjusted based on the test results of the pre-adjustment application AP1. This prompt will be referred to as the "sixth prompt," and the process of generating the sixth prompt will be referred to as the "sixth prompt generation process." The application obtained through the adjustment process will be referred to as the "adjusted application AP2." In the following explanation, if the pre-adjustment application AP1 and the adjusted application AP2 are not distinguished, they will simply be referred to as "application AP."

[0098] In other words, the response acquisition unit F2 implements processing to solve problems related to the spatial SP to be sensed by repeatedly generating prompts and acquiring responses.

[0099] Although the generated prompts were divided into smaller steps here, it is also possible to generate a single prompt that simultaneously instructs the analysis of metadata and identification of issues, the search for solutions, the generation of the pre-adjustment application AP1 that implements the solutions, the testing of the pre-adjustment application AP1, and the adjustment of the pre-adjustment application AP1, and then obtain a response from the large-scale language model M. In other words, the number of times prompt generation and response retrieval are performed is not limited.

[0100] The presentation processing unit F3 is responsible for presenting information to the user terminal 2. For example, the presentation processing unit F3 may issue an instruction to the display device 57b of the user terminal 2 to display predetermined information.

[0101] The presentation processing unit F3 may, for example, present multiple tasks to the user if there are multiple tasks that can be obtained by inputting a first prompt into the large-scale language model M. In this case, the receiving unit F1 receives the user's selection information for the multiple tasks.

[0102] The presentation processing unit F3 may present the user with information about the requirements definition obtained by inputting a third prompt into the large-scale language model M, such as a requirements definition document.

[0103] The presentation processing unit F3 may present the user with the program of the unadjusted application AP1 obtained by inputting the fourth prompt into the large-scale language model M.

[0104] The presentation processing unit F3 may present the user with test data and test results obtained by inputting the fifth prompt into the large-scale language model M.

[0105] The presentation processing unit F3 may present the user with the program of the adjusted application AP2 obtained by inputting the sixth prompt into the large-scale language model M.

[0106] In other words, the presentation processing unit F3 may appropriately present the user with the response information obtained by inputting the generated prompt into the large-scale language model M.

[0107] The response acquisition unit F2 may also input user information along with each prompt to the large-scale language model M.

[0108] For example, the response acquisition unit F2 inputs metadata and user information to the large-scale language model M along with the first prompt generated by the first prompt generation process. This allows the large-scale language model M to identify specific issues related to the user and spatial SP by taking into account not only the metadata but also the user information.

[0109] Various types of user information are conceivable. For example, when solving problems in the retail industry, user information could include purchase data such as POS (Point of Sales) data for each store, display data and display location information for advertising signage, location information for each store, information on other stores and facilities in the vicinity of each store, and customer count information.

[0110] Furthermore, when addressing challenges in the manufacturing industry, potential user information could include the daily number of defective products manufactured at the factory, factory map information, employee work shift information, and information on product inspection standards and inspection results.

[0111] Furthermore, when addressing challenges in the food and beverage industry, potential user information could include sales data for each product or store, location information for each store, information on other stores and facilities in the vicinity of each store, customer count information, employee work shift information, inventory information, and campaign information.

[0112] In other words, user information can be rephrased as information relating to the spatial SP of the sensing target. By inputting this various types of user information into a large-scale language model M, it becomes possible to obtain highly accurate responses from the large-scale language model M.

[0113] <5. Realization of Functions> The processing circuit 51 is realized by a circuit that performs calculations. That is, the circuit as the processing circuit 51 realizes the functions of the receiving unit F1, the answer acquisition unit F2, and the presentation processing unit F3 by performing predetermined calculations.

[0114] When the processing circuit 51 is a control unit implemented by arithmetic circuits such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit), various functions are realized by the processing circuit 51 executing a program to perform predetermined calculations using various memory or other storage areas. That is, predetermined programs and calculation results obtained during the execution of the program are appropriately written to memory or other storage areas provided inside or outside the processing circuit 51.

[0115] Furthermore, when the processing circuit 51 is implemented using a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), various functions can be realized without requiring memory or other storage areas provided outside the processing circuit 51, by designing and constructing the circuit in a manner that realizes predetermined functions. However, in realizing various functions using an ASIC or FPGA, the memory or other storage areas provided inside the processing circuit 51 may be utilized.

[0116] These examples of implementations can be applied to the camera's in-camera processing circuit 74, the sensor's in-sensor processing circuit 93, and the like.

[0117] <6. Processing Flow> The general flow of processing in each device will be explained with reference to Figure 6. Figure 6 shows the process up to the creation of the application AP requirements definition document, which is one of the steps involved in generating and presenting an application AP to solve the problem. Here, we will explain an example in which multiple imaging devices 3 are installed in a store in the retail industry. This store will be referred to as "Target Store St".

[0118] First, the user configures the imaging device 3, which is placed in the spatial SP, using the user terminal 2 or manually (step S1 in Figure 6). The user also uploads user information to the information processing device 1 using the user terminal 2 (step S2).

[0119] In Figure 6, the general processing order is shown in ascending order as steps S1, S2, ..., but the steps do not necessarily have to be executed in the order they are numbered. For example, the processing (operation) of step S2 may be executed before the processing of step S2.

[0120] The user information uploaded to the information processing device 1 may include, for example, location information of the target store St, information on other stores located around the target store St, information on the operating hours of the target store St, layout information of the spatial SP to be sensed at the target store St, information indicating the installation location of each imaging device 3 within the spatial SP, and information on the type of business of the target store St.

[0121] The imaging device 3 starts imaging the spatial SP based on the setting information specified by the user. Each imaging device 3 generates metadata for each image captured (step S3) and uploads it to the information processing device 1 as appropriate (step S4).

[0122] Metadata generation may be performed for each image captured, or it may be performed after a predetermined number of images have been captured. That is, when the imaging device 3 performs imaging of the spatial SP at 30 fps, metadata may be generated for all 30 images captured per second, or metadata may be generated for only one of the 30 images captured per second.

[0123] In the information processing device 1, primary processing is performed using metadata and user information (step S5). The metadata obtained from a single imaging device 3 is data that captures only events that occurred within the field of view of that imaging device 3. Therefore, it is not possible to analyze information that needs to be captured across the entire spatial SP, such as the movement of purchasing customers.

[0124] The primary processing involves organizing metadata generated by multiple imaging devices 3 based on the time of imaging and user information. Specifically, the primary processing can be rephrased as a process to identify the position of each imaging device 3 in the world coordinate system and to enable the interpretation of metadata from the perspective of the entire spatial SP, as well as more advanced analysis.

[0125] Furthermore, for example, if it is determined from the positional relationship in the world coordinate system that a subject captured small by one imaging device 3 is the same person as a subject captured large by another imaging device 3, then the erroneous metadata generated due to the small size of the subject can be corrected and modified in the initial processing.

[0126] Furthermore, the initial processing may include steps to increase the amount of metadata information using user information. For example, if metadata is obtained indicating that a customer picked up a product, the product picked up by the customer may be identified by considering POS data and product display information, or information indicating whether or not the picked-up product was ultimately purchased may be added to the metadata.

[0127] The data generated by the initial processing is considered processed data (step S6).

[0128] User information uploaded from user terminal 2 to information processing device 1 may include not only text data but also image data. User information as image data is analyzed as necessary (step S7) and then converted to text (step S8). The converted user information is used for problem identification processing (step S9). Note that if the user information is originally text data, steps S7 and S8 do not need to be performed.

[0129] The metadata, which has completed the initial processing as processed data, is analyzed (step S10) and used for problem identification processing (step S11). In the metadata analysis and problem identification processing, the information processing device 1 first performs a first prompt generation process to generate a first prompt. Furthermore, after inputting the metadata and text-based user information into the large-scale language model M, the problem identification result is obtained by inputting the first prompt into the large-scale language model M.

[0130] In Figure 6, "Analysis" and "Problem Identification" are shown as separate boxes. However, in reality, these processes may be performed integrally within the large-scale language model M, or they may be performed separately by inputting different prompts corresponding to each process into the large-scale language model M.

[0131] One or more issues obtained through the issue identification process are presented to the user (step S12). The results of the selection operation performed by the user on the presented issues are transmitted to the information processing device 1 (step S13). This allows the information processing device 1 to identify the issues selected by the user that it wishes to solve.

[0132] Figure 7 shows an example of a screen for presenting the identified issues to the user. The example shown in Figure 7 is a screen related to the application generation service for solving the issues, and is an example of a presentation screen G1 for presenting the answer information obtained by the information processing device 1 from the large-scale language model M to the user.

[0133] The display screen G1 comprises a title bar 21 and a display field 22. The title bar 21 displays the name of the service that the user will use and that is provided by the information processing device 1. In this example, the title "Application Builder" is displayed on the title bar 21 to indicate that it is a service that generates and provides an application AP to solve problems related to the spatial SP to be sensed.

[0134] The display area 22 shows information for each of the steps leading up to the generation of the application AP. In the example shown in Figure 7, the results of the problem identification process, specifically the three problems identified by the large-scale language model M, are presented.

[0135] Furthermore, in the presentation section 22, three options are presented for selecting which of the three issues the user wishes to solve.

[0136] The first option, Option Sel1, is selected as the issue you want to solve, which concerns "increased competition and price competition." The second option, Option Sel2, is selected as the issue you want to solve, which concerns "labor shortages and workforce management." The third option, Option Sel3, is selected as the issue you want to solve, which concerns "product display and inventory management."

[0137] Each option, Sel, has a corresponding radio button Rb, allowing the selected option to be easily identified.

[0138] The user can select a single problem to solve by performing a selection operation, choosing one of the options Sel1, Sel2, or Sel3.

[0139] As shown in Figure 7, the presentation screen G1 does not have text boxes or chat input fields. In other words, users can proceed through the process of generating an application AP that helps solve the problem without having to input any text.

[0140] This minimizes the burden on users, allowing even users with low digital literacy to enjoy appropriate services.

[0141] The information of the option Sel selected by the user is transmitted to the information processing device 1.

[0142] The information processing device 1 searches for solutions based on the identified multiple problems and the user's selection information (step S14 in Figure 6). In the solution search process, a second prompt is generated by a second prompt generation process, and a solution is obtained as an answer when the second prompt is input to the large-scale language model M.

[0143] In step S14, for example, past problems, solutions, and metadata used at the time, stored in the memory unit 58, are retrieved and compared with metadata that can be received from the current imaging device 3 to determine whether or not previously presented solutions are available.

[0144] The information processing device 1 creates a requirements specification document for the application AP based on the searched solutions, which are the answers from the large-scale language model M (step S15). In the requirements specification document creation process, a third prompt is generated by a third prompt generation process, and the requirements specification document is obtained as an answer when the third prompt is input to the large-scale language model M. The requirements specification document is a document that defines the inputs and outputs for the application AP. Furthermore, the requirements specification document created in step S15 does not need to be in a document format; it only needs to be in a format that can be input to the large-scale language model M.

[0145] Figure 8 shows the process following the creation of the requirements specification document. The information processing device 1 generates an application AP based on the created requirements specification document (step S16). In the application AP generation process, a fourth prompt is generated by the fourth prompt generation process, and the fourth prompt is input to the large-scale language model M to obtain the application AP program as the answer.

[0146] Figure 9 shows an example of a presentation screen G2 that is presented to the user during the application AP generation process. As shown in the figure, presentation screen G2 consists of a title bar 21 and a presentation field 22, similar to presentation screen G1.

[0147] The display area 22 contains text indicating that the application AP is under development, along with a download button BDL1 for downloading the requirements definition document.

[0148] The user can download the requirements definition document to user terminal 2 by performing a selection operation, such as clicking, on the download button BDL1.

[0149] The information processing device 1 obtains the pre-adjustment application AP1 through the application AP generation process (step S17). In the application AP generation process, data of previously created application APs stored in the storage unit 58 of the information processing device 1 may be used. By using an application AP that has been previously created and whose normal operation has been confirmed, the information processing device 1 can increase the likelihood that the pre-adjustment application AP1 will operate normally.

[0150] The information processing device 1 also performs a test data creation process using metadata uploaded from the imaging device 3 (step S18). This process yields the test data (step S19).

[0151] The information processing device 1 performs the test by inputting the generated test data into the pre-adjustment application AP1 (step S20).

[0152] The test performed in step S20 is, for example, an End-to-End (E2E) test. By performing an E2E test, it is possible to discover defects that depend on the user environment, which differs for each user and spatial SP that is the target of sensing. Furthermore, it is preferable to perform an E2E test when the performance of the large-scale language model M itself is guaranteed.

[0153] During the generation of test data and execution of tests, a fifth prompt is generated through a fifth prompt generation process. This fifth prompt is then input into the large-scale language model M, which yields test results and other information as answers.

[0154] The information processing device 1 obtains test results by executing the test (step S21).

[0155] The information processing device 1 performs a process to adjust the pre-adjustment application AP1 using the test results (step S22). The information processing device 1 obtains the adjusted application AP2 through the adjustment process of the pre-adjustment application AP1 (step S23). In the adjustment process, a sixth prompt is generated by a sixth prompt generation process, and the sixth prompt is input to the large-scale language model M to obtain the program of the adjusted application AP2 as the answer.

[0156] Furthermore, each process from step S18 to step S23 may be executed multiple times. That is, the information processing device 1 may perform further tests on the adjusted application AP2 obtained by adjusting the pre-adjustment application AP1 and perform further adjustments.

[0157] The adjusted application AP2 obtained through the adjustment process is provided to the user in various ways (step S24). An example of a presentation screen G3 presented to the user after the generation of the adjusted application AP2 is shown in Figure 10. As shown in the figure, the presentation screen G3 consists of a title bar 21 and a presentation field 22, similar to presentation screens G1 and G2.

[0158] The display area 22 contains text indicating that the creation of application AP has been completed, along with download buttons BDL2, BDL3, and BDL4 for downloading test data, test results, and the source code of the adjusted application AP2, respectively.

[0159] The test results available for download may include not only text and tables showing the test results, but also video files that demonstrate the application's behavior during the test execution.

[0160] Although the presentation screen G3 is configured by adding text, buttons, etc., to the presentation screen G2 shown in Figure 9, the presentation screen G3 may also be presented as a completely separate screen from the presentation screen G2.

[0161] As can be seen from the display screens G2 and G3, neither screen, like the display screen G1, has text boxes or chat input fields. In other words, the user can download the generated application AP and associated data to the user terminal 2 without having to input any text.

[0162] The user attempts to resolve issues related to spatial SP by downloading and executing the adjusted application AP2 generated by the information processing device 1 to the user terminal 2.

[0163] Furthermore, downloading the adjusted application AP2 to the user terminal 2 is not mandatory for resolving the issue. For example, the adjusted application AP2 may be provided to the user as a cloud application on the information processing device 1.

[0164] <7. Variations in Implementation> The general flow described above remains the same even if the industry or the spatial SP being sensed differs. Here, we will explain the parts that differ depending on the industry or spatial SP.

[0165] <7-1. Metadata> The metadata generated in the imaging device 3 in step S3 of Figure 6 and uploaded to the information processing device 1 in step S4 may differ depending on the industry and spatial SP. For example, the relative position information of the imaging device 3 is metadata that is uploaded in common regardless of the industry or spatial SP.

[0166] In the case of a retail business, metadata indicating the detection results of store employees and customers as subjects, metadata indicating the detection results of customers' gaze, metadata indicating the detection results of fixtures on which products are displayed, metadata indicating the detection results of store employees and customers' postures, and metadata indicating the detection results of products are uploaded to the information processing device 1.

[0167] In the case of a manufacturing company, metadata such as the detection results of manufactured products, metadata showing the detection results of normal products, metadata showing the detection results of products deemed defective, metadata showing the detection results of employees, metadata showing the detection results of manufacturing equipment, and metadata showing the detection results of normal or faulty operation of manufacturing equipment are uploaded to the information processing device 1.

[0168] In the case of a restaurant, metadata such as the detection results of waitstaff and customers, the detection results of tables and chairs, the detection results of the state of the table, such as the arrangement of tableware, utensils, and table condiments, and the detection results of customers' eating status are uploaded to the information processing device 1.

[0169] The metadata may also differ depending on the spatial SP. For example, the aforementioned example of the restaurant industry is an example where the spatial SP where the imaging device 3 is installed is the customer seating area. If the spatial SP where the imaging device 3 is installed is the kitchen area, metadata indicating the detection results of cooking staff, metadata indicating the detection results of finished dishes, metadata indicating the detection results of ingredients or seasonings, and metadata indicating the detection results of cooking utensils will be uploaded to the information processing device 1.

[0170] Furthermore, if the space SP where the imaging device 3 is installed is a store's parking lot, metadata indicating the detection results of parking spaces, metadata indicating the detection results of vehicles, metadata indicating the detection results of customers getting in and out of vehicles, and metadata indicating the detection results of pedestrians and cyclists walking on the sidewalk are uploaded to the information processing device 1.

[0171] In this way, by uploading metadata that differs for each industry or spatial SP to the information processing device 1, it becomes possible to identify specific issues corresponding to the installation configuration of the imaging device 3.

[0172] <7-2. User Information> The user information uploaded to the information processing device 1 in step S2 of Figure 6 may differ depending on the industry and spatial SP.

[0173] The differences in user information uploaded to the information processing device 1 in the retail, manufacturing, and food service industries have already been described, so we will avoid repeating this explanation.

[0174] Furthermore, even within the same industry, user information may differ depending on the more specific category. For example, even within the retail industry, user information may differ between convenience stores, specialty stores, and department stores.

[0175] In the case of a department store, information such as floor-by-floor map information, store information, and information on ongoing or planned events and exhibitions are uploaded to the information processing device 1 as user information.

[0176] Furthermore, in the case of a specialty store, information about the products it handles (such as "shoes," "vegetables," or "books") and more detailed information about the products (for example, for "books," "new books" or "used books") are uploaded to the information processing device 1 as user information.

[0177] <7-3. Identified Issues and Solutions> Due to differences in metadata and user information uploaded to the information processing device 1, a variety of issues and solutions can be identified by the information processing device 1 using the large-scale language model M. One example is described here.

[0178] In the retail industry, some challenges include the low effectiveness of advertising signage, poor customer flow within stores, frequent inquiries to staff, poor detection accuracy of the AI ​​model stored internally by the imaging device 3 when detecting customers, and a high incidence of shoplifting.

[0179] Possible solutions to the problem of low effectiveness of digital advertising include changing the type of digital advertising, or changing the placement and display locations. Furthermore, the resulting application application (AP) could be one that visualizes the viewing frequency, viewing time, and viewing rate of the digital advertising.

[0180] Possible solutions to the problem of poor customer flow within stores include optimizing the placement of fixtures, the location of sample corners, and the display of products. Furthermore, the resulting applications could include applications that visualize customer dwell time, such as heatmaps of customer movement, and applications that propose various optimizations.

[0181] Possible solutions to the problem of frequent inquiries to store staff include optimizing product display locations and optimizing the placement of promotional tools containing product descriptions. The resulting application APs could include application APs that suggest product placement and promotional tool placement locations, as well as application APs that suggest wording to be included in promotional tools.

[0182] Possible solutions to the problem of poor detection accuracy of the AI ​​model include optimizing the installation location of the imaging device 3 and optimizing the shooting settings of the imaging device 3. Furthermore, the generated application APs could include an application AP that optimizes the settings of the imaging device 3 over time, and an application AP that suggests installation locations.

[0183] Possible solutions to the problem of frequent shoplifting include creating stores that are difficult to shoplift from and quickly detecting shoplifting incidents. Potential applications that could be generated include applications that provide useful information about store design, applications that show the movement patterns of suspicious customers, applications that visualize areas prone to shoplifting, and applications that show times of day when shoplifting is most likely to occur.

[0184] In the manufacturing industry, some challenges include a higher number of defective products in certain areas compared to other areas, a higher incidence of contact accidents between employees, and a correlation between employee movement patterns and the areas where defective products are found.

[0185] Possible solutions to the problem of a higher number of defective products in a specific area compared to other areas include reviewing the work processes performed in the target area and optimizing the timing of maintenance of manufacturing equipment. The resulting application applications could include applications that visualize the ratio of defective products to work processes, applications that visualize the relationship between the timing of maintenance of manufacturing equipment and the timing of defective product occurrence, and applications that provide notifications and scheduling for maintenance timing.

[0186] Possible solutions to the problem of frequent contact accidents between employees include changing employee movement patterns and altering the location of manufacturing equipment. The resulting application applications could include those that visualize employee movement patterns and those that simulate changes to employee and manufacturing equipment locations.

[0187] One possible solution to the problem of a correlation between employee movement patterns and areas where defective products are found is to review employee placement. The resulting application applications could include those that simulate changes in employee placement and those that visualize quantitative data that changes as a result of these placement changes.

[0188] In the food and beverage industry, some of the challenges include a low customer turnover rate during certain times of the day, low table occupancy rates at certain tables, insufficient clearing of tableware, and long wait times between calling a waitstaff member and their arrival at the customer's table.

[0189] Possible solutions to the problem of low customer turnover during certain time periods include optimizing the matching of customers to tables and preventing the forgetting to refill tableware and condiments. The resulting application applications could include applications that visualize the changes in the refill rate of tableware and condiments on tables, and applications that notify customers when it is time to refill table condiments.

[0190] Possible solutions to the problem of poor tableware replenishment rates at certain tables include changing employee work shifts and making the timing of tableware replenishment clearer. The resulting application APs could include an application AP for generating employee work shifts and an application AP for notifying employees of the timing of tableware replenishment.

[0191] One possible solution to the problem of dishes and other items on tables not being cleared is to make it easier to identify which tables need to be cleared. The resulting application applications could include one that notifies users which tables need to be cleared, or one that notifies users in advance which tables will soon need to be cleared.

[0192] Possible solutions to the problem of long waiting times between calling a waiter and their arrival at the customer's table include making customer requests clearer and predicting customer requests. Furthermore, the resulting application application could be one that monitors each table and notifies the staff of tasks that need to be performed before customer requests arise.

[0193] It should be noted that the metadata, user information, issues, solutions, and application APs described here are merely examples, and many other diverse examples are possible.

[0194] <8. Database> The information stored in the storage unit 58, which functions as a database in the information processing device 1, will be described below.

[0195] Information stored in the storage unit 58, which serves as a database of past cases, and used to solve problems in the retail industry, includes, for example, metadata related to customer dwell time and fixtures, an application AP that visualizes sales floor information based on POS data, metadata generated by an AI model that performs people detection, metadata generated by an AI model that detects fixtures, and metadata indicating the position of the imaging device 3 in the world coordinate system.

[0196] Furthermore, information used to solve problems in the food and beverage industry includes, for example, an application AP that directs newly arrived customers to a table based on the availability of dishes on a table and the location information of each table in a world coordinate system; metadata generated by an AI model that calculates the occupancy rate of dishes on a table; metadata generated by an AI model that detects the occupancy rate of dishes on a table; and metadata generated by an AI model that detects the number of people.

[0197] Furthermore, if the large-scale language model M used by the information processing device 1 is provided by another server device 4, this information may be stored in the storage unit 58 of the other server device 4.

[0198] <9. Others> In the example described above, metadata was shown to be uploaded directly from the imaging device 3 to the information processing device 1. However, a fog server that manages multiple imaging devices 3 may be provided in the user environment, and the metadata may be uploaded to the information processing device 1 via the fog server.

[0199] In the examples mentioned above, examples for each industry were shown, but the industry may be automatically determined from the image data captured by the imaging device 3. In other words, manual input of industry information is not required.

[0200] <10. Summary> As described above, the information processing device 1 related to this technology includes a receiving unit F1 that receives metadata generated based on the sensing results, and an answer acquisition unit F2 that generates prompts to be input to a large-scale language model M and obtains answers by inputting prompts to the large-scale language model M. The answer acquisition unit F2 executes a first prompt generation process (steps S10, S11) that generates a prompt (first prompt) that instructs the analysis of metadata and the identification of a problem, a second prompt generation process (step S14) that generates a prompt (second prompt) that instructs the search for solutions to solve the identified problem, a third prompt generation process (step S15) that generates a prompt (third prompt) that instructs the definition of requirements for an application AP that realizes a solution selected by the user from among the solutions obtained by the search, and a fourth prompt generation process (step S16) that generates a prompt (fourth prompt) that instructs the generation of an application AP based on the requirements definition. By using the large-scale language model M, the answer acquisition unit F2 can acquire problems in the spatial SP that are obtained as a result of analyzing events and phenomena occurring in the spatial SP of the sensing target from the metadata. In other words, the information processing device 1 can identify problems that the user is not even aware of and present an application (AP) that solves those problems. For example, there is a solution that suggests a recipe by understanding the user's latent needs to satisfy their hunger, but even then, the user still needs to input their desire to "eat something" as a need. However, with this configuration, problems can be extracted without any input of needs at all. As a result, the user does not need to investigate and analyze what the problems are with the store they are involved in operating, for example. Therefore, various improvements related to the user or the spatial SP being sensed can be made without imposing an excessive burden on the user. Generally, the more skilled a user is at grasping problems, that is, the more they can grasp problems with a keen perspective, the better they can set appropriate problems and seek solutions. On the other hand, users who lack such ability cannot properly grasp problems and therefore cannot solve them.Therefore, for example, it can be difficult to appropriately increase sales at a specific store. For instance, a user attempting to enter a particular industry may not be familiar with the specific circumstances and common practices of that industry, making it difficult to identify challenges. However, by using the information processing device 1 equipped with this configuration, it becomes possible to identify appropriate challenges and explore solutions while considering the specific circumstances of the industry, and furthermore, to obtain application APs to implement those solutions, thereby enabling the user to perform at the same level as a specialist in that industry.

[0201] The response acquisition unit F2 in the information processing device 1 may execute a fifth prompt generation process (steps S18, S19, S20, S21) that generates a prompt (fifth prompt) instructing a test of the generated application (pre-adjustment application AP1). For example, the application presented by the large-scale language model M (pre-adjustment application AP1) does not necessarily operate normally. With this configuration, it is possible to instruct a process to test whether the input and output of application AP are normal, thereby ensuring the operation of application AP.

[0202] The response acquisition unit F2 in the information processing device 1 may execute a sixth prompt generation process (steps S22, S23) that generates a prompt (sixth prompt) instructing adjustment of the application (pre-adjustment application AP1) based on the test results of the application. If the normal operation of the application (pre-adjustment application AP1) presented by the large-scale language model M cannot be guaranteed, an improved application (adjusted application AP2) can be obtained by instructing adjustment of the application.

[0203] The information processing device 1 may include a presentation processing unit F3 that presents applications (pre-adjustment application AP1, post-adjustment application AP2) to the user. This allows the user to obtain an application AP to solve problems related to the spatial SP to be sensed with minimal effort. Furthermore, according to the example above, a user who has set up an environment to automatically upload metadata to the information processing device 1 can generate an application AP to solve a problem simply by selecting one problem from several problems presented by the information processing device 1. Moreover, that application AP can also be obtained simply by operating a download button. In other words, the user actions required to solve problems in the user environment are minimized to the greatest extent possible.

[0204] The information processing device 1 may include a presentation processing unit F3 that, when there are multiple identified problems, presents the problems to the user and obtains the user's selection information for the presented problems. For example, if multiple problems are discovered through metadata analysis, these multiple problems can be presented to the user, and the user can be allowed to select one. Therefore, if the problems that users want to solve are different, appropriate problem settings can be made for each user.

[0205] The information processing device 1 may include a presentation processing unit F3 that presents to the user test data obtained by inputting the prompt (fifth prompt) obtained by the fifth prompt generation process (steps S18, S19, S20) into a large-scale language model M, and output data of an application AP for the test data. The test data is data input to the generated application AP, and is, for example, metadata output from an imaging device 3 placed in the space SP to be sensed. In this case, the application AP could be an application AP that visualizes the movement of customers from the metadata output from the imaging device 3, or an application AP that summarizes the attributes of customers in an easy-to-understand manner. By presenting such an application AP to the user, the user can understand the status of the store using the application AP. This makes it possible for the user to resolve problems that occur in the store at an early stage.

[0206] The information processing device 1 may include a presentation processing unit F3 that presents the user with the answer obtained by inputting the prompt (third prompt) obtained by the third prompt generation process (step S15) into the large-scale language model M. The third prompt generation process is a process that generates a prompt instructing the user to define the requirements of the application AP to be created. For example, the presentation processing unit F3 presents the user with the requirements definition document generated by the large-scale language model M. This allows the user to understand the specifications of the application AP to be created while creating the application AP, which can help in the smooth operation of the application AP.

[0207] The response acquisition unit F2 in the information processing device 1 may obtain the problem identification result by inputting the prompt (first prompt) generated in the first prompt generation process (steps S10, S11) and user information into the large-scale language model M. By providing various user information to the large-scale language model M, it becomes possible to perform more accurate and appropriate analysis and problem identification.

[0208] In the information processing device 1, user information may be information relating to the spatial SP that is the target of sensing. For example, if the spatial SP to be sensed is the sales floor space of a store, the location information and industry information of that store are input into the large-scale language model M as user information. In addition, the information relating to the spatial SP may be information about other stores in the vicinity of a particular store. This makes it possible to identify appropriate issues even in cases that cannot be fully analyzed by metadata analysis alone. For example, even if the same metadata is obtained, it becomes possible to identify issues that are appropriate to the store's location conditions.

[0209] In the information processing device 1, user information may be information relating to products manufactured or sold in the spatial SP. For example, in the case of a retail business, POS data may be used as user information. This makes it possible to perform highly accurate analysis for each user, taking into account sales figures, and to identify user-specific issues.

[0210] In the information processing device 1, metadata may be generated by an imaging device 3 located in a predetermined space SP that is the target of sensing. For example, simply uploading metadata output from an imaging device 3 installed in a sales area to the information processing device 1, which acts as a server, automatically identifies the problem, generates an application AP to solve that problem, and presents the application AP. Therefore, users can obtain significant results with minimal effort. Furthermore, by configuring the imaging device 3 not to upload captured images to the information processing device 1, privacy can be protected and communication volume can be reduced.

[0211] The information processing method relating to this technology is an information processing method in which the information processing device 1 executes the following: receiving metadata generated based on the sensing results; a first prompt generation process (steps S10, S11) that generates a prompt (first prompt) that instructs the analysis of the metadata and the identification of a problem; a second prompt generation process (step S14) that generates a prompt (second prompt) that instructs the search for solutions to solve the identified problem; a third prompt generation process (step S15) that generates a prompt (third prompt) that instructs the definition of requirements for an application AP that realizes a solution selected by the user from among the solutions obtained through the search; and a fourth prompt generation process (step S16) that generates a prompt (fourth prompt) that instructs the generation of an application AP based on the requirements definition.

[0212] The program relating to this technology is a program that causes the information processing device 1 to execute the following: a process for receiving metadata generated based on the sensing results; a first prompt generation process (steps S10, S11) that generates a prompt (first prompt) instructing the analysis of the metadata and identification of a problem; a second prompt generation process (step S14) that generates a prompt (second prompt) instructing the search for solutions to solve the identified problem; a third prompt generation process (step S15) that generates a prompt (third prompt) instructing the definition of requirements for an application AP that realizes a solution selected by the user from among the solutions obtained through the search; and a fourth prompt generation process (step S16) that generates a prompt (fourth prompt) instructing the generation of an application AP based on the requirements definition.

[0213] The various effects and benefits described above can also be obtained through such information processing methods and programs.

[0214] Such programs can be pre-recorded on storage media such as HDDs (Hard Disk Drives) built into computer devices, or on ROMs within microcomputers with CPUs. Alternatively, programs can be temporarily or permanently stored (recorded) on removable storage media such as flexible disks, CD-ROMs (Compact Disk Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable storage media can be provided as so-called packaged software. In addition to being installed from removable storage media to personal computers, such programs can also be downloaded from download sites via networks such as LANs and the Internet.

[0215] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0216] Furthermore, the aforementioned examples can be combined in any way, and the various effects and benefits described above can be obtained even when using various combinations.

[0217] <11. The Technology> The technology can also be configured as follows: (1) An information processing device comprising: a receiving unit that receives metadata generated based on the results of sensing; and an answer acquisition unit that generates prompts to be input to a large-scale language model and obtains an answer by inputting the prompts to the large-scale language model, wherein the answer acquisition unit executes: a first prompt generation process that generates a prompt instructing the analysis of the metadata and the identification of a problem; a second prompt generation process that generates a prompt instructing the search for solutions to solve the identified problem; a third prompt generation process that generates a prompt instructing the definition of requirements for an application that realizes a solution selected by the user from among the solutions obtained by the search; and a fourth prompt generation process that generates a prompt instructing the generation of an application based on the requirements definition. (2) The information processing device according to (1) above, wherein the answer acquisition unit executes a fifth prompt generation process that generates a prompt instructing the testing of the generated application. (3) The information processing device according to (2) above, wherein the answer acquisition unit executes a sixth prompt generation process that generates a prompt instructing the adjustment of the application based on the test results of the application. (4) An information processing device according to any one of (1) to (3) above, further comprising a presentation processing unit for presenting the application to the user. (5) An information processing device according to any one of (1) to (4) above, further comprising a presentation processing unit for presenting the tasks to the user and obtaining the user's selection information for the presented tasks when there are multiple specified tasks. (6) An information processing device according to any one of (2) to (3) above, further comprising a presentation processing unit for presenting test data obtained by inputting the prompts obtained by the fifth prompt generation process into a large-scale language model and output data of the application for the test data to the user. (7) An information processing device according to any one of (1) to (6) above, further comprising a presentation processing unit for presenting the answers obtained by inputting the prompts obtained by the third prompt generation process into a large-scale language model to the user.(8) The response acquisition unit is an information processing device according to any one of (1) to (7) above, wherein the response acquisition unit obtains the result of identifying the problem by inputting the prompt generated in the first prompt generation process and user information into a large-scale language model. (9) The information processing device according to (8) above, wherein the user information is information relating to the space that was the target of sensing. (10) The information processing device according to (9) above, wherein the user information is information relating to goods manufactured or sold in the space. (11) The metadata is an information processing device according to any one of (1) to (10) above, generated in an imaging device placed in a predetermined space that was the target of sensing. (12) An information processing method in which an information processing device executes: a process for receiving metadata generated based on the results of sensing; a first prompt generation process for generating a prompt instructing the analysis of the metadata and identification of a problem; a second prompt generation process for generating a prompt instructing the search for a solution to solve the identified problem; a third prompt generation process for generating a prompt instructing the definition of requirements for an application that realizes a solution selected by the user from among the solutions obtained by the search; and a fourth prompt generation process for generating a prompt instructing the generation of an application based on the requirements definition. (13) A program in which an information processing device executes: a process for receiving metadata generated based on the results of sensing; a first prompt generation process for generating a prompt instructing the analysis of the metadata and identification of a problem; a second prompt generation process for generating a prompt instructing the search for a solution to solve the identified problem; a third prompt generation process for generating a prompt instructing the definition of requirements for an application that realizes a solution selected by the user from among the solutions obtained by the search; and a fourth prompt generation process for generating a prompt instructing the generation of an application based on the requirements definition.

[0218] 1 Information Processing Device 3 Imaging Device AP Application AP1 Application before adjustment AP2 Application after adjustment F1 Receiving Unit F2 Answer Acquisition Unit F3 Presentation Processing Unit M Large-scale Language Model SP Space

Claims

1. An information processing device comprising: a receiving unit that receives metadata generated based on the results of sensing; and an answer acquisition unit that generates prompts to be input to a large-scale language model and obtains an answer by inputting the prompts to the large-scale language model, wherein the answer acquisition unit executes: a first prompt generation process that generates a prompt instructing the analysis of the metadata and the identification of a problem; a second prompt generation process that generates a prompt instructing the search for solutions to solve the identified problem; a third prompt generation process that generates a prompt instructing the definition of requirements for an application that realizes a solution selected by the user from among the solutions obtained by the search; and a fourth prompt generation process that generates a prompt instructing the generation of an application based on the requirements definition.

2. The information processing apparatus according to claim 1, wherein the response acquisition unit executes a fifth prompt generation process that generates a prompt instructing the test of the generated application.

3. The information processing apparatus according to claim 2, wherein the response acquisition unit executes a sixth prompt generation process that generates a prompt instructing adjustment of the application based on the test results of the application.

4. The information processing apparatus according to claim 1, further comprising a presentation processing unit for presenting the application to a user.

5. The information processing apparatus according to claim 1, further comprising a presentation processing unit that, when there are multiple identified problems, presents the problems to the user and obtains the user's selection information for the presented problems.

6. The information processing apparatus according to claim 2, further comprising a presentation processing unit that presents to the user test data obtained by inputting the prompts obtained by the fifth prompt generation process into a large-scale language model, and output data of the application for the test data.

7. The information processing apparatus according to claim 1, further comprising a presentation processing unit that presents to the user an answer obtained by inputting the prompt obtained by the third prompt generation process into a large-scale language model.

8. The information processing apparatus according to claim 1, wherein the response acquisition unit inputs the prompt generated in the first prompt generation process and user information into a large-scale language model to obtain the result of identifying the problem.

9. The information processing apparatus according to claim 8, wherein the user information is information relating to the space that was the subject of the sensing.

10. The information processing apparatus according to claim 9, wherein the user information is information relating to goods manufactured or sold in the space.

11. The information processing device according to claim 1, wherein the metadata is generated in an imaging device located in a predetermined space that is the target of sensing.

12. An information processing method in which an information processing device performs the following steps: receiving metadata generated based on the results of sensing; a first prompt generation process that generates a prompt instructing the analysis of the metadata and identification of a problem; a second prompt generation process that generates a prompt instructing the search for solutions to resolve the identified problem; a third prompt generation process that generates a prompt instructing the definition of requirements for an application that implements a solution selected by the user from among the solutions obtained by the search; and a fourth prompt generation process that generates a prompt instructing the generation of an application based on the requirements definition.

13. A program that causes an information processing device to execute the following: a process for receiving metadata generated based on the results of sensing; a first prompt generation process for generating a prompt instructing the analysis of the metadata and identification of a problem; a second prompt generation process for generating a prompt instructing the search for solutions to resolve the identified problem; a third prompt generation process for generating a prompt instructing the definition of requirements for an application that implements a solution selected by the user from among the solutions obtained through the search; and a fourth prompt generation process for generating a prompt instructing the creation of an application based on the requirements definition.

Images