Data processing device, data processing method, and data processing program

Abstract

To appropriately respond to user inputs.SOLUTION: A data processing device includes: an acquisition unit for acquiring input data that indicates a user input; a processing unit for performing specific processing using a generation model that generates information corresponding to the input data; and an output unit for outputting the result of the specific processing to an electronic apparatus.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology disclosed herein relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art leaves room for improvement in responding appropriately to user input. [Means for solving the problem]

[0005] A data processing device of a first aspect includes an acquisition unit that acquires input data indicating a user input, a processing unit that performs a specific processing using a generative model that generates information according to the input data, and an output unit that outputs a result of the specific processing to an electronic device, wherein the acquisition unit acquires destination information that specifies a destination and time point information that specifies a time point as the input data, the processing unit inputs a prompt including the destination information and the time point information acquired by the acquisition unit and an instruction to inquire about an image of the destination at the time point indicated by the destination information and the time point information into the generative model as a first prompt to acquire an image of the destination corresponding to the first prompt, inputs a prompt including the destination information, the time point information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time point indicated by the destination information and the time point information into the generative model as a second prompt to generate a video of the destination corresponding to the second prompt, and the control unit outputs the video of the destination corresponding to the second prompt generated by the processing unit to the electronic device.

[0006] A data processing device of a second aspect is a data processing device of the first aspect, wherein the acquisition unit acquires a theme of a video as the input data, and the processing unit inputs a prompt including the theme, the destination information, the time information, and an image of a destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information, into the generative model as the second prompt, thereby generating a video of the destination corresponding to the second prompt.

[0007] A data processing device of a third aspect is a data processing device of the first or second aspect, wherein the acquisition unit acquires height information regarding the user's height as the input data, and the processing unit generates a video of a destination corresponding to the second prompt according to an eye line height derived based on the height information.

[0008] A data processing device of a fourth aspect is a data processing device of any of the first to third aspects, wherein the acquisition unit acquires the time point information specifying multiple time points as the input data, and the processing unit inputs a prompt including the destination information, the time point information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time point indicated by the destination information and the time point information into the generative model as the second prompt, thereby generating a video of the destination corresponding to the second prompt in which the multiple time points indicated by the time point information are arranged in chronological order.

[0009] A data processing device of a fifth aspect is a data processing device of any of the first to fourth aspects, wherein the acquisition unit acquires the camera's movement speed during mobile shooting as the input data, and the processing unit generates a video of the destination corresponding to the second prompt that reflects the movement speed.

[0010] A data processing method of a sixth aspect is performed by a computer to execute the following processes: acquire input data indicating a user input; perform a specific process using a generative model that generates information corresponding to the input data; output a result of the specific process to an electronic device; acquire as the input data destination information that specifies a destination and time point information that specifies a time point; input a prompt including the acquired destination information and time point information, and an instruction to inquire about an image of the destination at the time point indicated by the destination information and the time point information, into the generative model as a first prompt; acquire an image of the destination corresponding to the first prompt; input a prompt including the destination information, the time point information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time point indicated by the destination information and the time point information, into the generative model as a second prompt; generate a video of the destination corresponding to the second prompt; and output the generated video of the destination corresponding to the second prompt to the electronic device.

[0011] A data processing program of a seventh aspect causes a computer to execute the following processes: acquire input data indicating a user input; perform a specific process using a generative model that generates information corresponding to the input data; output a result of the specific process to an electronic device; acquire as the input data destination information that specifies a destination and time point information that specifies a time point; input a prompt including the acquired destination information and time point information, and an instruction to inquire about an image of the destination at the time point indicated by the destination information and the time point information, into the generative model as a first prompt; acquire an image of the destination corresponding to the first prompt; input a prompt including the destination information, the time point information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time point indicated by the destination information and the time point information, into the generative model as a second prompt; generate a video of the destination corresponding to the second prompt; and output the generated video of the destination corresponding to the second prompt to the electronic device. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Diagram 2] 1 shows an example of a headset type terminal. [Diagram 3] 2 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal. FIG. [Figure 4] 4 is a block diagram showing an example of a functional configuration of a specific processing unit. FIG. [Diagram 5] 13 is an example of an operational flow regarding an operation of the identification processing unit performing a first identification process and a second identification process. [Figure 6] 1 shows a first emotion map onto which multiple emotions are mapped. [Figure 7] 13 shows a second emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a data processing program according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained. In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0015] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0019] (First embodiment) First, a first embodiment of a data processing system 10 according to the embodiment will be described.

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment. 1, the data processing system 10 includes a data processing device 12 and a headset type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the headset type terminal 14 is an example of an "electronic device" according to the technology of the present disclosure.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0022] The headset type terminal 14 includes a computer 36, a microphone 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0023] The microphone 38 receives instructions and the like from the user 20 (see FIG. 2) by receiving voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20, converts the captured voice into voice data, and outputs it to the processor 46. The control unit 46A (see FIG. 3) of the processor 46 transmits input data indicating the user input received by the microphone 38 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the input data indicating the user input.

[0024] The output device 40 includes a display, a speaker, etc. (not shown), and presents data to the user 20 by outputting the data in an expression form (e.g., voice and / or text) that the user 20 can perceive. The display displays visible information such as text and images according to instructions from the processor 46. The speaker outputs voice according to instructions from the processor 46.

[0025] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0027] Fig. 2 is an explanatory diagram showing an example of the headset type terminal 14. As shown in Fig. 2, the headset type terminal 14 is a goggle-type smart device, and is worn to cover the eyes of the user 20 in the same manner as general goggles.

[0028] Fig. 3 shows an example of main functions of the data processing device 12 and the headset type terminal 14. As shown in Fig. 3, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0029] The specific processing program 56 is an example of a "data processing program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the headset type terminal 14, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the processing of the specific processing unit 290 when the data processing device 12 performs specific processing for generating information corresponding to input data indicating a user input. FIG.

[0033] As shown in FIG. 4, the specific processing unit 290 includes an acquisition unit 290A, a processing unit 290B, and an output unit 290C.

[0034] The acquiring unit 290A acquires input data indicating a user input accepted by the headset type terminal 14. Specifically, the acquiring unit 290A acquires input data based on a voice input of the user 20 accepted by the headset type terminal 14.

[0035] The acquisition unit 290A also acquires destination information specifying a destination and time information specifying a time as input data. For example, the acquisition unit 290A acquires destination information and time information as input data based on a voice input of the user 20 received by the headset type terminal 14. Here, the destination information is input data indicating a destination that exists in the real world (e.g., Tokyo, Hokkaido, Okinawa, etc.) or a destination that exists in a virtual world such as a game or anime. The time information is input data indicating a time in the past, present, or future. Note that the time that can be specified by the time information is not particularly limited, and may be in units of years, years and months, or years and months and days, and may also include hours and minutes.

[0036] The processing unit 290B performs a specific process using the data generation model 58. The data generation model 58 is a so-called generative AI (Artificial Intelligence). The data generation model 58 is configured by combining generative AIs such as ChatGPT, Stable Diffusion, and Gen-2. ChatGPT is used, for example, for Internet searches.<URL: https: / / openai.com / blog / chatgpt> and internet searches<URL: https: / / aismiley.co.jp / ai_news / chatgpt-tsukattemita / > Since the method is publicly known as disclosed in, for example, the Internet search, Stable Diffusion,<URL: https: / / ja.stability.ai / stable-diffusion> and internet searches<URL: https: / / aismiley.co.jp / ai_news / what-is-stable-diffusion / > Since this is publicly known as disclosed in, for example, the Internet search, Gen-2 is not described here.<URL: https: / / research.runwayml.com / gen2> and internet searches<URL: https: / / find-a.jp / seotimes / runway / > Since this is publicly known as disclosed in, for example, US Pat. No. 6,399,343, detailed description thereof will be omitted.

[0037] The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data, text data, image data, and video data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The data generation model 58 is an example of a "generation model" according to the technology of the present disclosure.

[0038] In addition, the above-mentioned identification process includes a first identification process for acquiring an image of the destination specified by the user 20 as information corresponding to the input data, and a second identification process for generating a video of the destination at the time specified by the user 20 as the information.

[0039] The processing unit 290B inputs a prompt including the destination information and time information acquired by the acquisition unit 290A and an instruction to inquire about an image of the destination at the time indicated by the destination information and time information as a first prompt to the data generation model 58, and acquires an image of the destination corresponding to the first prompt. At this time, the processing unit 290B may acquire an image searched for on the Internet by the data generation model 58 as the "destination image", or may acquire an image generated by the data generation model 58 as the "destination image".

[0040] In addition, the processing unit 290B inputs a prompt including destination information, time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and time information into the data generation model 58 as a second prompt, and generates a video of the destination corresponding to the second prompt.

[0041] The output unit 290C outputs the result of the specific processing to the headset type terminal 14. Specifically, the output unit 290C outputs a video of the destination corresponding to the second prompt generated by the processing unit 290B to the headset type terminal 14. In the headset type terminal 14, the control unit 46A causes the output device 40 to output the result of the specific processing. As a result, the video of the destination is played on the output device 40 of the headset type terminal 14.

[0042] Fig. 5 shows an example of an operation flow relating to an operation in which the specific processing unit 290 performs a first specific processing for acquiring an image of a destination specified by the user 20 and a second specific processing for generating a video of the destination at a time point specified by the user 20. The operation flow shown in Fig. 5 is performed by the specific processing unit 290 reading out the specific processing program 56 from the storage 32, expanding it in the RAM 30, and executing it. The operation flow shown in Fig. 5 is automatically executed repeatedly, for example, at regular intervals.

[0043] 5, the specific processing unit 290 acquires input data indicating a user input accepted by the headset type terminal 14. Then, the specific processing unit 290 proceeds to step S11.

[0044] In step S11, the identification processing unit 290 determines whether or not the input data acquired in step S10 satisfies the trigger condition. If the identification processing unit 290 determines that the trigger condition is satisfied (step S11: YES), the identification processing unit 290 proceeds to step S12. On the other hand, if the identification processing unit 290 determines that the trigger condition is not satisfied (step S11: NO), the operation flow shown in Fig. 5 ends. As an example, if the input data acquired in step S10 is destination information and time point information, the identification processing unit 290 determines that the trigger condition is satisfied.

[0045] In step S12, the specific processing unit 290 inputs a prompt including the destination information and time information acquired in step S10 and an instruction to inquire about an image of the destination at the time indicated by the destination information and time information as a first prompt to the data generation model 58, and acquires a result of the first specific processing based on the output of the data generation model 58. For example, the specific processing unit 290 acquires voice data of "Okinawa" as the destination information and voice data of "August 2020" as the time information. In this case, the specific processing unit 290 inputs text "Please search or generate an image of the destination at the time indicated by the following destination information and time information. Destination information: Okinawa Time information: August 2020" as the first prompt to the data generation model 58. Here, the specific processing unit 290 acquires an image of the destination corresponding to the first prompt as a result of the first specific processing. Then, the specific processing unit 290 proceeds to step S13.

[0046] In step S13, the specific processing unit 290 inputs a prompt including the destination information time point information acquired in step S10 and the destination image corresponding to the first prompt acquired in step S12, and an instruction to generate a video of the destination at the time indicated by the destination information and time point information, into the data generation model 58 as a second prompt, and acquires the result of the second specific processing based on the output of the data generation model 58. For example, the specific processing unit 290 acquires image data of "xxxyyyzzz.jpg" as the destination image corresponding to the first prompt in step S12. In this case, the specific processing unit 290 inputs the text "Please generate a video of the destination at the time indicated by the destination information and time point information below based on the image shown in the following image data. Image data: xxxyyyzzz.jpg Destination information: Okinawa Time information: August 2020" into the data generation model 58 as the second prompt. Here, the specific processing unit 290 acquires a video of the destination corresponding to the second prompt as a result of the second specific processing. Then, the specific processing unit 290 proceeds to step S14.

[0047] In step S14, the identification processing unit 290 outputs a result of the second identification processing to the headset type terminal 14. Specifically, the identification processing unit 290 outputs a video of the destination corresponding to the second prompt acquired in step S13 to the headset type terminal 14. As a result, the video of the destination is played on the output device 40 of the headset type terminal 14. Then, the identification processing unit 290 ends the operation flow shown in FIG.

[0048] As described above, in the data processing device 12, the specific processing unit 290 acquires destination information and time information as input data. The specific processing unit 290 also inputs a prompt including the acquired destination information and time information, and an instruction to inquire about an image of the destination at the time indicated by the destination information and time information, as a first prompt to the data generation model 58, and acquires an image of the destination corresponding to the first prompt. The specific processing unit 290 also inputs a prompt including the destination information, time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and time information, as a second prompt to the data generation model 58, and generates a video of the destination corresponding to the second prompt. The specific processing unit 290 then outputs the generated video of the destination corresponding to the second prompt to the headset type terminal 14.

[0049] With the above configuration, the data processing device 12 can output a video of the destination at the time specified by the user 20 to the headset type terminal 14 in appropriate response to a user input. This allows the user 20 to watch a video of the destination he or she desires at a time in the past, present, or future. Therefore, the user 20 can experience, for example, a simulated trip to a destination in the past or future through the headset type terminal 14.

[0050] Second Embodiment Next, a second embodiment of the data processing system 10 according to the present invention will be described while omitting or simplifying parts that overlap with the above embodiment.

[0051] The functional configuration of the specific processing unit 290 according to the second embodiment is, for example, the functional configuration shown in FIG. 4, similar to that of the first embodiment.

[0052] The acquisition unit 290A acquires a theme of the video as input data. For example, the acquisition unit 290A acquires a theme of the video as input data based on a voice input of the user 20 received by the headset type terminal 14. The theme of the video is, for example, a slide show, motion graphics, animation, an introduction video, a recruitment video, a general meeting of employees, a promotion, a branding video, a web CM (Commercial Message), etc.

[0053] The processing unit 290B inputs a prompt including the theme of the video, destination information, time information, and a destination image corresponding to the first prompt acquired by the acquisition unit 290A, and an instruction to generate a video of the destination at the time indicated by the destination information and time information, into the data generation model 58 as a second prompt, and generates a video of the destination corresponding to the second prompt. For example, it is assumed that the acquisition unit 290A acquires voice data of "Okinawa" as destination information, acquires voice data of "August 2020" as time information, and acquires voice data of "Introduction video" as the theme of the video. It is also assumed that the processing unit 290B acquires image data of "xxxyyyzzz.jpg" as an image of the destination corresponding to the first prompt. In this case, the processing unit 290B inputs, for example, the following text into the data generation model 58 as a second prompt: "Based on the image shown in the image data below, please generate a video of the destination at the time indicated by the destination information and time information below, taking into account the following theme. Image data: xxxyyyzzz.jpg Destination information: Okinawa Time information: August 2020 Video theme: Introduction video."

[0054] With the above configuration, the data processing device 12 can generate a destination video based on a theme desired by the user 20. This allows the user 20 to specify various themes and enjoy videos of the destination he or she desires at the time he or she desires from various perspectives.

[0055] (Third embodiment) Next, a third embodiment of the data processing system 10 according to the present invention will be described while omitting or simplifying parts that overlap with the above-described embodiments.

[0056] The functional configuration of the specific processing unit 290 according to the third embodiment is, for example, the functional configuration shown in FIG. 4, similar to that of the first embodiment.

[0057] The acquisition unit 290A acquires height information on the height of the user 20 as input data. For example, the acquisition unit 290A acquires height information as input data based on a voice input of the user 20 received by the headset type terminal 14. Note that the height information is not limited to the height of the user 20 himself / herself, and may be the height of another user (e.g., a child of the user 20).

[0058] The processing unit 290B generates a video of the destination corresponding to the second prompt according to the eye height derived based on the height information. As an example, the processing unit 290B derives a value obtained by multiplying the height indicated by the height information by 0.9 as the "eye height." In this case, the processing unit 290B adds text to the second prompt, for example, "Please generate a video from the viewpoint of a user whose eye height is 160 cm."

[0059] With the above configuration, the data processing device 12 can generate a video of the destination from the viewpoint desired by the user 20. This allows the user 20 to enjoy a video of the destination desired by the user 20 at the time desired by the user 20 from his / her own viewpoint. In addition, the user 20 can check the viewpoints of other users based on the generated video, and can check, for example, whether there are any dangers at the destination from the viewpoint of the other user.

[0060] (Fourth embodiment) Next, a fourth embodiment of the data processing system 10 according to the present invention will be described while omitting or simplifying parts that overlap with the above-described embodiments.

[0061] The functional configuration of the specific processing unit 290 according to the fourth embodiment is, for example, the functional configuration shown in FIG. 4, similar to that of the first embodiment.

[0062] The acquisition unit 290A acquires time point information that specifies a plurality of time points as input data. For example, the acquisition unit 290A acquires time point information that specifies a plurality of time points as input data based on a voice input of the user 20 accepted by the headset type terminal 14.

[0063] The processing unit 290B inputs a prompt including destination information, time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information into the data generation model 58 as a second prompt, and generates a video of the destination corresponding to the second prompt in which a plurality of time points indicated by the time information are arranged in chronological order. For example, it is assumed that the acquisition unit 290A acquires voice data of "Okinawa" as destination information, and voice data of "August 2020", "August 2021", and "August 2022" as time information. It is also assumed that the processing unit 290B acquires image data of "xxxyyyzzz111.jpg", "xxxyyyzzz222.jpg", and "xxxyyyzzz333.jpg" as images of the destination corresponding to the first prompt.

[0064] In the above case, the processing unit 290B inputs, for example, the following text into the data generation model 58 as a second prompt: "Please generate videos of destinations at multiple points in time indicated by the following destination information and time information in chronological order of image data 1, image data 2, and image data 3. Image data 1: xxxyyyzzz111.jpg Destination information: Okinawa Time information: August 2020 Image data 2: xxxyyyzzz222.jpg Destination information: Okinawa Time information: August 2021 Image data 3: xxxyyyzzz333.jpg Destination information: Okinawa Time information: August 2022."

[0065] With the above configuration, the data processing device 12 can generate a video of the destination in which multiple time points designated by the user 20 are arranged in chronological order. This allows the user 20 to check the change in the destination at the multiple time points designated by the user 20 based on the generated video.

[0066] Fifth embodiment Next, a fifth embodiment of the data processing system 10 according to the present invention will be described while omitting or simplifying parts that overlap with the above embodiments.

[0067] The functional configuration of the specific processing unit 290 according to the fifth embodiment is, for example, the functional configuration shown in FIG. 4, similar to that of the first embodiment.

[0068] The acquisition unit 290A acquires the moving speed of the camera during moving shooting as input data. For example, the acquisition unit 290A acquires the moving speed of the camera during moving shooting as input data based on the voice input of the user 20 accepted by the headset type terminal 14. In this case, the voice input of the user 20 may include a specific speed such as "the moving speed of the camera is 30 km per hour" or may not include a specific speed such as "the moving speed of the camera is the speed at which a person walks" or "the moving speed of the camera is the speed at which a car runs".

[0069] The processing unit 290B generates a video of the destination corresponding to the second prompt that reflects the moving speed acquired by the acquisition unit 290A. In this case, the processing unit 290B adds, for example, text "Please generate a video of the case where the video is shot while moving at a speed of 30 km per hour" to the second prompt.

[0070] With the above configuration, the data processing device 12 can generate a video of the destination captured with a camerawork desired by the user 20. This allows the user 20 to enjoy a video of the destination of his / her choice at a time point of his / her choice, which is generated with various camerawork desired by the user.

[0071] (others) In the above embodiment, the headset type terminal 14 is an example of the "electronic device" according to the technology of the present disclosure, but is not limited to this. For example, a device other than the headset type terminal 14, such as a smartphone or a tablet terminal, may be an example of the "electronic device" according to the technology of the present disclosure.

[0072] In the above embodiment, the identification processing unit 290 may input a prompt including an instruction to inquire about the state of the destination indicated by the acquired destination information in the identification process to the data generation model 58, and acquire the state of the destination corresponding to the prompt. For example, the identification processing unit 290 acquires text data such as "a lot of people", "the sea is rough", or "it's windy" as the state of the destination.

[0073] In the above case, the specific processing unit 290 may input a prompt including the destination appearance, destination information, time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and time information, as a second prompt to the data generation model 58, to generate a video of the destination corresponding to the second prompt. For example, the specific processing unit 290 inputs text "Based on the image shown in the following image data, please generate a video of the destination at the time indicated by the following destination information and time information, taking into account the appearance of the destination below. Image data: xxxyyyzzz.jpg Destination information: Okinawa Time information: August 2020 Destination appearance: Many people" into the data generation model 58 as the second prompt. With this configuration, the data processing device 12 can generate a video of the destination taking into account the appearance of the destination.

[0074] The processing unit 290B may estimate the emotion of the user 20 using the emotion identification model 59, and perform identification processing using the emotion of the user 20 and the data generation model 58.

[0075] The emotion identification model 59 may determine the emotion of the user 20 according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user 20 according to an emotion map (see FIG. 6), which is a specific mapping.

[0076] FIG. 6 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0077] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0078] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0079] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0080] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0081] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 20. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 7. Fig. 7 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0082] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0083] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0084] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0085] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0086] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0087] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0088] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0089] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The above specific processes are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0090] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0091] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0092] 10 Data Processing System 12 Data Processing Device 14 Headset-type terminal

Claims

1. an acquisition unit for acquiring input data indicative of a user input; A processing unit that performs a specific process using a generative model that generates information according to the input data; an output unit that outputs a result of the specific processing to an electronic device; The acquisition unit is Destination information specifying a destination and time information specifying a time are acquired as the input data; The processing unit includes: inputting a prompt including the destination information and the time information acquired by the acquisition unit and an instruction to inquire about an image of the destination at the time indicated by the destination information and the time information into the generative model as a first prompt, and acquiring an image of the destination corresponding to the first prompt; inputting a prompt including the destination information, the time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information into the generative model as a second prompt, thereby generating a video of the destination corresponding to the second prompt; The output unit is outputting, to the electronic device, a video of a destination corresponding to the second prompt generated by the processing unit; Data processing device.

2. The acquisition unit is Obtaining a theme of a video as the input data; The processing unit includes: inputting a prompt including the theme, the destination information, the time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information into the generative model as the second prompt, thereby generating a video of the destination corresponding to the second prompt; 2. A data processing apparatus according to claim 1.

3. The acquisition unit is Acquire height information regarding the height of a user as the input data; The processing unit includes: generating an animation of a destination corresponding to the second prompt in accordance with the eye level derived based on the height information; 2. A data processing apparatus according to claim 1.

4. The acquisition unit is acquiring the time point information specifying a plurality of time points as the input data; The processing unit includes: inputting a prompt including the destination information, the time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information into the generation model as the second prompt, thereby generating a video of the destination corresponding to the second prompt in which a plurality of time points indicated by the time information are arranged in chronological order; 2. A data processing apparatus according to claim 1.

5. The acquisition unit is The moving speed of the camera during moving shooting is acquired as the input data, The processing unit includes: generating an animation of a destination corresponding to the second prompt, the animation reflecting the moving speed; 2. A data processing apparatus according to claim 1.

6. Obtaining input data indicative of user input; performing a determination process using a generative model that generates information according to the input data; outputting a result of the specific processing to an electronic device; Destination information specifying a destination and time information specifying a time are acquired as the input data; inputting a prompt including the acquired destination information and time information, and an instruction to inquire about an image of the destination at the time indicated by the destination information and the time information, into the generative model as a first prompt, and acquiring an image of the destination corresponding to the first prompt; inputting a prompt including the destination information, the time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information into the generative model as a second prompt, thereby generating a video of the destination corresponding to the second prompt; outputting, to the electronic device, an animation of a destination corresponding to the generated second prompt; A data processing method in which processing is performed by a computer.

7. Obtaining input data indicative of user input; performing a determination process using a generative model that generates information according to the input data; outputting a result of the specific processing to an electronic device; Destination information specifying a destination and time information specifying a time are acquired as the input data; inputting a prompt including the acquired destination information and time information, and an instruction to inquire about an image of the destination at the time indicated by the destination information and the time information, into the generative model as a first prompt, and acquiring an image of the destination corresponding to the first prompt; inputting a prompt including the destination information, the time information, and an image of the destination corresponding to the first prompt, and an instruction to generate a video of the destination at the time indicated by the destination information and the time information into the generative model as a second prompt, thereby generating a video of the destination corresponding to the second prompt; outputting, to the electronic device, an animation of a destination corresponding to the generated second prompt; A data processing program for causing a computer to execute processing.

Images