Data processing device, data processing method, and data processing program

Abstract

The present invention provides a data processing device, a data processing method, and a program that can suggest to a user information corresponding to the content of an utterance relating to the user's memory or actions. [Solution] The data processing device includes an input unit that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor; a processing unit that performs specific processing using a data generation model that generates predetermined inference results according to the user data; and an output unit that plays the results of the specific processing from the speaker. The input unit inputs sound detected by the microphone, images captured by the camera, and biometric data acquired by the biosensor as user data, and the processing unit performs specific processing that proposes information corresponding to the content of speech to the user and acquires haptic feedback.

Description

Technical Field

[0001] The technology of the present disclosure relates to a data processing device, a data processing method, and a data processing program.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, when generating an utterance in response to the words spoken by a user, the actions taken by the user in daily life are not reflected, so there is room for improvement in proposing information corresponding to the content of the user's utterance.

Means for Solving the Problems

[0005] A first aspect of the technology of this disclosure includes an input unit that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor; a processing unit that performs identification processing using a data generation model that generates predetermined inference results according to the user data; and an output unit that plays the results of the identification processing from the speaker, wherein the earphones are canal-type earphones or inner-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones, and the earphones are equipped with a vibration function and are capable of outputting haptic feedback, and the input unit This data processing device inputs the sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the biosensor as user data. When the processing unit receives an utterance from the user wearing the earphones regarding the user's memories or actions, it proposes information corresponding to the content of the utterance to the user based on the user's life log in which the sounds and images associated with the user are recorded, and inputs a prompt to the data generation model instructing it to generate the haptic feedback using the earphones. The processing unit then performs the process of acquiring information corresponding to the content of the utterance and the haptic feedback as the specific processing.

[0006] A second aspect of the technology of this disclosure is a data processing method in which a computer performs a specific processing using a data generation model that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, and generates a predetermined inference result according to the user data, wherein the earphones are canal-type earphones or inner-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones, the earphones are equipped with a vibration function and are capable of outputting haptic feedback, and the sound detected by the microphone, the image captured by the camera, and the data acquired by the earphones This data processing method involves inputting the aforementioned biometric data as user data, receiving an utterance from the user wearing the earphones regarding the user's memories or actions, and then inputting a prompt into the data generation model instructing it to propose information corresponding to the content of the utterance to the user based on the user's life log in which sounds and images associated with the user are recorded, and to generate haptic feedback via the earphones. The computer then performs a process to acquire information corresponding to the content of the utterance and the haptic feedback as the specified process, plays the result of the specified process from the speaker, and outputs haptic feedback using the vibration function.

[0007] A third aspect of the technology of this disclosure is a data processing program that causes a computer to perform a specific processing using a data generation model that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, and generates a predetermined inference result according to the user data, wherein the earphones are canal-type earphones or inner-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones, the earphones are equipped with a vibration function and are capable of outputting haptic feedback, and the sound detected by the microphone, the image captured by the camera, and the data acquired by the earphones This data processing program inputs the aforementioned biometric data as user data, and when it receives an utterance from the user wearing the earphones regarding the user's memories or actions, it inputs a prompt into the data generation model instructing it to propose information corresponding to the content of the utterance to the user based on the user's life log in which sounds and images associated with the user are recorded, and to generate the haptic feedback via the earphones. This process is performed as the specific process to acquire information corresponding to the content of the utterance and the haptic feedback, and the computer is instructed to perform a process to play the result of the specific process from the speaker and output haptic feedback using the vibration function. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a conceptual diagram showing an example of the configuration of a data processing system. [Figure 2] Figure 2 is a conceptual diagram showing an example of the main functions of a data processing device and earphones. [Figure 3A] Figure 3A shows an example of an earphone configuration. [Figure 3B] Figure 3B shows the user wearing earphones. [Figure 3C] Figure 3C is a diagram illustrating the field of view of camera 42. [Figure 3D] Figure 3D shows the user wearing the earphones. [Figure 3E] Figure 3E shows the user wearing the earphones. [Figure 3F] Figure 3F shows the user wearing earphones. [Figure 4] The functional configuration of a specific processing unit of a data processing device is shown in general terms. [Figure 5] This diagram outlines an example of the operation flow of a specific process performed by a data processing device. [Figure 6] Figure 6 is a conceptual diagram showing an example of the configuration of a data processing system. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the data processing device, data processing method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0012] In the following embodiments, the tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by a processor.

[0013] In the following embodiments, the tagged storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the tagged communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] <First Embodiment> FIG. 1 shows an example of the configuration of a data processing system 10 according to an embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and earphones 14. An example of the data processing device 12 is a server. In the present embodiment, the data processing device 12 is an example of the "data processing device" according to the technology of the present disclosure.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of the "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The earphone 14 includes a computer 36, a microphone 38, a speaker 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Also, the microphone 38, the speaker 40, and the camera 42 are connected to the bus 52.

[0020] The microphone 38 receives an instruction or the like from the user 20 by receiving the voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20, converts the captured voice into voice data, and outputs the voice data to the processor 46. The speaker 40 outputs voice according to an instruction from the processor 46. Hereinafter, the microphone 38 may be simply referred to as the microphone 38.

[0021] The camera 42 is a small digital camera equipped with an optical system such as a lens, an aperture, and a shutter, and an imaging device such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and images the surroundings of the user 20 (for example, an imaging range defined by an angle of view corresponding to the field of view of a general healthy person).

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the earphone 14.

[0024] As shown in Figure 2, in the data processing device 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "data processing program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores the data generation model 58. The data generation model 58 is used by the specific processing unit 290.

[0026] (Earphones 14) In the earphone 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0027] The earphone 14 may be interpreted as a canal-type earphone that is fitted into the ear canal of the user 20, as shown in Figure 3A. However, the earphone 14 is not limited to a canal type; it may also be an inner-ear type earphone that is inserted into the inner ear of the user 20, or a headphone type earphone that covers the entire ear of the user 20. Each of the two earphones 14 is equipped with a microphone 38, a speaker 40, and a camera 42. The sound and images collected by the two earphones 14 fitted into the ears of the user 20 may be recorded as a life log in the database 24.

[0028] The life log can be interpreted as a history of the user 20's actions in daily life, and may include sounds and images associated with the user 20, specifically sounds collected by the microphone 38 and images taken by the camera 42 during daily life. The life log may record sounds and images associated with the user 20, along with the date, time, and location in which they were acquired.

[0029] The sounds collected by the microphone 38 may include the voice of the person the user 20 is talking to, and sounds that occur around the user 20 while walking or cycling (such as the sound of cars driving, birds chirping, the babbling of a stream, and the sound of trees swaying in the wind).

[0030] As shown in Figure 3C, the camera 42 may capture images of the scenery within its field of view that is in front of the user 20, or it may capture images of scenery within its field of view that is not in front of the user 20, for example, to the side, behind, below, or above the user 20. The images captured by the camera 42 may include images of the person the user 20 is talking to, the scenery around the user 20 when they are walking or cycling, and images of the pet the user 20 is walking with.

[0031] Since each of the two earphones 14 is equipped with a camera 42, the two earphones 14 worn on the user's ears 20 are positioned at a specific distance apart, one on the left ear and the other on the right ear, as shown in Figure 3B. Therefore, compared to cases where two cameras are arranged side by side in a single housing, such as in a video camera, the spacing between the two cameras 42 can be increased, making 3D sensing easier. 3D sensing can be interpreted as measuring three-dimensional shapes.

[0032] Furthermore, when the two earphones 14 are placed in the user 20's ears, the two cameras 42 are positioned close to the user 20's left and right eyes, allowing images (captured images) that are nearly identical to those seen with the naked eye to be recorded as a life log in the database 24. Consequently, in specific processing, it becomes easier to reproduce information corresponding to inquiries from the user 20, that is, information corresponding to the content of the user 20's speech.

[0033] While the two earphones 14 are attached to the user 20, all or part of the images captured by the camera 42 may be recorded in the database 24 as a life log. Specifically, when the two earphones 14 are attached to the user 20, the recording of images captured by the camera 42 to the database 24 may begin, and when the two earphones 14 are removed from the user 20, the recording of those images to the database 24 may end.

[0034] While the two earphones 14 are worn by the user 20, all or part of the sound collected by the microphone 38 may be recorded as a lifelog in the database 24. Specifically, when the two earphones 14 are worn by the user 20, the recording of the sound collected by the microphone 38 to the database 24 may begin, and when the two earphones 14 are removed from the user 20, the recording of the sound to the database 24 may end.

[0035] Next, we will describe the processing of the specific processing unit 290 when the data processing device 12 receives an utterance from the user 20 wearing the earphones 14 regarding the user 20's memories or actions, and performs specific processing to propose information corresponding to the content of the user 20's utterance to the user 20.

[0036] (Specific processing) In this embodiment, the specific processing involves inputting user data and performing specific processing using a data generation model that generates predetermined inference results corresponding to the input user data. Specifically, in the specific processing, when utterances related to the user's memories or actions are received as user data from a user 20 wearing earphones 14, the system refers to the database 24 and performs processing to propose information corresponding to the content of the utterances to the user 20. Specifically, after a life log is recorded in the database 24, if the user 20 wearing earphones 14 makes an utterance related to the user's memories or actions, the specific processing may involve referring to the database 24 and proposing information corresponding to the content of the utterances to the user 20.

[0037] (Example of specific processing) If the user wearing the earphones requests a message that will trigger the recall of a specific memory, the specific processing unit 290 may propose one or more messages selected based on the life log to the user who made the request, as information corresponding to the content of the utterance (request). For example, if the user wearing the earphones requests a message that will trigger the recall of a specific memory, the specific processing unit 290 may instruct the data generation model 58 in the prompt to propose one or more messages selected based on the life log to the user.

[0038] For example, if user 20, wearing earphones 14, tries to recall their memory and asks, "What did I say to person A around [date] at [time]?", the identification processing unit 290, as part of its identification process, inputs this message as a prompt to the data generation model 58. The identification processing unit 290 may refer to the life log in database 24 and, based on the output obtained from the data generation model 58, generate a message such as, "I think you said, 'I found a nice restaurant, let's make a reservation.'" This message may be interpreted as an example of information corresponding to the content of user 20's utterance.

[0039] For example, if user 20 wearing earphones 14 tries to recall their memory and asks, "Who was I talking to around [date] at [time]?", the identification processing unit 290 will input this message as a prompt to the data generation model 58 as part of its identification process. The identification processing unit 290 may refer to the life log in database 24 and, based on the output obtained from the data generation model 58, generate a message such as, "It seems you were talking with two friends at that time, probably B and C." This message may be interpreted as an example of information corresponding to the content of user 20's utterance.

[0040] For example, if user 20, wearing earphones 14, tries to recall their emotions and says, "How did I feel when I was talking to person A around [date] at [time]?", the identification processing unit 290, as part of its identification process, inputs this message as a prompt to the data generation model 58. The identification processing unit 290 may refer to the life log in database 24 and, based on the output obtained from the data generation model 58, generate a message such as, "At that time, you were laughing a lot, so it seems you had a good impression of your friend and were very happy." This message may be interpreted as an example of information corresponding to the content of user 20's utterance.

[0041] (Example of specific processing, part 2) If a user 20 wearing earphones 14 mutters a specific matter as part of their utterance, the specific processing unit 290 may suggest to the user 20 who requested the message, based on their life log, recommended actions for the user 20 regarding that matter, as information corresponding to the utterance (muttering). For example, if a user 20 wearing earphones 14 mutters a specific matter as part of their utterance, the specific processing unit 290 may instruct the data generation model 58 in the prompt to suggest to the user 20 recommended actions for the user 20 regarding that matter, based on their life log.

[0042] For example, when user 20 wearing earphones 14 is shopping at a specific retail store and says, "What should I buy?", the specific processing unit 290 inputs this message as a prompt to the data generation model 58 as a specific processing step. The specific processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "A few months ago, you purchased product A at this store and commented that it wasn't very tasty, so how about purchasing recently released products B and C this time?" This message may be interpreted as an example of information corresponding to the content of user 20's utterance.

[0043] (Third example of specific processing) As shown in Figure 3D, when user 20, wearing earphones 14, is operating a PC and says, "What was the name of product A that I searched for the day before yesterday?", the identification processing unit 290 inputs this message as a prompt to the data generation model 58 as part of its identification processing. The data generation model 58 refers to the life log in the database 24 and analyzes the video of the PC screen when user 20 was operating it in the past to generate a specific output. Based on the output obtained by the data generation model 58, the identification processing unit 290 may generate a message such as "Product A is ○○○". This message may be interpreted as an example of information corresponding to the content of user 20's utterance.

[0044] (Fourth example of specific processing) As shown in Figure 3E, if user 20, wearing earphones 14, says "There was a place nearby with a great view, but I wonder where it is?" while cycling, the identification processing unit 290 inputs this message as a prompt to the data generation model 58 as part of its identification process. The data generation model 58 refers to the life log in database 24 and analyzes places previously visited by user 20 and the route to those places to generate a specific output. Based on the output obtained by the data generation model 58, the identification processing unit 290 may generate a message such as "I think it's Cape XX, about 500m from here." This message can be interpreted as an example of information corresponding to the content of user 20's utterance.

[0045] (Example 5 of specific processing) As shown in Figure 3F, when user 20, wearing earphones 14, meets Mr. X at company A, the company he is visiting, and says, "Can you tell me this person's name?", the identification processing unit 290 inputs this message as a prompt to the data generation model 58 as part of the identification process. The data generation model 58 refers to the life log in database 24 and generates specific output from the history of people that user 20 met when he visited company A. Based on the output obtained from the data generation model 58, the identification processing unit 290 may generate a message such as, "I think his name is ○○." This message may be interpreted as an example of information corresponding to the content of user 20's utterance.

[0046] As shown in Figure 4, the specific processing unit 290 includes an input unit 291, a processing unit 292, and an output unit 293.

[0047] The input unit 291 acquires user input received through the earphone 14. Specifically, it acquires the user's voice received through the earphone 14.

[0048] The processing unit 292 performs specific processing using the data generation model 58. Specifically, it inputs voice from the user into the data generation model 58 and obtains a generation result. More specifically, when it receives an utterance from the user 20 wearing the earphones 14 regarding the user 20's memories or actions, it performs a specific processing step of proposing information corresponding to the content of the utterance to the user 20.

[0049] The output unit 293 transmits the result of the specific processing to the earphone 14. In the earphone 14, the control unit 46A causes the speaker 40 to output the result of the specific processing. The microphone 38 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0050] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0051] Next, the operation of the data processing system 10 will be explained.

[0052] An example of the flow of a specific processing method will be explained with reference to Figure 5. Note that the flow of a specific processing method shown in Figure 5 is an example of a "data processing method" related to the technology disclosed herein.

[0053] In step S300, the data processing device 12 receives user data, including sound and images collected by the two earphones 14.

[0054] In step S302, if the data processing device 12 receives an utterance from the user wearing the earphones 14 regarding the user's memories or actions, it executes a specific process to propose information to the user 20 that corresponds to the content of the utterance, based on the user's life log.

[0055] In step S303, the data processing device 12 executes a process to play back the result of a specific process from the speaker 40.

[0056] <Second Embodiment> The second embodiment relates to earphones that analyze a user's visual information in real time and generate music and haptic feedback. In particular, the second embodiment relates to a technology that provides a synesthesia experience in which visual, auditory, and tactile senses are fused.

[0057] Traditional earphones primarily focused on voice-related functions such as voice assistants, noise cancellation, or translation features. However, these earphones struggled to meet the needs of users seeking new experiences that integrated visual information. For example, when appreciating paintings in a museum or natural landscapes, there is a demand for a means to achieve a deeper sense of immersion and emotion by integrating visual information with other senses.

[0058] However, conventional technology has made it difficult to analyze visual information in real time and convert it into music or haptic feedback. This has resulted in the challenge of not being able to provide users with a new sensory experience that integrates sight, hearing, and touch.

[0059] Therefore, in the second embodiment, visual information (color, shape, and movement, etc.) acquired by the camera equipped in the earphones is analyzed in real time, and music or sound effects are generated using the data generation model 58. Furthermore, in the second embodiment, the vibration function equipped in the earphones is utilized to provide haptic feedback synchronized with the generated music or sound effects.

[0060] In the second embodiment, users can obtain a new sensory experience that fuses sight, hearing, and touch. For example, when viewing paintings in a museum or natural landscapes, the visual information the user receives is fed back as music and touch, allowing the user to enjoy a deeper sense of immersion and emotion. Furthermore, the second embodiment provides a multimodal experience that could not be achieved with conventional earphones.

[0061] The second embodiment will be described in detail below with reference to the drawings.

[0062] <System Configuration> Figure 6 shows an example of the configuration of the data processing system 210 according to the second embodiment. As shown in Figure 6, the earphones in the second embodiment are two earphones 214 that are worn in the user's ears. The earphones 214 are canal-type earphones or inner-ear type earphones. Therefore, only the user wearing the earphones 214 can hear the sound output from the earphones 214.

[0063] The earphone 214 is equipped with a computer 36, a microphone 38, a speaker 40, a camera 42, a biosensor 43A, a vibration device 43B, and a communication interface 44. The camera 42 is positioned at the same eye level as the user 20, making it possible to acquire information similar to the visual information obtained by the user 20.

[0064] The biosensor 43A in the earphone 214 is capable of acquiring the user's biometric data from the user's ear. This biometric data includes, for example, the user's body temperature, brainwaves, or heart rate.

[0065] The vibration device 43B in the earphone 214 vibrates in response to the received control signal. Therefore, the earphone 214 has a vibration function and is capable of outputting haptic feedback.

[0066] Next, we will describe the processing of the specific processing unit 290 when the data processing device 12 of the second embodiment receives an utterance from the user 20 wearing the earphones 214 regarding the user 20's memories or actions, and performs specific processing to propose information corresponding to the content of the user 20's utterance to the user 20.

[0067] (Specific processing) In the second embodiment, the identification process uses a data generation model that takes user data as input and generates a predetermined inference result corresponding to the input user data. In the second embodiment, the user data consists of sound detected by the microphone 38, images captured by the camera 42, and biometric data acquired by the biosensor 43A. In the identification process, when utterances related to the user 20's memories or actions are received as user data from the user 20 wearing the earphones 214, the process is executed to propose information corresponding to the content of the utterances to the user 20 by referring to the database 24. Specifically, after a life log is recorded in the database 24, if the user 20 wearing the earphones 214 makes an utterance related to the user 20's memories or actions, the identification process is executed to propose information corresponding to the content of the utterances to the user 20 by referring to the database 24. Furthermore, in the second embodiment, the output process of haptic feedback from the earphones 214 is performed as an identification process.

[0068] In the second embodiment, the identification processing unit 290 of the data processing device 12 performs identification processing using the data generation model 58.

[0069] In the second embodiment, the input unit 291 acquires user data received by the earphone 214. Specifically, it acquires the user's voice detected by the microphone 38 of the earphone 214, the image captured by the camera 42, and the biometric data acquired by the biosensor 43A as user data.

[0070] In the second embodiment, the processing unit 292 performs specific processing using the data generation model 58. Specifically, in the second embodiment, when the processing unit 292 receives an utterance from the user 20 wearing the earphones 214 regarding the user 20's memories or actions, it performs specific processing by inputting a prompt to the data generation model 58 that proposes information corresponding to the content of the utterance to the user 20 and instructs it to generate haptic feedback from the earphones 214, thereby acquiring information corresponding to the content of the utterance and haptic feedback.

[0071] More specifically, in the second embodiment, the processing unit 292 inputs a prompt to the data generation model 58 instructing it to generate haptic feedback via the earphone 214, thereby obtaining haptic feedback output from the data generation model 58. For example, the prompt may include a message such as, "Generate haptic feedback (e.g., vibration pattern) suitable for the input user data (voice, image, or biometric data)." This causes the data generation model 58 to output haptic feedback that matches the user data.

[0072] In the second embodiment, the processing unit 292 may acquire haptic feedback by a process different from the process described above. For example, in the second embodiment, the processing unit 292 analyzes video data, which is a sequence of images acquired from the camera 42, in real time and extracts color, shape, and motion features. Next, the processing unit 292 in the second embodiment inputs the extracted features into the data generation model 58 and generates music or sound effects corresponding to the features and biometric data. Alternatively, the processing unit 292 may input the image itself and the biometric data into the data generation model 58 without extracting features from the image and generate music or sound effects. Then, based on the generated music or sound effects, the processing unit 292 in the second embodiment generates a vibration pattern corresponding to haptic feedback.

[0073] The output unit 293 transmits the result of the specific processing to the earphone 214. In the earphone 214, the control unit 46A causes the speaker 40 to output the result of the specific processing. Specifically, the speaker 40 plays the generated music or sound effect. The vibration device 43B also outputs tactile feedback through its vibration function. Specifically, the vibration device 43B outputs tactile feedback synchronized with the music or sound effect. The microphone 38 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0074] (Example of specific processing) <Use in art museums> When user 20, wearing earphones 214, is viewing paintings in a museum, camera 42 captures images of the paintings, and the colors, shapes, etc., of the images are analyzed. In the second embodiment, the specific processing unit 290 generates classical music or unique sound effects based on this information and outputs them from speaker 40. Furthermore, vibration device 43B provides tactile feedback in the form of subtle vibrations in sync with the generated music. As a result, user 20 can feel the atmosphere of the artwork, including the paintings, with their whole body.

[0075] (Example of specific processing, part 2) <Appreciating natural scenery> When user 20 wearing earphones 214 is strolling in nature, camera 42 captures the colors and movement of the scenery. In the second embodiment, specific processing unit 290 generates refreshing music or natural sounds and outputs them from speaker 40. Furthermore, vibration device 43B provides tactile feedback in the form of subtle vibrations in sync with the generated music. This provides user 20 with a rich experience through both hearing and touch.

[0076] The other system configurations and operations of the data processing system 210 according to the second embodiment are the same as those of the first embodiment, so a detailed explanation is omitted.

[0077] As described above, the data processing system 210 according to the second embodiment can be used not only in the entertainment field but also in a wide range of fields such as education, rehabilitation, and tourism. According to the data processing system 210 according to the second embodiment, users can experience visual information in a multi-sensory way, and new value is created.

[0078] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0079] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0080] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0081] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0082] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0083] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0084] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0085] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0086] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0087] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0088] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0089] Furthermore, the following additional information is disclosed regarding the above explanation.

[0090] (Note 1) An input unit that receives user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, A processing unit that performs specific processing using a data generation model that generates predetermined inference results according to the user data, An output unit that reproduces the result of the specified processing from the speaker, Equipped with, The aforementioned earphones are canal-type earphones or in-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones. The aforementioned earphones are equipped with a vibration function and are capable of outputting haptic feedback. The input unit inputs the sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the biosensor as user data. The processing unit, upon receiving a speech from the user wearing the earphones regarding the user's memories or actions, performs the following as a specific process: proposing information corresponding to the content of the speech to the user based on the user's life log in which sounds and images associated with the user are recorded, and inputting a prompt to the data generation model instructing it to generate the haptic feedback using the earphones, thereby acquiring information corresponding to the content of the speech and the haptic feedback.

[0091] (Note 2) The data processing device according to Appendix 1, wherein, when the user wearing the earphones requests a message as the content of the utterance that will trigger the recall of a specific memory, the processing device instructs the prompt to propose to the user one or more messages selected based on the life log as information corresponding to the content of the utterance.

[0092] (Note 3) The data processing device according to Appendix 1 or 2, wherein, when the user wearing the earphones mutters a specific matter as the content of the utterance, the processing device instructs the prompt to suggest to the user, as information corresponding to the content of the utterance, the user's actions recommended for the matter based on the life log.

[0093] (Note 4) A data processing method in which a computer performs a specific processing using a data generation model that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, and generates a predetermined inference result according to the user data, The aforementioned earphones are canal-type earphones or in-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones. The aforementioned earphones are equipped with a vibration function and are capable of outputting haptic feedback. The sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the earphones are input as user data. When the system receives an utterance from the user wearing the earphones regarding the user's memories or actions, it performs the process of acquiring information corresponding to the utterance and the haptic feedback by inputting a prompt into the data generation model that suggests to the user information corresponding to the content of the utterance and instructs the model to generate the haptic feedback using the earphones, based on the user's life log in which the sounds and images associated with the user are recorded, as the specified process. The process involves playing back the results of the aforementioned specific processing from the speaker and outputting tactile feedback using the vibration function, A data processing method performed by the aforementioned computer.

[0094] (Note 5) A data processing program that causes a computer to perform specific processing using a data generation model that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, and generates predetermined inference results according to the user data, The aforementioned earphones are canal-type earphones or in-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones. The aforementioned earphones are equipped with a vibration function and are capable of outputting haptic feedback. The sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the earphones are input as user data. When the system receives an utterance from the user wearing the earphones regarding the user's memories or actions, it performs the process of acquiring information corresponding to the utterance and the haptic feedback by inputting a prompt into the data generation model that suggests to the user information corresponding to the content of the utterance and instructs the model to generate the haptic feedback using the earphones, based on the user's life log in which the sounds and images associated with the user are recorded, as the specified process. The process involves playing back the results of the aforementioned specific processing from the speaker and outputting tactile feedback using the vibration function, A data processing program to be executed by the aforementioned computer. [Explanation of symbols]

[0095] 10,210 Data Processing Systems 12 Data Processing Devices 14,214 earphones 290 Specific Processing Unit 291 Input section 292 Processing Unit 293 Output section< / url:>

Claims

1. An input unit that receives user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, A processing unit that performs specific processing using a data generation model that generates predetermined inference results according to the user data, An output unit that reproduces the result of the specified processing from the speaker, Equipped with, The aforementioned earphones are canal-type earphones or in-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones. The aforementioned earphones are equipped with a vibration function and are capable of outputting haptic feedback. The input unit inputs the sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the biosensor as user data. The processing unit, upon receiving a speech from the user wearing the earphones regarding the user's memories or actions, performs the following as a specific process: proposing information corresponding to the content of the speech to the user based on the user's life log in which sounds and images associated with the user are recorded, and inputting a prompt to the data generation model instructing it to generate the haptic feedback using the earphones, thereby acquiring information corresponding to the content of the speech and the haptic feedback.

2. The data processing device according to claim 1, wherein, when the user wearing the earphones requests a message as the content of the utterance that will trigger the user to recall a specific memory, the processing device instructs the prompt to propose one or more messages selected based on the life log to the user as information corresponding to the content of the utterance.

3. The data processing device according to claim 1, wherein, when the user wearing the earphones mutters a specific matter as the content of the utterance, the processing device instructs the prompt to suggest to the user, as information corresponding to the content of the utterance, an action recommended for the user in relation to the matter based on the life log.

4. A data processing method in which a computer performs a specific processing using a data generation model that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, and generates a predetermined inference result according to the user data, The aforementioned earphones are canal-type earphones or in-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones. The aforementioned earphones are equipped with a vibration function and are capable of outputting haptic feedback. The sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the earphones are input as user data. When the system receives an utterance from the user wearing the earphones regarding the user's memories or actions, it performs the process of acquiring information corresponding to the utterance and the haptic feedback by inputting a prompt into the data generation model that suggests to the user information corresponding to the content of the utterance and instructs the model to generate the haptic feedback using the earphones, based on the user's life log in which the sounds and images associated with the user are recorded, as the specified process. The process involves playing back the results of the aforementioned specific processing from the speaker and outputting tactile feedback using the vibration function, A data processing method performed by the aforementioned computer.

5. A data processing program that causes a computer to perform specific processing using a data generation model that inputs user data including sound, images, and biometric data collected by two earphones worn on the user's ears, which include a microphone, speaker, camera, and biosensor, and generates predetermined inference results according to the user data, The aforementioned earphones are canal-type earphones or in-ear type earphones, and only the user wearing the earphones can hear the sound output from the earphones. The aforementioned earphones are equipped with a vibration function and are capable of outputting haptic feedback. The sound detected by the microphone, the image captured by the camera, and the biometric data acquired by the earphones are input as user data. When the system receives an utterance from the user wearing the earphones regarding the user's memories or actions, it performs the process of acquiring information corresponding to the utterance and the haptic feedback by inputting a prompt into the data generation model that suggests to the user information corresponding to the content of the utterance and instructs the model to generate the haptic feedback using the earphones, based on the user's life log in which the sounds and images associated with the user are recorded, as the specified process. The process involves playing back the results of the aforementioned specific processing from the speaker and outputting tactile feedback using the vibration function, A data processing program to be executed by the aforementioned computer.

Images