Information processing device, information processing system, program and method

Abstract

To support users' exercise by engaging in appropriate dialogue according to their exercise status. [Solution] The information processing device 100 includes a motion information input means 214 into which motion information relating to the user's physical movements is input, and a reaction information input means (e.g., 210) into which reaction information from the user, which is different from motion information, is input. The information processing device 100 further includes a speech information output means 204 that outputs speech information relating to the user's motion support based on the reaction information input to the reaction information input means (e.g., 210) and the motion information input to the motion information input means 214.

Description

Technical Field

[0001] The present invention relates to information processing, and more particularly, to an information processing apparatus, an information processing system, a program, and a method.

Background Art

[0002] Conventionally, there has been known a speech promotion device that can be used for cognitive function training even with a small number of people (Japanese Patent No. 6833209 (Patent Document 1)). Also, there has been known an exercise support system that provides a field experience by video and audio in order to promote light exercise and brain activity of users such as the elderly (Japanese Unexamined Patent Application Publication No. 2023-008123 (Patent Document 2)).

[0003] Patent Document 1 discloses an information processing apparatus including a provided topic data creation unit that creates provided topic data based on user speech data, a question data creation unit that creates question data based on the provided topic data, and an answer data creation unit that creates answer data based on user speech data. Patent Document 2 discloses an information processing system for supporting a user's exercise, having a sheet-like member including a predetermined pressurized area, a pressure detection unit that detects the pressure applied to the pressurized area when a user places their body on the sheet-like member, and an output unit that outputs an image that is changed according to the walking state information of the user obtained based on the detection result by the pressure detection unit. Also, Patent Document 1 mentions that an exercise support system can display a reproduced image showing a scenery at a tourist destination on a television receiver according to the walking state information of the user, whereby the user can perform a walking exercise as if they were walking around a tourist destination.

[0004] When providing exercise support to the elderly using an exercise support system like the conventional technology described in Patent Document 2, typically, an assistant such as a caregiver or care staff member speaks to the elderly person to encourage them to exercise or to set the rhythm of their movements. It is conceivable that the burden on the assistant could be reduced by using a dialogue generation system, such as the speech facilitator described in Patent Document 1, in an exercise support system. However, when attempting to have the dialogue generation system communicate with the user, simply generating speech from the user's spoken words, as in the conventional technology, is not possible because it is not possible to produce speech that is appropriate to the user's exercise situation, and therefore exercise support cannot be adequately provided. [Overview of the project] [Problems that the invention aims to solve]

[0005] This disclosure is made in light of the above points, and aims to support users' exercise by engaging in appropriate dialogue according to their exercise situation. [Means for solving the problem]

[0006] This disclosure provides an information processing device having the following features in order to solve the above problems. The information processing device includes a motion information input means into which motion information relating to the user's physical movements is input. The information processing device also includes a response information input means into which response information from the user, which is different from motion information, is input. The information processing device further includes a speech information output means that outputs speech information relating to the user's movement support based on the response information input to the response information input means and the motion information input to the motion information input means. [Effects of the Invention]

[0007] With the above configuration, it becomes possible to support users' exercise by engaging in appropriate dialogue according to their exercise situation. [Brief explanation of the drawing]

[0008] [Figure 1] A schematic diagram of an exercise support system according to one or more embodiments of the present disclosure. [Figure 2] A functional block diagram of an exercise support system according to one or more embodiments of the present disclosure. [Figure 3] A diagram illustrating a screen displayed on the display unit of an exercise support system according to one or more embodiments of the present disclosure. [Figure 4] A diagram illustrating the data structure of various input information generated by an exercise support system according to one or more embodiments of this disclosure. [Figure 5] A schematic diagram illustrating the learning process of a machine learning model in an exercise support system according to one or more embodiments of this disclosure. [Figure 6] A flowchart illustrating walking motion support processing performed by an information processing device in an exercise support system according to one or more embodiments of this disclosure. [Figure 7] A diagram illustrating the speech information generation process based on motion information and speech information, which is performed by an information processing device according to the first embodiment of this disclosure. [Figure 8] A diagram illustrating a conversation generated based on motion information and voice information, as performed by an information processing device according to a first embodiment of the present disclosure. [Figure 9] A diagram illustrating the speech information generation process based on motion information, voice information, and image attribute information, which is performed by an information processing device according to a second embodiment of the present disclosure. [Figure 10] A diagram illustrating a conversation generated based on motion information, audio information, and image attribute information, as performed by an information processing device according to a second embodiment of the present disclosure. [Figure 11] A diagram illustrating motion information-based video playback control performed by an information processing device according to a third embodiment of this disclosure. [Figure 12] A diagram illustrating the abnormality reporting process based on biological information performed by an information processing device according to the fourth embodiment of this disclosure. [Figure 13] A diagram illustrating the abnormality reporting process based on biological information performed by an information processing device according to the fifth embodiment of this disclosure. [Figure 14] A diagram illustrating an example of a user screen displayed on the display unit of an exercise support system according to another embodiment of the present disclosure. [Figure 15] Functional block diagram showing an example of distributed implementation of a motion assistance system according to an embodiment of the present disclosure. [Figure 16] Schematic diagram showing a motion assistance system 100a according to a first modification example. [Figure 17] Diagram showing an example of the configuration of a motion assistance system 100b according to a second modification example. [Figure 18] Block diagram showing an example of the hardware configuration of the information processing apparatus 1. [Figure 19] Overview diagram of a motion assistance system according to yet another embodiment of the present disclosure. [Figure 20] Functional block diagram of a motion assistance system according to yet another embodiment of the present disclosure. [Figure 21] Overview diagram of a motion assistance system according to yet another embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0009] Hereinafter, an information processing apparatus, an information processing system, a method, and a program according to an embodiment of the present disclosure will be described in detail with reference to the drawings. However, the information processing apparatus, the information processing system, the method, and the program according to the embodiment of the present disclosure are not limited to those described below. In the following description, the same or equivalent members or functions are referred to by the same names and reference numerals, and detailed descriptions thereof are omitted as appropriate.

[0010] Hereinafter, with reference to FIGS. 1 to 6, the overall configuration related to motion assistance of a motion assistance system 100 as an information processing apparatus or an information processing system according to one or more embodiments of the present disclosure will be described.

[0011] FIG. 1 is a diagram showing an overview of a motion assistance system 100 according to one or more embodiments of the present disclosure. The motion assistance system 100 is a system that promotes the physical movement of a user U. In the embodiment to be described, the case where "walking motion" is performed as the physical movement will be described as an example. Here, the "walking motion" is assumed to include a stepping motion of raising and lowering the feet at a fixed position. Note that the walking motion is a preferred example of a physical movement to which the control according to the embodiments of the present disclosure can be suitably applied, but is not necessarily limited thereto.

[0012] As shown in FIG. 1, the motion assistance system 100 includes an information processing device 1 and a walking sensor unit 20 (an example of a motion information acquisition device) that outputs motion information related to the walking motion of the user U. In the environment of the user U, a mat 2 on which each user U performs a walking motion may be further provided. A printed display 21 serving as a guide for the stepping position may be formed on the mat 2. The walking sensor unit 20 is disposed on or adjacent to the mat 2, detects the states of the left and right feet of the user U on the mat 2, and outputs, as motion information, operation information regarding the grounding or liftoff operation of the user U's feet, or the walking or stopping of the user (indicating grounding operation, liftoff operation, stepping, stepping stop, etc.) to the information processing device 1.

[0013] In the embodiment to be described, a separate walking sensor unit 20 from the mat 2 is used for description. However, the configuration of the walking sensor unit 20 is not particularly limited, and an integrated one with the mat 2 may be used. For example, a stepping detection mat using a pressure-sensitive sheet may be used. Since various configurations can be used for the configuration of the walking sensor unit 20 that detects the walking motion, further detailed description will not be provided. <000009​​The exercise support system 100 shown in Figure 1 is configured to allow multiple users U to use it in parallel. In the example shown in Figure 1, four users U are using the exercise support system 100, and four sets of mats 2 and walking sensor units 20 are provided to correspond to the four users U. The information processing device 1 can acquire exercise information for each of the four users U from each of the four walking sensor units 20.

[0015] In the example shown in Figure 1, one user U is performing walking exercises while standing, while the other three users U are performing walking exercises while sitting on chairs 24. Thus, the exercise support system 100 disclosed herein allows users U to choose whether to perform walking exercises while standing or while sitting, according to their preferences and health condition.

[0016] The information processing device 1 acquires information about the user U's walking state and walking pitch from the motion information input from the walking sensor unit 20. Based on the motion information, the information processing device 1 can determine the period during which the user U's feet make contact with the mat 2 or the like, and calculate the user U's walking pitch from this period. Alternatively, instead of the period of contact with the ground, the walking pitch (the time from when one foot steps down to when the other foot steps down) may be determined, and the walking pitch may be calculated from this.

[0017] The exercise support system 100 shown in Figure 1 further includes a display unit 4, on which a video Mv is displayed. The display unit 4 is a display device such as a liquid crystal display, an organic EL (ElectroLuminescence) display, or a plasma display. The information processing device 1 controls the playback speed of the video Mv displayed on the display unit 4 according to the walking pitch of the user U. More specifically, the information processing device 1 can control the playback of the video Mv displayed on the display unit 4 according to the movement state of one or more users U detected by one or more walking sensor units 20. In the exercise support system 100 shown in Figure 1, the video Mv viewed by each of the four users U can be displayed on a single display unit 4, thereby allowing the videos viewed by each user U to be shared among the four users U.

[0018] In the embodiment described, multiple users U will be described as performing walking exercises at their respective positions within the detection range of the walking sensor unit 20 on the mat 2 while viewing a single display unit 4. The exercise support system 100 can control the playback of the video Mv while the user U is walking and stop the playback of the video Mv when the user U stops walking. However, when multiple people use the exercise support system 100, it is difficult to play the video at each person's speed if there is only one display unit 4. Also, the user U may get tired and stop walking midway through.

[0019] Therefore, video playback may be controlled according to the walking status of the majority of users. For example, playback may be advanced when the majority of users are walking (or when at least one user is walking), and stopped when the majority of users have stopped walking (or when all users have stopped walking). Then, video playback may be resumed when the majority of users U start walking (or when at least one user starts walking). In addition, by displaying foot marks F on the display unit 4 to show the stepping status of multiple users U and changing them in accordance with their stepping, multiple users can check each other's stepping status and encourage or call out to each other.

[0020] User U can view the video Mv, which is played according to their walking state, through the display unit 4, thereby creating a simulated experience of taking a walk in a tourist spot (for example, by watching video content filmed while strolling through a tourist spot). In particular, in the embodiment shown in Figure 1, multiple users U can use the system in parallel, allowing them to share the simulated experience of walking with each other. As a result, users U can enjoy exercising even more than when using it alone. By multiple users U using the exercise support system 100 in parallel, they can recall and talk about the scenes and share their impressions while performing exercises that contribute to rehabilitation.

[0021] In the embodiment described, the system is configured to be usable by multiple users U in parallel. However, the configuration of the exercise support system 100 is not limited to this, and it may be constructed as a system for a single user.

[0022] In the example shown in Figure 1, some or all of the user U may be equipped with a biosensor unit 23 (an example of a reaction information acquisition device), such as a smart device, smartwatch, or pedometer (registered trademark), which has functions such as a heart rate monitor, blood oxygen saturation monitor, and activity tracker, in addition to the walking sensor unit 20. The biosensor unit 23 measures the wearer's heart rate, blood oxygen saturation, activity level, etc., and outputs this as biometric information to the information processing device 1. The information processing device 1 receives the biometric information from the biosensor unit 23 and may perform control according to the biometric information when playing back videos.

[0023] The motor support system 100 shown in Figure 1 further includes a camera 3 that captures the usage environment, including one or more users U, within its field of view. The camera 3 may be provided separately from the information processing device 1, as shown in Figure 1, or it may be provided integrated into the information processing device 1. The camera 3 can be used for user identification. The camera 3 detects the face region from the captured image and outputs the user U's face image or image features to the information processing device 1. The information processing device 1 identifies the user U from the face image or image features, thereby enabling individual responses during video playback. The camera 3 may also detect the user U's gaze by using an image processing circuit to identify the position of the pupil image region included in the user U's face image captured by the camera, and the information processing device 1 may perform control according to the gaze during video playback.

[0024] The exercise support system 100 shown in Figure 1 further includes a speaker 5 that generates sound. The speaker 5 may be provided separately from the information processing device 1, as shown in Figure 1, or it may be provided integrated into the information processing device 1. The exercise support system 100 can generate sound from the speaker 5 in accordance with the video Mv displayed on the display unit 4. For example, if the video Mv includes scenery visible when walking on cobblestones, the exercise support system 100 will generate the sound of footsteps walking on cobblestones using the speaker 5. This allows the user U to have a more realistic simulated experience by utilizing their hearing. The speaker 5 can also output predefined narration, etc. The speaker 5 can also output speech information generated by the information processing device 1 through processing described in detail later. The speaker 5 may, for example, have a multi-channel (e.g., 5.1ch) sound field creation function and be able to generate sound with directionality.

[0025] In the example shown in Figure 1, the motor support system 100 further includes a microphone array 7 (an example of a response information acquisition device), which is a sound acquisition device for collecting sound. The microphone array 7 may be provided separately from the information processing device 1, as shown in Figure 1, or it may be provided integrated into the information processing device 1. The motor support system 100 receives audio input from the surrounding environment via the microphone array 7. By applying beamforming technology, the microphone array 7 can decompose sound into sounds from each direction and acquire audio signals for each direction. This makes it possible to recognize the speech of user U even in noisy environments. Furthermore, since there is an audio signal for each direction, it is possible to distinguish users who are speaking in each direction (for example, estimating that speech from the same direction belongs to the same user). By using the microphone array 7, it is expected that the words spoken by multiple people simultaneously can be heard individually.

[0026] User U can perform walking exercises while enjoying the scenery by receiving motor stimuli, visual stimuli, auditory stimuli, etc., through a video Mv that changes based on movement information from the walking sensor unit 20. However, the stimuli provided to User U are not limited to those of the embodiment described above. In other embodiments, in addition to the video and audio described above, other sensory stimuli such as tactile stimuli and olfactory stimuli may be provided. For example, in addition to the display unit 4 and speaker 5, other external devices such as a lighting device, air conditioning device, odor generator and blower may be provided. The lighting device, air conditioning device, odor generator and blower perform actions that act on (influence) User U's senses. Sensation is the function or awareness of feeling stimuli from the outside, and is, for example, at least one of a person's senses: sight, hearing, smell, and touch. Actions that act on the user's senses may include actions related to at least one of illuminance, wind, smell, water droplets, smoke, and temperature around User U. Illuminance can be controlled by the lighting device, wind by the blower, smell by the odor generator, and temperature by the air conditioning device. Some of the audio emitted from speaker 5 mentioned above, as well as the control of other external devices such as lighting, air conditioning, odor generators, and ventilation systems, can be defined in a script prepared in advance along with the video data.

[0027] Furthermore, Figure 1 shows an operator O in addition to the user U. Operator O is, for example, a caregiver who provides care to the care recipient, who is the user U. Operator O holds a remote controller (remote control) 6, and the exercise support system 100 is configured to be operable via the remote control 6. By operator O operating the exercise support system 100 using the remote control 6, the effort required for user U to operate the system is reduced, and the operation of the exercise support system 100 can be made smoother. This increases user U's motivation to use the exercise support system 100 and promotes user U's exercise.

[0028] Operator O uses the remote control 6 to select video content to be displayed on the display unit 4 during walking exercises, and to give instructions such as starting, stopping, or resuming playback of the video content. The remote control 6 can use a touch panel, operation buttons, a keyboard, a joystick, or a combination thereof. The remote control 6 may be a remote control that can be attached to or detached from the exercise support system 100 or the information processing device 1. Alternatively, the remote control 6 may be integrated into the information processing device 1. Furthermore, the remote control 6 may be an information processing terminal such as a tablet or smartphone.

[0029] The following will provide a more detailed explanation, assuming that the exercise support system 100 described above is used in a facility for the elderly.

[0030] In exercise support using the exercise support system 100, an operator O, such as a caregiver or care staff member, performs basic operations such as selecting and playing video content to initiate walking exercises (for example, recreation or rehabilitation). However, users may find it difficult to maintain interest simply by watching videos while walking. Therefore, it is desirable for operator O to encourage users to continue exercising throughout the walking exercise by providing verbal cues, expanding on conversations, and engaging with them. On the other hand, caregivers and care staff members who have other duties do not have much time to dedicate to tasks other than their primary caregiving duties. Therefore, it is expected that using a dialogue generation system in the exercise support system 100 will reduce the burden on assistants. However, when attempting to have the dialogue generation system communicate with the user, simply generating utterances from the user's speech alone is insufficient to produce utterances appropriate to the user's exercise situation, making it difficult to provide adequate exercise support.

[0031] Therefore, the exercise support system 100 according to this embodiment aims to provide exercise support to the user by automatically generating speech information according to the user U's walking exercise status and interacting with the user U, thereby engaging in appropriate dialogue according to the exercise situation. Here, the exercise situation refers to the user's condition (state) when performing physical exercise, such as the amount of physical exercise the user is performing and the user's reactions to physical exercise.

[0032] The functional configuration of the exercise support system 100 for achieving the above objectives will be described in more detail below with reference to Figure 2. Figure 2 shows a functional block 200 of the exercise support system 100 according to one or more embodiments of this disclosure.

[0033] The functional block 200 shown in Figure 2 includes an operation unit 202, a speech information output unit 204, a voice information input unit 210, a voice recognition unit 212, a motion information input unit 214, a motion information analysis unit 216, a motion recognition unit 218, a facial expression input unit 219, a user identification unit 220, a biometric information input unit 222, a biometric information analysis unit 224, a video playback unit 226, a notification unit 228, a control unit 230, a video storage unit 240, a video analysis unit 242, a conversation example storage unit 244, and a user information storage unit 246.

[0034] The control unit 230 performs overall processing and control for assisting the user U's movement, including the process of generating speech information, which will be described later. Based on input information received via input units such as the voice information input unit 210, the movement information input unit 214, the facial expression input unit 219, and the biometric information input unit 222, the control unit 230 generates speech information and outputs it to the speech information output unit 204. The control unit 230 can also control video playback by the video playback unit 226 based on this input information. Furthermore, the control unit 230 can store the content of conversations during walking, the user's engagement with walking, and topic information extracted from the conversation content in the user information storage unit 246, which will be described later, in response to the user U's walking movements.

[0035] The control unit 202 receives input from the remote control 6 (operating device) operated by operator O and receives the selection of video content to be played during walking exercise. Operator O also inputs information via the remote control 6, such as the registration of participating users U and the correspondence information between users U and walking sensor units 20, including which walking sensor unit 20 each participating user U is using. The control unit 202 then passes this information to the control unit 230. Note that the registration of participating users U may be partially or entirely performed by user identification using the camera 3 as described above. Similarly, the correspondence between users U and walking sensor units 20 may be partially or entirely performed by specifying the user by voice and having them perform stepping exercises.

[0036] Figure 3 illustrates a screen displayed on the display unit 4 of an exercise support system 100 according to one or more embodiments of the present disclosure. Figure 3(A) shows a content selection screen 300 for selecting a video content to play from a plurality of video contents prepared to support walking exercises. The operation unit 202 outputs an operation screen for operator O to operate using a remote control 6, and determines instructions from operator O based on the operation performed by the remote control 6 and the content of the output operation screen. In the content selection screen 300 shown in Figure 3(A), thumbnails 302 of a plurality of video contents are arranged in a tile-like manner, and operator O can select a video content to play via the operation unit 202 by operating the remote control 6. The plurality of prepared video contents are stored in the video storage unit 240. In addition, on the content selection screen 300, for example, the video content options may be rearranged or narrowed down according to the past history information of the participating user U (conversation content during past walking exercises and the status of their engagement with walking exercises) stored in the user information storage unit 246, which will be described later, in order to select a specific video content. Alternatively, instead of using the remote control 6, users may select the video content to play using voice recognition (such as using voice recognition to select the content name or move the cursor).

[0037] When a video content to be played is selected, a user screen 310 is displayed, as shown in Figure 3(B), which schematically shows the walking state of each user with foot marks F while playing the video Mv. The control unit 230 reads the selected video content from the video storage unit 240 via the operation unit 202 and causes the video playback unit 226 to play the video. The video playback unit 226 constitutes the image information output means in this embodiment. Here, it is assumed that the user will start walking when the user screen 310 is displayed. The user screen 310 shows the name N of the identified user, which will be described later, and displays an animation in which the foot marks F simulate walking in response to the movement state of each user.

[0038] Referring again to Figure 2, the speech information output unit 204 receives speech information (text) generated by the control unit 230 and performs output processing to output the speech information to output units such as the display unit 4 and the speaker 5. More specifically, the speech information output unit 204 is composed of a caption creation unit 206 and a speech synthesis unit 208.

[0039] The display unit 4 can display captions or subtitles corresponding to the spoken information, either superimposed on the video (Mv) being played or in the subtitle area at the bottom of the video. The caption creation unit 206 creates captions with a predetermined font based on the spoken information from the control unit 230, adds images such as illustrations as necessary, and displays them on the screen of the display unit 4.

[0040] Figure 3(C) shows a user screen 310 in which generated speech information is displayed as a caption 312 and an illustration image 314 superimposed on the video Mv. In addition, questions and answers, such as quizzes based on a script prepared along with the video content, may also be displayed as captions.

[0041] Speaker 5 can output an audio signal corresponding to the speech information. The speech synthesis unit 208 performs text-to-speech (TTS) conversion based on the speech information input from the control unit 230, generates an audio output corresponding to the speech information with predetermined settings (specified by language, gender, pitch, tone / speech style, or a specific voice model name), and outputs it to speaker 5.

[0042] In the embodiments described, the speech information output unit 204, the caption creation unit 206, or the speech synthesis unit 208 constitute speech information output means for outputting speech information. In the embodiments described, speech information is output as either voice or captions (text), but the invention is not limited to these. For example, other output formats such as video output, such as sign language expression using computer graphics, or expression through the movement of a sign language robot, etc., may also be used. Details of the speech information generation process will be described later.

[0043] The voice information input unit 210 receives voice information from the operating environment via the microphone array 7 and outputs it to the voice recognition unit 212. First, the voice information is input to the voice information input unit 210 in the form of a digital voice signal obtained by sampling the analog voice signal input to the microphone array 7. At this time, since the microphone array 7 can receive voice input for each direction, a voice signal may be generated for each direction. The voice information input unit 210 constitutes the voice information input means in this embodiment.

[0044] The speech recognition unit 212 applies speech-to-text (STT) conversion to the digital speech signal format speech information from the speech information input unit 210, converting it into text format speech information (for example, "It's a nice day") and outputting it to the control unit 230. The speech recognition unit 212 may also perform speaker identification based on the voiceprint profile of a specific user U that has been pre-registered, and add speaker information to the text format speech information (for example, "Yoshida-san, it's a nice day"). Furthermore, as mentioned above, since speech signals are directional, direction information identifying the direction from which the sound came may be added instead of, or along with, speaker identification ("Right: 60 degrees: Yoshida-san, it's a nice day"). In addition, the speech recognition unit 212 may perform speech emotion recognition and add attributes representing the nature of the sound and voice (voice tone, positive, negative, anger, excitement, enjoyment, boredom, calmness, sadness, etc.) to the text format speech information. In addition to extracting from the audio signal, the speech recognition unit 212 may also extract emotion-related attributes by performing natural language analysis (sentiment analysis) on the text obtained through speech recognition. The speech recognition unit 212 may further use other speech analysis to detect and add information such as the number of breaths from breathing sounds. The speech recognition unit 212 may further assign an ID to the recognized audio information and attach a timestamp (for example, "Right: 60 degrees: 2024 / x / xx / 10:04.432 / Yoshida-san, it's a nice day").

[0045] Figure 4 illustrates the data structure of various input information generated by the exercise support system 100 according to one or more embodiments of the present disclosure. Note that the data structure shown in Figure 4 is merely illustrative, and the data fields will be designed appropriately according to the specific implementation. Figure 4(A) illustrates the data structure of voice information. As shown in Figure 4(A), the voice information includes an ID field, a timestamp field that holds the time information of the voice input, a conversation text field that holds the conversation content, a voice / breath field that holds the tone of the sound and the number of breaths, and a voiceprint / direction field that holds information for identifying the speaker and direction.

[0046] Referring again to Figure 2, the motion information input unit 214 receives motion information related to the user U's walking motion via the walking sensor unit 20 and inputs it to the motion information analysis unit 216. The motion information depends on the output format of the walking sensor unit 20, but it may be a raw waveform signal, or, as described above, it may be provided in the form of information indicating that a predetermined action (event) has occurred, such as the user U's ground contact, lift-off, stepping, or stopping stepping. The motion information analysis unit 216 converts the motion information in the input format into an output format that the control unit 230 can process. For example, the motion information analysis unit 216 may receive a series of inputs of the user U's foot ground contact or lift-off, add a timestamp (date and time), generate motion information in a format indicating the timing of a left / right step (for example, in the format "Right / 2024 / x / x / 10:02:30.002" or "Left / 2024 / x / x / 10:02:30.450"), and output it to the control unit 230.

[0047] The motion information analysis unit 216 may also add information indicating the strength of stepping if the walking sensor unit 20 can detect it. The motion information analysis unit 216 may also use past motion information to calculate the number of steps and speed within a predetermined period and generate motion information in a format that associates the average number of steps and average speed with a predetermined period or point in time. The motion information input unit 214 constitutes the motion information input means in this embodiment, and the motion information analysis unit 216 constitutes the motion information analysis means in this embodiment. The motion information analysis unit 216 generates data from motion information that includes at least one piece of information: the time of walking, the number of steps over a predetermined period, the average walking speed at a predetermined point in time or over a predetermined period, and the intensity of walking.

[0048] Figure 4(B) illustrates the data structure of the motion information. As shown in Figure 4(B), the motion information includes a sensor ID field that holds an ID that identifies the walking sensor unit 20, a timestamp field, a stepping strength field that holds information indicating the strength of the stepping motion, a stepping position field that holds information indicating whether it is left or right, or the position on the pad, a step count field, and a speed field.

[0049] The walking sensor unit 20 described above is a sensor that directly detects the user's walking motion, but the method of acquiring motion information is not particularly limited. For example, the motion recognition unit 218 can analyze images input from the camera 3 (an example of a motion information acquisition device), detect the user U's skeleton, and perform motion analysis to detect walking motion. In addition to using the walking sensor unit 20 and the camera 3, it is also possible to use a device with an acceleration sensor or gyroscope (motion information acquisition device) such as a smartphone to analyze walking from acceleration and angular velocity patterns and obtain motion information. Furthermore, in addition to detecting the state by distinguishing between the left and right feet, count-up information from a device that counts steps without distinguishing between the left and right feet, such as a pedometer (registered trademark), (for example, in the format of "1 time:2024 / x / x / 10:02:30.002" or "1 time:2024 / x / x / 10:02:30.450") may also be used as motion information.

[0050] The facial expression input unit 219 detects the face region of a person from the image captured by the camera 3 (an example of a reaction information acquisition device), recognizes the person's facial expression (such as joy, anger, sadness, etc.) from the face image, and passes the facial expression information to the control unit 230.

[0051] The user identification unit 220 detects the face region of a person from the image captured by the camera 3, identifies a specific user U from the face image, and passes the information to the control unit 230. Face images for identifying each user U are stored in the user information storage unit 246. The user identification unit 220 constitutes the user identification means according to this embodiment.

[0052] The biometric information input unit 222 receives biometric information about the user U's body (such as the wearer's heart rate, blood oxygen saturation, and activity level) from the biosensor unit 23. The biometric information input unit 222 constitutes the biometric information input means in this embodiment.

[0053] The biometric information analysis unit 224 analyzes the biometric information, adds a timestamp, user information, etc., and passes it to the control unit 230. Various types of biosensor units 23 can be exemplified, such as the smartwatch and pedometer (registered trademark) mentioned above, as well as activity trackers, sleep trackers, blood pressure monitors, small brain activity sensors, and cameras on smartphones.

[0054] Figure 4(C) illustrates the data structure of biometric information. As shown in Figure 4(C), the biometric information includes a sensor ID field that identifies the biosensor unit 23, a timestamp field, and fields for step count, heart rate, blood oxygen concentration, stress level, body temperature, blood pressure, sleep information, and brain state.

[0055] Figure 4(D) illustrates the data structure of biometric information that can be acquired by a camera. For example, there is a known technique for estimating heart rate and respiratory rate by taking a picture of a face with a camera. As mentioned above, it is also possible to recognize facial expressions (joy, anger, sadness, etc.) from facial images and detect body movements. As shown in Figure 4(D), the biometric information includes a sensor ID field to identify the camera, a timestamp field, a heart rate field, a blood flow field, a facial expression field, and a body movement field.

[0056] The notification unit 228 notifies various types of information to pre-registered contacts (email, messaging system, social networking system, external care system, etc.) based on input from the control unit 230. The notification unit 228 is called in response to the control unit 230 detecting an abnormality based on biometric information, such as the heart rate measured by the biosensor unit 23 exceeding the upper limit, or an abnormality based on voice information, such as the voice information output from the voice recognition unit 212 containing a phrase requesting SOS. The notification unit 228 constitutes the notification means according to this embodiment.

[0057] The video storage unit 240 stores multiple video contents for playback during walking exercises, and also holds scripts and attribute information associated with the video contents. The script is data that describes the operation control of external devices such as lighting and air conditioning systems during video playback, as well as text overlays to be displayed on the screen in accordance with the progress of the video, and narration to be output as audio. The control unit 230 receives the selection of the video content to be played via the operation unit 202, reads the video data from the video storage unit 240, and passes it to the video playback unit 226 to play the video.

[0058] The video analysis unit 242 analyzes the video data of the video content in advance or in real time, recognizes objects (things) contained in the image using image analysis such as NER (Named Entity Recognition), and can add tags to identify objects (things, places, buildings, plants and animals, food, people, pictures, signs, etc.), or add descriptive information about the content of each frame ("a person is walking," "a dog is barking," etc.) using video captioning. These tags and texts are stored in the video storage unit 240 as image attribute information associated with the video content, and specific frames and time information within it. The image attribute information added to the video content may be automatically added by the analysis performed by the video analysis unit 242, or it may be added manually. In addition, the geographical coordinates of the shooting location may be attached to the video content.

[0059] Furthermore, in videos shot while strolling through tourist areas, the photographer may stop and continue shooting. In such cases, the video analysis unit 242 may estimate the photographer's walking speed from the movement speed of the scenery in the image and add speed information to each frame so that the video moves at a standard speed. By using such speed information or time information, in video playback control, for example, in frames with zero speed, such as when the photographer is standing still, playback control corresponding to the pitch of walking motion can be temporarily canceled, or the playback speed can be corrected (for example, in a video where the photographer's movement speed differs between sections, correction can be made to match the photographer's movement speed between sections).

[0060] Figure 4(E) illustrates the data structure of image attribute information added to video data by the video analysis unit 242 or manually. As shown in Figure 4(E), the image attribute information includes a video ID field that identifies the video, a file name field that holds the file name, a frame number field that identifies the frame number, a location information field that indicates the position or place associated with the frame number (a place name such as Asakusa, or geographic (GPS) coordinates indicating that location), and a related information field that holds related information related to the location (famous landmarks, celebrities, history, topics, buildings, etc.). Note that the frame number field, location information field, and related information field may be provided for each frame, or for each group consisting of multiple frames (for example, each scene (section) when the entire video is divided into multiple scenes (sections)).

[0061] The conversation example memory unit 244 stores conversation examples that have been anticipated and prepared by the developers, as well as actual conversation examples resulting from the interaction between the computer and user U by the control unit 230 (automatic questions to user U and responses from user U, and questions from user U and automatic responses thereto).

[0062] The user information storage unit 246 stores user information (name, gender, age, background, residential history, work history, travel history, hobbies, sports experience, and preferences) associated with each user. The user information storage unit 246 also stores a facial image referenced by the user identification unit 220, associated with the user, and provides it to the user identification unit 220. In addition, the user information storage unit 246 stores, in association with the user, the content of conversations that took place during past walking exercise sessions performed by the user, and information based on exercise information from the walking sensor unit 20 (average pitch and number of steps in past walking exercises, etc.). The user information storage unit 246 constitutes the user information storage means, the information storage means, or both in this embodiment.

[0063] Figure 4(F) illustrates the data structure of user information stored in the user information storage unit 246. As shown in Figure 4(F), user information includes an ID field, a name field, a nickname field, a face photo field, an age field, a gender field, an address / place of origin field, a family information field, a care level field, a cognitive status field, a hobby information field, an occupation information field, and a history information field. Note that the data structure with specific fields shown in Figure 4(F) is illustrative and will be designed according to the specific implementation.

[0064] In the embodiments described, the system will primarily use information stored in the video storage unit 240, the conversation example storage unit 244, and the user information storage unit 246, but external information may also be used. For example, the exercise support system 100 may be provided with modules for external integration, such as importing or exporting external data or using search results from external search engines.

[0065] The following describes the more specific functions of the control unit 230, including the speech information generation process performed by the control unit 230.

[0066] The control unit 230 is more specifically composed of a speech information generation unit 232, a video control unit 234, and a report information generation unit 236. The speech information generation unit 232 generates questions for the user and responses to questions from the user using input information (voice information, motion information, image attribute information, biometric information, facial expression information) input to the control unit 230 from each input unit, as well as pre-stored conversation examples, past conversation examples, and user information. The video control unit 234 generates playback speed, stop instructions, and restart instructions for video content based on the voice information, motion information, image attribute information, biometric information, and facial expression information input to the control unit 230 from each input unit, and controls video playback in cooperation with the video playback unit 226.

[0067] The speech information generation unit 232 can generate speech information related to the user's movement support by taking user response information and movement information input to the movement information input unit 214 and generated by the movement information analysis unit 216 as input. In the embodiment described, the user response information is voice information input by the user in response to an interaction from the movement support system 100 to the user, such as speech information previously generated and output by the speech information generation unit 232. More specifically, it is voice information in text format, where the user's speech is input to the voice information input unit 210 and recognized by the voice recognition unit 212.

[0068] Furthermore, speech information related to user exercise support refers to speech information that contains content that encourages the user to engage in physical movement. Speech information may include speech that promotes exercise (e.g., "Shall we pick up the pace?", "Let's try to keep going for a few more minutes with this adjustment"), speech that inhibits exercise (e.g., "Let's slow down a bit," "Let's take it slow"), and speech that stops exercise (e.g., "Let's stop for today").

[0069] For example, if the audio information indicates silence and the motor information indicates a decrease in walking pace, the speech information generation unit 232 generates speech information encouraging a break (for example, "Your walking pace has slowed down. Shall we take a break?"). In addition to the audio and motor information, the speech information generation unit 232 can also generate speech information by taking image attribute information output by the video playback unit 226 as further input. For example, if the audio information indicates silence, the motor information indicates a decrease in walking pace, and the image attribute information indicates a "viewpoint," the unit generates speech information encouraging a break (for example, "The view is nice. Shall we take a short break here?").

[0070] Furthermore, as mentioned above, image attribute information includes location information, and location information may also have related information attached. The speech information generation unit 232 can generate speech information based on the location information and its related information. For example, if the audio information indicates that a conversation is continuing, the motion information indicates that the walking pace is within the normal range, the location information indicates that the scene of "Asakusa" is being displayed, and the location information "Asakusa" has related information "Kaminarimon" attached, the unit uses this information as input to generate speech information (for example, "Everyone still seems to have a lot of energy. Speaking of Asakusa, Kaminarimon is famous, isn't it? Have you ever been there?"). The generated speech information is passed to the speech information output unit 204 and output in the form of a caption, audio, or both.

[0071] The speech information generation unit 232 may further generate speech information using user information stored in the user information storage unit 246, which corresponds to a specific identified user among multiple users, as input. For example, if the voice information indicates the user with the most utterances, and the user information for that user indicates that they are from the downtown area, the speech information generation unit 232 will generate speech information (for example, "Mr. / Ms. XX, you're from downtown, aren't you? Do you recognize the scenery around here?").

[0072] The speech information generation unit 232 can also generate speech information that encourages reducing the amount of exercise based on at least one of the response information (voice information), exercise information, and biometric information (for example, if the exercise information indicates that the "walking pace" has exceeded the recommended range, it may say, "You're really putting in the effort. You can walk a little slower if you like."). The speech information generation unit 232 may also generate speech information that encourages stopping exercise based on at least one of the information (for example, if the biometric information indicates that the "heart rate" has exceeded the recommended range, it may say, "Is your heart beating a little fast? Shall we take a break now?"). The speech information generation unit 232 can further generate speech information that encourages changing the topic based on at least one of the information (for example, if the exercise information indicates that the amount of walking activity of multiple users has fallen below a certain level, it may say, "Do you all like shopping?" based on other related information "shopping street" that is associated with the location "Asakusa" and related information "Nakamise Street" based on general knowledge). The speech information generation unit 232 may further generate speech information (for example, "Your steps seem lighter than usual today. Did something good happen?" based on past walking pitch information of a particular user) using past information stored in the user information storage unit 246 as input. Note that the speech information generation method described here is merely an example and is not limited to this method.

[0073] In the embodiment described, user response information was input to the voice information input unit 210 in response to an interaction with the user (for example, within a predetermined period) and recognized as voice information of the user's speech. However, user response information is not limited to this. In other embodiments, it may be facial expression information identified from facial image information of the user's facial expression, which is input to the facial expression input unit 219 in response to an interaction with the user (for example, within a predetermined period), or it may be biometric information about the user's body, which is input to the biometric information input unit 222 in response to an interaction with the user (for example, within a predetermined period). Hereafter, the explanation will continue assuming that the response information is voice information.

[0074] The video control unit 234 controls the video playback unit 226 to change the video (image information) it outputs based on the input motion information. Here, changing the video may mean changing the playback speed of the video. For example, the motion information can be adjusted using the average pitch of multiple users (or a trimmed average value obtained by excluding the maximum and minimum values ​​(or values ​​above a predetermined upper and lower limit) across multiple users) so that the average value matches the video's progression speed. Changing the video may also mean changing the content of the image information, or inserting either or both a caption and an inserted image into the image. The video control unit 234 constitutes the image control means in this embodiment. Note that such adjustments to the video playback speed, changes to the content of the image information, and insertion of either or both a caption and an inserted image into the image may also fall under the user interaction described above.

[0075] The report information generation unit 236 compares past information or predefined reference information stored in the user information storage unit 246 with the currently input information, generates report information based on the comparison result, and can call the notification unit 228 to send the report information to a predetermined recipient. The report information generation unit 236 constitutes the report information generation means in this embodiment. The generation of report information by the report information generation unit 236 will be described in detail later.

[0076] The notification unit 228 can notify external care systems, email addresses, messaging system accounts, or social networking service (SNS) accounts, but these are stored, for example, in the user information storage unit 246. The notification unit 228 transmits via the network 25 to predetermined destinations stored in the user information storage unit 246.

[0077] The control unit 230 can also, in response to interaction with the user during walking, store the conversation information as specific user information, record the status of the walking exercise (number of steps, average walking pace, walking time, etc.), and record newly extracted user information (for example, based on a conversation that the user used to play badminton, it can add badminton to the user's sports history).

[0078] The speech information generation unit 232 described above may generate speech information using a machine learning model 260 that takes motion information and response information (speech information) as input and outputs speech information, or it may generate speech information based on logic branching that maps motion information and response information (speech information) to speech information under conditions. Similarly, the video control unit 234 can change the playback speed of the motion using the machine learning model 260 or logic branching.

[0079] The following describes in more detail the process of generating speech information using a machine learning model 260 with motor information and response information (speech information) as input. Here, the functional block 200 shown in Figure 2 further includes a learning unit 248, a training data storage unit 250, and a machine learning model 260.

[0080] The training data storage unit 250 stores training data for supervised learning of the machine learning model 260. The training data associates correct output data (labels, values, or text) with predetermined input data. If learning conversations, a large number of sets of questions and their responses are prepared as training data. The learning unit 248 updates the parameters of the machine learning model 260 by applying a predetermined machine learning algorithm. The learning unit 248 may build the machine learning model to be used from scratch, or it may prepare the machine learning model 260 to be used by retraining a pre-trained model.

[0081] Figure 5 is a schematic diagram illustrating the learning process of a machine learning model 260 in an exercise support system 100 according to one or more embodiments of the present disclosure.

[0082] The training data for training the machine learning model 260 is expected to include (a) pre-training data prepared in advance by the developers and (b) historical training data generated while using the exercise support system 100 and obtained with permission from relevant parties, including users and facility managers, within the scope of the specified purpose of use. (a) Pre-training data is expected to include, for example, (a-1) training data obtained by manually writing a script of a conversation during walking training that serves as a model case, and (a-2) training data obtained by recording the interactions that took place while a collaborator actually watched a video and performed walking exercises in a test environment of the exercise support system 100 (for example, with the conversation function disabled). The training data in (a-1) consists of conversation examples, and in addition to conversation examples, exercise information, biometric information, and image attribute information of the video being viewed can be obtained from the training data in (a-2) and (b). (i) The historical training data may be used to retrain the exercise support system 100 in a form limited to use within a specific facility, or it may be used to retrain a shared model of the exercise support system 100 after obtaining permission from the relevant parties, including the purpose of training a shared model for use with other facilities.

[0083] The training data described in (A-2) above can be obtained as follows. For example, a caregiver or care staff member acts as a collaborator on the operator O side, and an elderly monitor acts as a collaborator on the user side. In the test environment, while watching a predetermined video content and conversing while walking, audio information, motor information, and biometric information are collected. Audio information is recorded separately for the collaborator on the operator O side and the collaborator on the user side. The video data to be viewed is prepared with image attribute information associated with the frames based on video analysis, and attribute information is modified or added manually as needed. User information can also be prepared for the collaborator on the user side.

[0084] As shown in Figure 5, motion information 402, audio information 404, user information 406, conversation examples 408 such as (A-1), image attribute information 408, and general knowledge 412 are prepared in the test environment. Here, general knowledge 412 is information that associates local information, dialects, local products, famous things, historical sites, celebrities, topics, etc., in a predetermined topology. Time information is linked to the motion information 402, audio information 404, user information 406, conversation examples 408, and image attribute information 408 by timestamps and frame numbers. For motion information 402 and audio information 404, a timestamp indicating the time is assigned. For image attribute information, a frame number is assigned, and from the frame number, it is possible to convert to the same time as the timestamp based on the video playback start time and video playback speed.

[0085] Since temporal information is associated with this information, the audio information, motion information, and image information are ordered together in the order of their occurrence. Using this temporally linked information, a collection of training data for the machine learning model 260 can be generated. The collection of training data is provided to the learning unit 248, which prepares the machine learning model 260 based on the provided information. The learning unit 258 may, for example, prepare a dedicated machine learning model for each video content, but more preferably, it prepares a machine learning model that is general-purpose and applicable to various video content.

[0086] Through the learning process described above, the system learns examples of conversations in video scenes, such as what the caregiver said, how the user responded, or what the user said and how the caregiver responded. The training data also includes exercise information and data based on image attribute information that matches the content displayed in the video, enabling learning that is tailored to what kind of conversations took place in what kind of video, and to the participants' responses (e.g., increased walking activity, increased number of conversations).

[0087] The machine learning model 260 is not particularly limited, but large-scale language models (LLMs) such as GPT (Generative Pre-trained Transformer)-2, GPT-3, GPT-3.5, GPT-4, GPT-4o, BERT (Bidirectional Encoder Representations from Transformers), XLNET, and ChatGPT can be used. LLMs can be trained through (I) pre-training using a large corpus for large-scale parameter learning, (II) supervised fine-tuning (SFT) using instructions, and (III) reinforcement learning from human feedback (RLHF) where the model is run on a large number of instructions and humans provide feedback on the quality of the outputs. For simplicity, for example, the learning unit 248 can retrain the machine learning model using the above-mentioned conversation example with SFT as the instruction (II).

[0088] The learning methods (I) to (III) described above are learning processes that involve updating the parameters of the LLM itself. However, when using an LLM, it is also possible to perform in-context learning. In-context learning involves giving the LLM instructions for a specific task in advance using so-called prompt statements to obtain the optimal output. This differs from learning in the usual sense, which involves updating the model's parameters, but by providing prior information, the LLM can be optimized for a specific task.

[0089] Existing LLMs such as GPT, while GPT-4 accepts images as input, primarily use text as input and output, and many utilize a conversational format for communication. On the other hand, the motor support system 100 in one or more embodiments of this disclosure uses motor information, biological information, and image attribute information as input information. In order to use motor information and biological information as input to the LLM, the format of the input information can be defined in the initial prompt statement as needed, and conversational motor information and biological information can be generated based on the motor information and biological information.

[0090] For example, exercise information is provided as a timestamp and information such as whether the left or right foot strikes. From this series of exercise information, the walking pitch is calculated, and when the walking pitch meets predetermined conditions (for example, the walking pitch exceeds a threshold, or the rate of change in walking pitch per unit time exceeds a threshold), a text statement containing the exercise information is generated, either together with a normal conversational sentence or as a separate conversational sentence (for example, "Your walking pitch has reached 70 steps / minute."). Similarly, biometric information is provided as a timestamp and information such as heart rate. When predetermined conditions are met (for example, the heart rate exceeds a threshold, or the rate of change in heart rate per unit time exceeds a threshold), a text statement containing biometric information is generated, either together with a normal conversational sentence or as a separate conversational sentence (for example, "Your heart rate has risen to 130 beats / minute."). Regarding image attribute information, at corresponding timings such as scene changes, text sentences containing information describing the video are generated either alongside normal conversational sentences or as separate conversational sentences (for example, if "Kaminarimon" is recognized as a result of analyzing the video's images, the speech information "I can see Kaminarimon." is generated). By inserting such text sentences based on motion information or image attribute information into the conversation, it becomes possible to learn speech generation logic that takes motion information, biometric information, and image attribute information into account, and to generate speech information.

[0091] Furthermore, by adding information such as key conversation examples and user information to the prompt sentence before starting walking movements and performing the above-mentioned in-context learning, it becomes possible to generate speech information that takes this prior information into account.

[0092] Although Figure 2 shows and explains various components, these are merely examples, and some components shown in Figure 2 may be omitted, or other components not shown in Figure 2 may be added. Also, in the embodiments shown in Figures 1 and 2, for convenience, the information processing device 1 is described as handling the majority of the functional blocks, as indicated by the dashed rectangle 1. However, since machine learning models such as LLMs consume a large amount of resources, they may be provided outside the information processing device 1. For example, as indicated by the dotted rectangle 10 in Figure 2, the learning unit 248, the training data storage unit 250, and the machine learning model 260 may be provided outside the information processing device 1. In this case, the information processing device 1 is equipped with a communication function to interact with the external LLM's application programming interface (API), and by receiving API calls and responses, most of the computational load can be outsourced externally, thereby easing the hardware resource requirements for the information processing device 1.

[0093] Furthermore, components such as the speech synthesis unit 208, speech recognition unit 212, motion recognition unit 218, and video analysis unit 242 may also be outsourced to external parties that communicate via APIs. In other embodiments, the information processing device 1 may be equipped with only the minimum components necessary for communication with the walking sensor unit 20 and the display unit 4, and the remaining components may be implemented by communicating with an external server device or a server application deployed on a cloud infrastructure (including a form in which the server application further communicates with the LLM via APIs).

[0094] By using machine learning models such as large-scale language models, it becomes possible to efficiently build dialogue models, and updates and extensions become easier.

[0095] Furthermore, although the above description assumes that video content is played in conjunction with walking, it is not necessarily required to play video. For example, in other embodiments, the user may simply display a television program on a separate monitor and perform walking exercises while watching the television program. In that case, the implementation would not involve providing image attribute information corresponding to the video content, but if the audio from the television program can be captured and processed by the microphone array 7, it would be possible to effectively incorporate the content of the television program into the conversation.

[0096] The walking motion support process with automated dialogue described above will now be explained in more detail with reference to Figure 6. Figure 6 is a flowchart showing the walking motion support process executed by the information processing device 1 according to one or more embodiments of this disclosure. The process shown in Figure 6 starts, for example, from step S100 in response to the startup of the system.

[0097] In step S101, the information processing device 1 displays a list of videos on the screen via the operation unit 202 and accepts the selection of a video to be played. In step S101, a content selection screen 300 as shown in Figure 3(A) is displayed, and video content is selected on this screen 300. It is assumed that prior to the start of walking exercise, registration of participating users, association of each participating user with the walking sensor unit 20, and voiceprint profiling of participating users have been completed. The period from the start to the end of walking exercise is referred to as a session.

[0098] In step S102, the information processing device 1 uses the video analysis unit 242 to analyze the video data of the video content read from the video storage unit 240 and acquires positional information and related information for each frame (for each frame group).

[0099] In step S103, the information processing device 1, using the control unit 230, determines whether or not the walking exercise session using the video content has ended. If it is determined in step S103 that the walking exercise session has not yet ended (NO), the process proceeds to steps S104 and S110, where the processing of audio information shown in steps S104 to S109 and the processing of motion information shown in steps S110 to S114 are executed in parallel.

[0100] To explain the processing of the audio information, in step S104, the information processing device 1 receives audio information input via the audio information input unit 210. In step S105, the information processing device 1 converts the audio information in the form of an audio signal into audio information in text format using the speech recognition unit 212. A timestamp is added as appropriate, and user identification information is added by speaker identification. In step S106, the information processing device 1 determines whether meaningful text has been generated by speech recognition.

[0101] If, in step S106, meaningful text is generated and speech recognition is determined to have been performed (YES), the process proceeds to step S115. On the other hand, if, in step S106, meaningful text is not generated and speech recognition is determined to have failed (NO), the process proceeds to step S107, waits temporarily, and then proceeds to step S108. Here, the intention is to exclude meaningless text generated due to ambient noise or speech recognition failures. This allows the system to wait for another utterance from the user. In step S108, the information processing device 1 performs speech recognition again and determines whether meaningful text has been generated or not. If, in step S108, meaningful text is generated and speech recognition is determined to have been performed (YES), the process proceeds to step S115.

[0102] On the other hand, if meaningful text is not generated again in step S108 and it is determined that speech recognition has not occurred, the process proceeds to step S109. This is because if there has been no utterance for a certain period of time, it is unlikely that the user will spontaneously utter anything if waiting any longer, and it is effective to proactively initiate a topic of conversation or ask the user to utter something. In step S109, the information processing device 1 records that it is necessary to create an opportunity for conversation, passes this information to the speech information generation unit 232, and proceeds to step S115.

[0103] Next, to explain the processing of motion information, in step S110, the information processing device 1 receives motion information input from the motion information input unit 214. In step S111, the information processing device 1 determines whether or not motion information has been input. If it is determined in step S111 that there is no motion information input (NO), the information processing device 1 records that there is no motion information input, passes that information to the speech information generation unit 232, terminates the processing related to motion information, and returns to step S103.

[0104] On the other hand, if it is determined in step S111 that there is input of motion information (YES), the process proceeds to step S112. In step S112, the information processing device 1 uses the motion information analysis unit 216 to analyze the input motion information, including previously input motion information, and evaluate the motion. For example, walking pitch is calculated, and it is evaluated whether the walking pitch is within an appropriate range. For example, an evaluation result is obtained indicating that the walking pitch is within the range of predefined lower and upper limits, that the walking pitch is below the lower limit, or that the walking pitch exceeds the upper limit. The motion evaluation result is passed to the speech information generation unit 232. In this way, the motion information analysis unit 216 generates evaluation information of body motion, and the speech information generation unit 232 can use this generated evaluation information as motion information to generate speech information. In step S113, the information processing device 1 uses the video control unit 234 to determine whether a control change is necessary. If it is determined in step S113 that a control change is not necessary (NO), the processing related to motion information ends, and the process returns to step S103.

[0105] On the other hand, if it is determined in step S113 that a control change is necessary (YES), the process proceeds to step S114. In step S114, the information processing device 1 changes the motion speed using the motion control unit 234, and the processing related to motion information is completed, returning to step S103.

[0106] Once the processing of the audio information shown in steps S104 to S109 and the processing of the motion information shown in steps S110 to S114 are completed, the process proceeds to step S115.

[0107] In step S115, the information processing device 1 generates speech information based on the information obtained so far using the speech information generation unit 232. Regarding speech information, if speech recognition is successful in step S106 or step S108, the speech information generation unit 232 receives the recognized speech information in text format. If speech recognition is unsuccessful again in step S108 (NO), it receives information in step S109 that it is necessary to create a conversation trigger. Regarding motor information, if it is determined in step S111 that there is no motor input, the speech information generation unit 232 receives information that there is no motor input. If it is determined in step S111 that there is motor input, it receives the motor evaluation result in step S112.

[0108] In step S115, the speech information generation unit 232 generates speech information based on the processing results of the obtained speech information (information indicating that text or a trigger is needed) and the processing results of the motion information (information indicating that there is no walking pitch or motion input), based on the machine learning model 260 or based on logic branching. The generation of speech information is as described above, but when using LLM, a request statement is created for LLM based on the processing results of the speech information and the processing results of the motion information, and is sent to LLM, for example, via an API, to obtain a response statement. Depending on the configuration of LLM, the request statement includes information on past interactions.

[0109] If a trigger is needed, and there is no text for the audio information, if there is evaluation information for exercise, for example, include the evaluation information in the text (e.g., "I am currently walking at an average pace of 70 steps / minute.") in the request and send it to the LLM. Then, a response such as "You are walking at a good pace" will be obtained, which will be used as the speech information. Alternatively, although omitted in the flowchart, if there is biometric information (heart rate, etc.), include the biometric information in the text (e.g., "My heart rate is 110 beats / minute.") in the request and send it to the LLM. Then, a response such as "This is moderate exercise. You are doing well" will be obtained, which will be used as the speech information. Alternatively, if there is location information and related information associated with the currently displayed video frame, include the location information and related information in the text (e.g., "I can see Kaminarimon." "I can see a souvenir shop.") in the request and send it to the LLM. Then, a response such as "The official name of Kaminarimon is apparently 'Furaijinmon'" will be obtained, which will be used as the speech information. Furthermore, the process of obtaining text from attribute information such as keywords like "Kaminarimon" and "Nakamise-dori" can be generated, for example, by linking with LLM separately, or by generating text from keywords using logic branching. Also, if multiple related pieces of information are associated, in that case, it is best to evaluate the relevance with the content described in the user information of participating users and select the one with the highest relevance.

[0110] In step S115, when speech information is generated by the speech information generation unit 232, processing proceeds to step S116. In step S116, the text-formatted speech information is converted to TTS to generate speech information in voice signal format, and in step S117, the voice signal is output to the speaker 5. In step S118, the information processing device 1, via the control unit 230, stores the content of the speech recognition and speech information, associating it with the user if the speaker has been identified, or as a general conversation example otherwise, and returns processing to step S103. Based on the speech information generated in step S115, a voice signal is generated in step S116, and when the voice signal is output in step S117, this voice signal corresponds to the interaction from the motor support system 100 to the user. Based on the output of that voice signal, in the next cycle, the voice information obtained in steps S104 to S108 represents the user's response information (including cases where there is no voice information and no response).

[0111] If it is determined in step S103 that the walking exercise session has ended, such as when video playback has finished or when operator O gives an instruction to end the session using remote control 6 (YES), the process proceeds to step S119 and this process ends.

[0112] At the end of a walking exercise session, an end screen 320 is displayed, for example, as shown in Figure 3(D), showing the results of the walking exercise session. On the end screen 320, result information 322, such as the number of steps taken by each user U, is displayed in a tiled format, and a button 324 is displayed to return to the content selection screen. When button 324 is selected, the user returns to the content selection screen 300 shown in Figure 3(A).

[0113] Furthermore, upon termination, the user information storage unit 246 may record usage history information associated with each user who participated in the walking exercise session. Usage history information may include the video content used, the number of steps taken in the session, the average, walking pitch (statistical values ​​such as maximum, minimum, and average), and information extracted from conversations during the session (hobbies, experiences, friends, acquaintances, hometown, occupation, sports, other experiences, interests, etc.).

[0114] The process of generating speech information based on motion information and audio information will be described in more detail below with reference to Figures 7 and 8. Figure 7 is a diagram illustrating the speech information generation process based on motion information and audio information performed by the information processing device 1 according to the first embodiment of this disclosure. Note that in Figure 7, no video is played, and for example, user U performs walking exercises while watching a general television program on a television.

[0115] Figure 7 shows a flowchart illustrating the sequence of steps from the start (S200) to the end (S205) of the walking motion support process, with the data flow between blocks shown next to the flowchart. As shown in the flowchart of Figure 7, the walking motion support process starts at step S200, and in step S201, the user's selection is accepted. User registration may be performed manually using the remote control 6, or automatically from a facial image using the camera 3. At this stage, the remote control 6, etc., is used to specify the usage time, register the user, and register the association between the user and the walking sensor unit 20. Once registration is complete, in step S202, use begins, and the usage status is displayed (on a usage screen similar to that in Figure 3(B) excluding the video Mv). In step S203, use ends due to the elapsed usage time. In step S204, the usage results shown in Figure 3(D) are displayed, and in step S206, the process ends, and the system waits for further instructions.

[0116] From step S202, when use begins, until step S203, when use ends, in the flowchart shown in Figure 7, motion information 420 is input from each of the one or more walking sensor units 20, and voice information 422 for multiple directions is input from the microphone array 7. User information 420a and time information 420b are added to the motion information 420 by the motion information analysis unit 216. User information 422a, time information 422b, and content text 422c are added to the voice information 422 by the voice recognition unit 212. This input information is passed to the speech information generation unit 232.

[0117] The speech information generation unit 232 generates speech information based on the input information. If there is utterance from the user, it inputs text-formatted audio information to generate a conversation, and if there is motor input, it inputs motor evaluation information to generate a conversation. In this process, the conversation example collection 432 and user information 434 may be used.

[0118] Figure 8 illustrates a conversation generated based on motor information and voice information by the information processing device 1 according to the first embodiment of this disclosure. In the table shown in Figure 8, the columns indicate the state of motor input, illustrating four states: (1) when the walking pitch of the motor input is below the recommended lower limit, (2) when the walking pitch of the motor input is within the recommended range, (3) when the walking pitch of the motor input exceeds the recommended upper limit, and (4) when there is no motor input. In the table shown in Figure 8, the rows indicate the state of voice information input, illustrating two states: (A) when there is no conversation, and (B) when there is voice input.

[0119] In the case of no conversation (A), the conversation consists only of utterances from the system, as indicated by "automatic." These utterances correspond to the case where, in step S109 in Figure 6, it is recorded that a conversation trigger needs to be created, and the utterance is generated in step S115 based on that information. This conversation may be generated by a machine learning model 260 such as LLM, or one or more examples of speech information (432 in Figure 7) may be predefined, as shown in the table in Figure 8, and a probabilistic selection may be made from these examples of speech information.

[0120] Furthermore, (4) if there is no motor input, step S111 will communicate that there is no motor input, and step S115 will correspond to the case where speech information is generated. (1) if the walking pitch of the motor input is below the recommended lower limit, (2) if it is within the recommended range, and (3) if it exceeds the recommended upper limit, it will correspond to the result of the motor evaluation in step S112. The recommended range may be a value common to all users, or it may be a value obtained from user U's past performance, stored in user U's user information (434 in Figure 7), and based on that information.

[0121] In the case of a conversation (B), the conversation is an example of a conversation in which the system responds to human utterances, as indicated by "human" and "automatic". This utterance corresponds to the case where text-based speech information is obtained in step S106 or 108 of Figure 6 and generated based on that information in step S115. Such utterances may be generated by logic branching, but are preferably generated by a machine learning model 260 such as an LLM.

[0122] The generated speech information is converted into an audio signal by the speech synthesis unit 208 and output from the speaker 5. Alternatively, instead of, or in conjunction with, the audio output is output as an image to the display unit 4 as a caption by the caption creation unit 206. Images such as illustrations may be displayed along with the caption.

[0123] The exercise support system 100 according to the embodiment of this disclosure has been described above. In the above configuration, it is possible to output speech information related to the user's exercise support in accordance with the user's response information and exercise information. This makes it possible to provide an information processing device, information processing system, method, and program that provides exercise support to the user by engaging in appropriate dialogue according to the exercise situation.

[0124] Simply applying a conventional dialogue system to the exercise support system 100 would not allow for adequate exercise support, as it would be unable to communicate with the user in a way that corresponds to the user's exercise status. For example, while a conventional dialogue system might be able to respond to a user's utterances while taking the conversational context into account, it would be difficult to consider the user's exercise state.

[0125] In contrast, the exercise support system 100 according to the embodiment of this disclosure generates speech information based on the user's exercise information in addition to response information such as voice information. This allows the content of the speech to be tailored to the user's exercise state, enabling appropriate exercise support by engaging in dialogue appropriate to the exercise situation. In other words, it is possible to generate speech information that includes not only speech content that responds to the user's speech, but also speech content that corresponds to the user's exercise state. For example, if the user's speech is "The weather is nice today," speech information can be generated by adding a conversation ("That's a pretty good pace") that corresponds to the exercise state (for example, whether the walking pace is within the appropriate range or outside the appropriate range) to a normal response to the user's speech (for example, "Yes, it's refreshing.").

[0126] Furthermore, in addition to utterances related to the user's response information, it is also possible to generate speech information that includes utterances corresponding to the user's exercise state. For example, if the user's utterance is "I'm feeling a bit sluggish today," this utterance suggests the user's physical condition. Speech information can then be generated by adding utterances corresponding to the exercise state ("You're walking perfectly fine now") to utterances based on information about the user's condition (e.g., "Please slow down and don't push yourself too hard").

[0127] The process of generating speech information based on motion information, audio information, and image attribute information will be described in more detail below with reference to Figures 9 and 10. Figure 9 is a diagram illustrating the speech information generation process based on motion information, audio information, and image attribute information executed by the information processing device 1 according to the second embodiment of this disclosure.

[0128] Figure 9 shows a flowchart illustrating the sequence of steps from the start (S300) to the end (S307) of the walking motion support process, with the data flow between blocks shown next to the flowchart. As shown in the flowchart of Figure 9, the walking motion support process starts at step S300, and in step S301, the user's selection is accepted, similar to step S201 in Figure 7. In step S302, the selection of video content by the remote control 6 is accepted. Alternatively, the selection of video content may be accepted by voice recognition of the video content name. In step S303, video playback control of the selected video content is started. In step S304, the video content is played and the user screen 310 shown in Figure 3(B) is displayed. In step S305, video playback ends. In step S306, an end screen 320 including the usage results as shown in Figure 3(D) is displayed, and the process ends in step S307, waiting for further instructions.

[0129] In the flowchart shown in Figure 9, from the start of video playback in step S304 until the end of video playback in step S305, motion information 420 and audio information 422 are input as needed, similar to Figure 7. Additional information is added to the motion information 420 and audio information 422.

[0130] Meanwhile, during video playback, video data 424 is identified, and related information 428 is extracted from the video data 424 by the video analysis unit 242. The video analysis unit 242 also measures the video's progression speed (pitch) 426. From the video analysis, the video's progression speed (i.e., the walking speed at the time of video recording) is calculated. Furthermore, frames within the video are analyzed, and image attribute information is extracted. The video's progression speed is used to compare with the user's walking pitch and to control the video's playback speed.

[0131] The speech information generation unit 232 generates speech information based on the input information and related information. Conversation example collections 432 and user information 434 may be used in this process.

[0132] Figure 10 illustrates a conversation generated by the information processing device 1 according to a second embodiment of this disclosure, based on motion information, audio information, and image attribute information. In the table shown in Figure 10, similar to Figure 8, the columns indicate the state of motion input and the rows indicate the state of audio information input.

[0133] In cases where there is no dialogue (A), the only dialogue consists of utterances from the system. In the example dialogue, [Location] is information obtained as image attribute information from video analysis, and includes specific names such as "Asakusa" or "Takao-san." The same applies to the sections labeled [Type of Shop] and [Person's Name]. The [Numerical Value] section is obtained, for example, from frame numbers associated with predetermined attribute information such as shop names, the video's playback speed, and the user's current walking pitch (converted from stride length to speed). Other underlined utterances are information generated based on user information 434, or information generated based on an analysis of the video's playback speed. For example, if the user describes places they have visited or videos they have played during walking exercises as user information in the initial prompt, it is expected that such dialogue will be generated.

[0134] The generated speech information is output similarly by the speech synthesis unit 208 and the caption creation unit 206, or both. Furthermore, with respect to videos, the system recognizes images within the video frames and, based on user information 434, may display explanatory information as captions or images of related information if there is something of interest to the user (for example, displaying an image of what is beyond the temple gate before the user exits). At the end of use, the conversation content may be fed back into the conversation example collection and user information for updating.

[0135] The following will provide a more detailed explanation of the video playback control process, which controls the video (image information) to be output to the video playback unit 226 based on motion information, with reference to Figure 11. Figure 11 is a diagram illustrating the video playback control based on motion information performed by the information processing device 1 according to the third embodiment of this disclosure.

[0136] Figure 11 shows a flowchart illustrating the sequence of steps from the start (S400) to the end (S408) of the walking motion support process, with the data flow between blocks shown next to the flowchart. The flowchart in Figure 11 is similar to that shown in Figure 9, but the difference is that between the start of video playback in step S404 and the end of video playback in step S406, there is a step in step S405 for controlling the video playback speed. Figure 11 also shows the video control unit 234, where, as in the first and second embodiments, speech information is generated by the speech information generation unit 232 based on input information and related information. In addition, the video playback unit 226 changes the video playback speed based on motion information.

[0137] For example, as mentioned above, the cameraman's walking speed can be estimated by analyzing the video. Then, by assigning a standard stride length from the user's walking pitch, the user's walking speed can be determined. Alternatively, by assigning a standard stride length from the cameraman's walking speed, the cameraman's walking pitch (referred to as the walking pitch) can be determined. In the following, for simplicity, calculations will be performed using pitch. The difference between the user's walking pitch and the walking pitch extracted from the video (user pitch / walking pitch) is calculated according to the user's walking motion. If there are multiple users, the average of the multiple users is calculated, and the video playback speed can be adjusted so that the average pitch (speed) and the walking pitch (speed) match the average value (and other maximum and minimum values).

[0138] The abnormality reporting process, in which the reporting information generation unit 236 generates reporting information based on biological information, will be described in more detail below with reference to Figure 12. Figure 12 is a diagram illustrating the abnormality reporting process based on biological information executed by the information processing device 1 according to the fourth embodiment of this disclosure.

[0139] Figure 12 shows a flowchart following the sequence of steps from the start (S500) to the end (S507) of the walking motion support process, with the data flow between blocks shown next to the flowchart. The flowchart in Figure 12 is the same as that shown in Figure 9, the difference being that the biosensor unit 23 inputs biometric information 436 as it progresses from the start of video playback in step S504 until the end of video playback in step S505. User information 436a and time information 436b are added to the biometric information 436 by the biometric information analysis unit 224 and passed to the speech information generation unit 232. Figure 12 also shows the report information generation unit 236, which, as in the first to third embodiments, generates speech information based on the input information and related information by the speech information generation unit 232, but in addition, the report information generation unit 236 detects abnormalities based on the biometric information and creates a report.

[0140] If the biosensor unit 23 detects an abnormality, such as the heart rate exceeding the upper limit for each user, the speech information generation unit 232 generates speech information to the user instructing them to stop using the device. The report information generation unit 236 generates emergency report information, calls the notification unit 228, and sends the emergency report information to a predetermined destination. The device's operation is also stopped as necessary. The upper limit that serves as the criterion for such emergency reports is provided as user information for each user, for example, based on advice from a doctor.

[0141] With the above configuration, by using real-time biometric information to capture changes in the user's physical condition, exercise can be promoted more smoothly. Alternatively, instead of using the above biometric information, the system may also stop and report in the same manner as above if it detects that the text-formatted voice information from the voice recognition unit 212 contains words or phrases (such as words requesting SOS) that belong to a predetermined list, based on voice information.

[0142] The following will provide a more detailed explanation of the video content suggestion process, which suggests multiple video content based on past information stored in the user identification unit 220, with reference to Figure 13. Figure 13 is a diagram illustrating the video content suggestion process executed by the information processing device 1 according to the fifth embodiment of this disclosure.

[0143] Figure 13 shows a flowchart illustrating the sequence of steps from the start (S600) to the end (S607) of the walking motion support process. The flowchart in Figure 13 is similar to that shown in Figure 9, the only difference being that the operation unit 202, which acts as a selection means in this embodiment, is shown, and the video selection in step S62 is based on the suggestion of recommended video content by the operation unit 202. User information 434 stores videos used in past sessions, the number of steps taken in a session, the average walking pitch, and information extracted from conversations in past sessions (hobbies, experiences, friends, acquaintances, hometown, occupation, sports, other experiences, interests, etc.).

[0144] The operation unit 202 selects video content as suggested content based on the information in the user information 434, such as video content with a similarity to the user information. If multiple users participate, the similarity between the user information of each user is calculated, an overall similarity is calculated, and the video with the highest similarity is selected. Furthermore, it is preferable to select suggested content from multiple perspectives. For example, if there are multiple users, the system can analyze each user's preferences to select videos, and also select several videos that suit the common preferences of multiple users, as well as several that suit the individual preferences. Regarding the display method, a score can be calculated and the videos can be displayed in score order. In this case, suggested videos selected from the perspective of the overall preferences of multiple users are displayed at the top, followed by suggested videos selected from each user's individual preferences, and also include recently played videos. Users may manually decide which of the multiple suggested content to play. Alternatively, if the user requests a "leave it to us" option, the system will select one video to play, either the top-rated video or one with a probability corresponding to its rating.

[0145] In this way, by recording the user's exercise activity and conversational status from previously used video content in the user's information, and by selecting content that matches the user's preferences from multiple video contents and suggesting it in a list, users can quickly find content that suits their preferences.

[0146] The following explains the speech generation process using specific conversation examples.

[0147] In one or more embodiments, the speech information generation unit 232 can generate speech information based on user information. The process of generating speech information based on user information will be described below using specific examples.

[0148] The speech information generation unit 232 can generate speech information using user information stored in the user information storage unit 246 as input. For example, from information such as the sports history in the user information 434, it can generate speech information such as, "Mr. / Ms. ○△, you really love sports, don't you? Badminton is quite strenuous exercise, isn't it?" This sports history may be information that was entered in advance, for example, and may have been mentioned in a conversation during a past walking exercise session. By storing the user's prior information or information spoken and exercised by the user during use, it becomes possible to make speech appropriate to the user's situation using that information, thereby eliciting conversation from the user.

[0149] In one or more embodiments, the speech information generation unit 232 can generate speech information prompting the user to reduce or stop exercising, based on at least one piece of information: voice information and exercise information. Another example of conversation is that, upon detecting that the walking pitch indicated by the exercise information exceeds the upper limit of the recommended range, and in response to the inclusion of a predefined word or phrase (e.g., "I'm getting tired") in the voice information, it may generate speech information such as, "Let's stop here for today. You walked a lot yesterday, so I think you've had enough exercise." In this case, if there is only one participant, the walking exercise session may be terminated. If there are multiple participants, the status of the corresponding user may be changed to "rest."

[0150] By using voice and movement information, the system can speak based on the user's real-time physical condition and situation, allowing it to suppress or stop movement, and enabling users to stop using the system without overexerting themselves.

[0151] In one or more embodiments, the speech information generation unit 232 can further generate speech information prompting a change of topic based on at least one piece of information, such as voice information and motor information. The speech information generation unit 232 can generate speech information to change the topic based on the motor information, in accordance with situations such as exceeding the upper limit of the recommended range of walking pitch, falling below the lower limit, or maintaining the recommended range for a long period of time. The speech information generation unit 232 can also generate speech information to change the topic if it detects an excited attribute, such as "I went here!" or "That's my favorite thing," through voice emotion analysis or sentiment analysis of the voice information.

[0152] Topic-changing speech information is generated from conversation examples, user information, and location and related information provided as image attribute information, as well as associative information linked to location and related information through general knowledge. In another example of a conversation, in response to a user's utterance, "The cherry blossoms at the castle ruins park were amazing," the system can detect the attribute of excitement and prompt a topic change, such as "Is the Imperial Palace around here?" based on associative information from "castle ruins park." Furthermore, based on movement information, in response to the walking pace falling below a certain threshold, the system can generate topic-changing speech information such as, "It's great that everyone has so many memories. It feels so good to rest under the cherry trees, doesn't it? Shall we take a break?"

[0153] In this way, based on audio and motor information, it is possible to detect whether the user is engrossed in walking, or if they are losing interest, bored, or have a decreased level of concentration. By changing the topic accordingly, the user's level of concentration can be increased.

[0154] In one or more embodiments, the user information storage unit 246 stores at least one piece of information consisting of voice information and motion information from past walking exercise sessions, and the speech information generation unit 232 can generate speech information using the stored past information as input.

[0155] The user information storage unit 246 generates speech information from past session dates and times, video content used, conversations within sessions, and historical information (such as cumulative statistical information). In another conversation example, it may generate speech information such as, "Last time, Mr. / Ms. XX, Mr. / Ms. YY, and Mr. / Ms. YY walked at a good pace." from past exercise information (number of steps and walking pace in past walking exercise sessions). In yet another conversation example, it may generate speech information such as, "During your session before last, Mr. / Ms. YY talked a lot about the typhoon. Were you affected in any way?" from past audio information (past topic "typhoon"). In this way, by generating speech information from conversations the user has made in past sessions and exercise information from those sessions, it becomes possible to have conversations that are tailored to the user and allow for greater understanding.

[0156] In one or more embodiments, the image attribute information includes location information, and the speech information generation unit 232 can generate speech information using relevant information corresponding to the location information as input.

[0157] The video data can store geographical coordinates such as GPS information associated with each frame, and the speech information generation unit 232 can collect information (e.g., shop information or historical building information) from the internet based on these geographical coordinates, and create spoken information by combining this information with conversation information and user information. In yet another conversation example, the spoken information "The ★★ shop on the right seems to have been recently renovated and started selling dango. Their mitarashi dango and other items are said to be delicious." can be generated based on information obtained from the internet based on location information associated with the video frames. In addition to spoken information, image information such as photos of the shop may be added to the on-screen text.

[0158] In this way, by using location information from video data to collect information from external sources and displaying it in conversations and images, it is possible to give users a more immersive experience of being at the actual location.

[0159] In one or more embodiments, the user identification unit 220 can identify a specific user from among multiple users, and the speech information generation means can generate speech information corresponding to the user identified by the user identification means. In other conversation examples, speech information such as "Mr. / Ms. ○△, Mr. / Ms. ○?, Mr. / Ms. ○◇, let's have a great day today," or "Mr. / Ms. ○△, you walked a lot yesterday, didn't you? Do you think you'll be able to walk a lot today too?" can be generated based on the user information of a specific user. "Mr. / Ms. ○?, it's been a week, hasn't it? How have you been lately? You said you had a cold, but are you feeling better now?" can be generated based on the user information of a specific user.

[0160] In this way, by using each user's usage history, it becomes possible to have conversations with each individual about their health and recent activities. By having conversations with each user based on their individual user information, conversation history, and exercise data, it is possible to give each user a sense of participation and being acknowledged.

[0161] The process for generating speech information has been described above. Below, the reporting process according to other embodiments will be described regarding the functions of the reporting information generation unit 236.

[0162] As described above, the report information generation unit 236 compares past information or predefined reference information stored in the user information storage unit 246 with the input current information and generates report information based on the comparison result.

[0163] Examples of reported information include emergency reports and periodic reports. As explained with reference to Figure 12, the report information generation unit 236 can generate emergency reports and call the notification unit 228 when it detects an abnormality based on biological information, such as when the heart rate of the biosensor unit 23 exceeds the upper limit given as the criteria information for emergency reporting, or when it detects that the text-formatted voice information from the voice recognition unit 212 contains words or phrases that correspond to the list given as the criteria information for emergency reporting (such as words requesting SOS).

[0164] In addition to the above, the reporting information generation unit 236 can also send information to registered recipients (e.g., family, assigned caregiver, etc.) if the walking pitch indicated by the exercise information or the number of conversations indicated by the voice information is below a predetermined percentage of the user's past average value or the overall average value stored in the user information storage unit 246, along with information indicating that there was a change, such as the exercise pitch not being as high as usual or the conversation not being as lively. Furthermore, if the walking pitch indicated by the exercise information or the number of conversations indicated by the voice information is above a predetermined percentage of the user's past average value or the overall average value stored in the user information storage unit 246, it can send information to registered recipients along with information indicating that the user was more active than usual. Such reports may be sent during a walking exercise session to encourage human intervention by caregivers or staff, or they may be sent as a summary of the session after the walking exercise is completed, or they may be sent at predetermined intervals, such as once a month, as a report to the family.

[0165] By comparing past conversations and exercise information with the current or results of the activity, changes in the user's physical condition and emotional state can be detected and reported, thereby reducing the time caregivers and other staff spend on interviews, observations, and report writing.

[0166] In the embodiments described above, the spoken information was output as audio from the speaker 5 or as text or images from the display unit 4. Further embodiments in which audio and video are output in conjunction will be described below.

[0167] Figure 14 illustrates a user screen displayed on the display unit of an exercise support system according to another embodiment of the present disclosure. Figure 14 differs from Figure 3(B) or Figure 3(C) in that a virtual human VH is displayed. Here, a virtual human refers to a person created using three-dimensional computer graphics. By using the virtual human VH to output speech information while lip-syncing, a more realistic conversation becomes possible. It is also possible for the virtual human VH to express itself with gestures and blink in accordance with speech. In this case, since the microphone array 7 has directional information, the face and gaze of the virtual human VH can be turned towards the direction of the user who is speaking or towards the direction in which a specific user is estimated to be located. Display control of the virtual human VH is performed, for example, by the video control unit 234.

[0168] In this way, displaying characters such as virtual humans (VH) in images makes it possible to draw users into more realistic conversations. Alternatively, animations or other superimposed images could be used instead of virtual humans.

[0169] Thus, in the embodiment shown in Figure 14, the video control unit 234 can change the content of the video's image information by controlling the character information in conjunction with the speech information output by the speech information output unit 204.

[0170] (Example of a distributed implementation of an exercise support system) In the above description, the components shown in Figure 2, indicated by the dashed rectangle 1, were described as being implemented in the information processing device 1. However, the implementation method of the motor support system 100 is not particularly limited, as mentioned above, and various distributed implementation methods may be adopted. Hereinafter, with reference to Figure 15, an example of a distributed implementation of the motor support system according to the embodiment of this disclosure will be described.

[0171] In the embodiment shown in Figure 15, the information processing device 1 includes an operation unit 202, a speech information output unit 204, a voice information input unit 210, a voice recognition unit 212, a motion information input unit 214, a motion information analysis unit 216, a motion recognition unit 218, a facial expression input unit 219, a user identification unit 220, a biometric information input unit 222, a biometric information analysis unit 224, a video playback unit 226, a notification unit 228, and a control unit 230. In contrast, a video storage unit 240, a video analysis unit 242, a conversation example storage unit 244, a user information storage unit 246, a learning unit 248, a training data storage unit 250, and a machine learning model 260 are implemented outside the information processing device 1.

[0172] By implementing the information processing device 1 in a distributed manner as shown in Figure 15, the resource requirements of the information processing device 1 can be reduced. The video storage unit 240, the video analysis unit 242, the conversation example storage unit 244, the user information storage unit 246, the learning unit 248, the training data storage unit 250, and the machine learning model 260 may be implemented on the same server, some or all of them may be located on different servers, or they may be realized by multiple servers in which a single storage unit or functional unit is distributed.

[0173] The above explanation has been given with reference to the specific configuration of the exercise support system 100, based on the configuration shown in Figure 1. Below, we will describe some variations of the exercise support system 100.

[0174] (First modified example of the exercise support system) Figure 16 is a schematic diagram showing an exercise support system 100a according to the first modified example. In the first modified example, the display unit 4 is a VR (Virtual Reality) glass, which differs from the embodiment in which the display unit 4 was a planar device. VR glasses are an example of a head-mounted display device. By using VR glasses for the display unit 4, the user U can be given a more realistic simulated experience, and the user U's exercise can be promoted. The VR glasses may also be a head-mounted display. Furthermore, the VR glasses are not limited to the goggle type illustrated in Figure 16, but may also be eyeglasses. In addition, the image displayed on the display unit 4 may be a normal planar image or a 360-degree image. For example, when using VR glasses for the display unit 4, an image cropped from a 360-degree image may be displayed according to the orientation of the VR glasses.

[0175] In addition to the above, the VR glasses may be equipped with a camera (eye-tracking camera) that detects the user's gaze from the tilt and movement of their eyes, in which case the video playback unit 226 may have a function to switch the video in the direction of the gaze. Furthermore, it may generate speech by inputting gaze information. For example, speech information can be changed according to the user's gaze information. For example, in a video of walking down a tree-lined street, if it is estimated from the gaze information and image information that the user's gaze is focused on the planted trees, speech related to the trees can be generated. In that case, the functions of the information processing device 1 shown in Figure 1 may also be incorporated into the VR glasses.

[0176] (Second variation of the exercise support system) Next, an information processing device or information processing system according to a second modification will be described. Figure 17 is a diagram showing an example of the configuration of the exercise support system 100b according to the second modification. The exercise support system 100b differs from the embodiments and modifications described above in that multiple users U, who are located remotely from each other, can perform walking exercises while sharing the same video.

[0177] In the example shown in Figure 17, the information processing device 1 and the display units 4 used by each of the multiple users U are connected to each other via a network so that they can communicate with one another. The display units 4 are PCs, tablets, or smartphones, etc. The information processing device 1 can acquire the movement information of the users U via the network and distribute video data and data to be displayed on the display units 4 in streaming format to each of the multiple display units 4.

[0178] The exercise support system 100b can provide users U, who are located remotely from each other, with a sense of presence as if they were exercising in the same location. In the example shown in Figure 17, the walking sensor unit 20 is connected to the display unit 4 wirelessly and can transmit exercise information to the information processing device 1 via the display unit 4. However, the walking sensor unit 20 can also be directly connected to the network and transmit exercise status information to the information processing device 1 without going through the display unit 4.

[0179] Furthermore, the data capacity of the video distributed in the exercise support system 100b may be appropriately changed depending on the device that makes up the display unit 4. For example, the information processing device 1 can obtain information from the display unit 4 about the type of device used by user U, as well as the performance of the CPU and memory, and distribute the video with a data capacity appropriate for each device. The data capacity can be adjusted by adjusting the resolution, frame rate, and bitrate. This allows users U to participate in exercise while watching a common video, even if they use devices with varying processing speeds, by reducing the data capacity for display unit 4 devices that do not have a fast processing speed.

[0180] Figure 18 is a block diagram showing an example of the hardware configuration of the information processing device 1. The information processing device 1 is constructed, for example, by a computer. The information processing device 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an HDD (Hard Disk Drive) / SSD (Solid State Drive) 104, and an I / F (Interface) 105. These are connected to each other via a system bus B so that they can communicate with one another.

[0181] The CPU 101 performs control processing, including various arithmetic operations. The ROM 102 is a non-volatile memory that stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is a volatile memory used as the work area of ​​the CPU 101. The HDD / SSD 104 is a non-volatile memory capable of storing various information and programs used for control by the information processing device 1.

[0182] I / F105 is an interface for communication between the information processing device 1 and devices or equipment other than the information processing device 1. I / F105 can also communicate with external devices other than the information processing device 1 via a network or the like. External devices include a walking sensor unit 20, a camera 3, a display unit 4, a speaker 5, a remote control 6, a microphone array 7, a lighting device (not shown), an air conditioning device, an odor generator, and a blower, each of which can output control signals. External devices may also be servers S that are connected to the information processing device via a network for communication.

[0183] The functions of the information processing device 1 can also be realized by one or more processing circuits. Here, "processing circuit" includes devices such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), DSPs (Digital Signal Processors), and conventional circuit modules designed to perform the functions described above. Furthermore, some of the functions of the information processing device 1 can also be realized by external devices such as an external PC (Personal Computer) or a server S that are connected to the information processing device 1 in a communicative manner. In addition, some of the functions of the information processing device 1 can also be realized through distributed processing between the information processing device 1 and these external devices.

[0184] Furthermore, embodiments of this disclosure include a program. The program causes a computer to perform the processes described above. Such a program can provide the same effects and benefits as the information processing device 1 and the motor support system 100 described above.

[0185] The exercise support system 100 for assisting walking in elderly care facilities has been described above with reference to Figures 1 to 18. However, the exercise support system according to the embodiments of this disclosure is not limited to assisting walking as described above. Hereinafter, an exercise support system 500 according to yet another embodiment for assisting other types of physical exercise will be described with reference to Figures 19 and 20.

[0186] Figure 19 is a schematic diagram of an exercise support system 500 according to yet another embodiment of the present disclosure. The exercise support system 500 is a system that supports indoor running exercise of a user U using a running machine (sometimes called a treadmill or running runner) 550. In the embodiments described, the case in which "indoor running exercise" is performed as physical exercise will be explained as an example, and here, "indoor running exercise" will include a simulated running motion that involves raising and lowering the feet in a fixed position as the running surface of the running machine 550 rotates.

[0187] As shown in Figure 19, the exercise support system 500 comprises a display terminal 520, such as a smartphone or tablet PC owned by the user, and a wearable terminal 530, such as a smartwatch. The display terminal 520 has a display, as will be described later, and is installed, for example, on the treadmill 550 in a position where the user U can easily view the display. In the embodiment described, the display terminal 520 is described as being installed on the treadmill 550, separate from the treadmill 550. This assumes a use case in which the user installs their own display terminal 520 on the treadmill 550. However, the embodiment is not limited to this, and the display terminal 520 may be a device that includes a display installed on the treadmill 550. The user U wears the wearable terminal 530 and performs indoor running exercises using the treadmill 550 while watching a video (for example, a video of exemplary movements by an instructor) displayed on the display terminal 520's screen. A wireless connection (pairing) is established between the display terminal 520 and the wearable terminal 530 using Wi-Fi (registered trademark) or Bluetooth (registered trademark) 506.

[0188] During indoor running exercise, the wearable terminal 530 acquires exercise information related to the exercise (information on running and walking movements) as an example of an exercise information acquisition device, and the user's biometric information (heart rate, blood oxygen saturation, and activity level) as an example of a reaction information acquisition device, and outputs it to the display terminal 520 via the wireless connection 506. The wearable terminal 530 may also acquire voice information of speech uttered by user U as an example of a reaction information acquisition device during indoor running exercise, and output it to the display terminal 520 via the wireless connection 506.

[0189] The display terminal 520 is further connected to a network 502, such as the Internet, via a mobile communication network 504, such as 4G or 5G. A server device 510 is located on the network 502. The display terminal 520 communicates with the server device 510 via the mobile communication network 504 and the network 502 to provide the aforementioned exercise support functions to the user U.

[0190] The display terminal 520 is also equipped with a speaker, as described later, and outputs voice based on the generated speech information to interact with user U. Furthermore, during user U's indoor running exercise, the display terminal 520 may, as an example of a reaction information acquisition device, acquire voice information of speech uttered by user U using its own microphone. The display terminal 520 is also equipped with a camera that guides the user to capture images with the user environment, including user U, within its field of view. During user U's running exercise, as an example of a movement information acquisition device or a reaction information acquisition device, the display terminal 520 may, using its own camera, analyze the input image to detect user U's facial expressions and acquire facial expression information, or detect user U's skeleton, perform motion analysis, and acquire movement information.

[0191] In the exercise support system 500 shown in Figure 19, user U performs indoor running exercises using a treadmill 550 while watching a video. During the indoor running exercise performed by user U, the exercise support system 500 generates speech information based on user U's exercise information and reaction information (voice information, biometric information, or facial expression information from user U within a predetermined time in response to the generated speech output), and interacts with the user. This provides appropriate exercise support according to the user's exercise state.

[0192] In the embodiments described, it is assumed that one user U uses one display terminal 520 and one wearable terminal 530, but the embodiments are not limited to this. In other embodiments, the exercise support system 500 may be configured to be usable by multiple users U.

[0193] Furthermore, while indoor running was used as an example of physical exercise in the example shown in Figure 19, the types of physical exercise that the exercise support system 500 according to the embodiment of this disclosure can support are not particularly limited. In addition to simulated running exercises such as indoor running mentioned above, examples of physical exercises that can be included are various exercises performed for the purpose of maintaining and improving health and physical fitness, such as weight training, aerobics, dance, stretching, yoga, and fitness.

[0194] The functional configuration of the exercise support system 500 will be described in more detail below with reference to Figure 20. Figure 20 shows a functional block 600 of the exercise support system 500 according to yet another embodiment of the present disclosure.

[0195] Figure 20 shows that the hardware of the display terminal 520 includes a touchscreen sensor 521, a display 522, a speaker 523, a microphone 524, a camera 525, and a wireless network interface 526. The functional block 600 of the display terminal 520 shown in Figure 20 includes an operation unit 602, a speech information output unit 604, a voice information input unit 610, a voice recognition unit 612, a motion information input unit 614, a motion information analysis unit 616, a motion recognition unit 618, a facial expression input unit 619, a biometric information input unit 622, a biometric information analysis unit 624, a video playback unit 626, and a control unit 630.

[0196] Figure 20 also shows the hardware of the wearable terminal 530, which includes a step sensor 531, a biosensor 532, and a wireless network interface 533.

[0197] Figure 20 further includes external functional blocks to the display terminal 520, such as functional blocks on a server device 510 to which the display terminal 520 is connected via the network 502, which include a video storage unit 640, a video analysis unit 642, a conversation example storage unit 644, a user information storage unit 646, a learning unit 648, a training data storage unit 650, and a machine learning model 660.

[0198] The functional parts described with reference to Figure 20 are the same as those shown in Figure 2, and the differences will be explained below.

[0199] The control unit 630 performs overall processing and control for supporting the user U's exercise, including the process of generating speech information. The control unit 630 includes a speech information generation unit 632, similar to the embodiment shown in Figure 2. The speech information generation unit 632 generates speech information based on input information received via input units such as the voice information input unit 610, the exercise information input unit 614, the facial expression input unit 619, and the biometric information input unit 622, and outputs the speech information to the speech information output unit 604. The control unit 630 can also control video playback by the video playback unit 626 based on this input information. Furthermore, the control unit 630 can store the content of conversations during running exercises, the user's engagement with walking exercises, and topic information extracted from the conversation content in a user information storage unit 646, which will be described later, in response to the user U's running exercises.

[0200] The voice information input unit 610 receives voice information from the usage environment via the microphone 524 and outputs it to the voice recognition unit 612. The voice recognition unit 612 applies STT conversion to the voice information in digital voice signal format from the voice information input unit 610, converts it into text format voice information, and outputs it to the control unit 230. As described above, the voice recognition unit 212 may also perform voice emotion recognition, natural language analysis, etc. In the embodiment described, the display terminal 520 is equipped with a microphone 524, and the voice information input unit 610 receives voice information input via the microphone 524. However, the embodiment is not limited to this, and voice information may be acquired by a microphone equipped in the wearable terminal 530. In that case, the voice information received from the wearable terminal 530 is input to the voice information input unit 610 via the wireless network interfaces 526 and 533.

[0201] The exercise information input unit 614 receives exercise information related to user U's running exercise from the step sensor 531 of the wearable terminal 530 via the wireless network interface 526 and inputs it to the exercise information analysis unit 216. The exercise information is the same and will not be explained further. Furthermore, as mentioned above, the method of acquiring the user's exercise information is not particularly limited. For example, the wearable terminal 530 can use a device with an acceleration sensor and a gyroscope sensor to analyze gait patterns from acceleration and angular velocity to obtain exercise information. In this case, the exercise information input unit 614 receives the exercise information via the wireless network interfaces 526 and 533. In addition, the camera 525 of the display terminal 520 can also be used to acquire exercise information. In this case, the motion recognition unit 618 analyzes the image input from the camera 525, detects the user U's skeleton, performs motion analysis to detect walking movement, and the exercise information input unit 614 receives this exercise information.

[0202] The facial expression input unit 619 detects the face region of a person from the image captured by the camera 525, recognizes the facial expression of the person from the face image, and passes the facial expression information to the control unit 630.

[0203] The biometric information input unit 622 receives biometric information about the user U's body (such as the wearer's heart rate, blood oxygen saturation, and activity level) from the biosensor 532 of the wearable terminal 530 via wireless network interfaces 526 and 533. The biometric information analysis unit 624 analyzes the biometric information, adds a timestamp, and passes it to the control unit 630. Various types of biosensors 532 can be exemplified, such as activity trackers, sleep trackers, blood pressure monitors, and small brain activity sensors. The method of acquiring biometric information is not limited to the above. As mentioned above, techniques for estimating heart rate and respiratory rate by photographing a face with a camera are known, and the heart rate and respiratory rate estimated from the image of the person's face detected from the image captured by the camera 525 may be input to the biometric information input unit 622.

[0204] The video storage unit 640, video analysis unit 642, conversation example storage unit 644, user information storage unit 646, control unit 630 (speech information generation unit and video control unit), learning unit 648, training data storage unit 650, and machine learning model 660 have the same configuration as the embodiment described with reference to Figure 2, so a detailed explanation will be omitted. The video storage unit 640, video analysis unit 642, conversation example storage unit 644, user information storage unit 646, learning unit 648, training data storage unit 650, and machine learning model 560 may be implemented on the same server, some or all of them may be on different servers, or they may be realized by multiple servers in which one storage unit or functional unit is distributed. Furthermore, the learning unit 648, training data storage unit 650, and machine learning model 560 may be configured as an external configuration of the motor support system 500, communicating with the server device 510 via an API.

[0205] According to the exercise support system 500 described with reference to Figures 19 and 20, user U performs physical exercises such as indoor running using a treadmill 550 while watching a video, and the system provides support for user U's exercise by engaging in appropriate dialogue according to user U's exercise status during these physical exercises.

[0206] In the exercise support system 500 described with reference to Figures 19 and 20, typically, a biosensor 532 on the wearable terminal 530 constitutes the first sensor for acquiring user response information. When the biosensor 532 on the wearable terminal 530 constitutes the first sensor, the wireless network interface 533 constitutes a transmission means for transmitting the acquired response information to the display terminal 520. In addition to the biosensor 532 on the wearable terminal 530, at least one of the microphone 524, camera 525, and separate microphone on the wearable terminal 520 may also constitute the first sensor for acquiring user response information, either together with the biosensor 532 or in place of the biosensor 532.

[0207] Furthermore, in the embodiment described with reference to Figures 19 and 20, at least one of the step count sensor 531 of the wearable terminal 530 and the camera 525 of the display terminal 520 may constitute a second sensor for acquiring exercise information. When the step count sensor 531 of the wearable terminal 530 constitutes the second sensor, the wireless network interface 533 constitutes a transmission means for transmitting the acquired exercise information to the display terminal 520.

[0208] In a specific embodiment, while a user is running indoors, the wearable terminal 530 worn by the user detects exercise information such as steps and distance traveled from a step sensor 531, and further detects biometric information (response information) about the user's body from a biosensor 532 equipped with the wearable terminal 530. The wireless network interface 533 of the wearable terminal 530 then transmits this exercise information and biometric information to the display terminal 520. The exercise information from the step sensor 531 is input to the exercise information input unit 614 of the display terminal 520, and the biometric information from the biosensor 532 is input to the biometric information input unit 622. As explained in Figure 2, speech is generated by the speech information generation unit 632 included in the control unit 630, and the generated speech information is output from the speech information output unit 604 via the speaker 523 and display 522. This makes it possible to have a conversation that corresponds to the user's exercise status, as explained in Figure 8.

[0209] In the embodiments described with reference to Figures 19 and 20, indoor running was described as an example of physical exercise. However, as stated above, the types of physical exercise that can be supported in the embodiments of this disclosure are not particularly limited and can also be applied to fitness exercises such as stretching and yoga, which do not involve walking, running, or stepping. Hereinafter, with reference to Figure 21, an exercise support system 700 according to yet another embodiment that supports yoga (or stretching) as another type of physical exercise, which does not involve walking, running, or stepping, will be described.

[0210] Figure 21 is a schematic diagram of an exercise support system 700 according to yet another embodiment of the present disclosure. The exercise support system 700 is a system that supports user U in performing yoga exercises. In the embodiments described, the case in which “yoga exercises” are performed as physical exercises is described as an example, where “yoga exercises” typically include full-body gymnastics exercises on the floor that do not involve walking, running, or stepping.

[0211] As shown in Figure 21, the exercise support system 700 consists of a display terminal 720, such as a smartphone or tablet PC owned by the user, and a wearable terminal 730, such as a smartwatch. The display terminal 720 is equipped with a display 720a, as will be described later, and the user U is to position the display 720a in a location where they can easily see it. The user U wears the wearable terminal 730 and performs yoga exercises while watching a video (for example, a video of exemplary movements by instructor I) displayed on the display terminal 720. The display terminal 720 and the wearable terminal 730 are connected (paired) via a wireless connection 706 such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0212] The wearable terminal 730 acquires exercise information related to user U's yoga exercise (for example, information on arm movement, assuming it is linked to arm movement, using motion sensors such as an accelerometer and a gyroscope) and the user's biometric information (heart rate, blood oxygen saturation, and activity level), and outputs this information to the display terminal 720 via the wireless connection 706. The wearable terminal 730 may also acquire voice information of speech uttered by user U during the yoga exercise and output this information to the display terminal 720 via the wireless connection 706.

[0213] The display terminal 720 is further connected to a network 702, such as the Internet, via a mobile communication network 704, such as 4G or 5G. A server device 710 is located on the network 702. The display terminal 720 communicates with the server device 710 via the mobile communication network 704 and the network 702 to provide the aforementioned exercise support functions to the user U.

[0214] The display terminal 720 may also be equipped with a speaker and output voice based on the generated speech information to interact with user U. The display terminal 720 may also acquire voice information of speech made by user U using its own microphone while user U is performing yoga exercises. The display terminal 720 may also be equipped with a camera that guides users to take pictures with the usage environment, including user U, within its field of view. While user U is performing running exercises, the display terminal 720 may analyze the input image using its own camera to detect user U's facial expressions and acquire facial expression information, or detect user U's skeleton, perform motion analysis, and acquire motion information related to the user's body movements as motion information.

[0215] In the exercise support system 700 shown in Figure 21, user U performs yoga exercises while watching a video. During the yoga exercises performed by user U, the exercise support system 700 generates speech information based on user U's exercise information and response information (voice information, biometric information, or facial expression information from user U within a predetermined time in response to the generated speech output), and interacts with the user. This allows the system to provide appropriate exercise support according to the user's exercise state.

[0216] In the embodiments described, it is assumed that one user U uses one display terminal 720 and one wearable terminal 730, but the embodiments are not limited to this. In other embodiments, the exercise support system 700 may be configured to be usable by multiple users U. In the embodiments described with reference to Figure 21, the functional blocks are the same as those described with reference to Figure 20, except for the different reference numerals, and their description is omitted.

[0217] In the exercise support system 700 described with reference to Figure 21, the biosensor provided by the wearable terminal 730 constitutes a first sensor for acquiring user response information. When the biosensor provided by the wearable terminal 730 constitutes the first sensor, the wireless network interface provided by the wearable terminal 730 constitutes a transmission means for transmitting the acquired response information to the display terminal 720. In addition to the biosensor provided by the wearable terminal 730, at least one of the microphone provided by the display terminal 720, the camera provided by the display terminal 720, and a separate microphone provided by the wearable terminal 730 may constitute a first sensor for acquiring user response information, either together with the biosensor or in place of the biosensor.

[0218] Furthermore, in the embodiment described with reference to Figure 21, at least one of the motion sensor provided by the wearable terminal 730 and the camera provided by the display terminal 720 may constitute a second sensor for acquiring motion information. When the step sensor provided by the wearable terminal 730 constitutes the second sensor, the wireless network interface provided by the wearable terminal 730 constitutes a transmission means for transmitting the acquired motion information to the display terminal 720. When the motion sensor provided by the wearable terminal 730 constitutes the second sensor, the detected acceleration and angular velocity, or motion analysis information obtained by analyzing these, becomes the motion information. When the camera provided by the display terminal 720 constitutes the second sensor, the information of the user's body movements detected by analyzing the camera image becomes the motion information.

[0219] In a specific embodiment, while the user is performing yoga exercises, the wearable terminal 730 worn by the user detects movement information such as arm movements from an activity sensor, or detects movement information from analysis of images captured by a camera on the display terminal 720, and further detects biometric information (response information) about the user's body from a biosensor on the wearable terminal 730. The wireless network interface of the wearable terminal 730 then transmits this movement information and biometric information to the display terminal 720. Movement information from the activity sensor on the wearable terminal 730 or the camera on the display terminal 720 is input to the movement information input section of the display terminal 720, and biometric information from the biosensor on the wearable terminal 730 is input to the biometric information input section. As explained in Figure 2, and similar to Figure 19, speech is generated by the speech information generation section included in the control unit, and the generated speech information is output from the speech information output section via a speaker or display. This makes it possible to have a conversation that corresponds to the user's movement status, as explained in Figure 8.

[0220] According to the embodiments described above, it is possible to output speech information related to supporting the user's movement in response to user response information and movement information. This makes it possible to provide an information processing device or information processing system that effectively supports the user's movement by engaging in dialogue suitable for movement support.

[0221] Furthermore, the exercise support system 100 was used as part of rehabilitation or recreation at the aforementioned elderly care facility, allowing users to perform walking exercises while watching videos. In particular, by having elderly people simulate the leg-based locomotion they have been doing for many years and combining this with stimuli such as videos, it is possible to enhance the users' sense of immersion and add an element of physical exercise. Ultimately, this makes it possible to implement rehabilitation and recreation to maintain the brain and physical health of the elderly while reducing the need for caregiver intervention.

[0222] In the embodiments described above, walking was used as an example of physical exercise, and its applications were primarily explained as rehabilitation or recreation in elderly care facilities. However, physical exercise is not limited to walking, nor are its applications limited to rehabilitation or recreation in elderly care facilities. It can be applied to various physical movements, and its applications include experiential event tools in events, exhibitions, tourist facilities, and public facilities; medical rehabilitation systems in post-illness maintenance and recovery rehabilitation facilities and nursing care facilities; facility tour experience tools in education; and online games.

[0223] Although preferred embodiments have been described in detail above, the embodiments of this disclosure described above are not limited to those described above, and various modifications and substitutions can be made to the embodiments of this disclosure described above without departing from the scope of the claims.

[0224] The ordinal numbers, quantities, and other figures used in the description of the embodiments of this disclosure are all illustrative to specifically illustrate the technology of this disclosure, and this disclosure is not limited to the illustrative figures. Furthermore, the connection relationships between components are illustrative to specifically illustrate the technology of this disclosure, and are not limited to the connection relationships that realize the functions of this disclosure.

[0225] The division of blocks in the functional block diagram is just one example; multiple blocks may be implemented as a single block, one block may be divided into multiple parts, or some functions may be moved to other blocks. Furthermore, the functions of multiple blocks with similar functions may be processed in parallel or time-sharing by a single piece of hardware or software. Also, some or all of the functions may be distributed across multiple computers.

[0226] The aspects of this disclosure are, for example, as follows: <1> An information processing device, A means for inputting exercise information related to the user's physical movements, A response information input means that receives response information from the user, which is different from exercise information, A speech information output means outputs speech information related to user motor support based on reaction information input into a reaction information input means and motor information input into a motor information input means. It is an information processing device equipped with [a specific feature]. <2> An information processing device, The system further comprises speech information generation means that generates speech information using response information input to response information input means and motion information input to motion information input means as input. <1> This is the information processing device described above. <3> An information processing device, Response information includes at least one of the following: audio information of the user's speech, facial expression information showing the user's facial expression, and biometric information about the user's body. <2> This is the information processing device described above. <4> An information processing device, Video playback unit controls the playback of videos displayed on the display unit. Furthermore, The speech information generation means is characterized by generating speech information by further inputting at least one of the image information contained in the video and the attribute information associated with the image information. <2> This is the information processing device described above. <5> An information processing device, Image control means that controls the playback speed of a video, the content of the video's image information, or the insertion of either or both of text overlays and / or insert images into the video's image, based on motion information input to the motion information input means. Equipped with, <4> This is the information processing device described above. <6> An information processing device, User information storage means for storing user information about a user Equipped with, The speech information generation means is characterized by generating speech information using user information stored in the user information storage means as input. <2> ~ <5> It is an information processing device described in any one of the following. <7> An information processing device, The speech information generation means is characterized by generating speech information that prompts the user to reduce their movement, speech information that prompts the user to stop moving, or speech information that prompts a change of topic, based on at least one piece of information consisting of response information and movement information. <2> ~ <6> It is an information processing device described in any one of the following. <8> An information processing device, A biometric information input means into which biometric information about the user's body is entered, A notification means that notifies when an abnormality in biometric information is detected. Furthermore, <1> ~ <6> It is an information processing device described in any one of the following. <9> An information processing device, Information storage means for storing at least one piece of information: reaction information and motion information. Equipped with, The speech information generation means is characterized by generating speech information using past information stored in the information storage means as input. <2> ~ <8> It is an information processing device described in any one of the following. <10> An information processing device, Information storage means for storing at least one piece of information, such as reaction information and motion information, A selection means that selects proposed video content based on past information stored in an information storage means. Equipped with, <1> ~ <9> It is an information processing device described in any one of the following. <11> An information processing device, Information storage means for storing at least one piece of information, such as reaction information and motion information. The system includes a reporting information generation means that compares past information stored in an information storage means with current information that has been input, and generates reporting information based on the comparison result. <1> ~ <10> It is an information processing device described in any one of the following. <12> An information processing device, Image information includes location information, The speech information generation means is characterized by generating speech information using relevant information corresponding to location information as input. <4> This is the information processing device described above. <13> An information processing device, The image information includes character information that is superimposed on the image. The image control means is, This method is characterized by changing the content of the image information of a video by controlling character information in conjunction with the speech information output by the speech information output means. <5> This is the information processing device described above. <14> An information processing device, The system further includes a means for identifying a specific user among multiple users, The speech information generation means is, It generates speech information corresponding to the user identified by the user identification means. <2> ~ <13> It is an information processing device described in any one of the following. <15> An information processing device, The speech information generation means generates speech information based on a machine learning model. <2> ~ <14> It is an information processing device described in any one of the following. <16> An information processing device, The system further comprises speech information generation means that generates speech information based on logic branching, taking response information input to response information input means and motion information input to motion information input means as input. <1> This is the information processing device described above. <17> An information processing device, The speech information output means outputs speech information through voice output, text display in the video, sign language display in the video, or mechanical operation. <1> ~ <16> It is an information processing device described in any one of the following. <18> An information processing device, A motion information analysis means generates data from motion information input into a motion information input means, including at least one piece of information such as the time of walking, the number of steps over a predetermined period, the average walking speed at a predetermined point in time or over a predetermined period, and the intensity of walking. Furthermore, <1> ~ <17> It is an information processing device described in one of the following entries. <19> An information processing device, Motion information analysis means that generates evaluation information of physical movement from motion information input to motion information input means. Furthermore, The speech information generation means generates speech information using evaluation information as motion information. <2> ~ <18> It is an information processing device described in any one of the following. <20> An information processing system, A device for acquiring movement information related to the user's physical movements, A response information acquisition device that acquires response information from users, which is different from motor information, An information processing device that outputs speech information related to user movement support based on reaction information acquired by a reaction information acquisition device and movement information acquired by a movement information acquisition device. It is an information processing system equipped with [the following features]. <21> An information processing system, A wearable device worn by the user, A display terminal equipped with a display unit that shows videos related to physical exercise and Equipped with, wearable devices, It includes a transmission means for transmitting response information obtained from the user to a display terminal, The display terminal is, A motion information input means that receives motion information related to the user's physical movements, which is different from reaction information, A response information input means that receives response information from a wearable device, A speech information output means outputs speech information related to user motor support based on reaction information input into a reaction information input means and motor information input into a motor information input means. An information processing system equipped with the following features. <22> An information processing system, Wearable devices are A first sensor that acquires response information from the user, It is equipped with a second sensor that acquires movement information related to the user's physical movements, The transmission means transmits the reaction information acquired by the first sensor and the motion information acquired by the second sensor to the display terminal. <21> This is the information processing system described in [the relevant document]. <23> An information processing system, The display terminal is equipped with an imaging means for capturing images that include the user. Motion information is motion information about the user's movements obtained by analyzing images that include the user. <21> This is the information processing system described in [the relevant document]. <24> It is a program, and a computer, A means for inputting exercise information related to the user's physical movements, A response information input means that receives response information from the user, which is different from motor information, and Speech information output means outputs speech information related to user motor support based on reaction information input to reaction information input means and motor information input to motor information input means. This is a program designed to function as such. <25> A method using computer processing, Steps include inputting exercise information related to the user's physical movements, A step in which user response information, which is different from exercise information, is entered, A step of outputting speech information related to user motor support based on input response information and motor information. This method includes [something]. [Explanation of Symbols]

[0227] 1...Information processing device, 2...Mat, 3...Camera, 4...Display unit, 5...Speaker, 6...Remote control, 7...Microphone array, 20...Walking sensor unit, 21...Print display, 23...Biometric sensor unit, 24...Chair, 100, 100a, 100b, 500, 700...Exercise support system, 101...CPU, 102...ROM, 103...RAM, 104...HDD / SSD, 105...I / F, 200...Function block, 202, 602...Operation unit, 204, 604...Speech information output unit, 206, 606...Terror Speech generation unit, 208,608... Speech synthesis unit, 210,610... Speech information input unit, 212,612... Speech recognition unit, 214,614... Movement information input unit, 216,616... Movement information analysis unit, 218,618... Motion recognition unit, 219,619... Facial expression input recognition unit, 220... User identification unit, 222,622... Biometric information input unit, 224,624... Biometric information analysis unit, 226,626... Video playback unit, 228... Notification unit, 230,630... Control unit, 232,632... Speech information generation unit, 234... Video control unit, 236...Report information generation unit, 240, 640...Video storage unit, 242, 642...Video analysis unit, 244, 644...Conversation example storage unit, 246, 646...User information storage unit, 248, 648...Learning unit, 250, 650...Training data storage unit, 260, 660...Machine learning model, 300...Content selection screen, 310...Usage screen, 320...Exit screen, 402...Motion information, 404...Audio information, 406...User information, 408...Conversation example, 410...Image attribute information, 412...General knowledge, 502, 702...Ne Network, 504, 704…Mobile communication network, 506, 706…Wireless connection, 510, 710…Server device, 520, 720…Display terminal, 521…Touchscreen sensor, 522…Display, 523…Speaker, 524…Microphone, 525…Camera, 526, 533…Wireless network interface, 530, 730…Wearable device, 531…Pedometer, 532…Biometric sensor, 550…Treadmill, S…Server, B…System bus, Mv…Video, U…User [Prior art documents] [Patent Documents]

[0228] [Patent Document 1] Patent No. 6833209 specification [Patent Document 2] Japanese Patent Publication No. 2023-008123

Claims

1. A means for inputting exercise information related to the user's physical movements, A response information input means into which response information from the user, which is different from the aforementioned motor information, is input. A speech information output means outputs speech information related to user movement support based on the reaction information input to the reaction information input means and the movement information input to the movement information input means. An information processing device equipped with the following features.

2. Speech information generation means that generates speech information using the reaction information input to the reaction information input means and the motion information input to the motion information input means as input. The information processing apparatus according to claim 1, further comprising:

3. The information processing apparatus according to claim 2, wherein the response information is at least one of voice information of the user's speech, facial expression information showing the user's facial expression, and biometric information relating to the user's body.

4. Video playback unit controls the playback of videos displayed on the display unit. Furthermore, The information processing apparatus according to claim 2, wherein the speech information generation means generates the speech information by taking at least one of the image information contained in the video and attribute information associated with the image information as input.

5. Image control means control the playback speed of the video, the content of the image information of the video, or the insertion of either or both of the text overlay and / or insert images into the video, based on the motion information input to the motion information input means. The information processing apparatus according to claim 4, comprising:

6. User information storage means for storing user information relating to the aforementioned user Equipped with, The information processing apparatus according to claim 2, characterized in that the speech information generation means generates the speech information using the user information stored in the user information storage means as input.

7. The information processing apparatus according to claim 2, characterized in that the speech information generation means generates speech information that prompts the reduction of motion, speech information that prompts the stopping of motion, or speech information that prompts a change of topic, based on at least one of the response information and the motion information.

8. A biometric information input means into which biometric information relating to the user's body is input, Notification means for notifying when an abnormality in the aforementioned biological information is detected. The information processing apparatus according to claim 1, further comprising:

9. Information storage means for storing at least one piece of information, such as the reaction information and the motion information. Equipped with, The information processing apparatus according to claim 2, characterized in that the speech information generation means generates the speech information using past information stored in the information storage means as input.

10. Information storage means for storing at least one piece of information, including the reaction information and the motion information, Selection means for selecting proposed video content based on past information stored in the information storage means, The information processing apparatus according to claim 1, comprising:

11. Information storage means for storing at least one piece of information, including the reaction information and the motion information. A report information generation means that compares past information stored in the information storage means with the currently input information and generates report information based on the comparison result. The information processing apparatus according to claim 1, comprising:

12. The aforementioned image information includes location information, The information processing apparatus according to claim 4, characterized in that the speech information generation means generates the speech information using relevant information corresponding to the location information as input.

13. The aforementioned image information includes character information superimposed on the image, The information processing apparatus according to claim 5, characterized in that the image control means controls the character information in conjunction with the speech information output by the speech information output means to change the content of the image information of the video.

14. The system further includes a means for identifying a specific user among multiple users, The aforementioned speech information generation means is The information processing apparatus according to claim 2, which generates speech information corresponding to a user identified by the user identification means.

15. The information processing apparatus according to claim 2, wherein the speech information generation means generates the speech information based on a machine learning model.

16. The information processing apparatus according to claim 1, further comprising a speech information generation means that generates the speech information based on logic branching, taking the reaction information input to the reaction information input means and the motion information input to the motion information input means as input.

17. The information processing apparatus according to claim 1, wherein the speech information output means outputs speech information by voice output, display of characters in the video, display of sign language in the video, or mechanical operation.

18. A motion information analysis means generates data from the motion information input to the motion information input means, which includes at least one piece of information including the time of walking, the number of steps over a predetermined period, the average walking speed at a predetermined point in time or over a predetermined period, and the intensity of the walking motion. The information processing apparatus according to claim 1, further comprising:

19. Motion information analysis means for generating evaluation information of physical movement from motion information input to the motion information input means Furthermore, The information processing apparatus according to claim 2, wherein the speech information generation means generates the speech information using the evaluation information as the motion information.

20. A device for acquiring movement information related to the user's physical movements, A reaction information acquisition device that acquires reaction information from the user that is different from the aforementioned motor information, An information processing device that outputs speech information related to user movement support based on the reaction information acquired by the reaction information acquisition device and the movement information acquired by the movement information acquisition device. An information processing system equipped with the following features.

21. A wearable device worn by the user, A display terminal equipped with a display unit that shows videos related to physical exercise and The wearable device is equipped with, The system includes a transmission means for transmitting response information obtained from the user to the display terminal, The aforementioned display terminal is A motion information input means into which motion information relating to the user's physical movements, which is different from the aforementioned reaction information, is input. A response information input means into which the response information received from the wearable terminal is input, A speech information output means outputs speech information related to user movement support based on the reaction information input to the reaction information input means and the movement information input to the movement information input means. An information processing system equipped with the following features.

22. The aforementioned wearable device is A first sensor that acquires response information from the user, The system includes a second sensor for acquiring the motion information relating to the user's physical movements, The information processing system according to claim 21, wherein the transmitting means transmits the reaction information acquired by the first sensor and the motion information acquired by the second sensor to the display terminal.

23. The display terminal is equipped with an imaging means for capturing an image including the user, The information processing system according to claim 21, wherein the motion information is motion information relating to the user's movements obtained by analyzing an image including the user.

24. Computers, A means for inputting exercise information related to the user's physical movements, A response information input means into which response information from the user, which is different from the aforementioned movement information, and Speech information output means that outputs speech information related to user movement support based on the reaction information input to the reaction information input means and the movement information input to the movement information input means. A program designed to function as such.

25. A method using computer processing, Steps include inputting exercise information related to the user's physical movements, A step in which response information from the user, which is different from the aforementioned motor information, is input, A step of outputting speech information related to user motor support based on the input response information and motor information. Methods that include...

Images