Mobile assistance device, mobile assistance method, and storage medium

Abstract

A mobile assist device includes: a photographing section that acquires a photographed image; a control section that calculates a safety degree based on the photographed image; a storage section that stores a necessary safety degree table that stores, in correspondence with an action candidate decided in advance, a necessary safety degree required for a user to safely perform an action indicated by the action candidate; an input section that accepts input of a selection operation in which the user selects a desired action candidate from among the action candidates; and an output section. The control section acquires the necessary safety degree corresponding to the selected action candidate from the necessary safety degree table in a case where there is input of the selection operation. The control section does not assist the user in achieving the selected action candidate in a case where the calculated safety degree is less than the necessary safety degree, and controls the output section to assist the user in achieving the selected action candidate in other cases.

Description

Technical Field

[0001] This invention relates to a mobility assistance device that assists users in their movement. Background Technology

[0002] Patent Document 1 discloses a walking assistance system for visually impaired individuals. This system extracts information about the presence and location of obstacles from a stereoscopic image, and an information transmission device transmits this information via sound or tactile feedback. As a result, visually impaired users can more specifically grasp information related to navigation, such as the presence, location, and distance of obstacles.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2001-318594 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, the invention described in Patent Document 1 does not accept input of actions desired by the user, but only performs navigation. Furthermore, the invention described in Patent Document 1 does not calculate the level of security representing the degree of security relevant to the user based on images.

[0008] Methods for solving problems

[0009] One aspect of this disclosure is a mobility assistance device that assists a user's mobility. The mobility assistance device includes: a camera unit that captures images of the user's surrounding environment; a control unit that calculates a security level based on the captured images, representing a degree of security related to the user; a storage unit that stores a necessary security level table, which stores action candidates predetermined as actions that the user can perform, corresponding to necessary security levels, each necessary security level representing a degree of security required for the user to safely perform an action indicated by a predetermined action candidate; an input unit that accepts an input from the user performing a selection operation, whereby the user selects a desired action candidate from the predetermined action candidates; and an output unit, wherein, when the selection operation input is present, the control unit retrieves the necessary security level corresponding to the selected action candidate from the necessary security level table; if the calculated security level is less than the necessary security level corresponding to the selected action candidate, the control unit does not assist the user in implementing the selected action candidate; and if the calculated security level is greater than or equal to the necessary security level corresponding to the selected action candidate, the control unit controls the output unit to assist the user in implementing the selected action candidate.

[0010] Invention Effects

[0011] One aspect of the mobile assistance device disclosed herein can ensure user safety and assist the user in performing desired actions based on the level of safety. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating an application example of a mobility assistance device according to an embodiment of the present disclosure.

[0013] Figure 2 This is a block diagram illustrating a structural example of the mobility assistance device according to this embodiment.

[0014] Figure 3 This is a flowchart illustrating an example of the operation of a mobile assistive device in normal mode.

[0015] Figure 4 This is a diagram representing an example of a necessary safety degree table.

[0016] Figure 5 This is a flowchart illustrating an example of the actions of a mobile assistive device in a required execution mode.

[0017] Figure 6 This is a flowchart illustrating a variation of the operation of a mobility aid in normal mode. Detailed Implementation

[0018] Hereinafter, embodiments of the mobility assistance device of this disclosure will be described with reference to the accompanying drawings. Furthermore, in each of the following embodiments, the same reference numerals will be used to denote the same constituent elements.

[0019] 1. Application Examples

[0020] Figure 1 This is a schematic diagram illustrating an application example of the mobility assistance device 100 according to this embodiment. The mobility assistance device 100, for example, guides the user 90 to the desired location by walking, thereby performing navigation actions to assist the user 90's movement. Navigation includes not only guiding the user 90 to the destination but also guiding the user 90 to a safe location.

[0021] The mobile assistance device 100 includes, for example, a control unit 4 such as a CPU, an input unit 1 such as buttons, a camera 2, and a sound output unit 32 such as headphones. The camera 2 is an example of the imaging unit disclosed herein. The control unit 4 is, for example, mounted on a robot 91 that is driven to move on its own. The robot 91 may also have a suitcase-like shape as shown in the figure. The input unit 1, the camera 2, and the sound output unit 32 are connected to the control unit 4 via wired or wireless means.

[0022] User 90, while moving, not only needs to reach their destination but may also have various other action requests. For example, user 90 may sometimes wish to talk to acquaintances. If robot 91 only ever moves to the destination, it cannot fulfill user 90's action requests, such as stopping en route to talk to acquaintances. To fulfill such action requests, the mobility assistance device 100, for example, should automatically stop or guide user 90 to the vicinity of an acquaintance if they encounter one during movement. On the other hand, in dangerous situations such as when user 90 is at a crosswalk, prioritizing user 90's action requests in the past would compromise user 90's safety.

[0023] Therefore, the mobility assistance device 100 accepts the action request of the user 90 via the input unit 1. The user 90, for example, operates the input unit 1 to select the action "to talk to an acquaintance" from a pre-determined action candidate. Alternatively, if the user 90 inputs "want to talk to an acquaintance" using the input unit 1, the control unit 4 can also select the action corresponding to the input information from a table stored in the storage unit. The control unit 4 of the mobility assistance device 100 calculates a security level S, representing the degree of security related to the user 90, based on the image captured by the camera 2. If the calculated security level S is less than the necessary security level T, representing the level of security required for the user 90 to safely "talk to an acquaintance," the control unit 4 does not assist in the realization of "talking to an acquaintance," for example, it continues navigation directly. Alternatively, if the security level S is greater than or equal to the necessary security level T, the control unit 4 can automatically stop or guide the user 90 to the vicinity of the acquaintance, thus assisting in the realization of "talking to an acquaintance." In this way, the mobility assistance device 100 seeks to realize the user 90's action request while ensuring the user 90's safety.

[0024] This disclosure can be applied to, for example, a mobile assistance device 100 that assists a visually impaired user 90 in their movement. For instance, the mobile assistance device 100, based on images captured by the camera 2, identifies a pre-registered acquaintance and notifies the user 90 of the acquaintance's presence via the voice output unit 32. The user 90 inputs the action request information—the intention to converse with the acquaintance—to the control unit 4 through the input unit 1. If the safety level S is less than the necessary safety level T required to converse with the acquaintance on the spot, the control unit 4 does not assist in fulfilling the action request. Alternatively, the control unit 4 may assist in "talking to an acquaintance" if the safety level S is greater than or equal to the necessary safety level T. In this way, the mobile assistance device 100 seeks to enable the user 90 to engage in social activities such as communication while ensuring the user 90's safety.

[0025] User 90 can also be a robot capable of autonomous movement. In this case, the components of the mobility aid 100, such as camera 2, control unit 4, and sound output unit 32, can also be assembled into the robot. In this case, the robot can, in principle, walk to its destination on its own, but it can also select from pre-determined action candidates to perform an action based on the image captured by camera 2 and the safety level S. Thus, even in a society where humans and robots coexist, it is possible to ensure the robot's safety while enabling smooth communication between the robot and humans.

[0026] 2. Structural Example

[0027] Figure 2 This is a block diagram illustrating a structural example of the mobility assistance device 100 according to this embodiment. The mobility assistance device 100 includes an input unit 1, a camera 2, an output unit 3, a control unit 4, a storage unit 5, and a communication interface (I / F) 6.

[0028] Input unit 1 may be an input device such as a keyboard, touchpad, touch panel, button, switch, or voice input device. For example, user 90 can operate input unit 1 to input action request information, which will be described later.

[0029] Camera 2 is a photographing device that captures images of the user 90's surrounding environment. Camera 2 forms images using a solid-state imaging element such as CMOS (Complementary MOS) or CCD (Charge Coupled Device). Camera 2 can be a wearable camera that the user 90 can wear. Camera 2 can also be mounted on a robot 91 or on the user 90's belongings, such as a backpack.

[0030] Output unit 3 is an output device that outputs data according to the control of control unit 4. Output unit 3 may include, for example, a display unit 31, a sound output unit 32, and a drive unit 33. Display unit 31 may be a display device such as a liquid crystal display, an organic EL display, or a projector. Sound output unit 32 may include audio devices such as speakers, headphones, or headphones. Drive unit 33 may include drive devices such as motors, wheels, and steering wheels for moving the mobility aid 100.

[0031] The control unit 4 is an information processing device that includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and controls the various components of the mobility aid 100 according to information processing. The control unit 4 includes, for example, an image acquisition unit 41, an image analysis unit 42, a security calculation unit 43, a necessary security determination unit 44, a judgment unit 45, and an output control unit 46. The image acquisition unit 41, image analysis unit 42, security calculation unit 43, necessary security determination unit 44, judgment unit 45, and output control unit 46 can also execute necessary programs through the control unit 4 to perform their respective processing tasks. Such programs can also be stored in the storage unit 5. When the control unit 4 executes the necessary programs, the program to be executed stored in the storage unit 5 is expanded in RAM. The control unit 4 interprets and executes the program expanded in RAM through the CPU, controlling each component. Examples of the operation of each component will be described later.

[0032] Storage unit 5 is a medium that stores recorded program information in a manner that allows computers and other devices or machines to read it, using electrical, magnetic, optical, mechanical, or chemical means. Storage unit 5 may be an auxiliary storage device such as a hard disk drive or a solid-state drive. Storage unit 5 may store, for example, a necessary safety level table 51, judgment result information 52, and programs executed by control unit 4. Storage unit 5 may also store map data used for mobility assistance up to the destination. Storage unit 5 may also include a main storage device such as RAM.

[0033] The communication interface 6 includes interface circuitry for enabling the mobile assistance device 100 to communicate with external devices. The communication interface 6 may communicate according to standards such as IEEE 802.3, IEEE 802.11, Wi-Fi (registered trademark), LTE, 3G, 4G, 5G, etc. The communication interface 6 may also be an interface circuitry that communicates according to standards such as USB (Universal Serial Bus), HDMI (registered trademark), IEEE 1394, Bluetooth (registered trademark). For example, the control unit 4 can obtain map data from an external server via the communication interface 6 and a network.

[0034] The mobility assistance device 100 can be mounted on, for example, a suitcase, a self-propelled robot, etc. The mobility assistance device 100 can also be a wearable device worn by the user, such as one carried on their back. The mobility assistance device 100 can also be installed in a backpack or other bag.

[0035] In this embodiment, an example of implementing the functions of the control unit 4 using a CPU will be described. However, some or all of the above functions can also be implemented by one or more dedicated processors. Furthermore, the components of the control unit 4 can be appropriately omitted, substituted, or added depending on the embodiment. The control unit 4 can also be composed of various semiconductor integrated circuits such as CPUs, MPUs, GPUs, microcomputers, DSPs, FPGAs, and ASICs.

[0036] 3. Action Examples

[0037] The mobility assistance device 100 operates in principle in a normal mode, assisting the user 90's movement by providing route guidance, etc. This normal mode can also be called a "navigation mode." The mobility assistance device 100 switches between the normal mode and the request execution mode based on the safety level S and input information from the user 90 regarding their action requests. Hereinafter, using... Figure 3 as well as Figure 4 The actions in each mode and the switching between modes are explained.

[0038] 3-1. Normal Pattern

[0039] Figure 3 This is a flowchart illustrating an example of the operation of the mobility assistance device 100 in normal mode. Figure 3 The processes shown are executed, for example, by control unit 4.

[0040] (ST11)

[0041] First, the image acquisition unit 41 acquires the image (ST11) captured by the camera 2. The camera 2, for example, captures images of the user 90's surroundings at a predetermined frame rate, generating image data sequentially. For example, the control unit 4 executes the control unit per frame. Figure 3 The process is shown in the flowchart.

[0042] (ST12)

[0043] Next, the image analysis unit 42 analyzes the image acquired in step (ST) 11 and detects objects such as obstacles, targets, and people within the image (ST12). The image analysis unit 42 can also analyze the position and state of the detected objects. For example, the image analysis unit 42 detects the direction of travel of moving objects such as cars and the direction of a person's gaze.

[0044] (ST13)

[0045] Next, the security calculation unit 43 calculates the security level S (ST13) based on the image. Specifically, the security calculation unit 43 calculates the security level S based on the image analysis results of ST12. For example, the security calculation unit 43 comprehensively considers the first to fifth indicators S1 to S5 to calculate the security level S. Here, the first to fifth indicators S1 to S5 can, for example, take values ​​from 0 to 1, and are determined by the security calculation unit 43 as follows.

[0046] For example, the first indicator S1 is determined based on factors such as weather and brightness in the user 90's surrounding environment. For instance, the safety calculation unit 43 sets the first indicator S1 to 0 when the weather is snowy and the surroundings are dark, and sets it to 1 when the weather is sunny and the surroundings are bright. The first indicator S1 is not limited to taking a higher value the brighter the surroundings are. For example, if the surrounding environment is too bright and the user 90 feels dazzled, the safety calculation unit 43 may also set the first indicator S1 to a low level, such as 0.1.

[0047] For example, the second indicator S2 is calculated based on the condition of the user 90's feet. For example, the safety calculation unit 43 sets the second indicator S2 to a smaller value when the user 90's feet are wet or on a step, and sets the second indicator S2 to a larger value when the user 90's feet are on a dry, flat surface.

[0048] For example, the third indicator S3 is calculated based on the presence and nature of obstacles in the user 90's surrounding environment. For instance, if there are no obstacles in the user 90's surrounding environment, the safety calculation unit 43 sets the third indicator S3 to 1. Conversely, if there are dangerous obstacles such as heavy machinery in the user 90's surrounding environment, the safety calculation unit 43 sets the third indicator S3 to a smaller value.

[0049] For example, the fourth indicator S4 is calculated based on the presence, nature, speed, and direction of movement of moving objects in the user 90's surrounding environment. For instance, if there are no moving objects in the user 90's surrounding environment, the safety calculation unit 43 sets the fourth indicator S4 to 1. Conversely, if there are moving objects such as cars moving towards the user 90's surrounding environment, the safety calculation unit 43 sets the fourth indicator S4 to a smaller value; if such moving objects are moving towards the user 90 at high speed, the fourth indicator S4 is set to an even smaller value.

[0050] For example, the fifth indicator S5 is calculated based on the presence or absence of people around the user 90, their movement speed, and their direction of movement. For example, if the user 90 is on a congested road or at an intersection, the safety calculation unit 43 will set the fifth indicator S5 to a smaller value.

[0051] The safety calculation unit 43 calculates the safety score S based on the first to fifth indicators S1 to S5 as described above. For example, the safety calculation unit 43 may also use the sum of the first to fifth indicators S1 to S5 as the safety score S. The safety score S may also be standardized to a value of 0 to 1. For example, the safety calculation unit 43 may also use the value obtained by multiplying the sum of the first to fifth indicators S1 to S5 by 1 / 5 and then standardizing it as the safety score S. Alternatively, the safety calculation unit 43 may apply different weights to the first to fifth indicators S1 to S5 and use the sum of the weighted indicators as the safety score S. Furthermore, the safety calculation unit 43 does not necessarily have to use all of the first to fifth indicators S1 to S5 to calculate the safety score S; it may use one or more of these indicators to calculate the safety score S. In addition, the safety calculation unit 43 may use more than five indicators to calculate the safety score S.

[0052] (ST14, ST15)

[0053] After ST13, the Necessary Safety Determination Unit 44, if there is input of Action Requirement Information ("Yes" in ST14), obtains the Necessary Safety Level T (ST15) from the Necessary Safety Level Table 51, which is required to safely achieve the Action Requirement Information shown in the Action Requirement Information.

[0054] Here, the necessary safety level T is an indicator of the degree of safety required for user 90 to safely perform the action indicated by the action request information. Furthermore, the presence of action request information input ("yes" in ST14) includes the presence of a selection operation input by user 90, whereby user 90 selects the desired action candidate from a pre-determined list of action candidates that user 90 can perform. If user 90 is an autonomously moving robot, the presence of action request information input ("yes" in ST14) includes the user 90 performing a process of selecting one or more action candidates from the pre-determined list of action candidates that can perform the action. Additionally, obtaining the necessary safety level T (ST15) from the necessary safety level table 51 for safely fulfilling the action request indicated by the action request information includes obtaining the necessary safety level T corresponding to the action candidate selected through the selection operation input from the necessary safety level table 51.

[0055] The action request information is input by user 90 to control unit 4 via input unit 1. User 90 may operate input unit 1, such as a keyboard, touchpad, touch panel, button, switch, or voice input device, to input action request information indicating the desired action. Alternatively, user 90 may operate input unit 1 to select an action candidate corresponding to the desired action from a pre-determined list of actions that user 90 can perform. For example, the action request information or pre-determined action candidates may include information such as "want to stop and talk to an acquaintance," "want to rest," "want to sit down," "want to go to the restroom," "want to check current location," "want to find an acquaintance," or "want to catch the bus."

[0056] Figure 4 This is a diagram illustrating an example of the necessary safety level table 51. In the necessary safety level table 51, pre-determined action candidates are stored corresponding to necessary safety levels, which represent the degree of security required for the user 90 to safely perform the action represented by the pre-determined action candidate. Figure 4 The illustrated necessary safety level table 51 stores pre-defined action candidates such as "stop talking," "rest," "sit down," "go to the restroom," "confirm current location," "find an acquaintance," and "take the bus at the desired time." Each action candidate corresponds to a necessary safety level T1 to T7.

[0057] The necessary safety levels T1 to T7 are predetermined, for example, based on the difficulty of user 90 safely performing the corresponding action candidate and / or the same indicators as the safety level S calculated in ST13. For example, to confirm the current location, it is sufficient to simply provide information to user 90 using the voice output unit 32, and the difficulty of safely performing this action is low. Therefore, the necessary safety level T5 required to confirm the current location is set to a low level. On the other hand, for example, to board a bus at the desired departure time, user 90 needs to move at high speed, and the difficulty of safely performing this action is high. Therefore, the necessary safety level T7 required to board a bus at the desired departure time is set to a high level. Furthermore, for example, it can be assumed that as long as there are no moving objects such as cars or bicycles around, it is safe to stop and speak. Therefore, the necessary safety level T1 required to stop and speak can also be set as a threshold value of the fourth indicator S4, calculated only based on the presence, nature, speed, and direction of moving objects in the user 90's surrounding environment. In this way, the thresholds of the five indicators included in the necessary safety level T, as described above, can be weighted in different ways according to the nature of the action candidates.

[0058] (ST16~18)

[0059] After ST15, the determination unit 45 determines whether the safety level S calculated by the safety level calculation unit 43 in ST13 is greater than or equal to the necessary safety level T obtained in ST15 (ST16). If the safety level S is greater than or equal to the necessary safety level T ("Yes" in ST16), the operation mode of the control unit 4 shifts from the current normal mode to the request execution mode that assists the user 90 in realizing the action candidate selected in ST14 (ST17). The request execution mode performs actions to assist the user 90 in realizing the action corresponding to the action request information input in ST14. If the safety level S is less than the necessary safety level T ("No" in ST16), the operation mode of the control unit 4 remains unchanged compared to the current normal mode, and the control unit 4 performs normal actions to assist the user 90 in moving by providing route guidance, etc. (ST18).

[0060] Normal operations are controlled by the output control unit 46. For example, the output control unit 46 controls the display unit 31 of the output unit 3, displaying the destination and the route to the destination on a map displayed on the display unit 31 such as a monitor. Thus, the user 90 can confirm the destination and the route. The output control unit 46 can also control the sound output unit 32 of the output unit 3 to provide route guidance via sound. The output control unit 46 can also control the drive unit 33 of the output unit 3, such as the motor, wheels, and steering wheel, to move the mobility aid 100 toward the destination.

[0061] Thus, if the safety level S is less than the necessary safety level T ("No" in ST16), the control unit 4 does not assist the user 90 in implementing the action candidate selected in ST14. In other words, if the safety level S is less than the necessary safety level T ("No" in ST16), the mobility assistance device 100 does not execute the output requested by the user 90. In such a case, executing the output requested by the user 90 might compromise the user 90's safety. By not executing the output requested by the user 90, the mobility assistance device 100 ensures the user 90's safety.

[0062] After executing ST18, control unit 4 returns to ST11 and repeats the process. Figure 3 The process is shown below. That is, Figure 3 The process shown repeats at a fixed cycle as long as it does not enter ST17 and switch to the required execution mode. In this way, the image acquisition step of ST11 is repeated at a specified frame rate.

[0063] The input of the action request information determined in ST14 can also be performed by the user 90 during navigation of the mobility assistance device 100, but this disclosure is not limited to this. For example, the input of the action request information can also be performed before the navigation operation. For example, the user 90 can also input action request information indicating "I expect to be notified if I meet a friend" into the mobility assistance device 100 in advance at home. The input action request information is stored, for example, in the storage unit 5.

[0064] 3-2. Required Execution Mode

[0065] Figure 5 This is a flowchart illustrating an example of the operation of the mobile assistance device 100 in the required execution mode. Figure 5 The processes shown are executed, for example, by control unit 4.

[0066] Figure 5 ST21~23 and Figure 4 ST11 to ST13 are the same. That is, the image acquisition unit 41 acquires the image captured by the camera 2 (ST21). The image analysis unit 42 analyzes the image acquired in ST21 and detects objects such as obstacles, targets, and people in the image (ST22). The safety calculation unit 43 calculates the safety level S based on the image (ST23). Specifically, the safety calculation unit 43 calculates the safety level S based on the result of the image analysis in ST12.

[0067] Next, the judgment unit 45 determines whether the safety level S calculated by the safety level calculation unit 43 in ST23 is safely implemented. Figure 4 The necessary safety level T required for the action candidate selected in ST14 is above (ST24). The necessary safety level T is, for example, in... Figure 4 The safety level is obtained in ST15 of the normal mode. If the safety level S is less than the necessary safety level T ("No" in ST24), the control unit 4 shifts the operation mode from the current required execution mode to the normal mode (ST27). If the safety level S is greater than or equal to the necessary safety level T ("Yes" in ST24), the process proceeds to the next step, ST25.

[0068] In ST25, control unit 4 determines that... Figure 4 The control unit 4 determines whether the execution of the selected action candidate in ST14 has been completed (ST25). If the execution of the currently executed action request is determined to be completed ("Yes" in ST25), the control unit 4 shifts the operation mode from the current request execution mode to the normal mode (ST27). The operation mode can also be manually changed via the user 90 operation input unit 1.

[0069] If it is determined in ST25 that the execution of the action requirement has not been completed (ST25: No), the control unit 4 executes the processing to fulfill the action requirement, or continues the currently executed processing (ST26).

[0070] After executing ST26, control unit 4 returns to ST21 and repeats the process. Figure 3 The process is shown below. That is, Figure 3 The process shown repeats at a fixed cycle as long as it does not enter ST27 and switch to the required execution mode. In this way, the image acquisition step of ST21 is repeated at a specified frame rate.

[0071] The above explanation states that in Figure 4 The usual pattern and Figure 5 The example given is an instance where both parties in the required execution mode perform image acquisition, image analysis, and security level S calculation steps. However, this disclosure is not limited to this. For example, the control unit 4 may always execute the image acquisition, image analysis, and security level S calculation steps at a predetermined cycle, while simultaneously performing state transition processing such as a state machine that controls the transition between the normal mode and the required execution mode.

[0072] 4. Function / Effect

[0073] As described above, in this embodiment, a mobility assistance device 100 is provided to assist the movement of a user 90. The mobility assistance device 100 includes a camera 2, a control unit 4, a storage unit 5, an input unit 1, and an output unit 3. The camera 2 captures images of the user 90's surrounding environment, obtaining captured images (ST11). Based on the captured images, the control unit 4 calculates a security level S, representing the degree of security related to the user 90 (ST13). The storage unit 5 stores a necessary security level table 51. In the necessary security level table 51, action candidates predetermined as actions that the user 90 can perform are stored in correspondence with necessary security levels T, representing the degree of security required for the user 90 to safely perform the actions represented by the predetermined action candidates. The input unit 1 accepts input from the user 90's selection operation. Through the selection operation, the user 90 selects the action candidate desired by the user 90 from the predetermined action candidates. When there is input of a selection operation (in the case of "yes" in ST14), the control unit 4 obtains the necessary security level T corresponding to the selected action candidate from the necessary security level table 51 (ST15). If the calculated safety level S is less than the necessary safety level T corresponding to the selected action candidate (in the case of "No" in ST14), the control unit 4 does not assist the user 90 in implementing the selected action candidate. If the calculated safety level S is greater than or equal to the necessary safety level T corresponding to the selected action candidate (in the case of "Yes" in ST14), the control output unit 3 assists the user 90 in implementing the selected action candidate.

[0074] If the user 90 performs the selected action when the safety level S is less than the necessary safety level T, the user's safety may not be guaranteed. With the structure described above, the mobility assistance device 100 does not assist the user 90 in performing the selected action when the safety level S is less than the necessary safety level T, thus ensuring the user 90's safety. On the other hand, the mobility assistance device 100 assists the user 90 in performing the selected action when the safety level S is greater than or equal to the necessary safety level T. In this way, the mobility assistance device 100 seeks to fulfill the user 90's requirements while ensuring the user 90's safety.

[0075] The control unit 4 can also operate in a manner that allows switching between a normal mode and a request execution mode. The normal mode controls the output unit 3 to assist the user 90 in moving, while the request execution mode controls the output unit 3 to assist the user 90 in implementing the selected action candidate. When operating in the normal mode, if the calculated safety level S is less than the necessary safety level T corresponding to the selected action candidate, the control unit 4 may not switch to the request execution mode. Conversely, the control unit 4 may switch to the request execution mode when the calculated safety level S is greater than or equal to the necessary safety level T corresponding to the selected action candidate. When operating in the request execution mode, if the calculated safety level S is less than the necessary safety level T corresponding to the selected action candidate, the control unit 4 may switch to the normal mode.

[0076] The mobile assistance device 100 operates by switching modes in this way, which can ensure the safety of the user 90 while fulfilling the user 90's requirements.

[0077] The control unit 4 can also calculate the safety level S based on at least one of the following: weather, brightness, the state of the user 90's feet, the presence or absence of obstacles in the user 90's surrounding environment, the nature of the obstacles, the presence or absence of moving objects in the user 90's surrounding environment, and the nature of moving objects in the surrounding environment.

[0078] In this way, by calculating the safety level S based on the specific conditions of user 90's surrounding environment, user 90's safety can be appropriately ensured according to the specific conditions. Furthermore, when the safety level S is high and appropriate, user 90's requirements can be met.

[0079] 5. Variations

[0080] The embodiments of this disclosure have been described in detail above, but the above description is merely illustrative in all respects. Various modifications and variations can be made without departing from the scope of this disclosure. For example, the following changes can be made. Furthermore, the same reference numerals are used below for the same constituent elements as in the above embodiments, and descriptions of points that are the same as in the above embodiments are appropriately omitted.

[0081] 5-1. First Variation

[0082] Figure 6 This is a flowchart illustrating a variation of the operation of the mobility assistance device 100 in normal mode. Figure 6 Flowcharts and Figure 3 Compared to the flowchart, it also includes ST31-33. Figure 6In the process, after ST13, the control unit 4 determines whether predetermined information has been detected in the captured image (ST31). Predetermined information includes, for example, familiar faces, destination markers, traffic lights, road signs, and various other information. Such information is, for example, pre-stored in the storage unit 5.

[0083] If predetermined information is detected (in the case of "Yes" in ST31), the control unit 4 determines whether the security level S is high enough (ST32). For example, the control unit 4 determines whether the security level S is above a preset threshold. If the security level S is high enough (in the case of "Yes" in ST32), the control unit 4 controls the output unit 3 to inform the user 90 of the meaning of the detected predetermined information and the content of the information (ST33). After the notification, the process proceeds to ST14.

[0084] In this way, when the control unit 4 detects predetermined information from the captured image (in the case of "yes" in ST31) and the calculated security level S is above the predetermined threshold (in the case of "yes" in ST32), the control output unit 3 informs the user 90 (ST33). The predetermined information is stored, for example, in the storage unit 5.

[0085] Thus, user 90 can learn about the pre-determined information by being informed, and based on that information, can select the desired action candidate from the pre-determined action candidates through a selection operation.

[0086] If the predetermined information is not detected in ST31 (if it is "No" in ST31), proceed to ST14.

[0087] If the safety level S is not determined to be sufficiently high in ST32 (if it is "No" in ST32), no notification is given, and the process proceeds to ST14. In such cases, control unit 4 prioritizes moving to a safe location, for example, by increasing the safety level S. Furthermore, if notification is given when the safety level S is not sufficiently high, user 90 may sometimes lose awareness of the danger, increasing the possibility that the hazard may affect user 90. To prevent such risks, control unit 4 does not notify user 90 unless the safety level S is sufficiently high.

[0088] 5-2. Second variation

[0089] like Figure 2As shown, the judgment unit 45 of the control unit 4 may also store the judgment result information 52 in the storage unit 5. The judgment result information 52 may also include information indicating the elements used in the judgment of the judgment unit 45 and the result of the judgment. For example, the judgment result information 52 is a judgment result table that stores the condition of the surrounding environment, the action requirements of the user 90 under that condition, the safety level S, and the necessary safety level T. The condition of the surrounding environment is, for example, information contained in the captured image obtained by the image acquisition unit 41 and the analysis result of the image analysis unit 42.

[0090] Performed by control unit 4 Figure 3 ST16 or Figure 5 ST24's judgment sometimes differs from human judgment. For example, from the perspective of user 90's companion, fulfilling user 90's request is clearly safe, but judgment unit 45 sometimes... Figure 3 ST16 or Figure 5 In ST24, the safety level S is determined to be less than the necessary safety level T (the judgment is "No"). In this case, the control unit 4 can also use the information in the judgment result table and the judgment made by the user 90's companion to perform machine learning and generate a learned model. The judgment made by the user 90's companion is input to the control unit 4, for example, through the input unit 1.

[0091] Such a learned model is generated, for example, by the control unit 4 performing machine learning on a model (learning algorithm) such as a Convolutional Neural Network (CNN). An example of machine learning is teacher-assisted learning, which uses the judgments of the user's 90 companions as teacher data to learn the relationship between the captured image obtained by the image acquisition unit 41 and the safety level S. Teacher-assisted learning of a model such as a CNN is performed by the control unit 4 using the error backpropagation method, for example.

[0092] In this way, by performing machine learning, the security level S can be appropriately calculated, and the judgment result of the judgment unit 45 becomes correct. For example, even if it is clear that fulfilling the user's 90% requirement is safe, the judgment unit 45 will not... Figure 3 ST16 or Figure 5 In ST24, it is determined that the safety factor S is less than the necessary safety factor T.

[0093] As another example of machine learning, the control unit 4 can also learn the relationship between the conditions of the surrounding environment and the action request information input in ST14. Thus, the mobility assistance device 100 can be optimized in a way that suits the user 90's preferences, and through continuous use, the number of times the user 90 needs to input their requests can be reduced. For example, suppose the user 90 consistently requests to take a bus that departs at a specific time every morning. In this case, the control unit 4 learns to navigate in a way that ensures the user 90 can catch the bus when leaving home within a specific timeframe in the morning.

[0094] After the learning process is complete, the machine learning-based generation results, such as the model, are stored in storage unit 5.

[0095] (Postscript)

[0096] The various methods of this disclosure will be described below. The reference numerals used below are merely illustrative.

[0097] The first aspect of the mobility assistance device 100 disclosed herein is a mobility assistance device 100 that assists the movement of a user 90, and includes:

[0098] The camera unit 2 captures images of the environment surrounding the user 90.

[0099] The control unit 4 calculates a security level S, which represents the degree of security related to the user 90, based on the captured image.

[0100] Storage unit 5 stores a necessary security level table, which stores action candidates that are predetermined as actions that the user 90 can perform, and a necessary security level T that represents the degree of security required for the user 90 to safely perform the actions shown by the predetermined action candidates.

[0101] Input unit 1, which accepts input from the user 90 to select the desired action candidate from the predetermined action candidates; and

[0102] Output section 3,

[0103] When the selection operation is input, the control unit 4 retrieves the necessary safety level T corresponding to the selected action candidate from the necessary safety level table.

[0104] If the calculated security level S is less than the necessary security level T corresponding to the selected action candidate, the user 90 will not be assisted in implementing the selected action candidate.

[0105] If the calculated security level S is greater than or equal to the necessary security level T corresponding to the selected action candidate, the output unit 3 is controlled to assist the user 90 in implementing the selected action candidate.

[0106] In the second aspect of this disclosure, a mobility assistance device 100 of the first aspect is provided, wherein,

[0107] The control unit 4 operates in a manner that allows switching between a normal mode and a request execution mode. The normal mode controls the output unit 3 to assist the user 90 in moving, while the request execution mode controls the output unit 3 to assist the user 90 in implementing the selected action candidate.

[0108] When the control unit 4 operates in the normal mode, if the calculated safety level S is less than the necessary safety level T corresponding to the selected action candidate, it will not switch the operation to the required execution mode. On the other hand, if the calculated safety level S is greater than or equal to the necessary safety level T corresponding to the selected action candidate, it will switch the operation to the required execution mode.

[0109] When the control unit 4 operates in the required execution mode, if the calculated safety level S is less than the necessary safety level T corresponding to the selected action candidate, the operation will be switched to the normal mode.

[0110] In the third aspect of this disclosure, a mobility assistance device 100 of the first or second aspect is provided, wherein,

[0111] The control unit 4 calculates the safety level S based on at least one of the following: the weather and brightness of the user 90's surrounding environment, the user 90's footing, whether there are obstacles in the user 90's surrounding environment, the nature of the obstacles, whether there are moving objects in the user 90's surrounding environment, and the nature of the moving objects.

[0112] In the fourth aspect of this disclosure, a mobility assist device according to any one of the first to third aspects is provided, wherein,

[0113] When the control unit detects predetermined information from the captured image, and the calculated security level is above a predetermined threshold, the control unit controls the output unit to notify the user 90.

[0114] In the fifth aspect of this disclosure, a mobility assistance method is provided to assist user 90 in moving, which includes the following steps:

[0115] Step ST11: Control unit 4 acquires the captured image of the surrounding environment of user 90.

[0116] Step ST13: Based on the captured image, the control unit 4 calculates a security level that represents the degree of security related to the user 90.

[0117] Control unit 4 accepts input from user 90 to select the desired action candidate from the pre-determined action candidates; and

[0118] In step ST15, when a selection operation input is present, the control unit 4 retrieves the necessary safety level corresponding to the selected action candidate from the necessary safety level table. The necessary safety level table consists of action candidates predetermined as actions that the user 90 can perform, and necessary safety levels that represent the degree of safety required for the user 90 to safely perform the actions indicated by the predetermined action candidates.

[0119] If the calculated security level is less than the necessary security level corresponding to the selected action candidate, the control unit 4 will not assist the user 90 in implementing the selected action candidate. If the calculated security level is greater than or equal to the necessary security level corresponding to the selected action candidate, the control unit 4 will output an output to assist the user 90 in implementing the selected action candidate.

[0120] In the sixth aspect of this disclosure, a program is provided for causing a computer to execute the mobile assistance method of the fifth aspect of this disclosure.

[0121] Label Explanation

[0122] 1: Input unit; 2: Camera (capturing unit); 3: Output unit; 4: Control unit; 5: Storage unit; 6: Communication interface; 31: Display unit; 32: Sound output unit; 33: Drive unit; 41: Image acquisition unit; 42: Image analysis unit; 43: Safety calculation unit; 44: Necessary safety determination unit; 45: Judgment unit; 46: Output control unit; 51: Necessary safety table; 52: Judgment result information; 90: User; 91: Robot; 100: Mobility assistance device.

Claims

1. A mobility assistance device mounted on a self-propelled robot, which assists a user's movement through self-propelled operation, the mobility assistance device comprising: The camera unit captures images of the user's surrounding environment. The control unit calculates a level of security, representing the degree of security related to the user, based on the captured image. The storage unit stores a necessary security level table, which stores action candidates predetermined as actions that the user can perform, corresponding to necessary security levels. The necessary security level indicates the degree of security required for the user to safely perform the actions indicated by the predetermined action candidates. The input unit accepts input from the user, which is a selection operation performed by the user to choose a desired action candidate from the predetermined action candidates; and The output section includes a drive section for moving the mobility aid. When the selection operation is input, the control unit retrieves the necessary safety score corresponding to the selected action candidate from the necessary safety score table. If the calculated safety score is less than the necessary safety score corresponding to the selected action candidate, the control unit does not assist the user in implementing the selected action candidate. If the calculated safety score is greater than or equal to the necessary safety score corresponding to the selected action candidate, the control unit controls the drive unit to assist the user in implementing the selected action candidate. The control unit operates in a manner that allows switching between a normal mode and a demand execution mode. The normal mode controls the drive unit to assist the user's movement, while the demand execution mode controls the drive unit to automatically stop or guide the user to the vicinity of an object to assist the user in realizing the selected action candidate. When the control unit operates in the normal mode, if the calculated safety level is less than the necessary safety level corresponding to the selected action candidate, it will not switch the operation to the required execution mode. On the other hand, if the calculated safety level is greater than or equal to the necessary safety level corresponding to the selected action candidate, it will switch the operation to the required execution mode. When the control unit operates in the required execution mode, if the calculated safety level is less than the necessary safety level corresponding to the selected action candidate, the operation will be switched to the normal mode.

2. The mobility assistance device according to claim 1, wherein, The control unit calculates the safety level based on at least one of the following: the weather and brightness of the user's surrounding environment, the state of the user's feet, the presence or absence of obstacles in the user's surrounding environment, the nature of the obstacles, the presence or absence of moving objects in the user's surrounding environment, and the nature of the moving objects.

3. The mobility assistance device according to claim 1, wherein, When the control unit detects predetermined information from the captured image, and the calculated security level is above a predetermined threshold, it controls the output unit to notify the user.

4. A mobility assistance method, executed by a control unit mounted on a self-propelled robot, the self-propelled robot assisting a user's movement by self-propelling, the mobility assistance method comprising the following steps: The control unit acquires images of the user's surrounding environment obtained by taking pictures. The control unit calculates a security level based on the captured image, representing the degree of security related to the user. The control unit accepts input from the user, who selects a desired action candidate from a pre-determined list of action candidates that the user is capable of performing; and When a selection operation is input, the control unit retrieves the necessary security level corresponding to the selected action candidate from the necessary security level table. The necessary security level table stores the predetermined action candidates and their corresponding necessary security levels, whereby the necessary security level represents the degree of security required for the user to safely perform the action indicated by the predetermined action candidate. If the calculated security level is less than the necessary security level corresponding to the selected action candidate, the control unit will not assist the user in implementing the selected action candidate; if the calculated security level is greater than or equal to the necessary security level corresponding to the selected action candidate, the control unit will activate the control unit to assist the user in implementing the selected action candidate. The control unit operates in a manner that allows switching between a normal mode and a demand execution mode. The normal mode is a mode that drives and assists the user's movement, while the demand execution mode drives and automatically stops or guides the user to the vicinity of an object to assist the user in realizing the selected action candidate. When the control unit operates in the normal mode, if the calculated safety level is less than the necessary safety level corresponding to the selected action candidate, it will not switch the operation to the required execution mode. On the other hand, if the calculated safety level is greater than or equal to the necessary safety level corresponding to the selected action candidate, it will switch the operation to the required execution mode. When the control unit operates in the required execution mode, if the calculated safety level is less than the necessary safety level corresponding to the selected action candidate, the operation will be switched to the normal mode.

5. A storage medium storing a program for causing a computer to execute the mobility assistance method of claim 4.

Images